PhotogrammetryWorkflows/VisualSFM CMVS-PMVS Meshlab

Note: I (Jacob Johnson, JJ) am currently using Windows. I will try to leave some leeway in these instructions for other OSs by including the overall links where you can find software for those versions, but the most detailed instructions at first will be for Windows versions (installation instructions at least, I think app usage will mostly be similar) until I get my Desktop back online with Ubuntu later and run through the process again for that, then document it.

Before going into the installation steps, here is a breakdown of what this set of software is doing / what we can do with it. This is what I have gathered so far, and it is still loose / rough around the edges. Please feel free to correct any mistakes I've made, or I will adjust as I learn more. VisualSFM takes in photos that you've taken around a 3D object and then runs an algorithm/program against them (SiftGPU, based on David Lowe's work - Chang Chang Wu wrote SiftGPU and VisualSFM) to find "feature points" (types of features in images that are easy to find in other images when the object is viewed from a different angle or distance) and from these, find the relative positions of where your camera was when you took each picture. Up to this point, it has performed "sparse reconstruction'. After this, it will turn to the CMVS-PMVS software (if you've copied it into the root VisualSFM install folder with the VisualSFM executable) to do dense reconstruction which involves finding many more features from the image data to fill out the point cloud. When you click on the dense reconstruction button, it'll ask you to provide a folder path where you would like it to save the results. Later, we will use Meshlab and open that folder/project. We will use Meshlab to construct a colored mesh around the point cloud. At that point, we'll have a 3D object with colors and texture. Once that's done, we can open it with any modelling software (or the main open source one, Blender), as well as import it into a slicer program for conversion to gcode for 3D printing.

After searching among many videos on Youtube, I found a good one that documents the installation process here: https://www.youtube.com/watch?v=GEAbXYDzUjU I will include timestamps into this video for each step below. Note that where my instructions differ it has to do with getting Windows binaries rather than Linux versions.

Final note before proceeding to installation steps: You might note that I am linking the same video over and over again. That's because I'm including the time stamps in the video for that piece of the instructions. If you already have the video up and do not want to open another tab, just look at the end. For instance, if I wanted the link to open the video at the 5 minute 30 second mark, I would add &t=5m30s to the end of the Youtube video link (stand for "time equals 5 minutes, 30 seconds"). You can just progress the video to that point in the tab you already have up if you want to move forward.

Installation: 1. https://www.youtube.com/watch?v=GEAbXYDzUjU&t=2m41s Download VisualSFM from this page : http://ccwu.me/vsfm/ (it has multiple OS versions, Windows, Linux, and MacOS, generally you'll want the 64 bit version - if that doesn't work you may have an old computer and need the 32 bit version) Installing this only consists of downloading the zip file and extracting it (there's no installer - once extracted, the executable is in the extracted folder and ready to run).

Next get CMVS-PMVS: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=2m52s https://github.com/pmoulon/CMVS-PMVS Click on the green button to the top right of the repo file list "Clone or Download", then select the option to clone download as a zip file. Extract the zip file and navigate to the subfolder that has the version for your operating system (for instance, for Windows the path from the top level "CMVS-PMVS-Master" extracted repo zip file to the Win64 version is: CMVS-PMVS-master.zip\CMVS-PMVS-master\binariesWin-Linux\Win64-VS2010). Copy everything from that subfolder into the folder where you put VisualSFM (further questions on how / what files to copy, watch the Youtube video starting at the 4 minute mark).

Next get Meshlab: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=3m15s https://sourceforge.net/projects/meshlab/ Sourceforge is no longer including adware automatically in all downloads. I installed this and had no issues (JJ, as of Apr 4th, 2020).

Finally, download and install Blender: https://www.blender.org/download/

Handling pictures: If you're using your cell phone, you'll want a way to easily get pictures from your phone to your computer so that you can run these apps on them to make the model. I had some initial difficulties, probably due to my phone being encrypted, where when I connected it to my computer, I could not see the phone's internal storage from Windows File Explorer. To overcome that, what I did was upload all the images to Google drive and installed Google Backup and Sync on my computer. Once you've uploaded photos to Google drive, when you sign into Google Backup and sync on the same Google account, it will download all of the files on Google drive to your computer. Its default directory is C:\Users\<YourUserNameHere>\Google Drive. I didn't do anything else with the files from there, only selected them in VisualSFM from where they were on my computer / where Google Backup and sync put them (see above). You may have to wait a few minutes after uploading for Backup and Sync to download all the pictures to your computer's hard drive. It will happen automatically as long as the app is signed into that Google account.

Running the pipeline:

VisualSFM steps: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=4m33s Open VisualSFM by going into the folder and double clicking the executable (the one with the green 'V' icon).

Open the images in VisualSFM: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=4m45s Click on the "Open Multiple Images" button (it's third from the left, just under the File tab in VisualSFM). Navigate to the location you stored your pictures on your computer (if you're using Google Backup and Sync, that's "C:\Users\Your User Name\Google Drive\Whatever you called the image folder"). Select all of the images by left clicking the first one, pressing shift and holding it down while you left click the last one. Finally, click the "open" button on the bottom right of that dialog window (while all the images still highlighted/selected). Now wait for a bit while it processes all the images. When it is done, you will see mini icons for all of the images in the main view and a summary in the log on the right.

Calculate missing matches: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m10s Select the "compute missing matches" icon. It's sixth from the right and looks like four arrows pointed at diagonal corners. Wait until it finishes.

Sparse reconstruction: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m23s (through 5m52s) Within VisualSFM, select "SfM" from the menu at top (2nd one from left). Within that menu, select "Reconstruct sparse". Wait until it finishes. You should then see the beginnings of a pointcloud with indicators where the cameras were located when the pictures were taken. Troubleshooting: Note, at this point when it has finished, if you notice that there are two few camera indicators, it is likely that one of them has a red "X" over it and is overall red. This means that while you were taking your pictures, the locations you took them from were likely too far (too many degrees) apart and it could not "connect the dots" between images. Try taking more images in between where there are more distant gaps in positions between the cameras, add the pictures and go back and do the prior step "calculate missing matches" again. This was happening to me and I got confused before really starting to think about it. I had taken about 20 images around a book on my couch. It wasn't nearly enough. Note that the video author took 70 pictures to capture his Millenium Falcon model. So I went back and took around 40 more (I had 64 when I was done, 5 were duplicates - this process doesn't care about those), being careful to move around the book no more than 15 degrees between images. Note that the time each step of this process takes will increase as the number of pictures goes up. After doing the missing matches step again, this time I had 2080 features instead of just over 100 and 434 seconds versus the 43 seconds it had taken prior (the last time it had stopped after getting confused after 3 pictures - if that step stops after less than a minute, you might have too few pictures). If the "compute missing matches" took enough time like this / you have plenty of pictures and features, you're probably going to be OK with the sparse reconstruction step that comes next. The next dense reconstruction for me after I took the extra pictures took 10 minutes, 55 seconds - note that unless you properly copied CMVS-PMVS into the VisualSFM folder as documented in the CMVS-PMVS installation step above, dense reconstruction will not work. Other: Make sure that the object you are taking pictures of has good lighting and contrast and there is detail on it. This process might not do well for scenes that don't have enough features/edges/contrast/detail - for instance a blank white wall wouldn't be captured well at all. For something as plain as a big piece of steel where the surface is very smooth, it might have to be enough to capture all the edges and for it to be dimensionally correct. Otherwise those surfaces will need detail somehow. (if you want a real challenge, try to do this to your cat when it happens to lay down - the process doesn't work at all for moving subjects, so move quickly - just kidding, probably a bad idea that will frustrate you to no end) Another thing I noticed was that the top cover of the book I was capturing was partially reflective. You'll want to avoid reflective surfaces if at all possible, because when you change your perspective, the reflected light changes as well, messing up the solver. It is very likely nothing will get captured at all from the reflective spots because matching of features can't occur between images there, causing gaps in the point cloud and resulting mesh.

Dense reconstruction (VisualSFM also uses the CMVS-PMVS software here): https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m52s Within VisualSFM, select "SfM" from the menu at top (2nd one from left). Within that menu, select "Reconstruct dense". It will pop up a dialog asking where it should save the files it creates. Give it a location on your hard drive by navigating, and type a file name you would like the project file to be called (you'll want to remember where you saved it so you can open it in Meshlab later - maybe write the path down). To see the results, go to View on the menu and select "Dense 3D points" (or just press tab). That's it, you're done with the VisualSFM and CMVS-PMVS portion of the pipeline. Next we will move on to Meshlab to connect the point cloud surface together with a mesh.

Meshlab:

Build the geometry: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=7m07s Open Meshlab and go to the File menu option and select "Open project". Navigate to where you saved the project through the dialog window in the "Dense reconstruction" step in VisualSFM above. Select the project file and wait until it finishes loading - you should see the point cloud. Go to the View menu option and select "Show Layer Dialog" - you should now see a pane to the right representing layers. Rotate the point cloud around until you have a good idea where your object is (left click and drag) and what points you can cut. Make sure that you have a good rotation for the cuts, then click the "Select vertexes" icon from the toolbar. Click and drag to create boxes to select points you don't want in your 3D model. You can hold control to click and drag again to add more boxes of points to your selection. When you're done, click on the first red X icon (it's an X over connected vertexes) to delete the selected points. Next, go to the File menu and select "Import mesh". Select the file (the dialog should already be in the correct folder since you loaded the project from there, and there will probably only be one file showing - it will end in the .ply extension) and click "open". Now you should have a much better view and be able to see which of the remaining points you want to delete. Repeat the select vertexes / delete vertexes step again until you have deleted those. Once you are happy that you have deleted all the points you want gone, go to the Filters menu, select "Point Set", then select "Surface Reconstructions Poisson". In the dialog window, set the octree depth to 12 and solver divide to 7, leaving the other two settings as is. Click "Apply". To get rid of the weird bubble that it creates while keeping the modifications to the model/mesh, on layers click on the model and .ply layers to make them invisible, then select the poisson layer (click on the name, not on the icon) highlighting it, then use the "select vertexes" and "delete vertexes" to get rid of the top part of the bubble. Extra notes: At this point, I went to File, "Export Mesh" and exported the mesh as a .obj file. It was 410KB, not too bad. When I loaded it into a slicer program, nothing came up. I think the origin of the 3D coordinate system for the model may have been too far from the model itself. Note: I was looking for a way to recenter the origin in Meshlab, but found an easy way to do it in Blender. For now, I'll leave that to a later step in Blender (just in case you need it).

Simplify the geometry: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=10m11s Look at the number of faces you have in your layers. If you have more than a million, you should probably do this step the quadric decimation: Go to the Filters menu, select "Remeshing, Simplification and Reconstruction", then select "Quadric Edge Collapse Decimation" (sounds intimidating and final). Give it a target number of faces of maybe 700,000 and apply. Next, go to the Filters menu, select "Cleaning and Repairing", and select "Remove Faces from Non Manifold Edges" (this prevents crazy things from happening in the next step where we do texturing).

Generate the Texture: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=11m06s Go to the Filters menu, Texture, and select "Parameterization + texturing from registered rasters" (not the one that is named almost exactly the same but without the '+'). In the dialog window, set the Texture size to 8192, change the texture name to what you want, and click apply. Wait for it to finish, may take a bit. You should be able to see a progress bar in the lower right.

Save the model: In Meshlab, go to the File menu, select "Export Mesh As", and in the dialog, change the file type to "Alias Wavefront Object (*.obj)" and give it a descriptive name, then click save. In the next dialog box, uncheck "normals" and click OK. Now to make things easier for the later step, copy the texture (.png file) and your latest .obj file into a new folder (name that something descriptive).

Create the scene in Blender: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=13m04s Open Blender. Go to File, select new, general. Import your model - go to File, import, wavefront (.obj). In the dialog, navigate to your model you saved in the "Save the model" step above. Once it has imported, take a look around the scene by rotating (in Blender, that is by default holding down the middle mouse button while you move the mouse).

If you need to recenter the model in the scene, select the model in the scene then press shift-ctrl-alt-c and select "geometry to origin". The model should move to the center. If when doing that, the model suddenly disappears, it's probably just inside the gray box that appears by default in Blender at the origin of the scene, which means you succeeded (you can scroll to zoom inside the box and check). You can delete the cube from the scene collection pane at the right by selecting it and selecting "delete".

TODO: At the end of this process, I had a model in Blender that still looked somewhat point-cloudish. I need to find more tutorials on Meshlab to include tips/tricks. The texture also didn't come through the process, so I need to go back over that and find a fix.