PhotogrammetryWorkflows/VisualSFM CMVS-PMVS Meshlab
Note: I (Jacob Johnson, JJ) am currently using Windows. I will try to leave some leeway in these instructions for other OSs by including the overall links where you can find software for those versions, but the most detailed instructions at first will be for Windows versions (installation instructions at least, I think app usage will mostly be similar) until I get my Desktop back online with Ubuntu later and run through the process again for that, then document it.
Summary:
Before going into the installation steps, here is a breakdown of what this set of software is doing / what we can do with it. VisualSFM takes in photos that you've taken around a 3D object and then runs an algorithm/program against them (SiftGPU, based on David Lowe's work - Chang Chang Wu wrote SiftGPU and VisualSFM) to find "feature points" (types of features in images that are easy to find in other images when the object is viewed from a different angle or distance) and from these, find the relative positions of where your camera was when you took each picture. Up to this point, it has performed "sparse reconstruction'. After this, it will turn to the CMVS-PMVS software (if you've copied it into the root VisualSFM install folder with the VisualSFM executable) to do dense reconstruction which involves finding many more features from the image data to fill out the point cloud. When you click on the dense reconstruction button, it'll ask you to provide a folder path where you would like it to save the results. Later, we will use Meshlab and open that folder/project. We will use Meshlab to construct a colored mesh around the point cloud. At that point, we'll have a 3D object with colors and texture. Once that's done, we can open it with any modelling software (or the main open source one, Blender), as well as import it into a slicer program for conversion to gcode for 3D printing.
After searching among many videos on Youtube, I found a good one that documents the installation process here: https://www.youtube.com/watch?v=GEAbXYDzUjU I will include timestamps into this video for each step below. Note that where my instructions differ it has to do with getting Windows binaries rather than Linux versions.
Final note before proceeding to installation steps: You might note that I am linking the same video over and over again. That's because I'm including the time stamps in the video for that piece of the instructions. If you already have the video up and do not want to open another tab, just look at the end. For instance, if I wanted the link to open the video at the 5 minute 30 second mark, I would add &t=5m30s to the end of the Youtube video link (stand for "time equals 5 minutes, 30 seconds"). You can just progress the video to that point in the tab you already have up if you want to move forward.
Installation:
Download VisualSFM from this page : http://ccwu.me/vsfm/ (it has multiple OS versions, Windows, Linux, and MacOS, generally you'll want the 64 bit version - if that doesn't work you may have an old computer and need the 32 bit version) Installing this only consists of downloading the zip file and extracting it (there's no installer - once extracted, the executable is in the extracted folder and ready to run).
- Next get CMVS-PMVS:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=2m52s https://github.com/pmoulon/CMVS-PMVS Click on the green button to the top right of the repo file list "Clone or Download", then select the option to clone download as a zip file. Extract the zip file and navigate to the subfolder that has the version for your operating system (for instance, for Windows the path from the top level "CMVS-PMVS-Master" extracted repo zip file to the Win64 version is: CMVS-PMVS-master.zip\CMVS-PMVS-master\binariesWin-Linux\Win64-VS2010). Copy everything from that subfolder into the folder where you put VisualSFM (further questions on how / what files to copy, watch the Youtube video starting at the 4 minute mark).
- Get Meshlab:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=3m15s https://sourceforge.net/projects/meshlab/ Sourceforge is no longer including adware automatically in all downloads. I installed this and had no issues (JJ, as of Apr 4th, 2020).
- Get Blender:
https://www.blender.org/download/
- Handling pictures:
If you're using your cell phone, you'll want a way to easily get pictures from your phone to your computer so that you can run these apps on them to make the model. I had some initial difficulties, probably due to my phone being encrypted, where when I connected it to my computer, I could not see the phone's internal storage from Windows File Explorer. To overcome that, what I did was upload all the images to Google drive and installed Google Backup and Sync on my computer. Once you've uploaded photos to Google drive, when you sign into Google Backup and sync on the same Google account, it will download all of the files on Google drive to your computer. Its default directory is C:\Users\<YourUserNameHere>\Google Drive. I didn't do anything else with the files from there, only selected them in VisualSFM from where they were on my computer / where Google Backup and sync put them (see above). You may have to wait a few minutes after uploading for Backup and Sync to download all the pictures to your computer's hard drive. It will happen automatically as long as the app is signed into that Google account.
Running the pipeline:
VisualSFM: https://www.youtube.com/watch?v=GEAbXYDzUjU&t=4m33s Open VisualSFM by going into the folder and double clicking the executable (the one with the green 'V' icon).
- Load the images into VisualSFM:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=4m45s Click on the "Open Multiple Images" button (it's third from the left, just under the File tab in VisualSFM). Navigate to the location you stored your pictures on your computer (if you're using Google Backup and Sync, that's "C:\Users\Your User Name\Google Drive\Whatever you called the image folder"). Select all of the images by left clicking the first one, pressing shift and holding it down while you left click the last one. Finally, click the "open" button on the bottom right of that dialog window (while all the images still highlighted/selected). Now wait for a bit while it processes all the images. When it is done, you will see mini icons for all of the images in the main view and a summary in the log on the right.
- Calculate missing matches:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m10s Select the "compute missing matches" icon. It's sixth from the right and looks like four arrows pointed at diagonal corners. Wait until it finishes.
- Sparse reconstruction:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m23s (through 5m52s) Within VisualSFM, select "SfM" from the menu at top (2nd one from left). Within that menu, select "Reconstruct sparse". Wait until it finishes. You should then see the beginnings of a pointcloud with indicators where the cameras were located when the pictures were taken.
- Dense reconstruction (VisualSFM also uses the CMVS-PMVS software here):
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=5m52s Within VisualSFM, select "SfM" from the menu at top (2nd one from left). Within that menu, select "Reconstruct dense". It will pop up a dialog asking where it should save the files it creates. Give it a location on your hard drive by navigating, and type a file name you would like the project file to be called (you'll want to remember where you saved it so you can open it in Meshlab later - maybe write the path down). To see the results, go to View on the menu and select "Dense 3D points" (or just press tab). That's it, you're done with the VisualSFM and CMVS-PMVS portion of the pipeline. Next we will move on to Meshlab to connect the point cloud surface together with a mesh.
Troubleshooting the VSFM process:
- Note, after you get done with the "Calculate missing matches" step, if you notice that there are two few camera indicators, it is likely that one of them has a red "X" over it and is overall red. This means that while you were taking your pictures, the locations you took them from were likely too far (too many degrees) apart and it could not "connect the dots" between images. Try taking more images in between where there are more distant gaps in positions between the cameras, add the pictures and go back and do the prior step "calculate missing matches" again. You should shoot for 50-60 pictures at least, both rotating around and above (below if possible) the object and perhaps even closeups. More pictures should give a better point cloud, but costs more processing time.
- Make sure that the object you are taking pictures of has good lighting and contrast and there is enough detail on it, matte surfaces work really well. This process might not do well for scenes that don't have enough features/edges/contrast/detail - for instance a blank white wall wouldn't work.
- The process doesn't work at all for moving subjects.
- You'll want to avoid reflective surfaces if at all possible, because when you change your perspective, the reflected light changes as well, messing up the solver, meaning you won't capture any points there and you'll likely have a gap in the mesh.
Meshlab:
- Build the geometry:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=7m07s Open Meshlab and go to the File menu option and select "Open project". Navigate to where you saved the project through the dialog window in the "Dense reconstruction" step in VisualSFM above. Select the project file and wait until it finishes loading - you should see the point cloud. Go to the View menu option and select "Show Layer Dialog" - you should now see a pane to the right representing layers. Rotate the point cloud around until you have a good idea where your object is (left click and drag) and what points you can cut. Make sure that you have a good rotation for the cuts, then click the "Select vertexes" icon from the toolbar. Click and drag to create boxes to select points you don't want in your 3D model. You can hold control to click and drag again to add more boxes of points to your selection. When you're done, click on the first red X icon (it's an X over connected vertexes) to delete the selected points. Next, go to the File menu and select "Import mesh". Select the file (the dialog should already be in the correct folder since you loaded the project from there, and there will probably only be one file showing - it will end in the .ply extension) and click "open". Now you should have a much better view and be able to see which of the remaining points you want to delete. Repeat the select vertexes / delete vertexes step again until you have deleted those. Once you are happy that you have deleted all the points you want gone, go to the Filters menu, select "Point Set", then select "Surface Reconstructions Poisson". In the dialog window, set the octree depth to 12 and solver divide to 7, leaving the other two settings as is. Click "Apply". To get rid of the weird bubble that it creates while keeping the modifications to the model/mesh, on layers click on the model and .ply layers to make them invisible, then select the poisson layer (click on the name, not on the icon) highlighting it, then use the "select vertexes" and "delete vertexes" to get rid of the top part of the bubble. Extra notes: At this point, I went to File, "Export Mesh" and exported the mesh as a .obj file. It was 410KB, not too bad. When I loaded it into a slicer program, nothing came up. I think the origin of the 3D coordinate system for the model may have been too far from the model itself. Note: I was looking for a way to recenter the origin in Meshlab, but found an easy way to do it in Blender. For now, I'll leave that to a later step in Blender (just in case you need it).
- Simplify the geometry:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=10m11s Look at the number of faces you have in your layers. If you have more than a million, you should probably do this step the quadric decimation: Go to the Filters menu, select "Remeshing, Simplification and Reconstruction", then select "Quadric Edge Collapse Decimation" (sounds intimidating and final). Give it a target number of faces of maybe 700,000 and apply. Next, go to the Filters menu, select "Cleaning and Repairing", and select "Remove Faces from Non Manifold Edges" (this prevents crazy things from happening in the next step where we do texturing).
- Generate the Texture:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=11m06s Go to the Filters menu, Texture, and select "Parameterization + texturing from registered rasters" (not the one that is named almost exactly the same but without the '+'). In the dialog window, set the Texture size to 8192, change the texture name to what you want, and click apply. Wait for it to finish, may take a bit. You should be able to see a progress bar in the lower right.
- Save the model:
In Meshlab, go to the File menu, select "Export Mesh As", and in the dialog, change the file type to "Alias Wavefront Object (*.obj)" and give it a descriptive name, then click save. In the next dialog box, uncheck "normals" and click OK. Now to make things easier for the later step, copy the texture (.png file) and your latest .obj file into a new folder (name that something descriptive).
Blender:
- Create the scene in Blender:
https://www.youtube.com/watch?v=GEAbXYDzUjU&t=13m04s Open Blender. Go to File, select new, general. Import your model - go to File, import, wavefront (.obj). In the dialog, navigate to your model you saved in the "Save the model" step above. Once it has imported, take a look around the scene by rotating (in Blender, that is by default holding down the middle mouse button while you move the mouse).
- Recentering the model (if necessary):
Press shift-ctrl-alt-c and select "geometry to origin". The model should move to the center. If when doing that, the model suddenly disappears, it's probably just inside the gray box that appears by default in Blender at the origin of the scene, which means you succeeded (you can scroll to zoom inside the box and check). You can delete the cube from the scene collection pane at the right by selecting it and selecting "delete".
TODO: At the end of this process, I had a model in Blender that still looked somewhat point-cloudish. I need to find more tutorials on Meshlab to include tips/tricks. The texture also didn't come through the process, so I need to go back over that and find a fix. Most of the tutorials tell you not to expect to have super nice looking models when you first start. I'll add the most helpful tips and tricks as I find them.
