11.1 Photogrammetry and volumetric capture

Photogrammetry and volumetric capture are powerful techniques for creating 3D models from real-world objects and scenes. These methods enable the digitization of physical environments, allowing for immersive experiences in virtual reality applications.

By capturing multiple images or using specialized sensors, these technologies reconstruct detailed 3D geometry and textures. From cultural heritage preservation to game development, photogrammetry and volumetric capture are revolutionizing how we create and interact with digital content.

Principles of photogrammetry

• Photogrammetry is a technique for creating 3D models from a series of 2D images, enabling the digitization of real-world objects and environments for use in immersive and virtual reality applications
• The process involves capturing multiple overlapping images of an object or scene from different angles and positions, then using specialized software to analyze and reconstruct the 3D geometry and texture information

Capturing overlapping images

Top images from around the web for Capturing overlapping images

• 

  AliceVision Association View original

  Is this image relevant?

• 

  Frontiers | A Review of Photogrammetry and Photorealistic 3D Models in Education From a ... View original

  Is this image relevant?

• 

  Frontiers | A Non-destructive Method to Create a Time Series of Surface Area for Coral Using 3D ... View original

  Is this image relevant?

• 

  AliceVision Association View original

  Is this image relevant?

• 

  Frontiers | A Review of Photogrammetry and Photorealistic 3D Models in Education From a ... View original

  Is this image relevant?

1 of 3

Top images from around the web for Capturing overlapping images

• 

  AliceVision Association View original

  Is this image relevant?

• 

  Frontiers | A Review of Photogrammetry and Photorealistic 3D Models in Education From a ... View original

  Is this image relevant?

• 

  Frontiers | A Non-destructive Method to Create a Time Series of Surface Area for Coral Using 3D ... View original

  Is this image relevant?

• 

  AliceVision Association View original

  Is this image relevant?

• 

  Frontiers | A Review of Photogrammetry and Photorealistic 3D Models in Education From a ... View original

  Is this image relevant?

1 of 3

• To ensure accurate 3D reconstruction, images must have sufficient overlap (typically 60-80%) to allow the software to identify and match common points between adjacent images
• Overlapping images provide multiple views of the same features, enabling the software to triangulate their positions in 3D space and estimate depth information
• Capturing more images with higher overlap can improve the quality and detail of the resulting 3D model, but also increases processing time and storage requirements

Camera positions and angles

• Camera positions and angles should be carefully planned to cover the entire object or scene with consistent spacing and minimal occlusions
• For small objects, a circular or hemispherical capture pattern is often used, with the camera positioned at regular intervals around the object and pointing towards its center
• For larger scenes or environments, a more complex capture pattern may be required, such as a grid or zigzag pattern, to ensure adequate coverage and overlap

Consistent lighting conditions

• Maintaining consistent lighting conditions throughout the capture process is crucial for achieving high-quality textures and minimizing artifacts in the reconstructed 3D model
• Ideally, the object or scene should be evenly lit from multiple directions to minimize shadows and specular highlights, which can confuse the reconstruction algorithms
• Diffuse, non-directional lighting (such as on an overcast day or in a softbox studio setup) is preferred, as it reduces the appearance of harsh shadows and reflections

Photogrammetry software

• Photogrammetry software is used to process the captured images and generate a 3D model of the object or scene, typically in the form of a point cloud, mesh, and texture map
• These software tools use computer vision algorithms to analyze the images, identify common features, and estimate the camera positions and 3D geometry of the captured subject

Open source vs commercial

• There are both open-source and commercial photogrammetry software options available, each with their own strengths and limitations
• Open-source tools (AliceVision, Meshroom) are often free and offer greater flexibility and customization, but may have steeper learning curves and less user support
• Commercial software (RealityCapture, Agisoft Metashape) generally provides more user-friendly interfaces, advanced features, and better performance, but come with associated costs and licensing requirements

Automatic vs manual alignment

• Many photogrammetry software packages offer automatic image alignment, which uses feature detection and matching algorithms to estimate the relative positions and orientations of the cameras without user intervention
• Manual alignment may be necessary in cases where automatic alignment fails or produces inaccurate results, such as when there is insufficient overlap or distinctive features between images
• Manual alignment involves manually identifying and marking corresponding points across multiple images, providing the software with additional constraints to help optimize the camera positions and 3D reconstruction

Point cloud generation

• Once the images are aligned, the software generates a dense point cloud representing the 3D structure of the captured object or scene
• Each point in the cloud represents a 3D position estimated from the corresponding pixels in the overlapping images, typically colored based on the average color of those pixels
• The density and accuracy of the point cloud depend on factors such as image resolution, overlap, and the quality of the camera calibration and alignment

Mesh reconstruction algorithms

• To create a continuous 3D surface from the point cloud, photogrammetry software employs mesh reconstruction algorithms that connect the points into a triangulated mesh
• Common algorithms include Poisson surface reconstruction and Delaunay triangulation, which create a watertight mesh by interpolating the surface between the points based on their local density and orientation
• The resulting mesh can be further optimized and simplified to reduce noise, fill holes, and improve its topology for downstream applications like texturing, rendering, and physics simulations

Capturing objects for photogrammetry

• When capturing objects for photogrammetry, it is important to consider factors that can affect the quality and accuracy of the resulting 3D model, such as the object's material properties, size, and surrounding environment
• Proper planning and setup of the capture process can help minimize potential issues and ensure the best possible results

Ideal object characteristics

• Objects with matte, diffuse surfaces and distinctive textures are generally easier to reconstruct accurately, as they provide clear features for the software to match across images
• Avoid objects with highly reflective, transparent, or translucent surfaces (glass, polished metal), as they can create confusing reflections and refractions that are difficult to interpret
• Objects with complex geometries, thin structures, or deep cavities may require more images and careful camera positioning to ensure adequate coverage and minimize occlusions

Minimizing reflections and shadows

• To minimize reflections, use diffuse, non-directional lighting and consider using polarizing filters on the camera and light sources to reduce specular highlights
• Position lights strategically to minimize shadows cast by the object onto itself or the background, which can obscure features and create artifacts in the reconstruction
• Use a neutral, matte background (green screen, white seamless) to isolate the object and make it easier for the software to distinguish foreground from background

Turntables for 360° capture

• For small to medium-sized objects, using a turntable can simplify the capture process and ensure consistent camera positions and angles
• Place the object on the turntable and rotate it in small increments (10-20°) while capturing images from a fixed camera position, repeating the process at different elevations to cover the top and bottom of the object
• Automated turntable systems (Foldio360, Orangemonkie) can streamline the capture process and ensure precise rotations, while also providing integrated lighting and background solutions

Capturing larger objects and environments

• For larger objects or environments that cannot be captured using a turntable, plan a capture path that covers the entire subject with sufficient overlap and multiple viewpoints
• Use a camera with a wide-angle lens to capture more of the scene in each image, reducing the total number of images required and minimizing parallax effects between foreground and background elements
• Consider using a drone or robotic arm to capture images from elevated or hard-to-reach positions, ensuring a more complete and detailed reconstruction of the subject

Volumetric capture techniques

• Volumetric capture refers to techniques for capturing and reconstructing dynamic 3D scenes, such as moving people or objects, in real-time or near-real-time
• Unlike traditional photogrammetry, which typically captures static subjects, volumetric capture enables the creation of interactive and immersive 3D content for applications like virtual reality, augmented reality, and holographic displays

Multi-camera arrays

• Multi-camera arrays consist of multiple synchronized cameras arranged in a circular or hemispherical configuration around the capture volume
• By capturing the scene from multiple viewpoints simultaneously, the system can reconstruct the 3D geometry and texture of the moving subject in real-time, creating a volumetric video stream
• Examples of multi-camera array systems include Microsoft Mixed Reality Capture Studios, 4DViews, and Volucap

Depth sensors and structured light

• Depth sensors, such as Microsoft Kinect and Intel RealSense, use infrared projectors and cameras to capture depth information directly, without the need for multiple viewpoints
• Structured light systems project a known pattern onto the scene and analyze the deformation of the pattern to estimate the depth and shape of the subject
• These techniques can be used alone or in combination with color cameras to create real-time 3D reconstructions of moving subjects, although they may have limitations in terms of resolution, range, and occlusion handling

Real-time vs offline processing

• Real-time volumetric capture systems process the captured data on-the-fly, generating a live 3D stream that can be viewed and interacted with immediately
• Offline processing involves capturing the data first and then processing it later using more computationally intensive algorithms, which can result in higher-quality reconstructions but requires more time and storage
• The choice between real-time and offline processing depends on the specific application and the trade-offs between latency, quality, and computational resources

Volumetric video formats

• Volumetric video requires specialized formats to store and transmit the time-varying 3D data efficiently, balancing compression, quality, and random access
• Point cloud formats, such as PLY and PCD, can be used to store the raw 3D point data, but lack connectivity information and can be inefficient for large sequences
• Mesh-based formats, like glTF and FBX, can provide a more compact representation by encoding the surface geometry and topology, but may require additional processing for level-of-detail and streaming
• Emerging volumetric video formats, such as Microsoft's LightField-Mesh and 8i's VoluMesh, aim to provide efficient compression, streaming, and rendering of dynamic 3D content

Integrating photogrammetry assets

• Once a 3D model has been created using photogrammetry, it must be processed and optimized for integration into immersive and virtual reality applications
• This involves tasks such as mesh cleanup, UV mapping, texture baking, level-of-detail generation, and file format conversion, to ensure the asset is compatible with the target platform and performs efficiently at runtime

Optimizing meshes and textures

• Raw photogrammetry meshes often contain excess geometry, holes, and other artifacts that need to be cleaned up before use in real-time applications
• Mesh optimization techniques, such as decimation, retopology, and hole filling, can be used to reduce the polygon count, improve the topology, and create a cleaner, more efficient mesh
• Texture maps may also require processing, such as color correction, sharpening, and compression, to improve their visual quality and reduce memory usage

UV mapping and texture baking

• UV mapping is the process of unwrapping the 3D mesh onto a 2D texture space, allowing textures to be projected onto the model's surface
• For photogrammetry assets, the UV layout should be optimized to minimize seams and distortion, while also making efficient use of the available texture space
• Texture baking involves rendering the high-resolution details from the original photogrammetry mesh onto a lower-resolution mesh using the optimized UV layout, creating a more efficient and portable asset

Level of detail (LOD) systems

• Level of detail systems create multiple versions of a 3D model at different resolutions, allowing the application to switch between them based on factors like camera distance and performance requirements
• For photogrammetry assets, LODs can be generated by progressively simplifying the mesh and texture maps, reducing the polygon count and texture resolution at each level
• Implementing LODs can significantly improve the performance and scalability of immersive applications, especially when dealing with large or complex photogrammetry assets

Asset import and rendering pipelines

• Integrating photogrammetry assets into immersive applications requires a robust pipeline for importing, processing, and rendering the 3D models and textures
• This may involve converting the assets into a compatible file format (FBX, OBJ, glTF), setting up materials and shaders, and configuring the rendering engine to handle the specific requirements of photogrammetry data
• Many game engines and 3D frameworks (Unity, Unreal Engine, Three.js) provide built-in support for importing and rendering photogrammetry assets, as well as tools for optimizing and visualizing the data

Applications of photogrammetry

• Photogrammetry has a wide range of applications across various industries, from cultural heritage and entertainment to scientific research and industrial design
• The ability to create detailed, photorealistic 3D models of real-world objects and environments makes photogrammetry a valuable tool for immersive and virtual reality projects

Virtual museums and heritage preservation

• Photogrammetry enables the digitization of historical artifacts, artworks, and architectural sites, allowing them to be preserved, studied, and shared with a global audience
• Virtual museums can use photogrammetry to create interactive 3D exhibits, where visitors can explore and interact with high-resolution models of objects and environments from different cultures and time periods
• Examples include the Smithsonian 3D Digitization Program, which has created detailed 3D models of artifacts from its collections, and the Scan Pyramids Project, which used photogrammetry to create a virtual tour of the Great Pyramid of Giza

Film VFX and virtual production

• Photogrammetry is increasingly used in film and television production to create realistic digital environments and assets for visual effects and virtual production
• By capturing real-world locations and props using photogrammetry, VFX artists can create highly detailed and accurate 3D models that can be seamlessly integrated into live-action footage or used as reference for digital set design
• Virtual production techniques, such as LED wall stages and real-time rendering, rely on photogrammetry assets to create immersive and interactive virtual environments for actors and crew to work within

Game development and virtual worlds

• Game developers use photogrammetry to create realistic and immersive game environments, characters, and objects, enhancing the visual quality and believability of their virtual worlds
• Photogrammetry assets can be used as a starting point for creating game assets, providing a high level of detail and realism that can be optimized and stylized to fit the game's aesthetic and performance requirements
• Examples of games that have used photogrammetry include Star Wars Battlefront, Battlefield V, and The Vanishing of Ethan Carter, which showcase the technology's potential for creating stunning and immersive game worlds

Scientific visualization and analysis

• Photogrammetry is used in various scientific fields, such as archaeology, geology, and biology, to create accurate 3D models of objects and environments for visualization, analysis, and documentation purposes
• By capturing and reconstructing real-world data in 3D, researchers can study and manipulate the models in ways that would be difficult or impossible with physical specimens, such as measuring, sectioning, and comparing different samples
• Examples include using photogrammetry to create 3D models of archaeological sites, fossils, and geological formations, allowing researchers to study and share their findings in a more engaging and accessible format

Challenges and limitations

• Despite its many applications and benefits, photogrammetry also presents several challenges and limitations that must be considered when using the technology for immersive and virtual reality projects
• These challenges relate to the capture process, reconstruction algorithms, and the inherent properties of certain types of objects and materials

Capturing thin or transparent objects

• Objects with thin or transparent surfaces, such as glass, foliage, and wire fences, can be difficult to capture accurately using photogrammetry
• Thin structures may not have enough volume or surface area to be reconstructed properly, resulting in gaps, holes, or incomplete geometry in the final model
• Transparent and translucent materials can create confusing reflections and refractions that are difficult for the reconstruction algorithms to interpret, leading to artifacts and distortions in the mesh and texture maps

Dealing with occlusions and holes

• Occlusions occur when parts of the object or scene are hidden from view in some of the captured images, due to obstacles, shadows, or limitations in camera positioning
• These occluded areas can result in holes or missing data in the reconstructed 3D model, requiring additional processing or manual intervention to fill in the gaps
• Strategies for dealing with occlusions include capturing more images from different angles, using algorithms that can infer missing data based on surrounding geometry, and manually editing the mesh to fill holes or replace missing sections

Texture blending and color correction

• Inconsistencies in lighting, exposure, and white balance between the captured images can result in visible seams and color variations in the final texture maps
• Texture blending techniques, such as multi-band blending and color correction, can help minimize these artifacts and create a more seamless and cohesive appearance
• However, these techniques may not always be able to fully eliminate texture inconsistencies, especially in cases where the lighting conditions vary significantly between images or there are strong shadows and highlights present

File sizes and performance considerations

• High-resolution photogrammetry assets can result in large file sizes and memory usage, which can impact the performance and scalability of immersive and virtual reality applications
• Mesh and texture optimization techniques, such as polygon reduction, UV layout optimization, and texture compression, can help reduce the file sizes and improve runtime performance
• However, there is often a trade-off between visual quality and performance, and finding the right balance requires careful consideration of the target platform, application requirements, and user experience goals
• Implementing efficient rendering techniques, such as level-of-detail systems, streaming, and occlusion culling, can further help manage the performance impact of large photogrammetry assets in real-time applications