5.2 Optical see-through and video see-through AR displays

AR displays come in two main flavors: optical see-through and video see-through. Optical displays let you see the real world directly, with virtual stuff overlaid on top. They're more natural but can struggle with alignment and occlusion.

Video displays show you a camera feed of the world instead. They give more control over the final image, making it easier to line things up and handle occlusion. But they can feel less natural and have more lag than optical displays.

Optical See-Through Displays

How Optical See-Through Displays Work

Top images from around the web for How Optical See-Through Displays Work

• 

  Frontiers | Optical See-Through Head-Mounted Displays With Short Focal Distance: Conditions for ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

• 

  Frontiers | Optical See-Through Head-Mounted Displays With Short Focal Distance: Conditions for ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

1 of 3

Top images from around the web for How Optical See-Through Displays Work

• 

  Frontiers | Optical See-Through Head-Mounted Displays With Short Focal Distance: Conditions for ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

• 

  Frontiers | Optical See-Through Head-Mounted Displays With Short Focal Distance: Conditions for ... View original

  Is this image relevant?

• 

  Frontiers | Virtual, Real or Mixed: How Surrounding Objects Influence the Sense of Embodiment in ... View original

  Is this image relevant?

1 of 3

• Optical see-through displays allow the user to see the real world directly through a transparent or semi-transparent display while virtual content is overlaid on top
• Employ optical combiners (half-silvered mirrors or holographic optical elements) to merge the real and virtual views
• Light from the real world passes through the combiner while light from the display is reflected into the user's eyes, creating a composite view
• Typically have a limited field of view (FOV) due to the size constraints of the optical combiner and display elements (Microsoft HoloLens, Meta 2)

Waveguide Displays in Optical See-Through Systems

• Waveguide displays are a type of optical see-through display that use a thin, transparent waveguide to guide light from a small display engine to the user's eyes
• Light is injected into the waveguide at one end and propagates through total internal reflection until it reaches the other end, where it is coupled out towards the user's eyes
• Allow for a more compact and lightweight design compared to traditional optical combiners
• Can achieve a wider FOV by using multiple waveguides stacked together or by employing a single waveguide with multiple input and output regions (Vuzix Blade, Google Glass)

Advantages and Challenges of Optical See-Through Displays

• Provide a direct view of the real world, resulting in a more natural and immersive experience
• Eliminate the need for complex camera systems and image processing required for video see-through displays
• Face challenges in achieving accurate registration between virtual content and the real world due to the lack of control over the real-world view
• Struggle with occlusion handling, as virtual content always appears overlaid on top of the real world, regardless of depth (virtual objects cannot be occluded by real objects)

Video See-Through Displays

How Video See-Through Displays Work

• Video see-through displays capture the real world using one or more cameras and display the video feed on a screen in front of the user's eyes
• Virtual content is rendered and composited with the video feed, creating an augmented view of the real world
• Allow for complete control over the final displayed image, enabling more accurate registration and occlusion handling compared to optical see-through displays
• Commonly used in smartphone-based AR applications and some head-mounted displays (Samsung Gear VR, Oculus Rift AR mode)

Registration in Video See-Through Displays

• Registration refers to the accurate alignment of virtual content with the real-world video feed
• Achieved through computer vision techniques, such as feature detection, tracking, and pose estimation
• Markers (fiducial markers) or natural features in the environment can be used as reference points for registration
• Accurate registration is crucial for creating a seamless and convincing AR experience, as misalignment can break the illusion of virtual objects being part of the real world

Occlusion Handling in Video See-Through Displays

• Occlusion handling involves correctly displaying the depth relationships between virtual and real objects
• In video see-through displays, occlusion can be handled by using depth information from cameras (stereo cameras, depth sensors) or by manually creating occlusion masks
• Depth information allows the system to determine which objects should be in front of or behind others, enabling proper occlusion rendering
• Occlusion masks define the regions where virtual objects should be visible or hidden based on the real-world geometry
• Accurate occlusion handling enhances the realism and immersion of the AR experience, as virtual objects appear to seamlessly integrate with the real environment