4.4 Mixed reality and holographic displays

Mixed reality blends physical and virtual worlds, creating immersive experiences where digital content interacts with real environments. From augmented reality to fully virtual experiences, MR technologies enable users to perceive and interact with virtual objects that appear to coexist within the real world.

Holographic displays create 3D images that float in mid-air, providing more realistic viewing experiences than traditional 2D displays. These displays use various techniques to manipulate light, enabling depth and parallax without special glasses or headsets. Advances in this technology are crucial for creating engaging MR experiences.

Mixed reality fundamentals

• Mixed reality (MR) combines elements of both the physical and virtual worlds, creating experiences where digital content interacts with and responds to the real environment
• MR technologies enable users to perceive and interact with virtual objects that appear to coexist within the real world, blurring the line between reality and virtuality
• Understanding the fundamentals of mixed reality is crucial for creating immersive and engaging experiences in virtual and augmented reality art

Merging physical and virtual

Top images from around the web for Merging physical and virtual

• 

  This is What the Future of Mixed Reality Looks Like View original

  Is this image relevant?

• 

  Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original

  Is this image relevant?

• 

  Building Industry 4.0 With Mixed Reality – Alice Bonasio – Medium View original

  Is this image relevant?

• 

  This is What the Future of Mixed Reality Looks Like View original

  Is this image relevant?

• 

  Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original

  Is this image relevant?

1 of 3

Top images from around the web for Merging physical and virtual

• 

  This is What the Future of Mixed Reality Looks Like View original

  Is this image relevant?

• 

  Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original

  Is this image relevant?

• 

  Building Industry 4.0 With Mixed Reality – Alice Bonasio – Medium View original

  Is this image relevant?

• 

  This is What the Future of Mixed Reality Looks Like View original

  Is this image relevant?

• 

  Immersive Tech Transforming Learning – AR/VR Journey: Augmented & Virtual Reality Magazine View original

  Is this image relevant?

1 of 3

• Mixed reality seamlessly blends the physical and virtual worlds, allowing digital content to interact with and respond to the real environment in real-time
• Virtual objects can appear to be anchored to specific locations in the physical world, maintaining their position and orientation relative to real-world objects
• Users can interact with virtual content using natural gestures, voice commands, and even physical objects, creating a more intuitive and immersive experience

Spectrum of mixed reality

• The mixed reality spectrum encompasses a range of experiences, from entirely virtual environments (virtual reality) to the integration of digital content into the real world (augmented reality)
• Augmented reality (AR) overlays digital information onto the real world, typically through a smartphone or tablet camera (Pokémon Go, Google Lens)
• Augmented virtuality (AV) brings real-world elements into a primarily virtual environment, such as incorporating live video feeds or physical objects into a VR experience
• True mixed reality experiences sit in the middle of the spectrum, allowing virtual and real-world content to interact and respond to each other in real-time (Microsoft HoloLens, Magic Leap)

Key characteristics of MR

• Spatial awareness: MR devices can understand and map the physical environment, allowing virtual content to interact with and respond to real-world objects and surfaces
• Real-time interaction: Users can interact with virtual content using natural gestures, voice commands, and even physical objects, with the virtual content responding in real-time
• Persistence: Virtual objects can maintain their position and orientation in the real world, even when not actively being viewed by the user
• Occlusion: Virtual objects can be occluded by real-world objects, enhancing the illusion that they exist within the physical environment

Holographic display technology

• Holographic displays create three-dimensional images that appear to float in mid-air, providing a more immersive and realistic viewing experience compared to traditional 2D displays
• These displays use various techniques to manipulate light, enabling the creation of depth and parallax without the need for special glasses or headsets
• Advances in holographic display technology are crucial for creating more realistic and engaging mixed reality experiences in virtual and augmented reality art

Light field displays

• Light field displays capture and reproduce the light rays emanating from a scene, allowing for more realistic and natural-looking 3D images
• These displays use an array of micro-lenses or other optical elements to create the illusion of depth and parallax, enabling users to view the image from different angles
• Examples of light field display technology include the Looking Glass and the Leia Display, which provide glasses-free 3D viewing experiences

Volumetric displays

• Volumetric displays create three-dimensional images by illuminating points in a volume of space, allowing users to view the image from any angle without the need for special glasses
• These displays use various techniques, such as rotating LED arrays, laser projection, or stacked liquid crystal panels, to create the illusion of a solid 3D object floating in mid-air
• Examples of volumetric display technology include the Voxon VX1 and the Holoxica 3D Display, which provide true 3D viewing experiences

Holographic optical elements

• Holographic optical elements (HOEs) are thin, transparent films that can manipulate light to create the illusion of three-dimensional images
• HOEs use diffraction patterns recorded on a photosensitive material to bend and shape light, enabling the creation of 3D images that appear to float in mid-air
• Examples of HOE technology include the HoloLens 2 and the WaveOptics waveguides, which use HOEs to project virtual images onto the user's retina

Challenges in holographic displays

• Creating high-resolution, full-color holographic displays with a wide field of view remains a significant challenge
• Current holographic display technologies often suffer from limited viewing angles, low brightness, and high power consumption
• Scaling holographic displays to larger sizes while maintaining image quality and reducing costs is another major hurdle
• Developing efficient and accurate methods for capturing and rendering real-world scenes as holographic content is an ongoing area of research

Interaction in mixed reality

• Interaction in mixed reality involves using natural and intuitive methods to manipulate and engage with virtual content seamlessly integrated into the real world
• MR systems must be able to understand and respond to user input in real-time, providing a sense of presence and agency within the mixed reality environment
• Developing effective interaction techniques is crucial for creating engaging and immersive mixed reality experiences in virtual and augmented reality art

Gaze, gesture, and voice input

• Gaze tracking allows MR systems to determine where the user is looking, enabling more natural and intuitive interaction with virtual content (eye tracking in HoloLens 2)
• Gesture recognition enables users to interact with virtual objects using hand and body movements, providing a more immersive and tactile experience (hand tracking in Oculus Quest)
• Voice input allows users to control and interact with virtual content using natural language commands, reducing the need for physical controllers or gestures (voice commands in Magic Leap)
• Combining gaze, gesture, and voice input can create more seamless and intuitive interaction experiences in mixed reality environments

Haptic feedback in MR

• Haptic feedback provides tactile sensations to users, enhancing the sense of presence and immersion in mixed reality experiences
• Haptic devices can simulate the feeling of touching, grasping, or manipulating virtual objects, providing a more realistic and engaging interaction experience
• Examples of haptic feedback in MR include the HaptX Gloves, which use microfluidic actuators to provide realistic tactile sensations, and the Teslasuit, which provides full-body haptic feedback

Spatial mapping and understanding

• Spatial mapping involves creating a digital representation of the physical environment, allowing MR systems to understand and interact with the real world
• MR devices use cameras, depth sensors, and computer vision algorithms to create 3D maps of the environment in real-time, enabling virtual content to be accurately placed and anchored within the real world
• Spatial understanding allows MR systems to recognize and respond to real-world objects and surfaces, enabling more natural and intuitive interactions between virtual and real content

Collaborative experiences in MR

• Mixed reality enables multiple users to interact with shared virtual content in real-time, fostering collaboration and social interaction
• Collaborative MR experiences can range from simple shared viewing of virtual content to complex multi-user interactions and co-creation of virtual environments
• Examples of collaborative MR experiences include Microsoft Mesh, which enables users to interact with shared 3D content in real-time, and Spatial, which provides a platform for virtual meetings and collaboration in MR

Mixed reality devices

• Mixed reality devices combine hardware and software components to create immersive experiences that blend virtual content with the real world
• These devices use various display technologies, sensors, and input methods to enable users to perceive and interact with virtual content seamlessly integrated into their physical environment
• Understanding the capabilities and limitations of current MR devices is essential for creating effective and engaging virtual and augmented reality art experiences

Microsoft HoloLens

• The Microsoft HoloLens is a self-contained mixed reality headset that uses holographic waveguide displays to project virtual images onto the user's retina
• The device features an array of sensors, including depth cameras, eye trackers, and inertial measurement units (IMUs), which enable spatial mapping, gaze tracking, and gesture recognition
• The HoloLens runs on the Windows Mixed Reality platform, which provides a framework for developing and deploying MR applications using tools like Unity and Unreal Engine

Magic Leap

• Magic Leap is a mixed reality headset that uses a proprietary lightfield display technology to create highly realistic and immersive 3D images
• The device features an array of cameras and sensors for spatial mapping and hand tracking, as well as a handheld controller for more precise input
• Magic Leap runs on the Lumin OS, a custom operating system designed specifically for mixed reality applications, and provides a suite of development tools for creating MR content

Differences vs VR and AR devices

• Mixed reality devices like the HoloLens and Magic Leap provide a more immersive and interactive experience compared to traditional AR devices, which typically rely on smartphone or tablet displays
• Unlike VR devices, which fully immerse users in a virtual environment, MR devices allow users to see and interact with the real world while simultaneously engaging with virtual content
• MR devices often feature more advanced spatial mapping and understanding capabilities compared to AR devices, enabling more seamless integration of virtual content with the real world

Limitations and future developments

• Current MR devices are often limited by their display resolution, field of view, and battery life, which can impact the quality and duration of MR experiences
• The high cost of MR devices like the HoloLens and Magic Leap can be a barrier to widespread adoption, particularly in consumer markets
• Future developments in MR technology are likely to focus on improving display quality, increasing field of view, and reducing the size and weight of MR devices
• Advancements in 5G networks and edge computing could enable more sophisticated and responsive MR experiences, particularly for multi-user and collaborative applications

Applications of mixed reality

• Mixed reality has the potential to revolutionize a wide range of industries and domains, from education and entertainment to healthcare and industrial design
• By blending virtual content with the real world, MR can provide more engaging, interactive, and effective experiences compared to traditional media and technologies
• Exploring the diverse applications of mixed reality is crucial for understanding its potential impact and identifying new opportunities for virtual and augmented reality art

Education and training

• Mixed reality can enhance learning experiences by allowing students to interact with 3D models, simulations, and virtual environments in a more immersive and hands-on way (HoloAnatomy for medical education)
• MR training applications can provide realistic and safe simulations for high-risk or complex tasks, such as surgery, equipment maintenance, or emergency response (HoloLens for military training)
• Collaborative MR experiences can enable remote learning and training, allowing students and instructors to interact with shared virtual content in real-time (Prisms for math education)

Entertainment and gaming

• Mixed reality gaming experiences can blend virtual content with the real world, creating more immersive and interactive gameplay (Minecraft Earth, Pokémon Go)
• MR can enhance traditional entertainment experiences, such as movies and theme parks, by providing interactive and personalized content that responds to the user's actions and environment (The Void, Dreamscape Immersive)
• Collaborative MR experiences can enable social gaming and entertainment, allowing users to interact with shared virtual content in real-time (Spatial for virtual concerts)

Industrial design and visualization

• Mixed reality can streamline the product design and development process by allowing designers and engineers to visualize and interact with 3D models in real-time (HoloLens for automotive design)
• MR can enable remote collaboration and design reviews, allowing teams to work together on shared virtual models and prototypes from different locations (PiXYZ Review for design collaboration)
• MR visualization tools can help communicate complex designs and concepts to stakeholders and customers, providing a more engaging and interactive experience compared to traditional media (Magic Leap for architecture visualization)

Healthcare and medical imaging

• Mixed reality can enhance medical imaging by allowing healthcare professionals to visualize and interact with 3D patient data, such as CT scans and MRIs, in a more intuitive and immersive way (HoloLens for surgical planning)
• MR can enable remote collaboration and consultation between medical experts, allowing them to share and discuss patient data and treatment plans in real-time (Medivis for surgical collaboration)
• MR training applications can provide realistic simulations for medical procedures and emergency response, allowing healthcare professionals to practice and refine their skills in a safe and controlled environment (CAE VimedixAR for ultrasound training)

Developing for mixed reality

• Developing for mixed reality involves creating applications and experiences that blend virtual content with the real world, taking into account the unique capabilities and constraints of MR devices and platforms
• MR development requires a combination of skills and tools, including 3D modeling, game engine programming, spatial computing, and user experience design
• Understanding the key concepts and best practices of MR development is essential for creating effective and engaging virtual and augmented reality art experiences

Unity and Unreal Engine for MR

• Unity and Unreal Engine are popular game engines that support mixed reality development, providing tools and frameworks for creating MR applications and experiences
• Both engines offer built-in support for MR devices like the HoloLens and Magic Leap, as well as plugins and extensions for additional functionality (Mixed Reality Toolkit for Unity, Unreal Engine 4 XR Plugin)
• Unity and Unreal Engine provide visual scripting tools (Bolt for Unity, Blueprints for Unreal) that can make MR development more accessible to non-programmers and artists

Spatial anchors and world locking

• Spatial anchors are virtual objects that are anchored to specific locations in the real world, allowing MR content to maintain its position and orientation relative to the physical environment
• World locking techniques ensure that virtual content remains stable and aligned with the real world, even as the user moves around or the environment changes
• Platforms like Azure Spatial Anchors and ARCore Cloud Anchors provide cloud-based services for creating and managing spatial anchors across multiple devices and sessions

Performance optimization techniques

• Optimizing performance is critical for creating smooth and responsive MR experiences, particularly on mobile devices with limited processing power and battery life
• Techniques for optimizing MR performance include reducing polygon count and texture resolution, using efficient shading techniques, and minimizing the use of physics and particle effects
• Asynchronous loading and streaming can help manage the memory footprint of MR applications, while spatial partitioning techniques like occlusion culling can reduce the rendering overhead

Best practices for MR design

• Designing for mixed reality requires a different approach compared to traditional 2D or even VR interfaces, taking into account the unique characteristics of MR devices and the user's physical environment
• Best practices for MR design include creating content at an appropriate scale and distance from the user, providing clear visual and audio feedback, and designing for 360-degree interaction
• Usability testing and user feedback are essential for refining MR designs and ensuring a smooth and intuitive user experience
• Accessibility considerations, such as providing alternative input methods and accommodating different user abilities and preferences, are important for creating inclusive MR experiences

Perceptual considerations in MR

• Perceptual considerations in mixed reality involve understanding how the human visual system processes and interprets virtual content blended with the real world
• MR experiences must take into account factors such as depth perception, stereoscopy, and the accommodation-convergence conflict to create a comfortable and convincing illusion of virtual objects coexisting with the real environment
• Addressing perceptual issues and ensuring user comfort and safety are critical for creating effective and engaging virtual and augmented reality art experiences

Depth cues and stereoscopy

• Depth cues are visual and non-visual signals that help the brain perceive the relative distance and position of objects in the environment, such as occlusion, perspective, and motion parallax
• Stereoscopy is the technique of presenting slightly different images to each eye to create the illusion of depth and 3D space, mimicking the way the human visual system naturally works
• MR devices like the HoloLens and Magic Leap use stereoscopic displays to create a convincing illusion of virtual objects existing in the real world, enhancing the sense of presence and immersion

Accommodation-convergence conflict

• The accommodation-convergence conflict occurs when the focus distance (accommodation) and the vergence distance (convergence) of the eyes do not match, leading to visual discomfort and fatigue
• In MR displays, the accommodation distance is fixed at the display plane, while the convergence distance varies depending on the virtual content, potentially causing a mismatch and visual strain
• Techniques for minimizing the accommodation-convergence conflict include using multiple focal planes, adjusting the virtual content to match the display plane, and providing user controls for adjusting the focal distance

Perceptual illusions in MR

• Perceptual illusions can occur in mixed reality when the virtual content and the real world do not align or interact in a convincing or consistent manner, breaking the illusion of presence
• Examples of perceptual illusions in MR include occlusion violations (virtual objects appearing to pass through real objects), incorrect shadows or lighting, and inconsistent scale or perspective
• Minimizing perceptual illusions requires careful design and calibration of the MR experience, taking into account factors such as the user's position, the lighting conditions, and the characteristics of the physical environment

User comfort and safety

• Ensuring user comfort and safety is essential for creating successful and enjoyable MR experiences, particularly for extended or repeated use
• Factors that can impact user comfort in MR include visual fatigue, motion sickness, and physical strain from prolonged use or awkward postures
• Safety considerations in MR include avoiding virtual content that could cause users to collide with real-world objects, providing clear boundaries and warnings, and allowing users to easily exit the experience if needed
• Designing for user comfort and safety requires testing and feedback from a diverse range of users, as well as adherence to established guidelines and best practices for MR development

Future of mixed reality

• The future of mixed reality holds tremendous potential for transforming the way we interact with digital content and the world around us, from education and entertainment to healthcare and industrial applications
• Advancements in display technology, spatial computing, and artificial intelligence are likely to drive the development of more sophisticated and immersive MR experiences in the coming years
• As MR