10.6 Virtual and augmented reality in photography

Virtual and augmented reality have revolutionized photography, offering immersive experiences that blur the line between real and digital worlds. These technologies build on early experiments in artificial environments and enhanced perception, evolving from panoramic paintings to sophisticated computer-generated graphics.

VR photography creates 360-degree images that simulate physical presence, while AR enhances real photos with digital elements. Both challenge traditional notions of framing and composition, opening new avenues for artistic expression and practical applications across various fields.

Origins of VR and AR

• Virtual and augmented reality technologies revolutionized photography by introducing immersive and interactive visual experiences
• VR and AR emerged from early experiments in creating artificial environments and enhancing real-world perception
• These technologies built upon foundational concepts in optics, computer graphics, and human-computer interaction

Early experiments in immersion

Top images from around the web for Early experiments in immersion

• 

  Realidade aumentada na educación View original

  Is this image relevant?

• 

  Panoramic painting - Wikipedia View original

  Is this image relevant?

• 

  DMS 423 - Class 6 View original

  Is this image relevant?

• 

  Realidade aumentada na educación View original

  Is this image relevant?

• 

  Panoramic painting - Wikipedia View original

  Is this image relevant?

1 of 3

Top images from around the web for Early experiments in immersion

• 

  Realidade aumentada na educación View original

  Is this image relevant?

• 

  Panoramic painting - Wikipedia View original

  Is this image relevant?

• 

  DMS 423 - Class 6 View original

  Is this image relevant?

• 

  Realidade aumentada na educación View original

  Is this image relevant?

• 

  Panoramic painting - Wikipedia View original

  Is this image relevant?

1 of 3

• Panoramic paintings in the 19th century provided 360-degree views, foreshadowing VR experiences
• Stereoscopes in the 1830s created illusions of depth using paired images
• Morton Heilig's Sensorama (1962) combined 3D video, audio, and other sensory inputs
• Ivan Sutherland's "Sword of Damocles" (1968) introduced head-mounted displays for computer-generated graphics

Precursors to modern VR

• NASA's Virtual Interface Environment Workstation (VIEW) in the 1980s developed VR for astronaut training
• Jaron Lanier coined the term "virtual reality" and founded VPL Research in 1984
• CAVE (Cave Automatic Virtual Environment) systems in the 1990s projected images on room-sized screens
• Nintendo's Virtual Boy (1995) attempted to bring VR gaming to consumers
  • Used red LED displays for 3D effects
  • Suffered from limited graphics and caused eye strain

Evolution of AR technology

• Ivan Sutherland's "The Ultimate Display" (1965) conceptualized computer-overlaid information in real-world views
• Steve Mann developed wearable computing devices in the 1980s, laying groundwork for AR
• Boeing researcher Tom Caudell coined the term "augmented reality" in 1990
• ARToolKit (1999) enabled marker-based AR on consumer devices
  • Used computer vision techniques to track fiducial markers
  • Allowed for real-time overlay of virtual objects on camera feed

Fundamentals of VR photography

• VR photography creates immersive, interactive visual experiences that simulate physical presence in real or imagined environments
• This field combines traditional photographic techniques with specialized equipment and software for capturing and displaying 360-degree imagery
• VR photography challenges conventional notions of framing and composition, requiring new approaches to visual storytelling

360-degree image capture

• Equirectangular projection maps spherical images onto rectangular frames
• Multiple cameras or specialized omnidirectional lenses capture full 360x180 degree field of view
• Stitching software aligns and blends overlapping images into seamless panoramas
• HDR techniques often employed to handle wide dynamic range in 360-degree scenes
  • Multiple exposures combined to preserve details in highlights and shadows

Stereoscopic vs monoscopic VR

• Monoscopic VR uses single 360-degree images viewed by both eyes
  • Simpler to produce but lacks depth perception
  • Suitable for many applications where immersion is primary goal
• Stereoscopic VR captures separate images for left and right eyes
  • Creates illusion of depth and 3D space
  • Requires specialized rigs with paired cameras or lenses
  • Viewing devices must support separate displays for each eye

VR camera types and setups

• Consumer 360 cameras (Ricoh Theta, Insta360) offer simple all-in-one solutions
• Professional VR rigs use multiple DSLR or mirrorless cameras in various configurations
  • Radial arrangements capture overlapping fields of view
  • Zenith and nadir cameras cover top and bottom views
• Light field cameras (Lytro Immerge) capture additional depth and parallax information
• Volumetric capture systems use multiple cameras to create 3D models of scenes or subjects
  • Enable six degrees of freedom (6DOF) movement in VR experiences

AR in photography

• Augmented reality enhances real-world photographic images with computer-generated elements
• AR photography combines physical and digital realms, creating interactive and dynamic visual experiences
• This technology opens new possibilities for creative expression and practical applications in various fields

Principles of augmented reality

• Computer vision algorithms detect and track real-world objects or markers
• 3D registration aligns virtual content with physical environment
• Real-time rendering overlays digital elements onto live camera feed
• Spatial audio and haptic feedback can enhance immersion in AR experiences
  • Sound sources tied to virtual objects' positions
  • Tactile feedback simulates interaction with AR elements

AR apps for photo enhancement

• Snapchat and Instagram filters apply real-time effects and overlays to selfies and photos
• Google Lens identifies objects and provides information overlays on camera view
• Adobe Aero allows creation and viewing of AR experiences using Photoshop assets
• Blippar enables interactive AR content triggered by printed images or products
  • Uses image recognition to identify trigger images
  • Overlays animations, 3D models, or additional information

Smartphone AR capabilities

• ARKit (iOS) and ARCore (Android) provide developer frameworks for mobile AR
• Depth sensors and LiDAR scanners improve AR object placement and occlusion
• Machine learning accelerates object recognition and tracking
• Simultaneous Localization and Mapping (SLAM) enables persistent AR experiences
  • Creates and updates 3D map of environment in real-time
  • Allows virtual objects to maintain position relative to real-world space

VR and AR in artistic expression

• Virtual and augmented reality technologies offer artists new mediums for creative exploration and expression
• These immersive technologies challenge traditional notions of artistic space, interactivity, and viewer participation
• VR and AR art often blurs boundaries between physical and digital realms, creating unique multisensory experiences

Virtual reality art installations

• Artists create fully immersive 3D environments explored through VR headsets
• Interactive elements respond to viewer movements and actions
• Time-based experiences can evolve or change during viewing session
• Tilt Brush and Medium allow artists to paint and sculpt directly in 3D space
  • Intuitive tools simulate traditional art materials in virtual environment
  • Artists can walk around and inside their creations

AR in conceptual photography

• Photographers incorporate AR elements to add layers of meaning to images
• QR codes or image recognition trigger additional content or animations
• AR apps reveal hidden elements or alternative versions of printed photographs
• Conceptual projects explore themes of reality, perception, and digital mediation
  • AR overlays can contrast with or complement physical images
  • Viewers become active participants in uncovering artwork's full content

Interactive photo exhibitions

• Gallery spaces enhanced with AR markers or beacons
• Visitors use smartphones or provided devices to access additional content
• Virtual docents or audio guides triggered by proximity to artworks
• AR wayfinding helps navigate complex exhibition layouts
• Social sharing features allow visitors to capture and share AR experiences
  • Encourages engagement and extends exhibition's reach beyond physical space

Technical challenges and solutions

• VR and AR photography face unique technical hurdles in capturing, processing, and displaying immersive content
• Ongoing research and development aim to improve image quality, reduce motion sickness, and enhance user experience
• Advancements in hardware, software, and computational techniques continually push the boundaries of what's possible in immersive media

Stitching and blending techniques

• Parallax errors occur when objects appear at different positions in overlapping images
  • Optical flow algorithms help align moving objects across frames
  • Multi-band blending smooths transitions between stitched images
• Ghosting artifacts caused by moving subjects in scenes
  • De-ghosting algorithms identify and remove duplicated elements
  • Temporal stitching techniques use information from multiple frames
• Exposure and color matching across different camera sensors
  • Global color correction algorithms harmonize overall image appearance
  • Local adjustment techniques address specific problem areas

Overcoming motion sickness

• Reducing latency between head movement and visual updates
  • Predictive tracking anticipates user movements
  • Asynchronous time warp renders intermediate frames
• Maintaining consistent frame rates (ideally 90 fps or higher)
• Implementing comfortable locomotion methods in VR
  • Teleportation avoids simulated motion
  • Vignetting or reducing field of view during movement
• Providing stable reference points in virtual environments
  • Virtual noses or cockpits can anchor user's view
  • Horizon lines help maintain sense of orientation

Improving image resolution

• Foveated rendering concentrates detail where user is looking
  • Eye-tracking determines gaze direction
  • Peripheral areas rendered at lower resolution to save processing power
• Light field displays create more natural focus cues
  • Multiple focal planes or holographic techniques
  • Reduces vergence-accommodation conflict
• Upscaling techniques enhance perceived resolution
  • AI-powered super-resolution algorithms
  • Temporal anti-aliasing smooths edges across frames

Applications in various fields

• VR and AR photography find diverse applications beyond artistic expression
• These technologies offer new ways to document, explore, and interact with real and virtual environments
• Immersive visual experiences enhance communication, education, and decision-making across industries

VR and AR in photojournalism

• 360-degree photos and videos provide context and immersion in news stories
• AR overlays add real-time data or historical context to live scenes
• Virtual reality experiences allow viewers to "witness" events firsthand
• Ethical considerations in immersive journalism
  • Potential for manipulation or misrepresentation
  • Balancing emotional impact with journalistic objectivity
• The New York Times' NYTVR platform showcases immersive reporting
  • Combines 360-degree video with traditional storytelling techniques
  • Covers topics from war zones to cultural events

Virtual tours and real estate

• 360-degree property tours allow remote viewing of homes and buildings
• AR apps overlay property information or visualize renovation possibilities
• VR enables exploration of unbuilt or historical structures
• Interactive floorplans with drag-and-drop furniture placement
• Matterport and similar platforms create detailed 3D scans of interiors
  • Allow for accurate measurements and space planning
  • Integrate with VR headsets for immersive walkthroughs

Education and training uses

• Virtual field trips to inaccessible locations or historical sites
• AR anatomy apps overlay internal structures on physical bodies
• VR simulations for hazardous or complex training scenarios
• Interactive AR textbooks with 3D models and animations
• Google Expeditions provides guided VR tours for classrooms
  • Covers diverse subjects from biology to art history
  • Teacher-led experiences with synchronized viewing

Impact on traditional photography

• VR and AR technologies challenge and expand conventional notions of photography
• These immersive media forms require new approaches to visual storytelling and composition
• The integration of interactive and dynamic elements transforms the relationship between image, creator, and viewer

Redefining composition techniques

• 360-degree images eliminate traditional framing constraints
• Consideration of viewer attention and movement in immersive spaces
• Guiding viewer focus through lighting, color, and point of interest placement
• Balancing visual information across entire spherical field of view
• Use of spatial audio cues to direct attention in VR environments
  • Sound sources can draw viewer's gaze to specific areas
  • Ambient audio enhances sense of presence and immersion

New perspectives on storytelling

• Non-linear narratives allow viewers to explore stories at their own pace
• Interactive elements enable branching storylines or deeper content exploration
• Combining still images, video, and computer-generated elements in hybrid experiences
• Challenges of maintaining narrative coherence in user-driven experiences
• Empathy-building through first-person perspectives in VR
  • Immersive documentaries place viewers in others' shoes
  • Ethical considerations in representing sensitive subjects or traumatic events

Ethical considerations

• Potential for manipulation and misrepresentation in immersive media
• Privacy concerns with capturing and sharing 360-degree environments
• Responsibility of creators to consider viewer comfort and safety in VR experiences
• Balancing artistic expression with ethical portrayal of subjects
• Transparency in disclosing use of AR enhancements or VR reconstructions
  • Clear labeling of computer-generated or altered elements
  • Providing context for historical or scientific visualizations

Future of VR and AR photography

• Rapid technological advancements continue to expand possibilities in immersive visual media
• Integration with other emerging technologies promises new forms of creative expression and practical applications
• The evolution of VR and AR photography will likely reshape how we capture, share, and experience visual information

Emerging technologies and trends

• Light field photography captures depth and directional information of light rays
  • Enables post-capture refocusing and perspective shifts
  • Potential for more realistic and interactive VR experiences
• Volumetric capture creates 3D models of scenes or subjects
  • Allows for six degrees of freedom (6DOF) movement in VR
  • Combines photogrammetry with video for dynamic 3D content
• Neural rendering uses AI to generate photorealistic images from minimal data
  • Could enable real-time creation of immersive environments
  • Raises questions about authenticity and ownership of AI-generated content

Potential for immersive narratives

• Interactive documentaries blend journalism with game-like exploration
• VR cinema experiments with viewer agency and multi-perspective storytelling
• AR-enhanced live performances merge physical and digital elements
• Location-based experiences tie virtual content to specific real-world places
• Social VR platforms enable shared immersive experiences
  • Virtual photo walks or collaborative art creation
  • Challenges of representing diverse avatars and ensuring inclusive spaces

Integration with other media forms

• Photogrammetry techniques create 3D models from 2D photographs
  • Enables reconstruction of real environments in virtual spaces
  • Applications in archaeology, preservation, and virtual tourism
• Combining VR/AR with haptic feedback for multi-sensory experiences
• Integration with brain-computer interfaces for direct neural interaction
• Holographic displays project 3D images without need for headsets
  • Potential for shared AR experiences in public spaces
  • Challenges in achieving high resolution and wide viewing angles

Notable VR and AR photographers

• Pioneering artists and technologists have pushed the boundaries of immersive visual media
• Their work explores the unique creative possibilities and challenges of VR and AR photography
• These innovators continue to influence the development and application of immersive technologies in visual arts

Pioneers in the field

• Scott Highton, early adopter of VR photography techniques in the 1990s
  • Developed methods for creating interactive panoramas
  • Authored guides on VR photography production
• Luc Courchesne, creator of interactive panoramic installations
  • "Landscape One" (1997) allowed viewers to navigate 360-degree video
  • Explored social interactions in shared virtual environments
• Michael Naimark, researcher and artist in immersive media
  • "Be Now Here" (1995) combined omni-directional video with motion platforms
  • Contributed to early development of Google Street View

Contemporary artists and innovators

• Marshmallow Laser Feast, collective creating immersive nature experiences
  • "Treehugger" uses VR to visualize tree biology and ecosystems
  • Combines scientific data with artistic interpretation
• Laurie Anderson, multimedia artist exploring VR storytelling
  • "Chalkroom" offers surreal VR environments with hand-drawn elements
  • Blends audio narration with interactive visual experiences
• Mat Collishaw, artist merging historical techniques with new technologies
  • "Thresholds" recreates 19th-century photography exhibition in VR
  • Combines physical installation with virtual reality overlay

Influential VR/AR photo projects

• "The Enemy" by Karim Ben Khelifa, VR journalism project
  • Presents interviews with combatants from opposing sides of conflicts
  • Uses volumetric capture to create lifelike 3D representations
• "Carne y Arena" by Alejandro González Iñárritu, VR installation
  • Immerses viewers in experiences of migrants crossing US-Mexico border
  • Combines physical set elements with virtual environments
• "Quantum Space" by Matteo Zamagni, AR-enhanced photography series
  • Explores visualization of quantum physics concepts
  • Uses smartphone app to reveal animated overlays on printed images