05 - Mobile Augmented Reality und Head Mounted Displays Flashcards
Was sind die Basiskomponenten eines AR Systems?
HW computational platform Display Tracking Wireless network Wearable input and interaction Software
Welche mobilen Displays werden für AR verwendet?
smartphones, tablets, konsolen, mobile projektoren
Was sind Backpack Setups?
Rucksäcke mit kompletten AR System. (semiportabel)
Warum werden Backpack Setups noch immer verwendet?
In VR more computational power is often required
Strong desire for untethered experiences
Strong desire for graphics workstations
Could be as well used for AR, especially when using AR video based HMDs
AR HMDs partially try to distribute the weight of the HMD on the head or also move the compute unit away from the head to the belt
Warum werden/wurden Mobile Devices in der AR verwendet?
Communication via Bluetooth or WiFi
Datasets are often stored externally
Processing power in PDAs and mobile consoles was sufficient for simple ARToolKit applications
If rendering power is to low
remote rendering and streaming of rendered image is possible as well
Possible to attach external screens, e.g. HMD, clip on systems,
With current devices setups are significantly more easy
Welche Mobile Devies werden verwendet?
Smart phones
iPhone, Android based phones
Often good sensor support
Tablets
Good API support
Similar development than smart phones
Mobile consoles (e.g. Gizmondo, PS Vita, Nintendo 3DS)
Provide good rendering performance
Additional controllers
Was war das Tango Projekt?
● Software architecture coupled with hardware devices
● Hardware is focused on smart phone and tablet market
● Relies on scene reconstruction with SLAM also using Time of Flight tracking
● Succeeded by Pixel series and terminated through AR Core
Welche Kommunikationsrechnologien werden bei Mobile Devices eingesetzt?
WPAN (Bluetooth, Infrared) - Short-range (10m)
WLAN - Mid-range (100m)
WWAN (LTE, GSM, UMTS, GPRS)- Wide coverage
Problems with latencies (e.g. UMTS round trip time 300-580ms ) and bandwidth
Wofür werden diese Kommunikationstechnologien in Mobile Devies verwendet?
GSM and UMTS
WLAN with the help of received signal strength indication (RSSI) by multiple access points
Was sind die Herausforderungen/Probleme bei den Mobile AR Devies?
Limited computational resources, battery life
Size, weight, robustheit(industrie)
Tracking and registration, 3D graphics and real-time performance
Social acceptance and mobility
Welche technischen Probleme treten bei Head-Mounted Displays auf?
- Systemlatenz
- Szenenauflösung
- Field of View
- Viewpoint Matchig
- Engineering and Cost Factors
hauptsächlich hardware abhängig
Auf welche Effekte des Auges muss man aufpassen bei der Enwicklung von VR/AR Brillen?
Binocular disparity - difference in image location of an object seen by the left and right eyes
Vergence - movement of the eyes to map an object on the corresponding side of the retina
Accommodation - change of lens to bring object at certain depth in focus
Welche Technologien gibt es im Stereo Display Bereich (Seperation of images)?
Spatial multiplexing - head-mounted displays
Colour multiplexing - anaglyph
Polarization multiplexing - polarisation filter
Time-sequential or shutter techniques - shutter glasse
Wie ist ein Head Mounted Display aufgebaut?
● Displays mounted inside a helmet or other kind of headgear
● Used to be Liquid Crystal Display (LCD) in earlier days Cathode Ray Tubes (CRT), today OLED and LCOS
● Head position and orientation is typically tracked (6DOF)
Warum gibt es bei Head Mounted Display ein Hygiene Problem?
Close contact to skin and eyes, users tend to sweat
For presentations or multiple users always clean the display ideally provide covers
Was sind typische technische Spezifikationen von Head Mounted Displays?
Weight (450g to 550g)
Resolution (1080×1200 pixels per eye to 1440x1600 per eye – 2560x1440 with dual display)
FOV (100° to 200° with dual display architecture)
Calibration of optics is important (interpupillary distance, exchangeable lenses)
Optics
F 14 Welche optischen Linsen werden in einem Head Mounted Display verwendet?
Fresnel lenses
Compresses space in HMD by keeping the same focal length
Leads to distortions
Special optics can be used to display the image as close as possible to the eye – Large Expanse Enhanced Perspective (LEEP)
F 14 Was ist der Screen Door Effect?
Depending on the resolution or better the distance between pixels (pixel gaps)
The determining factor is the gap size during magnification, resolution is not immediately related to it
Resolution is significantly increasing and the gap sizes are decreasing
Welche Head Mounted Displays gibt es?
Oculus Rift, HC Vive, Microsoft MR headsets
F 16 Was sind optical see through HMDs?
Partially transmissive/reflective optical combiners
Similar approach to Head-Up Displays (HUD)
Light from the real world is reduced (e.g. 30% transmitted)
Could be designed to omit real-world light at a certain wavelength
Level of blending is design issue
Optical blending requires no additional computing power
F 16 Was sind video see through HMDs?
Closed-view HMD with one or two cameras
Possible to fully occlude real world
Combination is performed by a video compositor
Much more flexible than optical see-through displays in compositing strategies
Used for diminished reality or mediated reality
F 16 Was sind video see through HMDs?
Closed-view HMD with one or two cameras
Possible to fully occlude real world
Combination is performed by a video compositor
Much more flexible than optical see-through displays in compositing strategies
Used for diminished reality or mediated reality
Chroma-keying often used for combination of real and virtual images (known from blue box, green box)
Use of depth information would be helpful in order to allow real objects cover virtual ones (e.g. combination Leap Motion and Oculus Rift)
Separate video streams are needed for real and virtual world, thus computationally more expensive than optical blending
Resolution is limited, thus resolution of the real world objects is reduced
Artificial offset due to position of recording cameras
Interpupillary distance (IPD) may not be exact
Additional registration strategies can be applied through the use of the video stream
Temporal mismatches between real and virtual image can be matched
Was sind die Vorteile von optical see through HMDs?
Simplicity
Optical blending is computationally much cheaper than video blending
One stream is rendered in optical blending, two with video based systems
Roughly one frame delay with video systems is added
Field Of View (FOV) is not a major issue with optical blending
Video cameras produce distortion, which has to be compensated
Resolution
Video blending limits the resolution to the display resolution
With optical combiners only the CG generated resolution is limited
Display of the real world is not degraded
Safety
In case of power loss a direct view on the real world is still available
Depending on the application area this can be an issue (e.g. fire fighting)
No eye offset
With video displays the cameras are located in most cases not directly in front of the eyes
IPD is often different with video see-through
Mirrors might be used to overcome the offset problem, adds complexity to the design of the HMD
Was sind die Vorteile von video see through HMDs?
Flexibility in compositing strategies
With optical see-through objects can not fully obscure real world
CG objects are always semi-transparent in optical see-through systems
With video blending composition can take place on a pixel-by-pixel basis
Additional registration strategies
Video stream provides additional source of information for registration
Easier to match brightness of real and virtual objects
With real world a high and dynamic range of brightness has to considered
If real environment is too bright the CG image will be washed out
If real world image is too dark CG image will wash out the real world image
Limited FOV provides a disadvantage
Complex optics add weight and can be expensive
Distortions of real world image have to be corrected optically with optical blending
Real and virtual view delays can be matched
Temporal mismatch problems can be avoided
Video stream can be delayed for matching CG stream
Was sind Sulon Cortex und Vrvana Totem?
Stereo cameras integrated for see-through mode
Camera calibration is non-trivial, offsets are to be considered
Vrvana acquired by Apple in 2017
HMD ist video based
Auf mixed reality ausgelegt
F 22 Was ist die Varjo XR1?
● Varjo XR1
Extension of the VR1 with additional cameras
● Illustrates advantages of video based see-through
Larger FOV Adaptation of lighting situation
Merged video stream provides better contrast
● Current trend to move away from dedicated MR displays and integrate functionality in existing VR displays
F 22 Was ist die Varjo XR1?
● Varjo XR1
Extension of the VR1 with additional cameras
● Illustrates advantages of video based see-through
Larger FOV Adaptation of lighting situation
Merged video stream provides better contrast
● Current trend to move away from dedicated MR displays and integrate functionality in existing VR displays
Was sind Smart Glasses?
- optical see through
Typically light weight and comfortable to wear
Small FOV, focus on information representation, rather than registered augmentations
Current devices applicable for specific use case
Welche Smart Glasses gibt es und welche Spezifikationen haben sie?
● Epson Moverio BT-300
Stereoscopic OLED display HD camera with 6h of battery life
Android based App development, external Android device
● Google Glass
Monoscopic 640×360 LCoS Display
Variety of sensors (e.g. gyro, accelerometer, compass)
Focus also on speech recognition Second Edition available from summer 2017 till November 2019
Neuer:
● Vuzix Blade
Uses touch display and voice recognition for interaction
Camera supports 720p 30fps or 1080p 24fps
Waveguide DLP based display
Android based OS, ARM CPU on device
Neuer:
● Vuzix M400 Smart Glasses
Monocular device using video blending
Powerbank as counter weight
Can be mounted on helmets or safety glasses
Ruggedised design
Welche Verfahren gibt es um optical HMD zu ermöglichen?
Prism Lenses
Output from miniature projector is projected into special prism lens, where light propagates via internal reflection and refraction
E.g. Google Glass
WaveGuide Arrays
Light is projected inside an array of wave guides
Different approaches of extracting light of wave guides into the eye
Holographic extraction
Diffractive extraction
Half-silvered mirrors
E.g. Meta2
Was macht die HoloLens aus?
Based on see-through waveguide lenses (optical)
Untethered
Big advance in terms of tracking and registration
Welche Viewing Area hat die HoloLens?
Focal plane at around 2m
Ideal object placement between 1.25m and 5m
Welche Field of View Rextrictions hat die HoloLens?
FOV restrictions (30°)
Object permanence might be broken due to limited FOV
Remember FOV constraint when designing applications
Peripheral vision is not available with HoloLens augmentations
Welche Interaktionsmöglichkeiten hat die HoloLens?
Gaze, Gesture, and Voice
Speech (with 4 microphones quite possible to filter out ambient noise)
Clicker (connected via Bluetooth)
Gesture recognition (abstract gestures)
Gaze based selection, AirTap for clicking
Welche technischen Spezifikationen hat die HoloLens?
Price 3.000$
Weight: 579g
Resolution: 2.3M total light points results in roughly 2x1268x720
FOV: 29.1×16.5° diagonal 33.4°
Sensors:
IMU
4 environment understanding cameras
1 depth camera
1 2MP photo / HD video camera
Mixed reality capture
4 microphones
1 ambient light sensor
Welche Features und Limitationen hat das headmounted display Meta2?
Tethered, try to become untethered Large FOV Different interface approach Uses Unity for development - optical blending - Haben versucht die Brille physikalischer zu machen, dass man direkt mit etlichen handgesten Sachen natürlicher verschiebt/verändert/damit interagiert
Welche technischen Spezifikationen hat die Meta2 Brille?
Price 1.000$
Weight: 420g (without straps and cables)
Resolution: 2x1280×1440
FOV diagonal 90°
Welche Interaktionsmöglichkeiten hat die Meta2?
Direct interaction (pushing buttons on menus, grabbing objects by showing a fist)
Supposedly still pretty basic
Problem with objects at distance
Issues with tracking stability and latencies
Was ist das DAQUIRI HMD?
Smart helmet (rugged design)
Comes with camera and sensing equipment like IR projectors and IR camera
Additional thermal camera
Intel RealSense
- optical hmd
Was ist das ODG R-7 HMD?
Dual 720p Stereoscopic See-through displays at up to 80fps
Variety of sensors including altitude and humidity
Full HD camera
- optical hmd
Was ist die MagicLeap One und was zeichnet sie aus?
Bi-focal display to reduce focus problem
Eye-tracking used to select appropriate focal plane
Compute unit external moved away from head (Lightpack)
Side areas blinded, which gives the impression of a larger FOV
6DOF Controller with magnetic tracking
Available in two versions for different IPDs
FOV slightly higher than MS HoloLens
128 GB (actual available storage capacity 95GB)
Up to 3 hours continuous use
Vast amount of funding (roughly 2.4 billion US$)
Support of many artists and companies (e.g. Sigur Rós, WETA)
- optical hmd
Was ist die HoloLens2?
● HoloLens2
Announced 24.02.2019 Pre-order: $3500
FOV 52° Diagonal
● Eye tracking for measurement and calibration of the IPD also interaction
● Available sometime 2019
● Weight with compute unit has been moved into the back of the device that the overall weight is balanced putting the centre of gravity in the middle of the head
● Semantical analysis of the environment
● Use of over 20 gestures for interaction
● Gesture tracking is reliable
XX Welche Brillen verwenden optical blending?
HoloLens 1 und 2, Magi Leap One, ODG R-7, DAQIRI, Meta2
Was sind die größten Unterschiede zwischen Smart Devices, Smart Glasses und HMDs?
● Smart Devices (tablets, phones)
Provide intuitive and to the user well kown interaction
Have to be held
Mainly monoscopic
● Smart glasses
Low weight
Reduced interaction possibilities
Very low FoV
● HMDs
Hands-free interaction
Encumbrance of the user
Stereoscopic
Low FoV
- noch nicht industrietauglich
