Lecture 3: History of VR

Question 1

When was the idea of stereoscopy already known? Who built the first stereoscope?

Answer 1

The idea of stereoscopy was already know
in the 19th century.

In 1838 Charles Wheatstone built his first
stereoscope.

Question 2

What is a stereoscope?

Answer 2

A stereoscope is a device used to view
stereoscopic images. Each eye is presented a different image to create the illusion of depth. Lenses help to make the image seem more distant.

(Remove the screens and tracking from
an modern HMD and you get a
stereoscope)

Question 3

What was the start of modern VR? Who is the inventor?

Answer 3

The “Sensorama” was proposed by
Morton Heilig in the 1950s.

Considered by many to be the first VR system!

Question 4

Which senses do Sensorama use?

Answer 4

Question 5

What is considered to be the first HMD?

Answer 5

Telesphere Mask: The “Telesphere Mask” proposed in the 1960s is considered to be the first HMD!

The Telesphere Mask is designed to:
* display stereoscopic content
* be an individual 3D TV
* use miniaturized TV tubes

Stereo audio is supported, but no
tracking of movements

Question 6

What was the Sketchpad? Who was the inventor?

Answer 6

Question 7

What was the Ultimate Display?
Who had the vision?

Answer 7

1965, Ivan Sutherland:

The ultimate display would, of course, be a room within which the computer can control the existence of matter. A chair displayed in such a room would be good enough to sit in. Handcuffs displayed in such a room would be confining, and a bullet displayed in such a room would be fatal. With appropriate programming such a display could literally be the Wonderland into which Alice walked.

Question 8

What was The Sword of Damocles?

Answer 8

HMD with Tracking:
A see through HMD that displays wireframe
models by Ivan Sutherland. The HMD is attached to the ceiling by the mechanical “sword of Damocles” tracking system.

Not strictly VR, but more of an AR system.
The ceiling carried the weight of the HMD!

Question 9

Who creates the Term “Virtual Reality”?

Answer 9

Commercial VR:
In 1984 Jaron Lanier founds VPL Research.
VPL is one of the first companies to sell
commercial VR products (dataglove, HMD, …).

In the late 1980s, Jaron Lanier also creates the
term “Virtual Reality”.

Question 10

Who conducted at the second half of the 1980s and the early 1990s several research projects into VR and why?

Answer 10

NASA

Primary interests for NASA were training simulations for astronauts.

Question 11

What was VIRTUALITY?

Answer 11

A company.

In the early 1990s several VR arcade machines were built. VIRTUALITY produced several games, including multiplayer games.

VIRTUALITY offered head- and hand-tracking and released a 2nd generation in the mid-90s.

Initial VIRTUALITY systems were powered by an Amiga 3000.

The resolution was 276x372 per LCD screen.

There was never a product for home-use

Question 12

What does CAVE stands for?
Briefly describe the concept behind it.

Answer 12

Question 13

What was the Virtual Boy?

Answer 13

Nintendo’s attempt.
In 1995 Nintendo released the Virtual
Boy as a standalone VR games console.

It could only display in red and was silently
discontinued. It was a commercial failure. SEGA tried something similar a few years
earlier. It never left the development phase.

Question 14

What turned more into focus from 2010? Give two product examples for the trend.

Answer 14

3D

Microsoft released the Kinect 3D camera.
Nintendo released the 3DS in 2011 featuring a autostereoscopic display.

Question 15

What were the beginnings of Oculus?

Answer 15

In 2012 Palmer Luckey started a Kickstarter campaign to fund Oculus and build a VR
headset designed for gaming.

Question 16

The first development kit of the Oculus Rift (DK1) was released in…
How many DOF were supported?

Answer 16

March 2013

It did not support full 6DOF tracking yet, but was limited to 3DOF (head rotation). Compared to the latest generation of HMDs,
the resolution was rather limited at 640x800
per eye.

Question 17

What were the improvements from Oculus Rift DK1 to DK2?

Answer 17

K2 was released in 2014 and featured several key improvements over DK1.

Improvements included:
* LEDs on the HMD to track positional changes
* Improved resolution at 960×1080 per eye
* One screen per eye

Question 18

Briefly sketch the history of modern HMDs from 2015 until 2019

Answer 18

Question 19

What is the difference between VR and AR in Stereoscopic Rendering?

Answer 19

VR:
For VR, conveying depth information is considered a crucial part!

A lot of VR hardware (HMDs, VR
displays, …) is designed around
presenting stereograms to users!

AR:
AR is less focused on depth information.
However, some dedicated AR devices (Microsoft Hololens, Magic Leap, …) rely on stereoscopy!

Question 20

What means Rendering a stereo-image-pair?

Answer 20

Stereoscopic rendering means rendering 2 images: One image for the left eye, and one image for the right eye.

Image parameters need to be set correctly for each image to create the illusion of depth!

Question 21

Which parameters (relevant to us) do we need to pay attention to?

Answer 21

Question 22

What should we keep in mind regarding Positioning the camera when rendering an image for a certain eye?

Answer 22

When rendering an image for a certain eye (left or right), we need to position the camera with an offset corresponding to the current eye.

• Head position may be retrieved from tracking.
• Offset can be adjusted according to IPD.
• Offset is not fixed along one axis, but takes head rotation into account.

Question 23

What is the Frustum?

Answer 23

For our purpose, a frustum is a pyramid with the top cut off.

In computer graphics, the frustum denotes the space in front of the camera that contains the area visible to the camera.

When rendering stereoscopic images, the frustum may or may not be symmetric.
What we have to use depends on the device we are rendering images for.

Question 24

Sketch a frustum for a single camera, with a projection plane normal to the viewing direction and a field of view centered around it.

Answer 24

The frustum defined by this setup is symmetric around the view vector.

Question 25

With stereoscopic rendering we have two frustums (one per image). So just double those? Sketch the frustums for this simplified case.

Answer 25

It depends on the output device!
We need to know where the displays
are located to know what our frustums
look like.

Worst case:
3D TV/screens where both images are
displayed on the same screen

Question 26

Toe-In

Let us assume we render for a 3D TV: a single screen plane shared for both eyes. Sketch the frustums for this case.

Why does the following sketch do not show the ideal solution for this case?

Answer 26

Rotating will not work in this case.
For a 3D TV or 3D display we are projecting onto a shared plane (=screen), not two planes at an angle! So this is not the ideal solution!

The solution is Assymmetry.
The viewing direction is parallel, but the frustum is defined by the output screen! We are rendering both images for a single
screen plane. Commonly used for projector or screen based systems with active or passive stereo separation.

Question 27

How do we manage to present content in a way, that each eye can only see the image intended for it? Describe the technique briefly.

Answer 27

With Multiplexing.
Multiplexing is a method or technology
that combines multiple signals into a single communication channel. Like presenting multiple images using only the visual channel.

Question 28

How can we multiplex images?
Give at least four examples.

Answer 28

• Colour multiplexing
• Polarisation multiplexing
• Spatial multiplexing
• Temporal multiplexing
  …

Question 29

Explain the technique behind Anaglyph / Colour filters. What are Pros and Cons ?

Answer 29

The images for left and right eye are
stored in a single image, each with a
different colour filter. The coloured “lenses” of the 3D-glasses separate the images.

Cheap glasses, easy to use, but brightness and colour can be an issue!

Question 30

Explain Polarisation Filters and the technique behind it

Answer 30

It is Common in 3D cinemas.
both images are projected onto a screen. Each image is filtered through a polarisation
filter. The 3D-glasses are equipped with polarisation filters as well, separating images.

Polarised 3D-glasses are cheap and easy
to use. Commonly uses circular polarisation as
linear polarisation would make images bleed into each other when the head is tilted!

Question 31

Describe Spatial Multiplexing

Answer 31

A screen for each eye..
Images are either shown on a single screen, but each eye can only see “its” region of the screen or there is a separate screen for each eye. Separation makes sure that each eye cannot see the screen or screen region for the other eye!

This is how it works for almost any modern HMD!

Question 32

Name two approaches for Autostereoscopic displays and briefly explain the term. What is the disadvantage of this technique?

Answer 32

Autostereoscopic display:
An autostereoscopic display can show stereograms without glasses, etc.

The displays have dedicated pixels for left
and right eyes. Lenses or barriers make sure
each eye only sees one image.

Disadvantage:
They only work well in the sweet spots!

Question 33

What means Temporal Multiplexing? What do we need to make sure? Give an example which device we can use for it.

Answer 33

A single screen is alternating between the image for the left and right eye, showing each image only for a few milliseconds.

We need to make sure that each image is only seen by the eye it is intended for!

Example Device:
Shutter glasses are active devices where each lens can switch to either be transparent or opaque. Glasses need to synchronise with
display! One eye is always “blanked out”, separating images.

Question 34

Describe Outside-In Tracking

Answer 34

Question 35

Describe Inside-Out Tracking

Answer 35

Question 36

How do we collect positional data?
What are requirements for a tracking system?

Answer 36

There are a number of different methods to collect positional data: mechanical,
magnetical, optical, inertial, etc.

Each method has their own advantages and disadvantages in terms of precision,
latency, ease-of-use, etc.

Ideally we want a tracking system that is easy to use, very precise, has very low latency
and does not restrict our movement!

Question 37

Give a example for Mechanical Tracking and describe it.

What are advantages and disadvantes?

Answer 37

The “Sword of Damocles” system was a
mechanical arm that not only carried the HMD, but also measured the HMD’s rotation.

Mechanical tracking usually relies on an arm with one or more joints that is attached at a reference position.

A mechanical tracking system is usually based around a mechanical arm. The “arm” has a number of static pieces connected by joints.
When the tracked device moves, the joints’ rotation changes. Given the reference point and the joints’ rotation, we can track the device or object.

Advantage: fast and accurate
Disadvantage: it can be quite restrictive!

Question 38

Describe Magnetic Tracking. What are here advantages and disadvantages?

Answer 38

Magnetic tracking works by measuring the strength of a magnetic field. Given three orthogonal magnetic fields from a known source, we can accurately measure the
position and orientation of a tracked device.

Advantage: relatively precise, does not need line of sight and allows for a relatively large tracking space.

Disadvantage: Magnets and metal objects can cause issues, though!

Question 39

Describe Optical Tracking

Answer 39

Most modern HMDs are equipped with at least an optical tracking system.

As the name indicates, it relies on optical sensors (cameras, photodiodes, etc.) to track devices.

An optical tracking system requires line-of-sight since it is based on light!

Question 40

Describe Inertial Tracking. What are advantages and disadvantages?

Answer 40

Another way to generate positional data is using inertial measurement units (IMUs).
IMUs like accelerometers, gyroscopes, etc. are very common today and can be found
in any smartphone.

Advantage: IMUs are cheap, small and have very low latency

They do have one big disadvantage: drift

Question 41

Explain the issue of IMUs drift.

Answer 41

IMUs do not generate absolute positional data. Each measurement gives us a value of change relative to our last measurement.
Each measurement has a tiny error. Over time these errors accumulate and cause a discrepancy between our positional data and the real world position.

Since IMUs can be polled very quickly (up to 1 000 times per second in modern VR systems), even a very small error in measurement can sum up over time!

Question 42

We know about different tracking methos and ideally we want a “best of both (all?) worlds” approach. How can we reach that?

Answer 42

Sensor Fusion (Combine it)

Sensor fusion is an approach that uses at least two, maybe even more tracking methods and combines them to exploit the advantages of multiple methods and hide the disadvantages.
Most modern HMDs rely heavily on tracking sensor fusion!

Question 43

Which two tracking methods are commonly used for sensor fusion in modern HMDs?

Answer 43

Question 44

What does sensor fusion allow us? (regarding optical and inertial sensors)

Answer 44

• Use the absolute, precise positional data from the optical system.
• Do the tracking via IMUs while we wait for the next measurement from the optical system!

The optical system does what it can do best: high precision. The IMUs compensate for the latency.

The IMUs do what they can do best: low latency. The optical system corrects the drift.