Input Devices

Universidade de Aveiro Departamento de Electrónica, Telecomunicações e Informática

Virtual and Augmented Reality 2017/18 Beatriz Sousa Santos

What is Virtual Reality?

“A high-end user interface that involves real-time simulation and interaction through multiple sensorial channels.” (vision, sound, touch, smell, taste) (Burdea and Coiffet., 2003)

2

Input Devices: Trackers, Navigation and gestures interfaces

3

Input devices • Trackers:

– Magnetic (AC, DC) – Optical – Ultrasonic – Inertial, – Mechanical – Hybrid ...

• Navigation and manipulation interfaces:

– Tracker-based – Trackballs – 3D mice, ...

• Gesture interfaces:

– Gloves – Depth cameras ...

4

Virtual objects have 6 degrees of freedom (D.O.Fs):

-three translations;

-three rotations.

- Roll – rotation around the longitudinal axis (Burdea and Coiffet., 2003)

Trackers measure the motion of “objects” (e.g. user head) in a fixed system of coordinates.

Tracker is a special purpose H/W to measure the real-time change in a 3D object position and orientation

5

Example: 3D magnetic sensor in a HMD

Without the head tracker - the image - the sound cannot change to match the head posture Required tracking accuracy: Image > sound

(Burdea and Coiffet., 2003)

6

Body Tracking:

• Head

• Hand and fingers

• Torso

• Feet

• A group of people, ...

Indirect tracking: Using physical objects (props and platforms)

Technologies:

• Electromagnetic

• Optical

• Ultrasonic

• Inertial

• Mechanical

• Hybrid ...

What is usually tracked? How?

7

Tracker characteristics:

• Measurement rate – Readings/sec

• Sensing latency

• Sensor noise and drift

• Measurement accuracy (errors)

• Measurement repeatability

• Tethered or wireless

• Work envelope

• Sensing degradation

8

Tracker performance parameters:

• Accuracy

• Jitter

• Drift

• Latency

• Tracker update rate

9

Tracker performance parameters should be analyzed to match a solution for sensorial channel and budget of an application

Tracker characteristics

Real object position

Accuracy

Tracker position

measurements

10

(Burdea and Coiffet., 2003)

Accuracy: Difference between the object’s actual 3D position and that reported by the measurement

Tracker characteristics

Real object position

Accuracy

Resolution

Tracker position

measurements

11

(Burdea and Coiffet., 2003)

Resolution:

“the smallest amount of the quantity being measured that the instrument will detect.”

(used by Ascension)

(Polhemus uses a different definition )

Real object fixed

position

Signal noise

Time

Tracker data

12

(Burdea and Coiffet., 2003)

Jitter: Change in tracker output when the tracked object is stationary

Real object fixed

position

Sensor drift

Time

Tracker data

13

Drift: Steady increase in tracker error with time

(Burdea and Coiffet., 2003)

Real object

position

Sensor latency

Time

Tracker data

14

Latency: Time delay between action and result: time between the change in object position/orientation and the time the sensor detects this change

(Burdea and Coiffet., 2003)

15 (Burdea and Coiffet., 2003)

Tracker update rate: Number of measurements that the tracker reports every second If the same tracker electronics is used to measure several objects, the sampling rate suffers due to multiplexing

Dedicated electronics to each tracked object

Same electronics to each tracked object

16

Most used trackers:

• Magnetic

• Ultrasonic

• Optical

• Inertial

• …

Magnetic Trackers A magnetic tracker is a non-contact position measurement device that uses a magnetic field produced by a stationary TRANSMITTER to determine the real-time position of a moving RECEIVER element may be AC DC

17

Magnetic Trackers

• Use low-frequency magnetic fields to measure position

• Fields are produced by a fixed source

• Size of source grows with the tracker work envelope

• The receiver is attached to the tracked object and has three

perpendicular antennas • Distance is inferred from the voltages induced in the antennas – needs

calibration…

18

19

(Burdea and Coiffet., 2003)

Magnetic tracker with the old Data Glove

Magnetic tracker accuracy degradation

23

(Burdea and Coiffet., 2003)

Comparison of AC and DC magnetic trackers

• DC trackers are immune to non-ferromagnetic metals

(brass, aluminum and stainless steel)

• Both DC and AC trackers are affected by the presence of ferromagnetic metals

(mild steel and ferrite)

• Both are affected by copper

• AC trackers have better resolution and accuracy

• AC trackers have slightly shorter range

24

A “standard” for motion tracking for years:

25

Polhemus

APPLICATIONS: High Accuracy Head Tracking EEG Localization Training and Simulation Eye Tracking Neuroscience Biomechanics

(proprietary AC electromagnetic technology)

Widely used and expensive:

26

Ascension

Tracking, guiding, and localizing medical instruments within a patient's body

3D measurement and analysis of human movement for biomechanical purposes: sports, performance, and design

Track head and objects for matching computer-generated imagery with head direction; weapon aiming and interactive instruction

Tracking of head, hands and 3D pointers for interaction with large scale and immersive displays

Example of a magnetic tracker

28

Aurora Adds customizable real-time spatial measurement technology to simulation systems to deliver realistic surgical navigation or weapons targeting simulation in all six degrees of freedom (6DOF).

29

Example of “Low cost” magnetic tracker:

30

• True six degree-of-freedom magnetic motion tracking

• Precise sensor for 1mm and 1 degree tracking

• No line of sight to controllers required

• Low-power magnetic field

• low power consumption

• Low latency feedback

Razer Hydra

31

Razer Hydra Technical Specifications - Per controller Thumb-ergonomic analog stick for fluid control - 4 Hyperesponse action buttons - Rapid-fire trigger and bumper for faster in-game response - Non-slip satin grip surface - True six degree-of-freedom magnetic motion tracking - Lightweight, anti-tangle braided cable Base station - Low-power magnetic field, low power consumption - Ultra precise sensor for 1mm and 1 degree tracking - No line of sight to controllers required - Low latency feedback - Approximate Size : 120 mm / 4.92” (Length) x 120 mm / 4.92” (Width) x 100 mm / 3.94” (Height) - Approximate Weight : 800 g / 1.76 lbs

Ultrasonic Trackers

A non-contact position measurement device that uses an ultrasonic signal produced by a stationary transmitter to determine the real-time position/ orientation of a moving receiver.

32

(Burdea and Coiffet., 2003)

Ultrasonic Trackers

• Use low-frequency ultrasound to measure position

• Ultrasound produced by a fixed triangular source (speakers)

• Number of sources grows with the tracker work envelope

• The receiver is triangular and attached to the tracked object and has

three microphones

• Distance is inferred from the sound time of flight

• Sensitive to air temperature and other noise sources

• Requires “direct line of sight”

• Slower than magnetic trackers (max 50 updates/sec)

33

Ultrasonic tracker (Logitech)

(Burdea and Coiffet., 2003)

http://www.vrdepot.com/vrteclg.htm

6 DOFs Update rate - 50 / sec Latency – 30 ms

34

Optical Trackers A non-contact position measurement device that uses optical sensing to determine the real-time position/ orientation of an object

35

(Burdea and Coiffet., 2003)

outside-looking-in inside-looking-out

Outside-looking-in

LaserBIRD optical tracker

36

Sub-degree and sub-millimeter accuracy Measurement rate of 240 meas/sec Tracking response: 7.17 ms

https://www.ascension-tech.com/products/

Outside-looking in Vicon MX https://www.vicon.com/

• Uses 4 Mpixel cameras with own 120 LED

array (infrared, or visible red). Accuracy 0.02 of a pixel

• Camera has real-time onboard image processing (masking and thresholding)

• Resolution 2352x1728 @ 160 fps

• 8 cameras are connected to a MX net unit which then communicates with the PC

• User wears reflective markers (small spheres)

(Burdea and Coiffet., 2003)

• The best accuracy is close to the work envelope • Very large tracking surface and

resistance to visual occlusions (line of sight)

38

Inside-out optical tracker advantages/ disadvantages

(Burdea and Coiffet., 2003)

39

HTC Vive “Lighthouses”

https://www.vive.com/eu/

Hybrid Ultrasonic/Inertial Trackers

• No interference from metallic objects

• No interference from magnetic fields

• Large-volume tracking

• “Source-less” orientation tracking

• Full-room tracking

40

But…

• Accelerometer errors a lead to decreased accuracy

• Errors grow geometrically in time!

• Gyroscope errors compound position errors

• Needs independent position estimation to reduce “drift”

41

44

https://www.virtalis.com/tracking-systems/

Hybrid Ultrasonic/Inertial Tracker

6 DOFs Utrasonic – position Inertial - orientation Sub-millimiter accuracy Head + hand units Used for: - Training - Assembly/ disassembly and design review - …

Mechanical Trackers A mechanical tracker consists of a serial or parallel kinematic structure composed of links interconnected by sensorized joints.

46

(Burdea and Coiffet., 2003)

Mechanical tracker - Push 1280 stereo display (Fakespace Inc)

Item is no longer available

Mechanical Trackers

Pros

• Use sensors imbedded in exoskeletons to measure position

• Have extremely low latencies

• Are immune to interference from magnetic fields and large metal objects

Cons

• Limit the user’s freedom of motion

• Can be heavy if worn on the body

47

Example of an exoskeleton (mechanical tracker)

48

http://www.youtube.com/watch?v=uJza6G-7tD4

Painting a virtual wall: example of a virtual rehabilitation task for a patient recovering from stroke or traumatic brain injury

49

Other devices can be used to track

(at low cost)

https://www.leapmotion.com/

https://developer.microsoft.com/en-us/windows/kinect

(Jerald., 2016)

Navigation and Gesture Input Devices

• Navigation interfaces allow relative position control of virtual objects

(including a virtual camera)

• Gesture interfaces allow dexterous control of virtual objects and interaction

through gesture recognition.

50

Navigation and manipulation Input Devices

• Trackballs

• 3D mice

• 3D probes

• Wiimote, …

• Perform relative position/velocity control of virtual objects

• Allow “fly-by” application by controlling a virtual camera

51

The Cubic Mouse (research, not commercial)

52

http://www.youtube.com/watch?v=1WuH7ezv_Gs http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.6127&rep=rep1&type=pdf

- Allows to intuitively specify 3D- coordinates

- The rods represent the X, Y, and Z axes of a given coordinate system

53

3D mouse Space controller

http://www.spacecontrol.us/3d-maeuse-spacecontroller-funktionsprinzip.html http://www.3dconnexion.com/products/what-is-a-3d-mouse.html http://www.youtube.com/3DMaus

• Sends relative coordinates

• Allows

- to move 3D objects in a more intuitive way than a 2D mouse

- allows rotation around each axis

• Price ~300 USD

3D-Spheric-Mouse

http://www.engadget.com/gallery/axsotic-3d-mouse-takes-a-spherical-stab-at-an-age-old-problem/#slide=616049

- Has no mechanical sensors that can generate unwanted behaviors

- The specially developed polymer

springbodies create a smoothly consistent workflow

Gesture Input Devices

• There are/ have been various sensing gloves such as:

- Fakespace Pinch Glove (switches)

- Immersion CyberGlove (stain gauges),

- Gloveone …

• Have larger work envelope than trackballs/3-D probes

• Most need calibration for user’s hand

55

5DT Data Glove

CyberGlove

Pinch Glove

Item is no longer available

Gloveone

http://www.cyberglovesystems.com/sdk-videos/2015/9/17/vhsdk-sample-app-video

~

The CyberGlove

http://www.cyberglovesystems.com/

57

Other devices can be used to detect gestures

https://www.leapmotion.com/

https://developer.microsoft.com/en-us/windows/kinect http://kinectvr.com/

58

Speech recognition is also an interesting possibility:

• Frees hands

• Allows multimodal input

• Specialized software

• Issues: recognition, ambient noise, training, false positives

An input device “providing an infinit VE”: a treadmill for VR

59 https://www.youtube.com/watch?v=oWIDqebGUqE

May have applications, behond gaming: promote physical exercise, train people, …

60

Virtusphere (“the VR hamster ball”)

https://www.youtube.com/watch?v=2e5Qvac3BB8

62

Conclusion

When choosing a device, consider:

• Cost

• Generality

• DOFs

• Ergonomics / human factors

• Typical scenarios of use

• Output devices

• Interaction techniques

Input + output CyberTouch Glove

63

http://www.cyberglovesystems.com/cybertouch/#photos

64

Input + output GloveOne

https://www.neurodigital.es/gloveone/

https://www.kickstarter.com/projects/gloveone/gloveone-feel-virtual-reality

Main bibliography

- Jerald, J., The VR Book: Human-Centered Design for Virtual Reality, ACM and Morgan & Claypool, 2016

- G. Burdea and P. Coiffet, Virtual Reality Technology, 2nd ed. Jonh Wiley and Sons, 2003

- Craig, A., Sherman, W., Will, J., Developing Virtual Reality Applications: Foundations of Effective Design, Morgan Kaufmann, 2009

- J. Vince, Introduction to Virtual Reality, Springer, 2004

65