Interfaces of the Future, Available NowAugmented Reality Displays, Google Glass, 3D Sensors bringing computers into the 3D World.

March 10th, 2014

Where’s My Flying Car?

So many future promises from movies, so little delivery!

Minority Report Hand Gesture Interface, Now Available

Leap Sensor ($70) Demonstration

Finger and palm tracking, object translation, scaling and rotation.

Talk Outline● Consumer Technology Enabled Experiences

○ Augmented Reality○ Wearable & Pervasive Computing○ Gestural Control

● Technology○ 3D Sensing○ Displays○ Wearable

● New device survey● Societal Implications

Terminator vision

Sentrinsic Background● Formed in 2006 on patented linear sensing tech from

GaTech lab● Developed embedded control

system to improve efficiency of Air-Operated Diaphragm Pumps

Sentrinsic Background● Mobile Integration○ Bluetooth Pump Monitoring○ Customer iPhone app

■ Scan pumps for documentation, order parts, track maintenance

■ Product: EquipCodes.com

● Augmented Reality○ Maintenance○ Operation Theory○ Training

● Google Glass Manufacturing Assembly Training app

EquipCodes app

Augmented Reality● What is it?● Examples

○ 1st & 10 yellow line○ Simple text

■ translations■ nearby attractions■ instructions

Augmented Reality Demos● Pump Maintenance

http://www.youtube.com/watch?v=NYpl0PD5Le4http://www.youtube.com/watch?v=nHfY56lHZjU

● Circuit Building Guidehttp://www.youtube.com/watch?v=AfVQ4N-u0sk

Vein VisualizationEvana Eyes-On™ Glasses multi-spectral imaging

Enabling Technology● Improved cameras, processors, memory● Pose estimation

○ Fiducial tracking○ Texture feature-based tracking○ SLAM - Simultaneous Location and Mapping○ 3D Sensors

Fiducial Tracking

http://www.hitl.washington.edu/artoolkit/Papers/ART02-Tutorial.pdf ARToolKit Algorithm

Goal: find pose of camera (Rotation and Translation matrix)

Texture Feature Matching● Find interesting points (corners)

○ independent of orientation, scale, lighting● Use surrounding area to develop fingerprint of feature

(128 pt vector, robust to skew).● Based on neuron

receptors in visualcortex, sensitiveto spatial gradients

Distinctive Image Features from Scale-Invariant Keypoints, Lowe 2004

Also used for Image Stitching - Using Feature Recognition

Texture Tracking

Tracking Texture Features tracked using Vuforia SDK

Texture Feature Matching● Compare feature vector against database (Euclidian

distance).● Can’t be regular pattern (checker board)● High contrast required● Glare problematic● Tracker must be large

relative to graphics toprevent jitter, misalignment.

Extensible Tracking● Environment unknown beforehand● SLAM (Simultaneous Localization and Mapping) or PTAM (Parallel

Tracking and Mapping)○ Structure from Motion - Autodesk’s 123D○ Scene should be static○ For overlays, requires initial registration, not for games○ Videos: http://www.youtube.com/watch?v=i7ierVkXYa8 (3D structure from 2D pictures)○ http://www.youtube.com/watch?v=Y9HMn6bd-v8#t=50 (real-time map generation on iPhone)

● For overlays that must match exactly, still need some initial alignment.○ Edge Based tracking using 3D Models

■ Uses extensible tracking, initialized with best fit edge of model. (Metaio SDK, Sept, 2013)

○ Small tracker can still be used

Cameraless TrackingGPS, compass, gryo,

accelerometerIlluminated Tracking

CastAR’s head tracking uses IR LEDsNokia City Lens

Depth Sensors● Skips over many challenges with regular

camera tracking○ Directly measures object depth○ Can handle “featureless” textures (everything same color)○ Still requires object recognition○ Primarily used for body / hand tracking

● Examples○ Kinect, Leap Motion, Laser range finder

How Kinect (ver 1) works● Structured Lighting (Infrared)

video: http://www.youtube.com/watch?v=uq9SEJxZiUg

Kinect (ver 1) Drawbacks● Affected by Sunlight● Multiple kinects would conflict● Minimum distance several feet - bad for

robotics and hand tracking (low resolution)● Very cheap, though!

Time of Flight Sensor● Uses speed of light and arrival time of reflected pulses

to calculate distance● Extremely small times:

○ D = t * c / 2, 1cm → 66ps○ 3 GHz cpu → 333ps

● Advantages: more compact (no separation between sensor and emitter), simplier calculation, higher resolution

● Kinect v2, Soft Kinetic sensor.

3D Sensing Applications● Augmented Reality● Body tracking

○ general computer gestural control, more natural○ gaming - Dance games○ head tracking - CastAR, Virtual Reality○ 3D design / printing

Google Project Tango● Adding depth sensors to phones● Incorporating mapping and tracking

technology● Applications

○ visually impaired navigation○ instant measurement of

home dimensions○ store & shelf navigation

video: http://www.youtube.com/watch?v=Qe10ExwzCqk

Wearable Phones?

Many of the applications make more sense if you don’t have to hold a phone!

● Maintenance - hard to hold iPad with wrench!

● Mapping

Wearable / Pervasive Computing

● Google Glass○ Bad for AR?

■ Battery life■ Camera can’t be on constantly■ Small field of View, not in direct

line of sight○ Can’t focus on glass and outside

world - looking on wiki mid conversationjust as disruptive on Glass as on phone. (driving?)

Google Glass Field of View - Vignette Greg Roberts GlassFAQ

Google Glass Pros● We check our phones 110 times / day● Better for contextual information

○ Incoming emails and texts○ nearby places of interest○ Instructions - following and creating (mfr DEMO)

● Voice recognition is great● Capturing spontaneous or 1st person videos● Looks cool, lightweight, viable wearable● No tether

Head Mounted Display HistorySteve Mann

Thad Starner - GaTech!from 1993

Wearable Tech not new! Just more compact and marketed! Battery & CPU finally getting small and cheap enough.

More head mounted options● GlassUP

○ $300 ○ small screen○ tethered

● Recon Jetdesigned for Sporting$600, shipping May?

● Meta Pro○ Does what people think Google Glass does○ built-in depth sensor, potential

Iron Man / Minority report int.○ 40 degree Field of View

(vs 14 degree on Glass)○ $3650, shipping Sept○ tethered controller

CastAR & Spacial AR● Projection on reflective mat● Head tracking via LEDs● Hologram effect● form of Spatial AR

interaction w/ projections

Oculus Rift / Crystal Cove● Primarily for gaming, Virtual Reality● Fully immersive 110 deg Field of View● Head tracking via accel, gyro sensors,

markers on ver2● Some have added cameras to achieve

Augmented Reality effect○ Solves delay problems with see-thru displays

● Motion sickness problem - display must react quickly to head movement.

● Requires fast frame rates to trick brain

Video: Oculus Rift with 2 Cameras achieves Augmented Realityhttp://www.youtube.com/watch?v=Bc_TCLoH2CA

Technology Hurdles● Latency: see-through AR must be faster.

○ Google glass small screen prevents motion sickness, but lag will be more jarring in large screens

● Dimming screen○ per-pixel dimming not viable yet, see-through text can

get washed out. (Video pass-thru helps)● Better, Smaller, Faster (Battery, CPU, screen)● Object & Gestural recognition algorithms

Societal Implications● Always on camera● How do we have a conversation with computer

distractions? Wearable manners?● Engineering - Design in 3D with hands, more natural

gestural control● Job Creation?

○ Reverse of robotic automation - use adept human dexterity ○ Enhance human capabilities via internet, AI or remote assistance.○ Remote diagnostics, field service, DIY repair

Conclusion & Final Thoughts

● Consumer Computers sense and project in 3D space

● Technology more pervasive / always on via wearable implementations

● Content is major bottleneck● If useful, appearance won’t matter

Contact● Scott Driscoll

○ Scott.Driscoll@sentrinsic.com○ plus.google.com/+ScottDriscoll○ blog: ImponderableThings.com ○ youtube: CuriousInventor

● Company Website: EquipCodes.com● Software Tools

○ Vuforia, Unity, Metaio, Google App Engine