1

2

Where ARWhere ARWe Heading?We Heading?

Steven Feiner

Computer Graphics & User Interfaces LabDepartment of Computer ScienceColumbia University New York, NY 10027

Supported in part by NSF, ONR, USMC, and gifts fromMicrosoft, Google, Vuzix

Augmented Reality Conference / eComm America 2010 San Francisco Airport Marriott, April 19–21, 2010

3

Over 40 Years of AR Research,Over 40 Years of AR Research,…… Ivan Sutherland,

Head-tracked VR/AR(Late 60s–early 70s) Stereo, head-tracked, optical

see-through, head-worn display Graphics overlaid on real world

I. Sutherland, A Head-Mounted Three-Dimensional Display, Proc. AFIPS Fall Joint Comp. Conf., 757–764, 1968

4

Over 13 Years of Mobile AR Over 13 Years of Mobile AR Research,…Research,…

Columbia“Touring Machine” 1996–

AR Campus Tour, 1997

5

Over 13 Years of Mobile AR Over 13 Years of Mobile AR Research,…Research,…

Columbia“Touring Machine” 1996–

AR Historical Hypermedia Documentary, 1999

6

Over 13 Years of Mobile AR Over 13 Years of Mobile AR Research,…Research,…

Columbia“Touring Machine” 1996–

AR Restaurant Guide, 2001

7

Flash Forward to 2008, 9, 10,…Flash Forward to 2008, 9, 10,…

Commodity mobile devices Small Powerful Inexpensive Networked

Sensors Ubiquitous

Content Proliferating

Wikitude et al., 2008 –

Columbia MobileAugmented RealitySystem, 1996–

2001

2001

8

Consider Mobile AR,…Consider Mobile AR,…

Where are We Now?Where are We Now?

9

Issues: Inaccurate TrackingIssues: Inaccurate Tracking

A-GPS position inaccuracy

Compass orientation inaccuracy

10

Issues: Inaccurate TrackingIssues: Inaccurate Tracking

Phone CPU/GPU/display better in 2010, but,…

Backpack position and orientation tracking better in 1999 RTK GNSS Hybrid IMU

11

Issues: Inaccurate TrackingIssues: Inaccurate Tracking

Better sensors Different sensors

(vision,…) Sensor fusion

E.g., using abetter orientation tracker

Y. Tokusho, M. Sukan, et al., 2009

12

Issues: Incorrect Field of ViewIssues: Incorrect Field of View

Display FOV too small

Camera FOV too large Not 1:1

Not parallax free

http://flickr.com/photos/click-see/25803640/

13

Cult of the hand-out “Excuse me while I whip

out my phone, turn it on, hold it at arm’s length, wait until it’s started tracking…”

Issues: Handheld LifestyleIssues: Handheld Lifestyle

Y. Tokusho et al., 2009

S. Güven et al., 2006

http://labs.lastminute.com/nru/ Nokia

14

Issues: Handheld LifestyleIssues: Handheld Lifestyle

The promise of eyewear Hands-free

Always on

Must be Small

Light

Bright / high contrast

Comfortable

Visually appealing

Affordable

Lumus

Konica Minolta

Sony

Prototype optical see-through eyewear

15

Eyewear-Based Task AssistanceEyewear-Based Task Assistancegraphics.cs.columbia.edu/projects/armargraphics.cs.columbia.edu/projects/armar

S. Henderson & S. Feiner, 2009

16

Eyewear-Based Mobile GamesEyewear-Based Mobile Gamesgoblinxna.codeplex.comgoblinxna.codeplex.com

O. Oda & S. Feiner, 2009

17

Issues: UI Design for ARIssues: UI Design for AR

Design presentations dynamically Respect the real

world!

E.g., automated layout accounting for occlusion

B. Bell et al., 2001

18

WhatWhat’s Next?’s Next?

19

UbiquitousUbiquitous

When we stop asking

“What’s the Killer App?”

20

HeterogeneousHeterogeneous N. Dedual & S. Feiner, 2010 N. Dedual & S. Feiner, 2010

Surface Publicly

visible Multitouch

+ See-through

HWD Private Stereo

Benefit from advantages of complementary displays and devices

21

CollaborativeCollaborative

E.g.,

Collaborative Tracking/Modeling

22

Collaborative Tracking/ModelingCollaborative Tracking/Modeling

Users Want to know their location and “interesting

information”

System Wants to model the world (including users and

their “interesting information”)

23

Collaborative Tracking/ModelingCollaborative Tracking/Modeling

User: Provides Current

Sensor data: Imagery, audio, (A-)GPS, compass, inertial measurements, radio signal strength, GNSS carrier phase,…

Historic Known position/orientation Sensor data: …

Relevant “Interesting information” System

Uses sensor data and history from user and others to match against database

Returns User’s (precise) current position/orientation Relevant “Interesting information”

Updates database

24

UsersUsers

“Just plain folk”

http://jonathan.tenkely.org/wordpress/wp-content/gallery/gallery/VectorPeople.png

25

SystemSystem

“The usual suspects”

Individual providers for private sites might make agreements with larger organizations

26

ResultsResults

Up-to-the-second Street {View, Side,…} Current shop window

contents Current pedestrian

traffic Not just those who opt

in, but others through face/gait/… recognition “Publicly observable

behavior”

pedestrians

27

ResultsResults

Up-to-the-second world model Made possible by

uploaded sensor data Fine-grained history

Provides past record Makes predictions

possible The ultimate social

network And, you will be

part of it

28

Privacy Issues?Privacy Issues?

http://foursquare.com Where are you now?

http://plancast.com Where are you going?

http://blippy.com What are you buying?

http://everythingithink.com What are you thinking?

29

AcknowledgmentsAcknowledgments

Blaine Bell Hrvoje Benko Gábor Blaskó Franz Coriand Nick Dedual Steve Henderson Sinem Güven Tobias Höllerer Eddie Ishak Levi Lister Blair MacIntyre Ohan Oda

John Pavlik Mengu Sukan Yoshitaka Tokusho Tony Webster Sean White Lauren Wilcox Ryuji Yamamoto Supported in part by NSF,

ONR, USMC, and gifts from Microsoft, Google, Vuzix

Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation

See you at ISMAR 2010, Oct 13-16Seoul, Korea, www.ismar10.org