User Interfaces: Disappearing, Dissolving, and Evolving

Brown University

July 2002

Andries van Dam

• Why user interfaces are critical

• Computing environment trends:

ubiquitous, pervasive, invisible

• Immersive Virtual Reality and

Augmented Reality

• Research agenda

• Conclusion

Roadmap

Computing Capacity

t

Use this power to increase bandwidth to the brain

Human Capacity

t

ComputeCompute GraphicsGraphics

Why Do We Need Dramatic Improvements in the User Interface?

GUIs are the “killer app” for graphics.

Even young children who can’t yet read and write can be productive computer users.

An Unanticipated and Revolutionary Result of GUIs

But Are Today’s GUIs Good Enough?

Dan Robbins

Impedance-matchingLimitations of WIMP GUIs

Limited Vision (Flat, 2D)

No Speech

No Gestures

Limited Audio

One Hand Tied Behind Back

Limited Tactile

• None! UIs are a necessary evil

• Counterpoint: the aesthetics of a good UI

• Want to communicate and control as we do in and with the real world– objects

– tasks

– other participants (real and software agents)

• Models: Jeeves, HAL-9000

• Best today: transparency

• Future: Raj Reddy’s SILK, ultimately cogito ergo fac?

The Ultimate User Interface

• Why user interfaces are critical

• Computing environment trends:

ubiquitous, pervasive, invisible

• Immersive Virtual Reality and

Augmented Reality

• Research agenda

• Conclusion

Roadmap

• Multimodal post-WIMP interfaces

– parallel sensory channels (sight, hearing, haptics)

• Ubiquitous, pervasive, invisible computing

– profusion of form factors - embedded in ordinary everyday devices, and even in/directed by our bodies

– smart appliances, furniture, rooms, vehicles, jewelry…

o MIT LCS’s Oxygen,

o MIT Media Lab’s Smart Rooms, Things that Think,…

Computing Environment Trends (1/2)

• Federation of devices mediating human-human interaction

Computing Environment Trends (2/2)

• User is video-tracked for identification, location, gaze, gesture (and in the future, affect?)

• Continuous speech recognition:

– + natural language understanding + intelligent information processing allows dialog with intelligent assistants (agents)

• Furniture: chair is instrumented to help detect posture, (affect?), adjust to the user’s preferred position

Smart Office Scenario

• Wall, displays + personal notepads to provide private space in addition to any shared space(s)

• Implanted delivery mechanisms

• Prostheses (today, heart pacemakers, hearing aids, cochlear implants, voice boxes, artificial joints and organs…)

• Avatars of– health-care providers– family and friends, support group

Medical Scenario

• Electro-chemical sensors/probes (increasingly less obtrusive), to monitor stress, body chemistry, etc.– smart toilet to monitor bodily wastes

• Scaled-down office computing environment

• Why user interfaces are critical

• Computing environment trends:

ubiquitous, pervasive, invisible

• Immersive Virtual Reality and

Augmented Reality

• Research agenda

• Conclusion

Roadmap

Immersive Virtual Reality (IVR) & Augmented Reality (AR)

•Psychophysical sensation created by hardware and software

•Semi-immersive VR

Fishtank VR on a monitor

GMD’s Responsive workbench,Barco, Immersadesk™

Head-mounted displays (HMD’s)

CAVE™ and its derivatives

•Fully-immersive VR

•Augmented Reality (AR)Video or optics superimposes computer-generated data on real worlde.g., Columbia’s MARS

Uses of IVR and AR

• Workshops for today

– increasing use in design studios and laboratories

– artistic, therapeutic, and educational use increasing

– smart environments will make use of tools developed for IVR and AR

• Testbeds for the future– time machines that will become personally affordable, e.g.,

Elumens’ VisionStationCMU CUBE

• Thad Starner, formerly MIT Media Lab, Wearable Computing group, now at Georgia Tech

• Micro Optical Corporation, 320x240x8 LCD Display

• High- resolution Virtual Retinal Display (laser-based)at Dr. Furness’ HITLab@ U. of Washington

Augmented Reality: LCD Projection

CFD Scientific Visualization

• Blood flow through an arterial bypass graft

– fully 3D, time-dependent, viscous Navier-Stokes

– 3562 Spectral Elements: 1064 Prisms and 2498 Tetrahedra.

– only 50K triangles for artery wall, never came close to visualizing the LOD available from simulation results.

– this is a driving force for hierarchical, level-of-detail visualization

• Post-WIMP interaction

ARCHAVEVirtual Archeology

• Archeological research and analysis tool

• Walkthrough of excavation of Great Temple at Petra, Jordan– show excavation layers, trenches, to aid understanding of digging process

– abstract codings of multi- valued data– multi-scale navigation and visualization

• Analysis– 3D in situ presentation of artifacts

• Why user interfaces are critical

• Computing environment trends:

ubiquitous, pervasive, invisible

• Immersive Virtual Reality and

Augmented Reality

• Research agenda

• Conclusion

Roadmap

Research Agenda (1/3)

• User interfaces to impedance-match all our senses

– component technologies are wholly inadequate

• User interfaces to seamlessly integrate a federation of devices and users, through different environments

• Interaction styles to be task, user, and computing device/environment specific, unlike desktop WIMP

• Integration of direct and indirect (through agents) manipulation to be at a higher level of abstraction than point-and-click

• Knowledge representation techniques to deal with federations of devices and users

• Toolkits for these new styles of interfaces

Research Agenda (2/3)

• Privacy, noise, sensory and cognitive overload issues

• “Universal design” principles to address needs of different cultures, physical, and mental capabilities

• User studies to show what is effective

– formal

– useful

– application-specific

Research Agenda (3/3)

• Why user interfaces are critical

• Computing environment trends:

ubiquitous, pervasive, invisible

• Immersive Virtual Reality and

Augmented Reality

• Research agenda

• Conclusion

Roadmap

• Perceptual, cognitive, and social sciences and engineering

• Design disciplines: Industrial Design, User Interface Design, …

• Storytelling and communication arts: theater, film & video, advertising,…

• Computer science and engineering

• Early examples

– Pixar

– CMU’s Entertainment Technology Center

– USC’s Institute for Creative Technologies

– NPS’ MOVES (Modeling, Virtual Environments and Simulation) Institute

– UCF’s Digital Media program

Need a New Interdisciplinary Design Discipline

“To Infinity and Beyond…”

Recommended Reading

•“The Unifinished Revolution,” Michael Dertouzos, HarperCollins, 2001

•“The Next 1,000 Years,” Communications of the ACM, March 2001

•“Virtually There,” Jaron Lanier, Scientific American, April 2001

•“The Invisible Future,” Peter J. Denning, editor, McGraw Hill, 2001

The End