Upload
damita
View
86
Download
0
Embed Size (px)
DESCRIPTION
Human Computer Interaction Institute School of Computer Science Carnegie Mellon University. Mobile Devices for Control. Brad A. Myers [email protected] http://www.cs.cmu.edu/~pebbles. Mobile Devices. Mobile Devices for communication As a “Personal Information Manager” - PowerPoint PPT Presentation
Citation preview
Mobile Devices for ControlMobile Devices for Control
Brad A. [email protected]
http://www.cs.cmu.edu/~pebbles
Brad A. [email protected]
http://www.cs.cmu.edu/~pebbles
Human Computer Interaction Institute Human Computer Interaction Institute
School of Computer ScienceSchool of Computer Science
Carnegie Mellon UniversityCarnegie Mellon University
Brad Myers 2
Mobile Devices
Mobile Devices for communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement
“PocketPC”
For Remote Control of other devices Remote Control of PCs Remote Control of appliances
Mobile Devices for communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement
“PocketPC”
For Remote Control of other devices Remote Control of PCs Remote Control of appliances
Brad Myers 3
Problem
Appliances are too complex Appliances are too complex
Brad Myers 4
Problem
Too many remotes Too many remotes
Brad Myers 5
Problem
User Interfaces of Mobile Devices themselves are too hard to use
User Interfaces of Mobile Devices themselves are too hard to use
Brad Myers 6
Problem
PC’s have only a keyboard and mouse as input devices
PC’s have only a keyboard and mouse as input devices
Brad Myers 7
Approach
Use a personal Mobile Device as an Interface to the PC and to the Appliances
Use a personal Mobile Device as an Interface to the PC and to the Appliances
Brad Myers 8
History
ParcTab – continuous communication
Early “organizers” – no communication
Newton – extra cost for synchronization
Palm – HotSync once a day
WindowsCE/PocketPC –ActiveSync once a day
ParcTab – continuous communication
Early “organizers” – no communication
Newton – extra cost for synchronization
Palm – HotSync once a day
WindowsCE/PocketPC –ActiveSync once a day
Brad Myers 9
Now, Handhelds will be Communicating
802.11 (“Wi-Fi”)
BlueTooth
Cell-phone network (G3)
(Infrared)
(Wires or cradle)
802.11 (“Wi-Fi”)
BlueTooth
Cell-phone network (G3)
(Infrared)
(Wires or cradle)
Toshiba e740
HP iPaq 3870
Microsoft SmartPhone
Brad Myers 10
Premises of our Research
“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event
for synchronization. Instead, the devices will frequently be in
close, interactive communication.”
“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event
for synchronization. Instead, the devices will frequently be in
close, interactive communication.”— Brad Myers, “Using Hand-Held Devices andBrad Myers, “Using Hand-Held Devices and
PCs Together,” PCs Together,” Comm. ACMComm. ACM,,Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Brad Myers 11
Pebbles Project
Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the
same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances
Multiple users with their devices Single user with multiple devices
Performed as part of the Pebbles Project Overall goal: Use of multiple devices at the
same time Palm and desktop PC Mobile Phone and “Smart Home” Multiple handhelds in a meeting Pocket PC and appliances
Multiple users with their devices Single user with multiple devices
Brad Myers 12
Pebbles is:
PEBBLES
PEBBLES
DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.
DAs forDAs forntry ofntry ofothothytes andytes andocations fromocations fromxternalxternalources.ources.
http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/
Controlling a PCControlling a PC Remote Commander Slide Show Commander “Semantic Snarfing” Scrolling Shortcutter
Remote Commander Slide Show Commander “Semantic Snarfing” Scrolling Shortcutter
http://www.pebbles.hcii.edu/http://www.pebbles.hcii.edu/
Brad Myers 14
Important Work by Others PARCTab [Want 95], [Weiser, CACM’93] Abowd: eClass using Handhelds [CHI’98] Greenberg: public and private data
[Personal Technologies, 3(1), 1999] Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …]
Pick and drop;handhelds withbig displays
PARCTab [Want 95], [Weiser, CACM’93] Abowd: eClass using Handhelds [CHI’98] Greenberg: public and private data
[Personal Technologies, 3(1), 1999] Jun Rekimoto: [UIST’97, CHI’98, CHI’99, …]
Pick and drop;handhelds withbig displays
Brad Myers 15
Remote Commander
Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and
grab mouse All mouse and keyboard functions
Use PDA like touchpad Graffiti or our own pop-up
keyboard Word prediction and completion
Allow PDAs to control a PC Can be used with any application Uses the standard (single) cursor Don’t have to jump up and
grab mouse All mouse and keyboard functions
Use PDA like touchpad Graffiti or our own pop-up
keyboard Word prediction and completion
Brad Myers 16
For People with Muscular Disabilities
Using handhelds as interface to PCs People with Muscular Dystrophy
have fine-motor control but losegross motor control
Difficulties with mouse and keyboard,but stylus OK
So use Remote Commanderas PC’s keyboard and mouse
Using handhelds as interface to PCs People with Muscular Dystrophy
have fine-motor control but losegross motor control
Difficulties with mouse and keyboard,but stylus OK
So use Remote Commanderas PC’s keyboard and mouse
Brad Myers 17
PocketPC version Get PC’s screen onto PocketPC Full view, or one-to-one zooming
Scroll with iPaq’s buttons Control or scribble
Get PC’s screen onto PocketPC Full view, or one-to-one zooming
Scroll with iPaq’s buttons Control or scribble
Brad Myers 18
Power Point Control Use PC to give the presentation Use hand-held to control the PC
Two-way communication Hand-held shows picture of slide,
notes, timer
Use PC to give the presentation Use hand-held to control the PC
Two-way communication Hand-held shows picture of slide,
notes, timer
Brad Myers 19
SlideShow Commander, cont.
See list of slide titles Scribble Tap on on-screen buttons
and links Preview other
slides Control
demonstrations
See list of slide titles Scribble Tap on on-screen buttons
and links Preview other
slides Control
demonstrations
Brad Myers 20
Control at a Distance
Controlling a PC fromacross the room Meetings, etc.
Use a laser pointer? Studies Difficult due to:
Jitter of hand (+/- 10 pixels) No button on the laser pointer People not accurate at where
beam will go on and off
Controlling a PC fromacross the room Meetings, etc.
Use a laser pointer? Studies Difficult due to:
Jitter of hand (+/- 10 pixels) No button on the laser pointer People not accurate at where
beam will go on and off
Brad Myers 21
Idea:“Semantic Snarfing” Grab contents to handheld
Picture “Semantic” because
reformat based on the meaning of the contents Menus Text Re-visualization
Grab contents to handheld Picture
“Semantic” because reformat based on the meaning of the contents Menus Text Re-visualization
Brad Myers 22
Command Post of the Future Large DARPA funded project Make commanders more effective Private drill-down of public information
Large DARPA funded project Make commanders more effective Private drill-down of public information
Brad Myers 23
Classroom Investigations
Investigate improving large lectures with frequent in-class “concept tests” Provide immediate feedback to instructor
Easy for instructor
Investigate improving large lectures with frequent in-class “concept tests” Provide immediate feedback to instructor
Easy for instructor
Brad Myers 24
Scrolling Desktop Applications
Scrolling using the non-dominant hand Parallel and efficient use of both hands Generates Windows
scrolling events Study showed can be
faster for some tasks
Scrolling using the non-dominant hand Parallel and efficient use of both hands Generates Windows
scrolling events Study showed can be
faster for some tasks
Brad Myers 25
Shortcutter
User-created panels of controls Create custom interfaces and
extensions to PC applications And then take them with you
Direct manipulationfor edit, then setproperties
PocketPC or Palm
User-created panels of controls Create custom interfaces and
extensions to PC applications And then take them with you
Direct manipulationfor edit, then setproperties
PocketPC or Palm
Brad Myers 26
Shortcutter Widgets
Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel
Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel
Brad Myers 27
Shortcutter Actions
Send any keyboard key, mouse button, scrolling action or string to PC
Open a file or URL Run an application Invoke any PC menu or button
Windows message Recorded
Switch to a different Shortcutter panel Control the Mouse
Send any keyboard key, mouse button, scrolling action or string to PC
Open a file or URL Run an application Invoke any PC menu or button
Windows message Recorded
Switch to a different Shortcutter panel Control the Mouse
Brad Myers 28
Shortcutter Actions, cont. Control external devices
through PC’s serial port Directly (e.g., projectors) X-10 for electrical devices
Macro Can be multi-application
Application-specific Same button, different messages Useful for application sets:
browsers, compilers
Control external devicesthrough PC’s serial port Directly (e.g., projectors) X-10 for electrical devices
Macro Can be multi-application
Application-specific Same button, different messages Useful for application sets:
browsers, compilers
Brad Myers 29
More Scenarios of Use
Lean-back mail reading
Controlling media players
… and many others
Lean-back mail reading
Controlling media players
… and many others
Controlling AppliancesControlling Appliances
PhD research of Jeffrey Nichols PhD research of Jeffrey Nichols
http://www.pebbles.hcii.edu/puc/http://www.pebbles.hcii.edu/puc/
Problem
Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock
Increasingly computerized Low usability
Too many complex devices, eachwith its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock
Increasingly computerized Low usability
Brad Myers 32
Problem
Brad Myers 33
Existing “universal” controls
Pre-programmed at the factory with a subset, Or, Laboriously hand-programmed by the user
Pre-programmed at the factory with a subset, Or, Laboriously hand-programmed by the user
Brad Myers 34
Important Work by Others
Xweb [Olsen Jr. 2000] INCITS V2 standardization effort: Alternative
Interface Access Protocol (AIAP) [Zimmermann, CHI’02]
Stanford iRoom, iCrafter[Ponnekanti, UBICOMP 2001]
Xweb [Olsen Jr. 2000] INCITS V2 standardization effort: Alternative
Interface Access Protocol (AIAP) [Zimmermann, CHI’02]
Stanford iRoom, iCrafter[Ponnekanti, UBICOMP 2001]
Brad Myers 35
Our Approach Handheld is a “Personal Universal Controller” (PUC) Two-way communication
Appliance describes its functions Handheld PUC
Automatically creates interface Controls the appliance Displays feedback about appliance status
Handheld is a “Personal Universal Controller” (PUC) Two-way communication
Appliance describes its functions Handheld PUC
Automatically creates interface Controls the appliance Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers 36
Automatic Generation of UIs
Benefits All interfaces consistent for a user
With conventions of the handheld Even from multiple manufacturers
Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)
A Hard Problem Previous automatic systems have not
generated high quality interfaces
Benefits All interfaces consistent for a user
With conventions of the handheld Even from multiple manufacturers
Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI)
A Hard Problem Previous automatic systems have not
generated high quality interfaces
Brad Myers 37
Research Challenges
1. Automatic Design of Graphical User Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software
1. Automatic Design of Graphical User Interfaces
2. Automatic Design of Speech User Interfaces
3. Connection with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software
Brad Myers 38
Hand-Generated Graphical Interfaces
First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering
Machine
Used Embedded Visual Basic
Ensured quality with heuristicanalysis and think-aloud studies
Compared with manufacturer’sinterfaces
First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering
Machine
Used Embedded Visual Basic
Ensured quality with heuristicanalysis and think-aloud studies
Compared with manufacturer’sinterfaces
Brad Myers 39
Results of Comparison
Using PUC, users took 50% less time & made 50% fewer errors
All differences are significant (p < 0.05)
Using PUC, users took 50% less time & made 50% fewer errors
All differences are significant (p < 0.05)
Brad Myers 40
Discussion of Comparison
Our hand designed interfaces succeeded for several reasons Good organization Each button has one function Good labels Only available functions are active
Others, hidden on tabs or grayed out Better feedback and error messages
Our hand designed interfaces succeeded for several reasons Good organization Each button has one function Good labels Only available functions are active
Others, hidden on tabs or grayed out Better feedback and error messages
Brad Myers 41
Current PUC Specification Language
XML Full documentation for the
specification language and protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample specification for a stereo
XML Full documentation for the
specification language and protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample specification for a stereo
Brad Myers 42
Properties of PUC Language
State variables & commands Each can have multiple labels
Useful when not enough room Typed variables
Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure
Groups
State variables & commands Each can have multiple labels
Useful when not enough room Typed variables
Base types: Boolean, string,enumerated, integers,fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure
Groups
Brad Myers 43
Dependency Information
Crucial for high-quality interfaces Expressed as <active-if> clauses
Operations: Equals, Less-Than,
Greater-Than Combined Logically
AND, OR Used for:
Dynamic graying out Layout Widget selection
Crucial for high-quality interfaces Expressed as <active-if> clauses
Operations: Equals, Less-Than,
Greater-Than Combined Logically
AND, OR Used for:
Dynamic graying out Layout Widget selection
Brad Myers 44
Specifications
Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator
Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator
Brad Myers 45
Examples of Generated GUIs
Stereo and X-10 Stereo and X-10
Brad Myers 46
More Examples
Elevator Elevator
Brad Myers 47
Generating Speech Interfaces “Universal Speech Interface” (USI) project
Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi
Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with disambiguation and explanation
Supports queries and spoken feedback Paraphrases as confirmation
“Universal Speech Interface” (USI) project Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi
Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with disambiguation and explanation
Supports queries and spoken feedback Paraphrases as confirmation
Brad Myers 48
PUC Architecture
Brad Myers 49
Adaptors
“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly
Custom hardware Audiophase Stereo
Custom software ARQ MP3 player SIMA MP3 player (future)
X-10 Light switches, etc.
“Adaptors” provide the interface to existing (and future) appliances If do not support specification language directly
Custom hardware Audiophase Stereo
Custom software ARQ MP3 player SIMA MP3 player (future)
X-10 Light switches, etc.
AV/C (standard protocol) Sony CamCorder PlayStation, etc. (future)
HAVi (current work) Mitsubishi HDTV 65” TV Mitsubishi HDTV VCR
UPnP (future) ??
AV/C (standard protocol) Sony CamCorder PlayStation, etc. (future)
HAVi (current work) Mitsubishi HDTV 65” TV Mitsubishi HDTV VCR
UPnP (future) ??
Brad Myers 50
Adaptor using Custom Hardware for Stereo
Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design
Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design
Brad Myers 51
Adaptor for AV/C
Hand-written spec of AV/C
devices
PUC Java Proxy
Our Java device code
Our C++ code
Microsoft DirectShowlibrary
AV/C protocol
Firewire cable(IEEE 1394)
PUC Java Library
AutomaticGUI
generation
Digital VideoCamera or VCR
Multiple devicecontrollersMultiple devicecontrollers
AutomaticSpeech UIgeneration
Brad Myers 52
Demonstration
X-10 Camcorder through AV/C protocol for
IEEE 1394 (Firewire) Two way communication:
When state changes from appliance, GUI is updated and can query state with speech
X-10 Camcorder through AV/C protocol for
IEEE 1394 (Firewire) Two way communication:
When state changes from appliance, GUI is updated and can query state with speech
Brad Myers 53
For More Information
Many papers on the Pebbles web site On PUC and other work
Most programs available for free downloading:
SlideShow Commander available commercially
Many papers on the Pebbles web site On PUC and other work
Most programs available for free downloading:
SlideShow Commander available commercially
http://www.cs.cmu.edu/~pebbleshttp://www.cs.cmu.edu/~pebbles
http://www.slideshowcommander.comhttp://www.slideshowcommander.com
Brad Myers 54
Conclusions
Study The Big Picture: How mobile devices fit into users’ entire
information and control space
As more and more electronics are computerized, mobiles devices can help control them
Mobile devices can improve the user interfaces of everything else Not just be another gadget to be learned
Study The Big Picture: How mobile devices fit into users’ entire
information and control space
As more and more electronics are computerized, mobiles devices can help control them
Mobile devices can improve the user interfaces of everything else Not just be another gadget to be learned
Brad Myers 55
Thanks to Our Sponsors!The Pebbles research project is supported by
grants from:DARPAMicrosoftNSFPittsburgh Digital GreenhouseGeneral Motors
And equipment grants from:Hewlett PackardLucent TechnologiesMitsubishiPalm ComputingSymbol Technologies
The Pebbles research project is supported by grants from:DARPAMicrosoftNSFPittsburgh Digital GreenhouseGeneral Motors
And equipment grants from:Hewlett PackardLucent TechnologiesMitsubishiPalm ComputingSymbol Technologies
IBMSMART TechnologiesVividLogicSynergy SolutionsHandango
IBMSMART TechnologiesVividLogicSynergy SolutionsHandango
Mobile Devices for ControlMobile Devices for Control
Brad A. [email protected]
http://www.cs.cmu.edu/~pebbles
Brad A. [email protected]
http://www.cs.cmu.edu/~pebbles
Human Computer Interaction Institute Human Computer Interaction Institute
School of Computer ScienceSchool of Computer Science
Carnegie Mellon UniversityCarnegie Mellon University