1 Past, Present and Future of User Interface Software Tools Brad A. Myers, Scott E. Hudson, and Randy Pausch Human-Computer Interaction Institute School

1

Past, Present and Future ofUser Interface Software ToolsPast, Present and Future ofUser Interface Software Tools

Brad A. Myers, Scott E. Hudson, and Randy PauschBrad A. Myers, Scott E. Hudson, and Randy Pausch

Human-Computer Interaction InstituteHuman-Computer Interaction InstituteSchool of Computer ScienceSchool of Computer ScienceCarnegie Mellon UniversityCarnegie Mellon University

http://www.cs.cmu.edu/~bamhttp://www.cs.cmu.edu/[email protected]@cs.cmu.edu

Myers, Hudson & PauschMyers, Hudson & PauschCMU - HCI InstituteCMU - HCI Institute 2HCIC’99HCIC’99

Introduction

User Interface Software Tools Help developers design and implement user interfaces

Focus on Tools, but influenced by future UIs Today’s tools are highly successful

Window Managers, Toolkits, Interface Builders ubiquitous Most software built using them Are based on HCI research

Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

Future tools must be different

User Interface Software Tools Help developers design and implement user interfaces

Focus on Tools, but influenced by future UIs Today’s tools are highly successful

Window Managers, Toolkits, Interface Builders ubiquitous Most software built using them Are based on HCI research

Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html

Future tools must be different


Talk Outline

Historical Perspective What worked What didn’t catch on Why Lessons Learned

Future Prospects and Visions UI Trends that will require new tools Important issues

Historical Perspective What worked What didn’t catch on Why Lessons Learned

Future Prospects and Visions UI Trends that will require new tools Important issues


Historical Perspective Themes

Address the useful & important aspects of UIs Tools that succeeded helped (just) where needed

Threshold / Ceiling Threshold = How hard to get started Ceiling = how much can be achieved

Path of Least Resistance Tools influence user interfaces created

Predictability If not predictable, then not accepted by programmers

Moving Targets Changing user interface styles makes tools obsolete

ThemesAddress the useful & important aspects of UIs

Tools that succeeded helped (just) where neededThreshold / Ceiling

Threshold = How hard to get started Ceiling = how much can be achieved

Path of Least Resistance Tools influence user interfaces created

Predictability If not predictable, then not accepted by programmers

Moving Targets Changing user interface styles makes tools obsolete


What Worked

Window Managers and Toolkits Event Languages Graphical, Interactive Tools Component Architectures Scripting Languages Hypertext Object Oriented Programming

Window Managers and Toolkits Event Languages Graphical, Interactive Tools Component Architectures Scripting Languages Hypertext Object Oriented Programming


Window Managers

Multiple (tiled) windows in research systems of 1960’s: NLS, etc.

Overlapping introduced in Alan Kay’s thesis (1969) Smalltalk, 1974 at Xerox PARC Successful because multiple windows help users

manage scarce resources: Screen space and input devices Attention of users Affordances for reminding and finding other work

Multiple (tiled) windows in research systems of 1960’s: NLS, etc.

Overlapping introduced in Alan Kay’s thesis (1969) Smalltalk, 1974 at Xerox PARC Successful because multiple windows help users

manage scarce resources: Screen space and input devices Attention of users Affordances for reminding and finding other work


Toolkits

A collection of widgets Menus, scroll bars, text entry fields, buttons, etc.

Toolkits help with programming Help maintain consistency among UIs

Key insight of Macintosh toolbox

Path of least resistance translates into getting programmers to do the right thing

Successful partially because address common, low-level features for all UIsAddress the useful & important aspects of UIs

A collection of widgets Menus, scroll bars, text entry fields, buttons, etc.

Toolkits help with programming Help maintain consistency among UIs

Key insight of Macintosh toolbox

Path of least resistance translates into getting programmers to do the right thing

Successful partially because address common, low-level features for all UIsAddress the useful & important aspects of UIs


Event Languages

Create programs by writing event handlers Many UIMSs used this style

Univ. of Alberta (1985), Sassafras (1986), etc.

Now used by HyperCard, Visual Basic, Lingo, etc. Toolkits with call-backs or action methods are related

Advantages: Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s

Discourages moded UIs

Won’t work well in future

Create programs by writing event handlers Many UIMSs used this style

Univ. of Alberta (1985), Sassafras (1986), etc.

Now used by HyperCard, Visual Basic, Lingo, etc. Toolkits with call-backs or action methods are related

Advantages: Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s

Discourages moded UIs

Won’t work well in future


Graphical Interactive Tools

Create parts of user interface by laying out widgets with a mouse Examples: Menulay (1983), Trillium (1986), Jean-Marie

Hullot from INRIA to NeXT Now: Interface Builders, Visual Basic’s layout editor,

resource editors, “constructors”

Advantages: Graphical parts done in an appropriate, graphical way

Address the useful & important aspects of UIs Accessible to non-programmers

Low threshold

Create parts of user interface by laying out widgets with a mouse Examples: Menulay (1983), Trillium (1986), Jean-Marie

Hullot from INRIA to NeXT Now: Interface Builders, Visual Basic’s layout editor,

resource editors, “constructors”

Advantages: Graphical parts done in an appropriate, graphical way

Address the useful & important aspects of UIs Accessible to non-programmers

Low threshold


Component Architectures

Create applications out of components which are separately developed and compiled In UI software, each component controls an area of the

screen Example: drawing component handles picture inside a

document

Invented by Andrew research project at CMU (1988) Now: OLE, OpenDoc, ActiveX, Java Beans Address the useful & important aspects of UIs

Just the “glue” to hold together components

Create applications out of components which are separately developed and compiled In UI software, each component controls an area of the

screen Example: drawing component handles picture inside a

document

Invented by Andrew research project at CMU (1988) Now: OLE, OpenDoc, ActiveX, Java Beans Address the useful & important aspects of UIs

Just the “glue” to hold together components


Scripting Languages

First GUIs used interpreted languages Smalltalk, InterLisp

Rapid development, supports prototyping Low threshold

Then C and C++ became popular Now, bringing back advantages in scripting

languages tcl/tk, Python, perl Visual Basic, Javascript

But language must contain general-purpose control structures

First GUIs used interpreted languages Smalltalk, InterLisp

Rapid development, supports prototyping Low threshold

Then C and C++ became popular Now, bringing back advantages in scripting

languages tcl/tk, Python, perl Visual Basic, Javascript

But language must contain general-purpose control structures


Hypertext

Ted Nelson named it in 1965 and developed Hypertext system at Brown University

Important systems: NLS (1967), Hyperties (1986) World-Wide Web

Phenomenal success due to: Ease of use of Mosaic browser Support for embedded graphics Support for easy authoring

Low threshold both for authoring and viewing

Ted Nelson named it in 1965 and developed Hypertext system at Brown University

Important systems: NLS (1967), Hyperties (1986) World-Wide Web

Phenomenal success due to: Ease of use of Mosaic browser Support for embedded graphics Support for easy authoring

Low threshold both for authoring and viewing


Object Oriented Programming

Success of OO owes much to UI software field Popularized by Smalltalk GUI elements (widgets) seem like objects

Have state, accept events (messages)

Rise parallels GUIs C++ with Windows 3.1 Java for behaviors in WWW

Success of OO owes much to UI software field Popularized by Smalltalk GUI elements (widgets) seem like objects

Have state, accept events (messages)

Rise parallels GUIs C++ with Windows 3.1 Java for behaviors in WWW


What Hasn’t Caught On

User Interface Management Systems Formal Language-Based Tools Constraints Model-Based and Automatic Techniques

User Interface Management Systems Formal Language-Based Tools Constraints Model-Based and Automatic Techniques


User Interface Management Systems

Original goal: like databases, provide high-level language that abstracts details of input and output devices

This separation has not worked in practice Good user interfaces must take into account the

pragmatics and detailed behavior of all objects

Standardization of GUI input and output devices has made goal somewhat mootDoesn’t address the useful & important aspects of UIs

Original goal: like databases, provide high-level language that abstracts details of input and output devices

This separation has not worked in practice Good user interfaces must take into account the

pragmatics and detailed behavior of all objects

Standardization of GUI input and output devices has made goal somewhat mootDoesn’t address the useful & important aspects of UIs


Formal Language Based Tools

Early UIMSs used grammars and state-transition diagrams

Focus on dialog management Moving Targets

Direct manipulation made dialog management less important

Path of Least Resistance State diagrams afford worse user interfaces

High threshold Formal languages are often hard to learn

Early UIMSs used grammars and state-transition diagrams

Focus on dialog management Moving Targets

Direct manipulation made dialog management less important

Path of Least Resistance State diagrams afford worse user interfaces

High threshold Formal languages are often hard to learn


Constraints

Declare a relationship and system maintains it Sketchpad (1963), ThingLab (1979), Higgens (85),

Garnet (1990), Amulet (1997), SubArctic (1996) Predictability

Constraint networks can be hard to debug Especially in multi-way constraints

Programmer must specify (or deduce) solving order High threshold

Constraints require thinking differently May be appropriate for graphical layout

Address the useful & important aspects of UIs

Declare a relationship and system maintains it Sketchpad (1963), ThingLab (1979), Higgens (85),

Garnet (1990), Amulet (1997), SubArctic (1996) Predictability

Constraint networks can be hard to debug Especially in multi-way constraints

Programmer must specify (or deduce) solving order High threshold

Constraints require thinking differently May be appropriate for graphical layout

Address the useful & important aspects of UIs


Model-Based and Automatic Techniques

Automatic techniques for generating UIs from a model or declarative specification of contents

Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993)

Try to separate specification of UI from content May provide automatic reformating, retargeting,

customization to users, etc.

Result is often unpredictable Often can be worse UI than hand-drawn Sometimes model is larger than the code it would

replace

Automatic techniques for generating UIs from a model or declarative specification of contents

Cousin (1985), Mike (1986), UIDE (1993), MasterMind (1993)

Try to separate specification of UI from content May provide automatic reformating, retargeting,

customization to users, etc.

Result is often unpredictable Often can be worse UI than hand-drawn Sometimes model is larger than the code it would

replace


Discussion of Themes

Address the useful & important aspects of UIs Narrower tools have been more successful than ones

that try to do “everything” Do one thing well

Threshold / Ceiling Research systems often aim for high ceiling Successful systems seem to instead aim for a low

threshold Impossible to have both?

Address the useful & important aspects of UIs Narrower tools have been more successful than ones

that try to do “everything” Do one thing well

Threshold / Ceiling Research systems often aim for high ceiling Successful systems seem to instead aim for a low

threshold Impossible to have both?


Discussion of Themes, cont.

Path of Least Resistance Tools should guide implementers into better user

interfaces Goal for the future: do this more?

Predictability Programmers do not seem willing to release control Especially when system may do sub-optimal things

Moving Targets Long stability of Macintosh Desktop paradigm has

enabled maturing of tools We predict a change soon

Path of Least Resistance Tools should guide implementers into better user

interfaces Goal for the future: do this more?

Predictability Programmers do not seem willing to release control Especially when system may do sub-optimal things

Moving Targets Long stability of Macintosh Desktop paradigm has

enabled maturing of tools We predict a change soon


Future Prospects and Visions

Important Trends Computers becoming a commodity Ubiquitous Computing Move to recognition-based interfaces 3-D interfaces End-user customization and scripting

Violate assumptions of today’s tools Assumptions limit what designers can do Often unrecognized Implications for future tools

Important Trends Computers becoming a commodity Ubiquitous Computing Move to recognition-based interfaces 3-D interfaces End-user customization and scripting

Violate assumptions of today’s tools Assumptions limit what designers can do Often unrecognized Implications for future tools


Computers Becoming a Commodity

There are no longer “high-end” computers Computer Science research’s trick of buying faster

computers to “time travel” may no longer be possible

Moore’s law continues to operate New opportunities Quantitative change makes qualitative change possible Enables the diversity of platforms UIs more cinematic

Smooth transitions, animation, visual

There are no longer “high-end” computers Computer Science research’s trick of buying faster

computers to “time travel” may no longer be possible

Moore’s law continues to operate New opportunities Quantitative change makes qualitative change possible Enables the diversity of platforms UIs more cinematic

Smooth transitions, animation, visual


Ubiquitous Computing

Computation embedded in many kinds of devices Digital pagers and cell phones, Palm Pilots,

CrossPads, laptops, wall-size displays, “smart” rooms Next wave: easy communication with radio

E.g., BlueTooth: www.bluetooth.com

Significant Implications for tools Tools for coordinating multiple, distributed,

communicating devices “Multi-computer” user interfaces

Moving target problem

Computation embedded in many kinds of devices Digital pagers and cell phones, Palm Pilots,

CrossPads, laptops, wall-size displays, “smart” rooms Next wave: easy communication with radio

E.g., BlueTooth: www.bluetooth.com

Significant Implications for tools Tools for coordinating multiple, distributed,

communicating devices “Multi-computer” user interfaces

Moving target problem


Varying Input and Output Today’s Desktop screens vary by a factor of 2.5 in

size and a factor of 4 in pixels Tomorrow’s screen will vary by factors of 100 in

size and a factor of 625 in pixels Cell phone to Stanford’s wall (3796 x 1436 pixels)

Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels

Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels Cell phone to Stanford’s wall (3796 x 1436 pixels)


Need New Interaction Techniques

Interaction techniques for desktop will not work No room on small devices Can’t reach menubar on wall-size devices

Want to run same application on different devices

Interaction techniques for desktop will not work No room on small devices Can’t reach menubar on wall-size devices

Want to run same application on different devices


Need for Prototyping Devices

User interface will be in hardware Rapid design and prototyping needed for hardware Pragmatics and usability cannot be evaluated from

a simulation on a screen

User interface will be in hardware Rapid design and prototyping needed for hardware Pragmatics and usability cannot be evaluated from

a simulation on a screen


Multiple, Distributed, Communicating

Computers more for communication, not for computation Already true for WWW, email, digital pagers, cell-phones

Computers as intermediaries between people CSCW But can’t assume have similar systems

Single person with multiple devices Room-area networks like BlueTooth or HomeRF People communicating with themselves

Tools will need to help with data sharing and synchronization

Computers more for communication, not for computation Already true for WWW, email, digital pagers, cell-phones

Computers as intermediaries between people CSCW But can’t assume have similar systems

Single person with multiple devices Room-area networks like BlueTooth or HomeRF People communicating with themselves

Tools will need to help with data sharing and synchronization


Limitations of Today’s Tools for UbiComp

Tools assume a Pointing Device Hidden reliance on specific characteristics of

common devices Size of display Many tools cannot handle a different number of mouse

buttons Change to a stylus on a touchpad requires different

techniques

Assumptions about the setting Assume user is sitting and looking at UI Assume has user’s full attention

Tools assume a Pointing Device Hidden reliance on specific characteristics of

common devices Size of display Many tools cannot handle a different number of mouse

buttons Change to a stylus on a touchpad requires different

techniques

Assumptions about the setting Assume user is sitting and looking at UI Assume has user’s full attention


Move to Recognition-Based Interfaces

Speech, gestures, camera-based vision Multimodal interaction

User will pick which modality to use Use multiple modalities at same time

Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. Need tools to hide these complexities

Speech, gestures, camera-based vision Multimodal interaction

User will pick which modality to use Use multiple modalities at same time

Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. Need tools to hide these complexities


Fundamental Differences ofRecognition-based UIs

Input is uncertain Recognition can make errors Requires monitoring, feedback, correction

Interpreting input requires deep knowledge of data Context of the application “Move the red truck to here”

Input is uncertain Recognition can make errors Requires monitoring, feedback, correction

Interpreting input requires deep knowledge of data Context of the application “Move the red truck to here”


Implications of Recognition-based UIs

GUI event model no longer works Do not produce discrete events

Separation of UI from application no longer works Need a architecture based on accessible

application data structures “Reflection”, “Open Data Model”

GUI event model no longer works Do not produce discrete events

Separation of UI from application no longer works Need a architecture based on accessible

application data structures “Reflection”, “Open Data Model”


3-D Interfaces

Difficult to design the right abstractions for tools Demise of VRML for Web

Need to settle on the 3-D widgets and interaction techniques that will be standard

Requirement for near-real-time interactivity Need to hide the mathematics

Difficult to design the right abstractions for tools Demise of VRML for Web

Need to settle on the 3-D widgets and interaction techniques that will be standard

Requirement for near-real-time interactivity Need to hide the mathematics


End-user Customization and Scripting

Spreadsheet enables end users to specify their own computation

Visual Basic, other “scripting” languages Needed in all applications Threshold for programming is too high Need “gentle slope systems”

Spreadsheet enables end users to specify their own computation

Visual Basic, other “scripting” languages Needed in all applications Threshold for programming is too high Need “gentle slope systems”


Gentle Slope Systems

Difficultyof

Use

Sophistication of what can be created

Goal

HyperCard

Visual BasicDirector (v6)

HyperTalk

xCmds

Basic

C Programming

Lingo

C Programming

Programming in C

MFC

Click and

Create


More Assumptions of Today’s Tools

Skill and Dexterity of users Older users Makes single, fixed library of widgets untenable

Non-overlapping and opaque components Preclude translucency, magic lens interactions

Fixed libraries of components (widgets) Creating new widgets is very difficult New devices will require new interaction techniques

Interactive tools provide freedom of design Aim for “Mechanism not Policy”

Skill and Dexterity of users Older users Makes single, fixed library of widgets untenable

Non-overlapping and opaque components Preclude translucency, magic lens interactions

Fixed libraries of components (widgets) Creating new widgets is very difficult New devices will require new interaction techniques

Interactive tools provide freedom of design Aim for “Mechanism not Policy”


Operating Systems Considerations

What is in the OS? Window Manager? Toolkit? Communication? Scripting

facilities?

Need ever increasing services for applications Need more access to low-level information

E.g., hardware buttons, whether on network

Ideally, API to support competition and research into these components

What is in the OS? Window Manager? Toolkit? Communication? Scripting

facilities?

Need ever increasing services for applications Need more access to low-level information

E.g., hardware buttons, whether on network

Ideally, API to support competition and research into these components


Some Design Guidelines for Future Tools

Many things require further research Organize around providing rich context

Of application and device state To inquire about data & methods; “reflection” Enables EUP, Recognition-Based UIs, data sharing for

UbiComp Rather than event-based

Many things require further research Organize around providing rich context

Of application and device state To inquire about data & methods; “reflection” Enables EUP, Recognition-Based UIs, data sharing for

UbiComp Rather than event-based


More Design Guidelines

Replaceable User Interfaces Ability to have multiple UIs Enabled by procedural interface to everything in UI Enables UbiComp devices, EUP

Aim for low threshold, rather than high ceiling But cover the right parts of the interface

Predictable for programmers rather than “smart” or automatic

Need for support for evaluation

Replaceable User Interfaces Ability to have multiple UIs Enabled by procedural interface to everything in UI Enables UbiComp devices, EUP

Aim for low threshold, rather than high ceiling But cover the right parts of the interface

Predictable for programmers rather than “smart” or automatic

Need for support for evaluation


Conclusions

Research in tools necessarily trails innovation in UI design

Due to consolidation on desktop metaphor, significant progress in tools

UI design poised for radical changes New opportunities and challenges for tools

Research in tools necessarily trails innovation in UI design

Due to consolidation on desktop metaphor, significant progress in tools

UI design poised for radical changes New opportunities and challenges for tools

Documents

1 Past, Present and Future of User Interface Software Tools Brad A. Myers, Scott E. Hudson, and Randy Pausch Human-Computer Interaction Institute School