Discussion: The Past, Present, and Discussion: The Past, Present, and Future of Programming in HCIFuture of Programming in HCI

John C. ThomasJohn C. ThomasIBM T. J. Watson Research CenterIBM T. J. Watson Research CenterSnow Mountain Ranch, ColoradoSnow Mountain Ranch, Colorado

February 6, 2009February 6, 2009

Praiseworthy Points Praiseworthy Points • An excellent overview of a complex field• WRT CMU work, a long-term effort

– Sometimes, it takes a long time to solve difficult problems

• Focus on a particular formulation of the problem (translation) shifting to broader concerns

• Focus on a stance --- “naturalness” as opposed to a particular technical approach: visualization.

Pickworthy NitsPickworthy Nits

• The phrase “everyone is computing”

• IT still not accessible to 5 of the 6 billion people on the planet

• Expanding in a useful way provides its own set of challenges: HCI4D

• If “good” HCI is a function of users, tasks, contexts, and technology, should we expect “an answer” to programming with good HCI?

A Formulation?A Formulation?

• The goal is to make it possible for people to express their ideas in the same way they think about them.

• By “natural,” we mean “faithfully representing nature or life,” which here implies it works in the way people expect. By “natural programming” we are aiming for the language and environment to work the way that nonprogrammers expect.

A Broader ViewA Broader View

• Translation of Internal Thinking into Computer Terms

• Making Implicit Knowledge Explicit

• Understanding How to Make the Computer Work Effectively and Efficiently

• Understanding How to Solve a Problem

• Understanding How the Program Might Interact with People, other Programs, Systems

Gary Larsson

Translation Difficulty is not Limited Translation Difficulty is not Limited to Human-to Human-ComputerComputer

• Dance N Funk instruction: Human to Human Interaction and common confusing comments:

• After doing a sequence like ABAB, CDCD, ABAB, EFEF, the instructor often says:– “Let’s do that part again.”– OR– “Everything repeats.”– “I forgot. There’s an insert in there. Like this.”

How do we “translate” from human How do we “translate” from human intention into horse action?intention into horse action?

• Some simple but highly evolved tooling helps (reins, saddle, stirrups).

• Humans need to be trained…but so do the horses.

• We do not try to get the horses to do brain surgery for us.

Making the implicit explicitMaking the implicit explicit

Ghost of Programming Past: QBE Ghost of Programming Past: QBE (early 1970’s)(early 1970’s)

• Query By Example allowed a one-hour training session to allow people to translate from natural language queries into QBE quickly with few errors.

• If we took into account “formal” complexity of resulting query, student’s IQ, time, mapping ease, and their own confidence rating, we could predict 95% of the variance in errors.

• In a later, more open-ended study, students were asked to generate their own relevant questions and the results were much poorer.

Lost Before TranslationLost Before Translation• Students were given table structure of university data

bases and then given “issues” such as: “Some of the younger faculty feel they are not paid enough relative to the older faculty.” Generate some questions relevant to this issue and translate them into QBE.

• Typical query: “Are the older faculty paid too much?” Print the names of people whose age is greater than 55 and whose pay is “too much.”

• As Gary Olson pointed out Thursday morning, we may get different results if we did this study today wherein most students are Google-savvy.

Ghost of Programming PresentGhost of Programming Present

• High Performance Computing Project

• Complexity Modeling

• Rice Trial

Rice TrialRice Trial

• Students given a problem and linear code and asked to parallelize in one of two languages.

• Videotaped

• Screen capture

• Real time observation/coding

• Blinded by paradigm

Ghost of Programming FutureGhost of Programming Future

• At Rice Trial, a lot of “expressive” gesturing from each participant

– Variations among participants

– Seemingly very regular within participant

• People spent a lot of time “looking for code” despite outline views and search facilities. Scanning even with fingers to keep place.

• People spent a lot of time using paper.

Ghost of Programming FutureGhost of Programming Future

• Instead of having the computer trying to infer the emotional state of users (fraught with issues) how about recognizing the appropriate sub-task or sub-goal?

• Understand and provide better search/viewing facilities. E.g., fisheye views?

• Provide better coordination between paper documents and notes and on-line environment.

Tools in Other DomainsTools in Other Domains

• “Blank Piece of Paper” phenomenon

• What do we do in designing house, for instance?

Given a Blank Slate …Given a Blank Slate …

Look at Worked ExamplesLook at Worked Examples

But the vast majority of people buy pre-made houses or use a professional builder.

Providing Different PerspectivesProviding Different Perspectives

• Training

• Perceptual Aids

• Cognitive Aids

• History

• Pattern Languages

• “Random” stimulation

Provide Tools to Allow New Provide Tools to Allow New PerspectivesPerspectives

Allowing new patterns to emerge

Using Physical Space to Provide Overview, Detail, Using Physical Space to Provide Overview, Detail, Continuous Zooming, Multiple ActivitiesContinuous Zooming, Multiple Activities

• Examples: Historical timeline of CHI conferences

• Constructed map of the world on which to perform exercises for CHI 2008 workshop on Human Computer Interaction for International Development

• 1929 Exposition Building, the Plaza de España

• The Plaza de España is one of Seville's most easily recognised buildings and the epitome of the Moorish Revival in Spanish architecture. In 1929 Seville hosted the Spanish-American Exhibition and numerous buildings were constructed for the exhibition in Maria Luisa Park, among them the Plaza.

Historical PerspectiveHistorical Perspective

Zooming, annotations

Viscosity?Viscosity?• Imagine that new historical research indicates that

ONE of the stories portrayed by a city tile is inaccurate?

• This could be remedied with almost no effect on the other displays.

• So…is visual programming, by necessity, highly viscous or have we just not found the right paradigm…or is it really computational paradigms themselves which are highly viscous and visual programming makes it obvious?

End User ProgrammingEnd User Programming

• Is EUP primarily a construction problem

• Or a search and sense-making problem?

• Wikipedia of programming possible?

• Finding the definition of Chinese characters.

Must Must allall our problems be solved on our problems be solved on a computer?a computer?

These are cats watching NOT These are cats watching NOT birds.birds.

There is no doubt some intrinsic pleasure in watching… but they will not get to eat any birds

ConclusionsConclusions• There might be important lessons for HCI and

programming from fields other than HCI and programming.

• Humans are both remarkably intelligent and remarkably flawed in their abilities to solve problems and communicate even with each other.

• We seem to have the capacity for “theory of mind” but do not always invoke it

• How to program may depend a lot on the domain and perhaps some domains are not best “done” by programming computers at all.

Is the glass half full or half empty?Is the glass half full or half empty?

• From the perspective of how much we’ve learned and how far we’ve come the glass is 90% full

ConclusionConclusion

• However, from the perspective of how much of the glass is actual “hard matter”, it is 99.999999999999 % empty.

• Similarly, if we consider the gap between the best that ordinary humans can do and how effectively and efficiently we can construct software, the glass is still 99.999999999999 % empty.

ReferencesReferences• Carroll, J., Thomas, J.C. and Malhotra, A. (1980). Presentation and representation in design

problem solving. British Journal of Psychology/,71 (1), pp. 143-155. • Thomas, J. C. & Richards, J. T. (2008). Achieving psychological simplicity: Measures and

methods to reduce cognitive complexity. In Handbook of HCI, A. Sears & J. Jacko (Eds.). Hillsdale, NJ: Erlbaum.

• Thomas, J.C., Lyon, D. & Miller, L. (1977). Aids for problem solving. IBM Research Report. RC-6468. Yorktown Heights, NY: IBM Corporation.

• Thomas, J. C. (1977). Cognitive psychology from the standpoint of wilderness survival. IBM Research Report, RC-6647. Yorktown Heights, NY: IBM Corporation.

• Thomas, J.C. (1989). Problem solving by human-machine interaction. In Gilhooly K.J., (Ed). Human and machine problem solving. London: Plenum Publishing.

• Thomas, J.C. (1988). Human factors and artificial intelligence. In H. Hartson and D. Hix (Eds.). Advances in human -computer interaction. Norwood, New Jersey: Ablex.

• Thomas, J.C. (1983). Psychological issues in the design of data-base query languages. In M. Sime and M. Fitter (Eds.), Designing for human-computer communication.. London: Academic Press.

• Thomas, J.C. and Carroll, J. (1978). The psychological study of design. Design Studies, 1 (1), pp. 5-11.

• Thomas, J. C. & Gould, J. D. (1975),A psychological study of Query By Example. National Computer Conference Proceedings, 44, 439-445. New York: AFIPS Press.