Multi-Touch: A New Tactile 2-D Gesture Interface for Human

PROCEEDINGS of the HUMAN FACTORS AND ERGONOMICS SOCIETY 45th ANNUAL MEETING- 2001

MULTI-TOUCH: A NEW TACTILE 2-D GESTURE INTERFACE FOR HUMAN-COMPUTER INTERACTION

Wayne Westerman, John G. EliasDepartmentof Electrical and ComputerEngineering, University of Delaware

140 Evans Hall, Newark, DE 19716-3130.and

Alan HedgeDepartment of Design and Environmental Analysis

Cornell University, College of Human Ecology, MVR Hall, Ithaca, NY 14853-4401.

The naturalnessand variety of a touch-based hand gesture interface offers new oppor-tunities for human-computer interaction.Using a new type of capacitive sensor array,aMulti-Touch Surface (MTS) can be createdthat is not limited in size, that can bepresented in many configurations, that is robust under a variety of environmentaloperating conditions, and that is very thin. Typing and gesture recognition built intothe Multi-Touch Surface allow users to type andperform bilateral gestures on thesame surface area and in a smaller footprim than is required by current keyboard andmouse technologies. The present approach interprets asynchronous touches on thesurface as conventional single-finger typing, while motions initiated by chords areinterpreted as pointing, clicking, gesture commands, or hand resting. This approachrequires learning only a few new chords for graphical manipulation, rather than avocabulary of new chords for typing the whole alphabet. Graphical manipulationseems a better use of chords in today's computing environment.

INTRODUCTION e) The physicalarea required for a keyboardand a mouse or other input device is an

Hand gestures play an integral role in human inefficient use of workspace.communication. Gestures are a physical expressionof mental concepts (Huang et al., 1996). As Consequently, several kinds of gesturalPavlovic et al. (1997) emphasize, when compared interface have been developed. This paper willwith current interfaces, the naturalness and variety describe the development and operation of a uniqueof hand gestures offers unique opportunities for new interface for multi-finger gestures performed on aforms of human computer interaction (HCI). surface.

Most personal computers require the opera-tion of at least two devices: a keyboard for bilateral Taxonomy of Hand Gesturestext and numeric input; and a pointing device such As used in HCI, the term 'gesture' is definedas a mouse, trackball, trackpoint, touchpad, pen, as a trajectory in a parameter space over a suitablyjoystick, etc. In terms of human performance, the defined interval, and a taxonomy of hand gesturestypical desktop computer hardware interface ar- has been developed (Pavlovic et al., 1997) as shownrangement is inefficient for several reasons: in Figure 1 (figure 1 is photo, not taxonomy!). In

a) The operation of the keyboard and pointing this taxonomy, hand movements are initially sepa-device requires non-productive movement rated into unintentional movements and gesturestimes to switch a hand between typing and (intentional movements are meant to communicatepointingactivities, the user's intent).Gesturesthen are dividedinto

b) The use of separate devices requires differ- manipulative or communicative forms. Examples ofent skill sets: typing movements for key- manipulative gestures are actions such as the move-boards, mostly point-and-click operations ment or rotation of an object. Communicativefor other input devices, gestures include the hand movements that are part

c) With the exception of the keyboard, most of the nonverbal components of speech. Communi-input devices can only be operated with one cative gestures are divided into symbols and acts.hand. Finally,symbolscanbereferentialactions,suchas

d) The physical arrangement of a keyboard and those movements that are symbolic representationsa mouse or other input device can be a risk (e.g. the circular movement of fingers to symbolizefactor in the development of musculoskel- opening an object) or modalizing actions, such asetal disorders (e.g. National Academy of imitating a wave movement up and down. Acts canSciences,2000). be eithermimetic(i.e.movementsthat imitatean

632


action) or deictic (acts of pointing). Temporal Multi-Touch: a new approach to a 2-D GIcomponents and patterns of gestures are also impor- Multi-touch surface (MTS)tam in the flow of communication. However, most To overcome the sensing limitations ofcurrent input devices restrict users to mainly deictic current touch surfaces and the typing limitations ofgestures and limit temporal components to se- free-space interfaces, a new technology, the Multi-quencesof keystrokesor clicks. Touch Surface (MTS) has been developed

(Westerman, 1999; Westerman and Elias, 1999).Research on Gestural Interfaces (GI) Using scalable arrays of a new capacitive proximityFree-Space 3-D GIs sensor, the MTS is not limited in size and can be

Most of the research on gestural interfaces presented in many configurations. The use ofhas attempted to develop efficient and economical different sensor densities allows the creation ofsystems for capturing three-dimensional hand surfaces with different resolutions. The Multi-Touchmovements in free-space. Approaches have either technology is robust under a variety of environmen-used instrumented gloves or vision-based systems tal operating conditions, and compared with a(Huang and Pavlovic, 1996). Glove systems tend to conventional keyboard, the surface depth can bebe restrictive since the gloves are wired to the very thin. A keyboard that uses the MTS allowscomputer, but these systems have been used in users to type and perform bilateral gestures on theapplications such as controlling 3-D molecular same surface area, and in a smaller footprint than ismodeling software in molecular biology work (e.g. required by current keyboard and mouse technolo-Pavlovic et al., 1996). Vision based approaches gies. Compared to free-space gesture systems, thetypically require the use of at least two video cam- MTS offers a mirror-like correspondence betweeneras and considerable computational power to surface contacts felt by the user and those detectedprocess the resulting signals (e.g. Martin and by the proximity sensors. This correspondenceCrowley., 1997). To date, the expense and complex- inherently ignores many postural adjustments andity of these approaches has limited their usefulness, ensures that the user is always aware of any touchesDistinguishing intentional gestures from uninten- (intentional or not) that the system may attempt totional postural adjustments is very difficult for free- interpret.space gesture recognition systems because surface In 1984, Lee at the University of Torontocontact and pressure, the simplest intention cues, first implemented a capacitive sensor array capableare missing. Also, because of the lack of tactile of tracking multiple touches simultaneously (Lee etfeedback and a surface to support the hands, these al. 1985), but this early prototype suffered fromapproaches are less practical for typing input than a interference between sensors. Also, relatively littleconventionalkeyboard, computationalpower for interpretinggestureswas

available at the time, and graphical user interfacesTouch-Based 2-1) Gls were not yet mainstream in the PC world. The

Perhaps the best example of a widespread 2- Toronto research group envisioned breaking sur-D GI is the touchpad. This surface allows for deictic faces up into multiple virtual windows whereingestures and it has become an integral component in gestures within each window would have differentlaptop computers and some keyboards, as well as effects (Brown et al. 1990; Buxton et al. 1985). In abeing a standalone device. Some types of touchpads different approach, McAvinney's SensorFrameallow the capture of basic symbolic representations, (Rubine and McAvinney, 1990) recognized gesturessuch as capturing a signature, but typically these involving up to three fingers that pass through a gridrequire an additional component, such as a stylus or of light beams. This approach has recently beenspecial pen. Touchpads currently on the market use extended by Newcom, Inc. of Japan (Newcom,row and column electrodes that span the entire 1999), whose projective optical sensors mounted atsensing area, which can be likened to projection the comers of a touchscreen can distinguish infraredscanning from the surface edges. Such projective reflective pen tips from fingertips, which absorb andscanning systems cannot distinguish all arrange- block the infrared beams.ments of multiple fingers, and are generally limited Previous attempts to implement both typingto tracking the centroid of all surface contacts (Lee and pointing on the same region of a touch surface1984). Furthermore, capacitance-based projective have assumed the user would press a button tosystems do not scale up well because signal-to-noise switch modes (Boie et al. 1995). The present MTSratios fall as the electrodes that must span the implementation creates a seamless, unified interfacesurface increase in length. Optical-based projective in which mode or tool selection is implicit in handsystems (Newcom, 1999) do not suffer this limita- configuration, not absolute placement of the hand intion. certainregionsofthesurfacenorperformanceofan

explicit mode-switching signal such as a button

633


press. Unifying typing, manipulative gestures, MTS mere contact with the surface, not explicitcommand gestures, and handwriting requires a very finger pressure, is sufficient to activate keys. Thecarefully designed gesture set. Rather than apply lightness of the touch should benefit users experi-the chord detection capability afforded by multi- encing tendonitis (one of the test subjects wastouch sensing to a chord keying scheme, chords are experiencing such pain prior to using the multi-reserved for hand resting, manipulative gestures, touch surface and this has been alleviated).and command (symbolic) gestures. Traditionaltouch or hunt-and-peck typing can then easily be MTS Researchdistinguished from chord gestures as long as finger To date, two studies of a MTS have beentouchdowns during typing are asynchronous, while conducted. The first study explored the use of afinger touchdowns at the beginning of a chord are MTS keyboard with children doing fingerpaintingroughly simultaneous. The initial combination of and its use in a kindergarten classroom (Browne ettouches in a manipulation chord selects the graphi- al., 2000). They began by designing a finger paint-cal manipulation 'channel' (i.e. point,drag, orscroll), ing program based on the input from six childrenThis way users are free to relax their remaining between the ages of 6 and 11, and who came to thefingers onto the surface while continuing the same laboratory twice a week after school and work withmanipulation. Hand configurations other than basic adults to design and test the MTS technology. Thefingertip combinations can also be recognized, e.g. final software design assigned colors based on thefistsandpengrips, temporalorderin whichthe 10digitstouchedthe

Thus, with the present approach, asynchro- multi-touch surface (Figure 2). Initially, the 10nous touches on a MTS are interpreted as conven- colors, always assigned in the same order, buttional single-finger typing, while motions initiated children could modify this by using menus toby chords are interpreted as pointing, clicking, and change color order for the 10 colors, brush type (5other gesture commands. This approach is prefer- choices), as well as clearing the screen. Changes inable because it requires learning only a few new contact pressure changed the painting action, a lightchords for graphical manipulation rather than a pressure painted the cursor mark being painted onvocabulary of new chords for typing the whole the screen for a brief period of time before disap-alphabet. Graphical manipulation seems a better use pearing; a stronger pressure painted as usual. Infor-of chords in today's graphics intensive computing mal testing was conducted with 7 children, 4 ofenvironment, whom had participated in pre-design questioning.

The MTS was used along with the painting programVocabulary of 2-D gestures by children in groups of 2 or 3 for about 10 minutesBy itself the MTS merely allows for the detection of for each group. The groups received minimalfinger and hand contacts on the surface. For this to instruction, mostly discovering how the MTS andbe an effective GI, software has been developed that software worked by exploration. Two adults ob-recognizes a number of 2-D gestures. For example, served the groups working. The contextual inquiryall current keyboard and 2-button and 3-button method was used to record time-synchronized notesmouse operations are included in this vocabulary of from one observer recording the children's activities2-D gestures. The gestures that represent specific and the other recording their verbal utterancesactions have been developed based upon both the (Druin, 1999). Results showed that all of the chil-naturalness of the finger movements, their concep- dren immediately liked and understood how to usetual associations, and results from user trials of the the MTS. Most groups wanted to continue whentechnology (Figure 1). In addition to the existing their allotted time was up and four children specifi-vocabulary, configurable gesture sets can be created, cally remarked that it was "cool". Although it waswhich allows users to create an interface that is used by small groups, the MTS was shared remark-naturally compatible with user expectations. The ably well by the children, dividing the surface upsoftware is able to distinguish unintentional hand equally according to sitting position and collaborat-movements, such as resting the hand on the surface, ing on the artwork. Working in pairs seemed tofromgestures, workbest,whereasthreesomesseemedcramped.

Prototypes of the multi-touch GI have been Most of the children just scribbled with their fingerstested with 5 subjects over a 3 months period in and enjoyed seeing the different colors fill thetheir normal work situations. Users report a fast screen, though some children said that they didn'tlearning curve for the new technology and they like how quickly the screen filled up. After this thequickly master the gesture vocabulary. The design children wrote and drew in journals about theof the enclosure for a MTS can also improve the experience. Many of the children drew both a touch-position of the hands during typing and the perfor- screen and a MTS in their illustrations. Interestingly,mance of input gestures. When users type on the the MTS can be integrated with display-screen

634


technology, somesimpletactileandvisualfeedbacktoAt the end of the study children worked on users

designing a "Classroom of the Future" for b) The use of a multi-touch surface allows userskindergarteners. Based on this, in August 2000 a to rest their hands to relieve shoulder loadsFinger Painting Table, about 5 feet (153 cm) in c) The use of a multi-touch surface allows alldiameter and covered it in 2-inch thick foam and keying and mousing activities to be per-incorporating MTS, was built. The table was formed using a single, space efficient unitpositioned beneath a ceiling-mounted projector that d) The use of a multi-touch surface providesprojected the finger-painting screen image onto a users with a more intuitive interface thatwhite foam area of the table surface about 2 x 1 ½ allows bilateral gestural controls and thatfeet (6lx46 cm). The rest of the table was decorated speeds human performancewith colorful cloth and fabric (Figure 3). Testing of e) The use of a multi-touch surface is a verythis table by the children found that it provided a cost-effective approach to the creation of amuch softer, friendlier, and inviting painting envi- new GI and it does not require the instru-ronment, and it was more conducive to collabora- mentation of users hands or the use of twintive creations because the children stood and easily video cameras and substantial computationalmoved around the table. A combination of physical operations.and virtual design tools using different coloredfoams were cut into shapes such as animals, people, In conclusion, the MTS represents a newhouses, and clouds and placed on the projection direction in thinking about HCI and it holds thesurface so that children could paint around them potential to create a new generation of more usable,using the MTS was immediately successful. This more effective HCI products.work determined that the MTS can readily beintegrated into alternative uses for computer tech- REFERENCESnology. Boie,R.A.,Ruedisueli,L.W.andE.R.Wagner

In a second study at the same HCI labora- (1995) Computer Mouse or Keyboard Input Devicetory, the usability of an MTS keyboard Utilizing Capacitive Sensors. U.S. Patent 5463388,(FingerBoard, see Figure 4) was compared to that of October 31.a standard keyboard and mouse setup using two Brown, E., Buxton, W.A.S. and K Murtaghidentical tests involving basic typing and mousing (1990) Windows on Tablets as a Means of Achievingactivities. Fourteen college students (10 men, 4 Virtual Input Devices, In D. Diaper et al. (Eds.)women) were tested but the results were inconclu- Human-Computer Interaction- INTERACT '90,sive in part because the learning curve for using the Elsevier Science Publishers B.V., North Holland,MTS exceeded the short test-time (30 minutes) 675-681.available for become proficient at using the MTS. Browne, H., Bederson, B., Druin, A., and L.However, in this study keyboard layout of the MTS Sherman (2000) Designing a Collaborative Fingerwas confounded with the surface characteristics of Painting Application for Children, Human-Com-the MTS, and the results obtained were similar to puter Interaction Laboratory, Institute for Advancedthose for other ergonomic split-keyboards (e.g. Computer Studies, University of Maryland, Collegekinesis), and this design change has been shown to Park, MD 20742 (unpublished report).decrease typing speed in the short-term (Chen et al., Buxton W., Hill, R. and P. Rowley (1985)1994). IssuesandTechniquesinTouch-SensitiveTablet

Input (Extended Summary), Graphics Interface '85,CONCLUSIONS Computer Systems Research Institute, Toronto,

Development of the MTS technology and Ontario, Canada, 147-149.supporting software allows the creation of an Cakir, A.E., Cakir, G., Muller, T. and P.efficient touch-based GI for computers systems. A Unema (1995) The Trackpad - A Study on Userbasic vocabulary of gestures has been created that Comfort and Performance, Proceedings of CHI '95,can be expanded to include a greater range of ACM.operations than is currently available with input Chen, C., Burastero, S., Tittiranonda, P.,devices. The multi-touch 2-D GI may overcome Hollerbach, K., Shih, M. and R. Denhoy (1994),limitations with existing 2-D interfaces, by harness- Quantitative evaluation of 4 computer keyboards:ing many of the benefits of a 3-D interface through wrist posture and typing performance, Proceedingsthe use of more intuitive 2-D gestures performed on of the Human Factors and Ergonomics Socie_ 38tha surface. This approach offers several advantages Annual Meeting, 2, HFES, Santa Monica, CA, 1094-over other attempts at creating effective GIs: 1098.

a) The use of a multi-touch surface affords Detwiler, B., Hun An, C. and P.Choi (2000)

635


The Battle of Input Devices: Examining the Pavlovic, V.I., Sharma, R. and T.S. HuangFingerWorks FingerBoard, Human-Computer (1996) Gestural Interface to a Visual ComputingInteraction Laboratory, Institute for Advanced Environment for Molecular Biologists, in Proceed-Computer Studies, University of Maryland, College ings, International Conference on Automatic FacePark, MD 20742 (unpublished report) and Gesture Recognition, pp. 30-35.

Druin, A. (1999) Cooperative Inquiry: Pavlovic, V.I.R. Sharma, T. S. HuangDeveloping New Technologies for Children with (1997) Visual interpretation of Hand Gestures forChildren. Proceedings of CH199, 592-599. Human-Computer Interaction: A Review. IEEE

Lee, S. A (1984) Fast Multiple-Touch- PAMI, 1__.99,(7),. 677-695.Sensitive Input Device, University of Toronto. Rubine, D. and MeAvinney, P. (1990)

Lee, S.K., Buxton, W. and K.C. Smith, A Programmable Finger-tracking Instrument Control-(1985) Multi-Touch Three Dimensional Touch- lers, Computer Music Journal, 14 (1), 26-41.Sensitive Tablet, Proceedings of CHI '85, ACM. Westerman, W.C. (1999) Hand Tracking,

Martin, J. and J. L. Crowley (1997) An Finger Identification, and Chordic Manipulation onappearance-based approach to gesture-recognition, a Multi-Touch Surface, University of Delaware PhDIn Proceedings Int'l Conf. on Image Analysis and Dissertation.Processing, Florence, Italy,September. Westerman,W.C. and J.G. Elias (1999) "A

National Academy of Sciences (2001) method and apparatus for integrated manual input,"Musculoskeletal disorders and the workplace: low U.S. Patent Application Serial No. 09/236,513,back and upper extremities. National Academy January 25.Press, Washington, DC.

Newcom, Inc. of Japan. http://newcom- ..................................................................................................................................................................................................................................................................................................................................uc.co.jp/, 1999.

636

Documents

Multi-Touch: A New Tactile 2-D Gesture Interface for Human