Tangible User Interfaces (TUI’s)

What are Tangible User Interfaces?

Physical World Digital worldTUI’s

2

GUI vs TUI

(Ishii 2008)

3

Precursors

• E.g. Marble answering machine, Durrell Bishop, Royal College of Art, Interaction Design, 1992. http://vimeo.com/19930744

4

Early Work

• Fitzmaurice et al. 1995 – Graspable user interfaces http://www.youtube.com/watch?v=V-TGEe-Imro

• Ishii et al. 1997 - Tangible bits

5

Recent work

• E.g. Lumino, Baudish et al, 2010 http://www.youtube.com/watch?v=tyBbLqViX7g

• E.g. Portico, Avrahami et al, 2011 http://vimeo.com/29359319

6

Benefits of TUI’s (Ishii 2008)

1. Double interaction loop -immediate tactile feedback

2. Persistency of tangibles

3. Coupled input/output space

4. Special vs generic purpose

5. Space-multiplexed vs time-multiplexed input

Also fun + engaging!

7

• Sense position & orientation on touch technology

• Add intelligent drawing support

• How can they be best combined with multi-touch surfaces for enjoyable and productive interaction?

Can we connect these physical drawing tools to the digital space?

8

Our Approach

• Design– Tangible hardware

• Implementation– Recognizer– Drawing application

• Usability evaluation

Capacitive Touch (CapTUI) Infrared Touch (TanGeo)– Ryan Tan & Bryan Chen

– Rachel Blagojevic

– Jacky Zhen

9

CapTUI Technology – Capacitive

• Small touch screens e.g. iPad, smart phones etc…

• Touch detection via electrical pulse from fingers/conductive material

10

Design: Tangible Hardware

11

Final Design

12

Implementation: Tangible Recognition

• Tangible ID

– 3 point (min) unique patterns

Valid patterns Invalid patterns13

Implementation: Tangible Recognition

• Learning phase

• Recognition phase

– Touch point detection

– Match point distances to saved tangible ID’s

– No way of knowing which part of the touch point is in contact (+/- error)

14

Implementation: Drawing Application

• Beautification

Ink-to-edge snapping Corner snapping

& Length visualization 15

Implementation: Drawing Application

• Visual drawing guides

Tangible outline Angle visualization16

Evaluation

• First iteration: usability

– Simple drawing tasks

• Second iteration: comparative study

– Recognizable vs non recognizable drawing tools on screen

17

Usability Evaluation

• Can users construct simple drawings using the tangibles? Is the system usable?

• 10 participants

• 5 simple drawing tasks

18

Usability Evaluation: Results

• First exploration– Technology works– Is usable for simple drawings

• Tangible detection problems– Stability– Consistent circuit– Finger to tangible contact– Friction with screen– Comfortable drawing

• Tangible outline helpful – recognition indicator• Drawing guides needed

19

Comparative Study

• Does CapTUI assist users to easily draw precise geometric drawings

• Recognizable vs non recognizable drawing tools on screen

• 12 Participants

20

Comparative Study: Results

• CapTUI rated significantly higher than Paint overall• Visual guides helpful for precise drawing

– significantly lower average angle error.

• Participants enjoyed using CapTUI significantly more • Participants believed that CapTUI produces significantly

more tidy drawings than Paint.• Making fine grained movements with the tangibles

difficult• Tangible design still needs work

– consistent detection– accurate positioning

21

Tangeo Technology – Infrared

• Table tops e.g. Microsoft Surface 2.0

• PixelSense - Touch detection via infrared reflection for each pixel

• Image processing on detected pixels

• Identifies finger/blobs/tags

22

Design: Tangible Hardware

23

Final Design

24

Implementation: Tangible Recognition

• Learning phase

• Recognition phase

– Detection via custom tags

– Use tag location to get tangible outline

– Use thresholds for blob sizes

25

Implementation: Drawing Application

26

Implementation: Drawing Application

• Visual Guides

– Tangible outline

– Angle visualisation

– Length visualisation

– Ink beautification

• corner snapping

• ink-to-edge snapping

27

Usability Evaluation

• Can users construct simple geometric drawings using Tangeo? Is the system usable?

• 2 phase cycle

• 8 participants

• 4 drawing tasks

28

Usability Evaluation: Results

• Enjoyable / easy to use the tangibles

• Good recognition

• Visual guides helpful and easy to understand

• Drawing accuracy – less positive perception

• Add stylus for drawing

29

Oroo’ Tangibles

• Don’t need accurate sensing

• Extendable set

30

Design Methodology

• Participatory design (ie users make decisions)

• First prototype

• 3 x elders review

– Need to be ‘real’

• Community meeting

– 10 core

31

Oroo’ tangibles

• Web based html5 + Css

– Works on safari (iPad) and Chrome (everything we have tested)

• Video

32

This year

• Tangrams – Nazish

– Spatial relationships between tangibles

33

This year

• Jenga (Simon and Ben)

– 3D-2D-3D interaction

34

References

• Marble answering machine - Crampton Smith, G. The Hand That Rocks the Cradle. I.D., May/June 1995, pp. 60-65.• Fitzmaurice G. W., H. Ishii, and W. Buxton. 1995. Bricks: laying the foundations for graspable user interfaces.

In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '95), ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 442-449.

• Ullmer B. and H. Ishii. 1997. The metaDESK: models and prototypes for tangible user interfaces. In Proceedings of the 10th annual ACM symposium on User interface software and technology (UIST '97). ACM, New York, NY, USA, 223-232.

• Ishii H., B. Ullmer, Tangible bits: towards seamless interfaces between people, bits and atoms, Proceedings of the SIGCHI conference on Human factors in computing systems, p.234-241, March 22-27, 1997, Atlanta, Georgia, United States

• Ishii H., 2008. Tangible bits: beyond pixels. In Proceedings of the 2nd international conference on Tangible and embedded interaction (TEI '08). ACM, New York, NY, USA, xv-xxv.

• Baudisch P., T. Becker, and F. Rudeck. 2010. Lumino: tangible building blocks based on glass fiber bundles. In ACM SIGGRAPH 2010 Emerging Technologies (SIGGRAPH '10). ACM, New York, NY, USA, Article 16 , 1 pages.

• Avrahami D., J. Wobbrock, and S. Izadi. 2011. Portico: tangible interaction on and around a tablet. In Proceedings of the 24th annual ACM symposium on User interface software and technology (UIST '11). ACM, New York, NY, USA, 347-356.

• Blagojevic R., X. Chen, R. Tan, R. Sheehan, and B. Plimmer. 2012. Using tangible drawing tools on a capacitive multi-touch display. In Proceedings of the 26th Annual BCS Interaction Specialist Group Conference on People and Computers (BCS-HCI '12). British Computer Society, Swinton, UK, UK, 315-320.

• Zhen, J. S., R. Blagojevic and B. Plimmer (2013). Tangeo: Geometric Drawing with Tangibles on an Interactive Table-Top. CHI 2013. Paris France, ACM. WIP: in press.

• Shaer O. and E. Hornecker (2010) "Tangible User Interfaces: Past, Present and Future Directions",Foundations and Trends® in Human-Computer Interaction: Vol. 3: No 1-2, pp 1-137.

• Plimmer, B., He, L., Zaman, T., Karunanayaka, K., Yeo, A.W., Jengan, G., Blagojevic, R., Do. E.Y-L., (2015). New Interaction Tools for Preserving an Old Language. Chi 2015. Seoul, ACM: pp 3493-3502

35