NFC-based User Interfaces

NFC-Based

User Interfaces

Jukka Riekki, Ivan Sanchez, Mikko Pyykkönen

Department of Computer Science and Engineering

University of Oulu

Oulu, Finland

http://www.oulu.fi/cse

NFC 2012 4th International Workshop on Near Field Communication

March 13th, 2012

Helsinki, Finland

These slides were modified slightly after the presentation. Three videos were replaced with text and figures. Two of the videos are available at YouTube, links are provided on the corresponding slides.

Table of Contents

• Introduction

• Previous work

• Interaction model

• Advertising NFC tags

• Conclusions

Introduction

• NFC can have more profound effect on our

everyday lives than has been discussed in public – Transit and Ticketing, Payment, and Advertising applications are

just the tip of the iceberg

• NFC-based user interfaces can be built to

any Internet or mobile phone application;

to any device connected to Internet – Generally, NFC can be used to replace manual typing, menu

selections, and other user interface actions with acts of touching.

– For example: household appliances with an Internet connection;

see the next slide

Introduction

When will this application be published?

Introduction

• We see significant potential in interactive

spaces – Environments equipped with rich user interfaces and

offering a large variety of services for users

– Focus on user control – instead of intelligent environments

observing the users and activating services autonomously

• The challenge is to provide easy-to-use user

interfaces – The available services need to be communicated clearly to

users; how they can be controlled, and the devices

the services use

– The actions for controlling the services need to be easy to

perform.

Introduction

• NFC excels in tackling this challenge – Graphical icons advertise services, commands & devices

– NFC tags are placed under icons

– Available services can be identified by visual scanning

– Commands performed by touching icons with NFC phones

• We study interactive spaces – We have built many prototypes containing NFC-based user

interfaces

• This talk presents some of these prototypes and

our current work on: – Interaction model for NFC-based user interfaces

– Graphical language for advertising NFC tags

Previous Work

• Advertising NFC tags

• Prototypes:

– Positioning Prototype 2006: NFC + WLAN positioning

– Information Board Prototype 2007: Information board map + tags

– Touch & Control Prototype 2007: Controlling a wall display

– Touch & Run Prototype 2008: Outdoor game

– Touch & Collect Prototype 2008: Pupils collect content from a

museum

– EPG Control Prototype 2009: UI for Electronic TV Program Guide

– PlaceMessaging Prototype 2009: Leaving messages on a place

– Touch & Learn Prototype 2009: Learning languages

– iPoster Prototype 2010: Poster presentation with NFC UIs

– Touch & Learn to Read Prototype 2010: Children learn to read

• Next, some of this work is presented in more detail

Previous Work: Icons

• Research on NFC icons is quite rare

• We have designed several sets of NFC icons

Previous Work:

Icons

EPG control board: over 30 icons & tags

iPoster: a cartoon contains the icons and gives instructions

Touch & Control Prototype 2007

• NFC tag is placed next to a wall display

• Touching the tag starts a multimedia

player on the wall display

• The player can be controlled – With phone UI (BOTTOM LEFT)

– By touching icons on the Control Panel (BOTTOM MIDDLE)

– By touching icons on the Control Cube (TOP RIGHT)

• Minimalistic phone UI suffices for the panel and

the cube – a single image (BOTTOM RIGHT)

• Video: http://www.youtube.com/isgoulu

ControlCube

Previous Work: Lessons learnt

• Carefully designed graphical icons facilitate recognizing:

• The services available in the local environment

• How the services can be controlled

• But not much has been done to develop the visual

appearance of NFC user interfaces

• Touching is a natural way of interacting with

our everyday environment

• Users can focus on their activities (calm computing!)

• This interaction model results in users being present

in the environment

• And interacting directly with other users

• Instead of each user focusing on own mobile terminals'

display (or on PCs’ or tablet PCs’ display)

Previous work: Lessons learnt cont’d

• Most of our prototypes have been tested with real users

• Feedback has been very positive

• In fact, phones with very modest resources can be used

to build impressive user interfaces

• Large touch displays are not mandatory

• Rather, a small phone easy to hold is better!

• Lots of different user interfaces can be built with

the current technology

• Many applications need software only for phones (no server);

the tag is the database!

• So applications can be kept simple and robust

• And using the applications is cheap (no communication costs)

Interaction model: Motivation

• Interactive space UIs differ from traditional GUIs – UI is spread in the environment

– Tangible UI, i.e. physical objects are handled

• An interaction model can be used to – Identify and define the UI components (objects, devices,…)

– Structure and model the interaction

– Study alternatives

• In addition – The model might be used to present

recommended interaction patterns,

for example

Interaction model: Elements

• User: a person in an interactive space

• Token: a physical object in the space

– Users can control and observe the states of tokens

• Service: a digital object consisting of a set of

functionalities and policies controlling their usage

– Users achieve their goals with these functionalities

• Resource: a device in the physical space

• Services use resources to realize

their functionalities

– To recognize the states of tokens

– To change the states of tokens and

other environment

– To present information to users

– To get input from a user (e.g. touch

display)

Interaction model: Tokens & Resources

• The mapping between tokens and resources

determines – How services experience the user-token interaction

– How users experience the service-resource interaction

• An object (e.g. an NFC phone) can be – From the user perspective a token (handled as physical object)

– From the service perspective a resource

• NFC tags are resources as well

• When NFC readers are

placed in the environment – User handles tokens equipped

with NFC tags

Interaction model: Stages

• Discovery Stage: a user scans visually the space

to find the tokens to interact with

• Composition Stage: the user composes the

application he or she wants to use – By selecting services and tokens

– E.g. by touching tokens with a phone; NFC tags in tokens

• Usage Stage: The tokens

are handled – User communicates with the

application by interacting

with the tokens in the physical

space

– Services interact with resources

in the digital space

The model elements in the environment

Advertising NFC tags

• We advertise NFC tags with service advertisements

– Graphical advertisements communicating information about

interaction possibilities

– And offering the means for realizing the interaction


• A service advertisement specifies – The exact position to touch

– All details a user needs to know when making

a decision on whether to touch a tag or not

• Attention element: draws a user's attention

to the advertisement – This element is included into each advertisement

• Technology element: indicates the technology

utilized in the interaction – Also other interaction technologies,

like 2D barcodes and Bluetooth,

can be used


• Interaction element: advertises the exact point

to touch – NFC tags are placed under these elements

• Action element: indicates the action the system

performs when a user touches the interaction

element – The related NFC tag stores the command to execute that action, or

the data the system maps to the command

• Context element: can be

used to present, for example,

the service to be commanded


• Instruction element: explains how the interaction

is to be performed; text, cartoon, etc – Separate instruction elements can give general instructions about

services in a certain area

• A service advertisement does not have to contain

all these elements

• When users become familiar with NFC and some

details are known otherwise,

less details suffice in

service advertisements

Advertising NFC tags: Examples

• Service advertisement for a photo album service presenting

photographs on a wall display:


• Advertisement for a voting service:


• Multimedia player service for a tourist attraction:


• Service advertisement for connecting a phone to a photo printer. Left:

printing is free. Right: printing costs 2,5 Euros:


• Service ad for downloading and joining a game shown on a wall display:


• Maps and contact information by touching:

Conclusions

• NFC can revolutionize the way we use applications

in our everyday life – a single touch to give a command

• We have good experiences on:

• Applications related to a place and/or using local resources

• Using objects in environment as data containers

• Learning applications

• NFC is ready for large scale usage – for a rich set of

different applications

• Simple phones are sufficient

• We continue building prototypes and developing the

model and the language

• The model can be used when designing applications and to

communicate good practices

• The visual language can advance common understanding

Contact information and acknowledgements

• Contact information: [email protected]

– http://www.oulu.fi/cse

– http://www.youtube.com/ISGOulu

• Acknowledgements

– This presentation was enabled by numerous researchers from

the University of Oulu and from our partners. The EPG

application was developed at the University of Tampere. The

work has been funded by Tekes, EU, regional funding agencies

and companies

SELECTED PUBLICATIONS

Riekki J, Sanchez I, Pyykkönen M (2012) NFC-Based User Interfaces. 4th International Research Workshop with focus on Near Field

Communication (NFC2012), March 13th, 2012, Helsinki, Finland. Invited paper. (THIS TALK)

Riekki J, Cortés M, Hytönen M, Sánchez I and Korkeamäki RL (2012) Touching Nametags with NFC Phones: A Playful Approach to

Learning to Read. LNCS Transactions on Edutainment. Springer Verlag. In press.

Sanchez I, Cortes M, Riekki J & Pyykkönen M (2011) NFC-Based Physical User Interfaces for Interactive Spaces. In: Near Field

Communications Handbook (Internet and Communications). Ed Syed a. Ahson and Mohammad Ilyas. Auerbach Publications. 175-230.

Pyykkönen M, Riekki J, Alakärppä I, Sánchez I, Cortés M & Saukkonen S (2012) Designing Tangible User Interfaces for NFC Phones. In

Kiyokawa, K. (Ed.) Advances in Human Computer Interaction. Hindawi Publishing Corporation. Volume 2012.

http://dx.doi.org/10.1155/2012/575463

Riekki J, Sanchez I & Pyykkönen M (2010) Remote control for pervasive services, Int. J. Autonomous and Adaptive Communications

Systems, Vol. 3, No. 1, pp.39–58. http://dx.doi.org/10.1504/IJAACS.2010.030311.

Riekki, J. (2007) RFID and smart spaces, Int. J. Internet Protocol Technology, Vol. 2, Nos. 3/4, pp.143–152.

http://inderscience.metapress.com/link.asp?id=d3t8lmph214k0tl2

Riekki J, Salminen T & Alakärppä I (2006) Requesting Pervasive Services by Touching RFID Tags. IEEE Pervasive Computing, Jan.-

Mar. 2006. 5(1)40-46. http://dx.doi.org/10.1109/MPRV.2006.12.

Riekki J, Cortés M, Hytönen M, Sánchez I & Korkeamäki R-L (2011) Learning to Read by Touching Nametags with NFC Phones. The

Fourth International eLearning Baltics Conference (eLBa 2011), 26-27 May 2011, Rostock, Germany.

Sanchez I, Davidyuk O & Riekki J (2009) Towards User-Oriented Application Composition, In Proc. the IEEE Workshop on Pervasive

Service Computing and Applications (PSCA'09), December 17-19, Shanghai, China.

Sanchez I, Riekki J, Rousu J & Pirttikangas S (2008) Touch & Share: RFID Based Ubiquitous File Containers. In Proceedings of 8th

International Conference on Mobile and Ubiquitous Multimedia (MUM08), December 3-5 2008, Umeå, Sweden. 57-63.

http://dx.doi.org/10.1145/1543137.1543148

Sanchez I, Riekki J, Pyykkönen M (2008) Touch & Control: Interacting with Services by Touching RFID Tags. In Proceedings of the 2nd

International Workshop on RFID Technology - Concepts, Applications, Challenges (IWRT 2008), In conjunction with ICEIS 2008.

Barcelona, Spain, June 12-13, 2008. pp 53-62.

Riekki J, Sanchez I, Pyykkönen M (2008) Universal Remote Control for the Smart World. F.E. Sandnes et al. (Eds.): UIC 2008, LNCS

5061, Springer-Verlag, pp. 563-577. http://dx.doi.org/ 10.1007/978-3-540-69293-5_44

Sánchez I, Cortés M & Riekki J (2007) Controlling Multimedia Players using NFC Enabled mobile phones. In Proceedings of 6th

International Conference on Mobile and Ubiquitous Multimedia (MUM07), Oulu, Finland, December 12-14 2007.

http://dx.doi.org/10.1145/1329469.1329485