Andrea Piras - Playing VKards to support cognitively impaired people on meal preparation

Playing VKards to support cognitively impaired peopleon meal preparation

Workshop on Interaction Techniques and Metaphors in Assistive Smart Environments, Salzburg – Austria, 18/11/2009

Andrea Piras

Playing VKards to support cognitively impaired peopleon meal preparation

Andrea [email protected], Italy

Sylvain [email protected] of Sherbrooke, Canada

Jeremy [email protected] & Management SudParis, France

Workshop on Interaction Techniques and Metaphors in Assistive Smart Environments, Salzburg – Austria, 18/11/2009

mailto:[email protected]

Salzburg, November 18th, 2009 IntTech09 3

Cognitive Deficiencies

They effect the abstract human faculty of information process and knowledge applying reasoning, inference, decision-making, comprehension,

planning, ... diseases examples: traumatic brain injury, dementia,

Alzheimer's disease, mental retardation, schizophrenia, autism

Millions of people lose temporarily or permanently the ability to rightly use objects and successful complete activities of daily living (ADLs)

People become dependent by families and Health Institutes


A solution

Augment the people's house introducing a set of not-intrusive technologic solutions to support in ADLs and promote independent living A net of sensors monitors user actions A net of effectors helps user to find objects A task assistant Change the traditional GUI-based interaction

Teaching and supporting at the same time Making some actions “automatic” Providing help when required


Archipel

It is an autonomous cognitive assistant for ADLs designed for smart homes and usable by user with lack of initiative who is alone in the kitchen

ADL is defined like a decision tree and includes the actions to address errors

It knows the position of sensors and objects


Archipel

The input received by sensors and by explicit user actions lets to determine if the user performs wrong actions and provide the right support

Output is exposed on appliances, furniture and screen

It is installed in the DOMUS Laboratory at the University of Sherbrooke


Focus our attention on ...

Kitchen It is the room where people spend a lot of time

performing actions and interacting with others

Meal preparation It is a privileged moment for promoting self-esteem

and enhancing autonomy

Recipe selection and search of ingredients and utensils The preliminary phases of meal preparation


Change the interaction

Mouse, keyboard and screen have drawbacks to any category of users but they are harder for cognitive impaired people

We focus on TUI and banish traditional GUI to a secondary rule

On preliminary steps of meal preparation,

Archipel mainly collects input from one

touch screen Enhance its tangible features Substitute the screen with cards


Virtual Kitchen card – VKard

We borrow something from four different fields: Picture cards and card games

Related to fun time of people ranging from child to aged To reuse such experiences and the pleasure to play to

help and train

Visual supports used in Picture Exchange Communication System (PECS) used in AAC For texts and images on the cards

Token+Constraint system To take decision when user moves cards

Visual tags (Amoeba) To recognize cards


VKard examples

From visual supports of PECS: one image, the name of its subject (eventually the quantity), Amoeba visual tag on the top right corner, white background


Searching an object ...

Token: VKard Constraint: Virtual Kitchen Table (VKTable) and

its four areas


The procedure

1. Go to VKTable

2. Select the recipe putting its VKard in the recipe area (recipe recognized by its visual tag)

3. For each ingredient/utensil:a) Place VKard on the search area (item recognized thanks to the visual tag,

related draw’s light on)

b) Move in the kitchen

c) Open draw

d) Check if she has it

e) Close draw (related draw’s light off)

f) Return on VKTable

g) Move VKard onto missing items area or on found items area (add item to the list of missing or found objects)

in italics, operations automatically performed by Archipel


Search completed:the system ...

... has helped to address user errors: user does not open/close related ingredient container user does not move VKard from search area to missing or found area user does not place VKard of next ingredient to search in search area user places two recipe VKards on the recipe area user places two ingredients on search area user places an ingredient VKard on recipe area

... implicitly knows: the chosen recipe if required ingredients are available and it is possible to prepare that

recipe ingredients/utensils are missing


Search completed: the user ...

... has successfully completed the ADL without striving to remember: the ingredients and utensils used for a recipe the names of the objects next object to search their position in the kitchen the quantity the list of missing objects the list of found objects

She just plays with VKards


Pros & Cons

Pros: Cheapness (just a colour printer and one

camcorder) Ingredients and utensils can be searched any

time and with in any order also without to be connected to a recipe

Cons: Need of constraint areas Occlusions could not let the camcorder to detect

the VKard


Future works

No constraint area Feedback direct on the VKard

Integrating electronics inside it

Use VKards in other ADLs to lie table, to shop ingredients, to place

bought items on the right position, to wash dishes, ...


Conclusions

Augmented kitchens can help cognitive impaired people to autonomously and successfully complete meal preparation

Sensor networks and cognitive assistants must be coupled with a user interaction extremely simple and intuitive

VKards can be used for recipe selection and search of ingredients and utensils in the kitchen For cognitive impaired people having lack of initiative and/or memory

To reduce their rejection we mainly borrow by card games and by PECS (well-known practice in autism)

This is an on-going work Tests with users must be performed

Archipel, VKard and VKTable are parts of the Virtual Collaborative Kitchen (VCK) vision


Grazie!

Gracias! Obrigado!Thank you! Merci!

Danke! Arigato!Xiè_Xiè! Efharisto!Aitäh! Kiitos!

Spassiba! 感謝

Technology

Andrea Piras - Playing VKards to support cognitively impaired people on meal preparation