Towards a Natural User Interface to Support People with ...€¦ · Towards a Natural User...

Towards a Natural User Interface to Support People with Visual

Impairments in Detecting Colors Sergio Mascetti (sergio.mascetti@unimi.it)1,2

Chiara Rossetti (chiara.rossetti@studenti.unimi.it)1

Andrea Gerino (andrea.gerino@unimi.it)1,2

Cristian Bernareggi (cristian.bernareggi@unimi.it)1,2

Lorenzo Picinali (l.picinali@imperial.ac.uk)3

Alessandro Rizzi (alessandro.rizzi@unimi.it)1

1 Università degli Studi di Milano, Department of Computer Science2 EveryWare Technologies

3 School of Design Engineering Imperial College London

From computer accessibility to real world accessibility

• Mainstream computers, mobile and wearable devices are accessible to people with visual impairments.

• Nowadays, assistive technologies aim to support daily activities of people with visual impairments through these devices.

Problem

Detecting the color of real world items (e.g., bottles, boxes, clothing) by blind and visually impaired people.

Major existing solutions• Applications for mobile devices that:

• sonify color images captured by the camera (e.g., EyeMusic, SeeColor);

• read the name of the color in the center of the field of view of the camera.

• Two issues:

• Sonification techniques require a lot of training

• It is difficult to point the camera on the right spot

Contribution

• JustPoint: an Android application that detects color through a natural user interface.

• Evaluation of JustPoint.

JustPoint works in 5 stages1.acquires images from the device camera;

2.detects a marker (a QR-code) on a ring placed on the user's fingertip;

3.computes the target area being pointed by the user;

4.identifies colors in the target area;

5.reads the colors to the user.

1 2 3 4 5

Evaluation

• Ongoing evaluation by a congenitally blind designer.

• Three sessions with blind subjects to evaluate:

• how they point the camera;

• the first prototype;

• a final prototype.

How to point the camera• Two blind subjects recorded videos while simulating the

use of JustPoint.

• Results

• The device is hold too close to the pointing hand.

• Blind people are often unaware of light conditions.

• Action

• On first run the application provides training

• Flash is automatically turned on when needed

First prototype• Three blind subjects used the prototype to detect the

color of three items.

• Results

• They needed hints in detecting the QR-code.

• The color name was not accurate in RGB.

• Action

• Audio hints are reproduced while pointing

• Use of the HSV color space

Final prototype

• Three blind subjects detected the color of different items by using JustPoint.

• Results

• They were able to detect the color correctly.

• The time for pointing and detecting was about one second.

Future work

• A natural user interface based on finger detection.

• Use of smart glasses.

• Sonification of colors.

Questions?

Thank you for the attention

Cristian Bernareggi (cristian.bernareggi@unimi.it)