Embed Size (px)
Citation preview
Towards a Natural User Interface to Support People with Visual
Impairments in Detecting Colors Sergio Mascetti ([email protected])1,2
Chiara Rossetti ([email protected])1
Andrea Gerino ([email protected])1,2
Cristian Bernareggi ([email protected])1,2
Lorenzo Picinali ([email protected])3
Alessandro Rizzi ([email protected])1
1 Università degli Studi di Milano, Department of Computer Science2 EveryWare Technologies
3 School of Design Engineering Imperial College London
From computer accessibility to real world accessibility
• Mainstream computers, mobile and wearable devices are accessible to people with visual impairments.
• Nowadays, assistive technologies aim to support daily activities of people with visual impairments through these devices.
Problem
Detecting the color of real world items (e.g., bottles, boxes, clothing) by blind and visually impaired people.
Major existing solutions• Applications for mobile devices that:
• sonify color images captured by the camera (e.g., EyeMusic, SeeColor);
• read the name of the color in the center of the field of view of the camera.
• Two issues:
• Sonification techniques require a lot of training
• It is difficult to point the camera on the right spot
Contribution
• JustPoint: an Android application that detects color through a natural user interface.
• Evaluation of JustPoint.
JustPoint works in 5 stages1.acquires images from the device camera;
2.detects a marker (a QR-code) on a ring placed on the user's fingertip;
3.computes the target area being pointed by the user;
4.identifies colors in the target area;
5.reads the colors to the user.
1 2 3 4 5
Evaluation
• Ongoing evaluation by a congenitally blind designer.
• Three sessions with blind subjects to evaluate:
• how they point the camera;
• the first prototype;
• a final prototype.
How to point the camera• Two blind subjects recorded videos while simulating the
use of JustPoint.
• Results
• The device is hold too close to the pointing hand.
• Blind people are often unaware of light conditions.
• Action
• On first run the application provides training
• Flash is automatically turned on when needed
First prototype• Three blind subjects used the prototype to detect the
color of three items.
• Results
• They needed hints in detecting the QR-code.
• The color name was not accurate in RGB.
• Action
• Audio hints are reproduced while pointing
• Use of the HSV color space
Final prototype
• Three blind subjects detected the color of different items by using JustPoint.
• Results
• They were able to detect the color correctly.
• The time for pointing and detecting was about one second.
Future work
• A natural user interface based on finger detection.
• Use of smart glasses.
• Sonification of colors.
