Blind-enT: Making objects visible for blind people* Waldemar Bartyna (speaker)

S. Ambroszkiewicz; M. Faderewski; S. Jakubowski; D. Kocieliński; D. Mikułowski; G. TerlikowskiInstitute of Computer Science, University of Podlasie, POLAND

*)The work was partially supported by a grant from the British Council

The goal of our project

• a blind person can perceive visual attributes of objects, e.g., door, pillar, ditch, elevator, passage, room, hall, etc.,

• having only a mobile cellular phone in his/her hand, e.g., Nokia 6600.

• can familiarize environment (e.g., building, street, town) by getting to know a cognitive map

• can be given a short and safe path on the map • Guidance along this path

What is needed?

• mobile phone alone (even as smart as Nokia 6600) is not enough

• special infrastructure is needed to provide information about objects and environment to the mobile

• the information is processed and delivered to the blind user as voice

• mobile is a computer connected (via GPRS, Bluetooth, and infrared) to other computers and devices

System architecture

Internet

R-server

elementaryobject

complexobject

Blue- server

mobile

IrDA

GPRS

Bluetooth

The basic idea

• unlike the famous „Talking Signs”, the object description as well as a map of complex object is not in a voice format

• it is a special data structure expressed in terms of generic attributes and types

• it can be processed automatically • only the result of such processing is delivered

to a blind user as voice• THE IDEA: generic object attributes and

types

Attributes and types

• object is of some type, and has some attributes

• type and attributes are specified in uniform and digital form

• once the object is identified, they can be delivered to the mobile

• how to get object’s identifier?• via infrared or/and Bluetooth transceivers

embedded into the objects

Object description

• Object’s identifier and short textual info• Type name • List of attribute names and their values.• Object position: X, Y, and Z coordinates • List of identifiers of superior objects • List of identifiers of sub objects• List of neighbours. A neighbour consists of an

identifier, the azimuth along with the distance from the object to this neighbour, as well as a short textual info.

Elementarny Objects

wallwall

Neighbours: straight safe and direct path between objects

azimuth and distance to neighbor

exit2exit1

Object oriented maps:

complex objects

……

platform1

Railway station

hall

ticket office exit2 exit1 tracklamp

postexit3

hall and platform1 are neighbors

Object hierarchy

• Object hierarchy: super and sub objects • Object oriented map: hierarchical collection of

object descriptions– the root object (e.g., a complex object: town or

large building), – its sub-objects, sub-sub-objects, and so on up to

the elementary objects.

• Map can be represented as a tree• Restriction of the tree to a branch of length n

can be viewed as the scale n of this map

Virtual tours

• having object oriented map, blind user can familiarize the surrounding area

• the mobile can display descriptions of neighbour objects, superior object, and sub objects

• this is delivered to blind user as voice• the blind user can navigate between the

objects on the map acquiring knowledge about them

XML data structures

• data structure for representing object attributes, types, identifiers, and descriptions are specified in XML Schema

• attribute, type, object identifier and description are instances of the corresponding schema

• these schemas are available from the project web site

The algorithm of hierarchical routing

…

building2

street1

Railway station

building1

…street2

• Graph of the neighbourhood of the immediate sub-objects of town

The algorithm of hierarchical routing

1-st floor

… stairs1

ground floor

2-nd floor

stairs2

• Graph of the neighborhood of the immediate sub-objects of a building

Prototype implementation

• object oriented maps constructed• the maps served for testing the routing

algorithm and virtual tours by blind and sighted users

• elementary objects were simulated by the second mobile phone (Nokia 6600).

• Blue-server was implemented on this very notebook

Prototype implementation

• R-server implemented on a PC-workstation: ii4.ap.siedlce.pl

• Bluetooth, infrared and GPRS connectivity has been verified

• the application (for blind user) on the mobile phone (Nokia 6600) implemented in C++ directly on the Symbian OS

Prototype implementation

experiments proved our claim:

mobile phone is enough for making objects visible for blind people

Conclusions

• THE PROBLEM: short range of infrared connectivity (up to 6 meters) and Bluetooth connectivity (up to 10 meters).

• Once this problem is solved (by a manufacturer) the system can be applied in the real environment.

• Prototype implementation is ready for demonstration after the session.

• Visit our web site, http://ii4.ap.siedlce.pl