Alternative communication & access to information

Alternative communication & access to information

Department of Computer and Information SciencesUniversity of Tampere, Finland

January – March, 2003

Grigori Evreinov

www.cs.uta.fi/~grse/

Alternative communication & access to information

Department of Computer and Information SciencesUniversity of Tampere, Finland

ACAI

TAUCHI MMIG Evreinov G. p 01_23 22.01.2003

information may be defined as the characteristics of the output of a

process; these characteristics enhance knowledge, being informative about

the process and its input

hierarchies of processes, linked together, provide a communication channel

between each of the corresponding functions and layers in the hierarchies

fundamentally information is not modality-specific (i.e. not tied to its form of

presentation, be it visually, as sounds or whatever)

but the end user (destination, addressee) for whom the information exists,

is a person who is trained to manipulate through modality-specific notions

(which are formed by percepts)

therefore, within ‘information technology’ modal aspect of information

maybe more essential for development of relations between person and

computer [1]

http://www.fst.ch/FST2/al/default.htm

communication is defined as the transfer of information from a

communicator-source /originator to a communicator-receiver

the communication process allows people to share information, ideas, and

feelings; this is the transfer of meaning

where no meaning is transferred, no communication has taken place

the goal of communicators is to accomplish this process efficiently

(rationally) and effectively

elements of communication process [3]

ACAI


human communication has no other purpose than to cause some kind of

action - to direct, to inform, to question, and to persuade [2]

the challenge is to come up with the right combination of codes, media, and

contexts in order to make the transfer of information fast, cost effective,

and accurate

ideas and feelings can be shared only if they are represented by symbols

symbols are things that stand for something else

signs are literal; symbols are not

a symbol means more than it literally says [4]

all of our communication messages are made up of two kinds of symbols:

verbal and nonverbal

symbols can have several kinds of association like personal, cultural or

universal

interpretation of symbols’ meaning depends on the set of parameters,

conditions and features of elements involved in communication process

ACAI


ACAI


the ideal message is one that is received & interpreted as it is intended

a receiver is a one or more individuals for whom the message is intended

to be an effective, communicator first gain and keep the receiver's attention

communicators must have similar knowledge /background with regards to

usage of the medium or get knowledge before during learning

communication conditions

noise is interference of signals with similar parameters in analyzers that

keeps a message from being understood or accurately interpreted

psychological noise occurs in the minds of the communicators and the

receivers when they are distracted by something

there are different types of communication: intrapersonal, interpersonal,

interviews, small group, and public

Your Appearance andPhysical and Psychological Condition

Your Moods and Feelings

Your Social RolesYour Social Traits

Talents YouPossess or Lack

Your Intellectual Capacity

Your Strong Beliefs

HealthyGood frame of mindNeatCleanStylishEtc.

HumorAngerHateLoveContentmentEtc.

OutgoingAssertiveWarmEmpathicEtc.

ParentFamily providerCommunity leaderProfessional personEtc.

ArtisticMusicalAthleticWritingSpeakingEtc.

LogicalReflectiveStudiousSpeculativeEtc.

Your Self

What you sayWhat you thinkWhat you feel How are you

ACAI


intrapersonal communication occurs within usit involves thoughts, feelings, and the way we look at ourselves [2, 5]

ACAI


communication messages are made up of two kinds of symbols:

verbal and nonverbal

voice set (pitch, volume, rate and rhythm) is defined by the speaker’s

physical and emotional condition

certain kinds of voice sets are strongly classified with gender and age

there are three types of nonverbal vocalizations [7]

vocal characterizers (yawning or belching )

vocal qualifiers (momentary variations of speech or volume,

personality traits )

vocal segregates (sounds or silences made between expression

of words, such as ahh, hah, mmmh, and include pauses during speech,

which can convey an attitude of calmness and reflection) [6]

however, the information and richness contained in paralanguage, music,

and background sounds is lost to the deaf viewer [10, 11]

Demo 1 Demo 2 Demo 3 Demo 4 Demo 5 Demo 6 Demo 7 Demo 8 Demo 9 Demo10

[9] http://www.esl-lab.com/para.htm

http://www.esl-lab.com/para.htm


on the contrary, for blind viewer… how to transform emotional

expression from the picture into sound/earcon or whatever [12]

there is not any system to reveal an emotional access to pictures for

blind persons

surveys show that it is very important for blind to transport emotions

independently of the content especially for modern images

the sonification system can enable an emotional access using the

similarity of feeling by watching a picture or hearing music and

transforming visual information in music compositions

results of psychological, physiological and emotional acoustic

researches, methods of image analysis and the rules of music

composition are basis for this translation and were describe by

Schwende in Auditory Emotional Access to Visual Information [33]

ACAI


on the contrary, for blind viewer… how to transform emotional

expression from the picture into sound/earcon or whatever [12]

there is not any system to reveal an emotional access to pictures for

blind persons

surveys show that it is very important for blind to transport emotions

independently of the content especially for modern images

the sonification system can enable an emotional access using the

similarity of feeling by watching a picture or hearing music and

transforming visual information in music compositions

results of psychological, physiological and emotional acoustic

researches, methods of image analysis and the rules of music

composition are basis for this translation and were describe by

Schwende in Auditory Emotional Access to Visual Information [33]

Sonification of facial expressions

ACAI


http://www.newag.org/funpics/weard/html/imagepages/image68.htmhttp://www.berghuis.co.nz/abiator/weird/ears.html, http://www.velvet.net/~poppy/kitch/, http://www.domainnamesanity.com/default.php, http://www.dogandponysound.com/ico.htm

propose or design a system of very brief eARmoticons which could

describe a facial expression and evoke the same reaction

Sonification of facial expressions

http://www.newag.org/funpics/weard/html/imagepages/image68.htm

http://www.newag.org/funpics/weard/html/imagepages/image68.htm

http://www.berghuis.co.nz/abiator/weird/ears.html

http://www.velvet.net/~poppy/kitch/

http://www.domainnamesanity.com/default.php

http://www.domainnamesanity.com/default.php

http://www.dogandponysound.com/ico.htm






tactile diagrams are relatively easy to produce yet there are no

standard guidelines to assist during the design process

attempting to include as much information as possible can lead to

diagrams that are simply too confusing to be of use

research into a project on non-visual access to music notation

(Challis, “Weasel”) has shown that an interactive multimodal

approach to presentation could be of benefit in these

circumstancesWeasel, http://www.benchallis.com/research.htm#WeaselNational Centre for Tactile Diagrams http://www.nctd.org.uk/GEWA Access Computer Keyboards - Remote Computer Access http://www.zygo-usa.com/accessir.htmlTellus und Mind Express http://www.weissenstein-bs.de/produkte/softw/mindexpr/mindexpr.htm

ACAI


http://www.benchallis.com/research.htm#Weasel

http://www.nctd.org.uk/

http://www.zygo-usa.com/accessir.html

http://www.weissenstein-bs.de/produkte/softw/mindexpr/mindexpr.htm

ACAI


communication of mathematics is usually visual - formulas, diagrams,

graphs etc.

that makes it very difficult for blind and partially sighted people /students

to do mathematics and is one of the biggest obstacles for them in school

and at the university [13-20]

6-dot Braille can represent 64 unique characters

through the use of special sequences Braille can support a much larger

set of characters, that is, the basic characters can have different meaning

in different contexts

while there is effectively no limit to the number of characters one can

represent in the fashion, the reading and understanding of these special

sequences becomes very involved [13]

the more new characters one needs, the more “escape” sequences one

requires

e.g., the letter “a” could be an “a”, an “A”, a “1”, the “first finger” in music

notation and so on

Access to Mathematics

http://presentations.animfactory.com/pmpjumbos.html

http://presentations.animfactory.com/pmpjumbos.html

[24]

ACAI



most of the work reported in developing techniques and deal with

mathematics can be presented through the next categories:

tactile as in Braille and other raised representations

audio aids that read equations with tools to help in the reading

process

tonal representations of equations and graphs

(sonification/audification)

haptic or forced feedback devices represent shapes of objects

and curves

integrated /multimodal approaches

Y = X Y = |sin X|

[25]

** http://www.gfai.de/sachsen/blind_i.htm

*** http://www.seeingwithsound.com/winvoice.htm

**

Y = 5sinX ***

http://www.gfai.de/sachsen/blind_i.htm

http://www.gfai.de/sachsen/blind_i.htm

http://www.seeingwithsound.com/winvoice.htm

http://www.seeingwithsound.com/winvoice.htm

ACAI


the Mathtalk command language is based on the concept of a action and a

target

that is the user might choose to hear the (whole) current expression, then

the current term (i.e. the first term of the expression), followed by the next

term (i.e. the second term of the expression) and so on

the command language is mnemonic, so that, the command to read an

expression is r e, to move to the next term it is n t, etc. [19]

[20]


ACAI


Stuttgarter Mathematikschrift für Blinde the Stuttgarter Mathematikschrift für Blinde (SMSB) is an 8-dot-Braille

which includes small and capital letters, the numbers and mathematical

symbols

SMSB provides three alphabets, the Latin, the old German and the Greek

alphabet, to write letters which are used in mathematics

capital letters differ from the small ones by the supplemented dot 7

each character of SMSB has a representation in Braille and one in print

no SMSB-character stands for a string, a group of characters or even a

word as it could be in contracted Braille

that means, that each SMSB-character can be shown on a Braille-output-

device as well as on a screen and may be printed [22, 23]

SMSB provides tactile characters for symbols like


ACAI


SMSB uses characters, which can be understood intuitively by a person,

who is familiar with mathematics

symmetric Braille symbols are symmetric in meaning:


ACAI


Demo 11

DotsPlus is a tactile font, but unlike Braille, it requires no translation from

ordinary literature [27, 28]

any computer file that uses fonts with screen characters having the right

size can be printed directly in DotsPlus on the Tiger tactile graphics and

Braille embosser

DotsPlus has many more symbols than Braille

DotsPlus and the Tiger can make graphics such as maps, floor plans,

pictures of animals, etc.

http://dots.physics.orst.edu/dotsplus.html


http://dots.physics.orst.edu/dotsplus.html

ACAI


http://www.braille-scs.com/tech_code.htmlhttp://www.tedub.org/Start_tab.html

Special Braille Technical Drawings the domain of the technical drawings includes electronics, architecture and mechanical eng. they contain graphical items that follow industrial norms, like ISO or other standardstransforming visual info in technical drawings into an understandable form for blind people is very important for vocational training digitized technical drawings can be presented and edited on standard PCs with appropriate software blind people need to access a user interface and presentation tool specially tailored for this type of info

ACAI

TAUCHI MMIG Evreinov G. p 17_23 22.01.2003Demo12


ACAI

TAUCHI MMIG Evreinov G. p 18_23 22.01.2003Demo13


H J K

L G F

D S A

ZX

C

987

4 5 6

1 2 3

1 1

1

7 8 9

4 5 6

1 2 3

input & graphics sonification based on numerical keyboard’s paradigm

1:1 2:1 3:1

TL, a language for creating games for visually impaired and blind children

[17, 18]

this language is a part of the TiM (Tactile Interactive Multimedia computer

games for visually impaired children) project whose overall aim is to offer to

young visually impaired children the possibility to play with computer

games

ACAI


Gaze Control

http://www.metrovision.fr/mv-vi-notice-us.html http://www.delta7.asso.fr/Deltavision%202001/ecrire3.html the Camera Mouse http://www.cs.bc.edu/~gips/CM/ or http://www.cameramouse.com/

ACAI


http://www.metrovision.fr/mv-vi-notice-us.html

http://www.delta7.asso.fr/Deltavision%202001/ecrire3.html

[1] Edwards, A. D. N, Carey, K., Evreinov, G. et al. Information and Communication Technology in Special Education: Analytical Survey. Moscow: UNESCO, Institute for Information Technology in Education, 2001

[2] Effective Communication Skills, http://www.srsd.org/jrotc/Text%20Books/LEII_U~1.doc[3] The Diffusion of Web Icons. Communication,

http://chrissnider.com/academic/icons/Theories/Communication/communication.html[4] Definition of Symbols, http://web.umr.edu/~gdoty/classes/concepts-practices/def-symbols.html[5] Intrapersonal Communication Processes. Joan E. Aitken & Leonard J. Shedletsky (Eds.)

http://www.usm.maine.edu/com/intrap.htm[6] Non Verbal Communication, https://webspace.utexas.edu/QUAHS/WWW/[7] Eisenberg, A.M., & Smith, Jr., R.R. Nonverbal communication. N-Y: The Bobbs-Merrill Company, Inc.,

1971[8] John Carey, Paralanguage in Computer Mediated Communication,

http://acl.ldc.upenn.edu/P/P80/P80-1018.pdf[9] Paralanguage, http://www.esl-lab.com/para.htm[10] Towards Emotive Captioning for Interactive Television, http://www.ryerson.ca/clt/towards_emotive_

captionin_1.doc[11] Charles Silverman and Deborah I. Fels, Emotive Captioning in a Digital World, 2002, LNCS 2398, p.

292, http://link.springer-ny.com/link/service/series/0558/papers/2398/23980292.pdf[12] Frank Deconinck and Patricia Verschueren, A model of the understanding of graphical information by

blind people, TIDE103/VUB/930531-2, 1993 http://minf.vub.ac.be/~fdconinc/underst.rtf[13] Karshmer, A.I., Gupta, G., Gillan, D. Architecting an Auditory Browser for Navigating Mathematical

Expressions, ICCHP 2002, LNCS 2398, p. 477http://link.springer.de/link/service/series/0558/papers/2398/23980477.pdf

[14] Gaura, P. REMathEx - Reader and Editor of the Mathematical Expressions for Blind Students, 2002, LNCS 2398, p. 486, http://link.springer-ny.com/link/service/series/0558/papers/2398/23980486.pdf

References

ACAI


http://www.srsd.org/jrotc/Text%20Books/LEII_U~1.doc





http://chrissnider.com/academic/icons/Theories/Communication/communication.html

http://web.umr.edu/~gdoty/classes/concepts-practices/def-symbols.html

http://www.usm.maine.edu/com/intrap.htm

https://webspace.utexas.edu/QUAHS/WWW/

http://acl.ldc.upenn.edu/P/P80/P80-1018.pdf


http://www.ryerson.ca/clt/towards_emotive_captionin_1.doc







http://link.springer-ny.com/link/service/series/0558/papers/2398/23980292.pdf

http://minf.vub.ac.be/~fdconinc/underst.rtf

http://link.springer.de/link/service/series/0558/papers/2398/23980477.pdf



[15] Fitzpatrick D. Speaking Technical Documents: Using Prosody to Convey Textual and Mathematical Material, ICCHP 2002, LNCS 2398, p. 494, http://link.springer.de/link/service/series/0558/papers/2398/23980494.pdfhttp://www.computing.dcu.ie/~dfitzpat/publications.html

[16] Math project, http://www.cs.york.ac.uk/maths/index.html[17] Prosody in Mathtalk http://www.cs.york.ac.uk/maths/robert/prosody.html[18] Mathematical Access for Technology and Science,

http://www.papenmeier.de/reha/research/mathe.htm[19] Edwards, A. D. N., Stevens, R. D. and Pitt, I. J. Représentation non visuelle des mathématiques,

(translated by A. Assimacopoulos) in A. B. Safran and A. Assimacopoulos (editors) Le Déficit Visuel, Éditions Masson, pp. 169–178 (1995), http://www.cs.york.ac.uk/ftpdir/pub/alistair/publications/ps/geneva.ps

[20] Karshmer, A.I., Gupta, G., Geiger, S., and Weaver, C.: Reading and Writing Mathematics: The MAVIS Project, BIT (Behaviour & Information Technology), January 1999

[21] Calle Sjöström, Using Haptics in Computer Interfaces for Blind People, http://www.certec.lth.se/doc/usinghapticsin/usinghapticsin.pdf

[22] W. Schweikhardt: 8-Dot-Braille for Writing, Reading and Printing Texts which Include Mathematical Characters. In: A.D.N. Edwards, A. Arato, W.L. Zagler: Computers and Assistive Technology, ICCHP98, Proceedings of the XV. IFIP World Computer Congress, Wien and Budapest, 31. August - 4. September 1998

[23] W. Schweikhardt: Requirements on a mathematical Notation for the Blind. In: R. Vollmar, R. Wagner (eds), CompurtersHelping Peoples with Special Needs ICCHP 2000, pp. 663-670, July, 17-21. 2000

[24] Typographic Semantics of Webpages Accessible for Visually Impaired Users - Mapping Layout and Interaction Objects to an Auditory Interaction Space, in Proceedings ICCHP 2000 - International Conference on Computers Helping with Special Needs. Karlsruhe 17-21 July 2000, http://www-cg-hci.informatik.uni-oldenburg.de/resources/ICCHP2000-gorny.pdf

ACAI




http://www.computing.dcu.ie/~dfitzpat/publications.html

http://www.cs.york.ac.uk/maths/index.html

http://www.cs.york.ac.uk/maths/robert/prosody.html

http://www.papenmeier.de/reha/research/mathe.htm

http://www.cs.york.ac.uk/ftpdir/pub/alistair/publications/ps/geneva.ps

http://www.certec.lth.se/doc/usinghapticsin/usinghapticsin.pdf

http://www-cg-hci.informatik.uni-oldenburg.de/resources/ICCHP2000-gorny.pdf

http://www-cg-hci.informatik.uni-oldenburg.de/resources/ICCHP2000-gorny.pdf

[25] Yu, W., Ramloll, R., Brewster, S.A. and Riedel, B. Exploring computer-generated line graphs through virtual touch. In Proceedings of IEEE ISSPA 2001 (Kuala-Lumpur, Malaysia), IEEE , pp 72-75, http://www.dcs.gla.ac.uk/~stephen/papers/ISSPA2001.pdf

[26] Oakley, I., Brewster S.A. and Gray, P.D. (2001). Communicating with feeling. Haptic Human-Computer Interaction. Brewster, S.A. and Murray-Smith, R. (Eds.), Springer LNCS, Vol 2058, pp 61-68, http://link.springer.de/link/service/series/0558/papers/2058/20580061.pdf

[27] Equalizing Information Access, http://www.viewplustech.com/[28] Tactile Graphics, An Overview and Resource Guide, http://dots.physics.orst.edu/tactile/tactile.html[29] TIM Project http://inova.snv.jussieu.fr/tim/[30] Tactile and Multimedia Tools for Young Visually Impaired People http://inova.snv.jussieu.fr/colloques

/BNet2001/uk/programme.php[31] Hammarlund, J. Computer Play for Children who are Severely Visually Impaired, TRC Rapport nr. 20

(1999)[32] Anita Hildén MUSSE version 2.0 http://www.sih.se/pdf/musse2_lathund.pdf[33] Schwende, H., Auditory Emotional Access to Visual Information,

http://link.springer-ny.com/link/service/series/0558/papers/2398/23980445.pdf[34] Gips James, DiMattia Philip A., Curran Marialice, Lees Debra, Accessing Internet Courses by Eye

Movement and Head Movement, http://link.springer-ny.com/link/service/series/0558/papers/2398/23980236.pdf

[35] World of Gestures, http://nonverbal.ucsc.edu/gest.html[36] Language Invention: Resources & Links,

http://www.linguistics.ucla.edu/people/grads/jforeman/collegium/resource.htm

ACAI


http://www.dcs.gla.ac.uk/~stephen/papers/ISSPA2001.pdf


http://www.viewplustech.com/



http://dots.physics.orst.edu/tactile/tactile.html





http://inova.snv.jussieu.fr/tim/











http://inova.snv.jussieu.fr/colloques/BNet2001/uk/programme.php














http://www.sih.se/pdf/musse2_lathund.pdf









http://nonverbal.ucsc.edu/gest.html



All references & presentations are available at:http://www.cs.uta.fi/~grse/ACAI_2003/

ACAI


http://www.bjorkasen.vgs.no/bcd/bcd0-99/ears2.htm