VIRTUALSIGNTRANSLATOR AS A BASE FOR A SERIOUS GAME

VIRTUAL SIGN

GILT

Games, Interaction and Learning Technologies

OBJETIVES

Implementation of a Portuguese Sign

Language(PSL) bidirectional

translating system.

PORTUGUESE SIGN LANGUAGE

• PSL is an official language used in learning,

leisure moments and in professional

environments.

• In PSL, the speaker should take into

consideration a series of parameters that

define the manual and non-manual

components.

LINGUISTIC ASPECTS- I

Manual Components:

– Hand configuration;

– Hand orientation;

– Articulation position;

– Hand movement;

LINGUISTIC ASPECTS- II

Non-manual component:

– Body movement

– Facial expression

VIRTUAL SIGN: PROJECT

Three main modules:

PSL translator to text;

Text translator to PSL;

Didatic game;

PSL TO TEXT

PSL TO TEXT - HARDWARE

• Sensor Gloves(5DT) • Kinect

• Provides information

about the hand

configuration

• Hands orientation and

movement, body movement

and facial expression

PSL TO TEXT – SOFTWARE I

“Divide and conquer”

– Bom Dia!

PSL TO TEXT – SOFTWARE II

Sign in PSL

• The beginning of a gesture is defined by a

configuration of the dominant hand.

• The ending of the gesture is determined

by the return of the dominant hand to a

neutral position or a configuration

change(new gesture).

• 57 configurations (letters,numbers, ...)

PSL TO TEXT – SOFTWARE III

• The gloves provide the information of each

hand allowing the software to estimate the

current hand configuration. The Kinect

provides infromation about the hands

movement and orientation

• The comparison between series of points is

done by applying a sequence alignment

algorithm.

PSL TO TEXT – SOFTWARE IV

The resulting sequence can represent one or more

words in sign langauge. The sequency is classified

using the database words as atoms.

LGP TO TEXT - SOFTWARE V

• Result sequency analysis :

PSL TO TEXT– IN PROGRESS

Processing not manual aspects:

– Body movement: Data retrieved by the kinect;

– Facial expression: Facial expression classifier;

TEXT TO PSL

TEXT TO PSL - AVATAR

• 3D animations created on

Blender

• Can be reused in other

avatars with similar bone

structure;

TEXT TO PSL - TRANSLATOR

The translator checks the

text for orthographic

mistakes and then

translates it to PSL.

TEXT TO PSL– INTEGRATING PPT

• Powerpoint add in

connected to the

translator;

• Text in the slides is

translated by the avatar;

TEXT TO PSL – CHAT

• Supporting long distance

communication using

PSL

SERIOUS GAME

SERIOUS GAME – ASPECTS I

Structure:

• 3 PSL learning scenes;

• 3 PSL evaluation scenes;

• Multiple minigames;

• Integration of the VirtualSign translator;

SERIOUS GAME – ASPECTS II

Large maps

Interactive environment

Intuitive gameplay

Inventory system

Checkpoint system

SERIOUS GAME – SCENE I

• Alphabet learning scene

• Interactive objects;

SERIOUS GAME – SCENE II

• Free roaming scene

• Teaches basic words

• The player chooses the

order of events;

SERIOUS GAME – SCENE III

• Futuristic scene;

• Teaches basic sentences

• Specific sentences trigger

minigames;

SERIOUS GAME – EVALUATION SCENES

SERIOUS GAME – INTEGRATING THE TRANSLATOR

SERIOUS GAME - MINIGAMES

• Memory • 2D Shooter

VIRTUAL SIGN

Thank you!