Craig Newman Curtin University of Technology Translating Australian Sign Language into Speech

Craig NewmanCurtin University of Technology

Translating Australian Sign Language into Speech

Translating Australian Sign Language into Speech

QuickTime™ and a Video decompressor are needed to see this picture.

• Introduction To Auslan• Elements Of Auslan Signs• Hardware Used• Hardware Control Algorithm• Elements Of Sign Recognition Algorithm• Future Intentions

Presentation OverviewPresentation Overview

• British and Irish origins• More than 4000 commonly used signs • 0.1% of Australians use Auslan

Introduction to AuslanIntroduction to Auslan

Two-handed fingerspelling alphabet

Two-handed fingerspelling alphabet

• Handshapes• Location• Orientation• Movement• Expression

Elements of Auslan signsElements of Auslan signs

HandshapesHandshapes


First Attribute: Degree to which fingers are bent


Second Attribute: Combination of fingers touching


Third Attribute: Pitch of the hand

• Neutral Space• Primary Locations• Secondary Locations

LocationLocation

LocationLocation

LocationLocation

OrientationOrientation

OrientationOrientation

Roll required to differentiate

Yaw required to differentiate

• Large Scale• Straight Line• Series of Straight Lines• Arcs• Circles

• Small Scale• Changes in Orientation• Changes in Fingers New Handshape

MovementMovement

• Head• Eyebrows• Eyes• Mouth• Cheeks

ExpressionExpression

P5 glove from Essential RealityP5 glove from Essential Reality

• Known through the P5• Degree of Finger Bend• Position Primary Locations, Movement• Orientation

• Required• Contacting Secondary Locations

What was Needed?What was Needed?

Acquiring Secondary LocationsAcquiring Secondary Locations

M16C/62 Single-chip MicrocontrollerM16C/62 Single-chip Microcontroller

The PrototypeThe Prototype

• Collect Data Every 17ms• 60 Hertz• Synchronisation with P5

• Test Pins Not Logged• Write Pin High, Read Remaining Pins• Log Connected Pins• Debounce: Test For Consistency• Encapsulate for Transmission• Transmit When Required

Contact Sensor AlgorithmContact Sensor Algorithm

Working Example 1Working Example 1



INSERT CLIP: IM


• 3 Samples Stored in stop_buffer• Position• Orientation• Finger Bend• Contacts

• Check Last 2 Samples within Range

When was the hand considered Stopped?

When was the hand considered Stopped?

• Wait Until Hands Stop• Take First Sample From stop_buffer• Filter Through if/else Structure to find

Correct Combination• Return Handshape

Identifying HandshapesIdentifying Handshapes


INSERT CLIP: GUNQuickTime™ and a Video decompressor are needed to see this picture.


INSERT CLIP: MOTHERQuickTime™ and a DV - PAL decompressor are needed to see this picture.

• Record Position of • Head• Shoulder• Bicep• Forearm

• Extrapolate Primary Locations• Else is Neutral Space

• Problem• Requires Dynamic Position Tracking

Identifying Primary LocationsIdentifying Primary Locations

• Wait Until “not stopped”• While “not stopped” copy stop_buffer

• Allow for Slight Pauses

• Filter Data By Testing For Inherent Geometric Structure

• Output Movement Type

Identifying Movement TypeIdentifying Movement Type

• Identify Initial Handshape, Location, Orientation

• Filter Through if/else Structure

• If Sign Recognised

• Output Sign

• Else

• Identify Movement

• Identify Final Handshape, Location, Orientation

• Filter through if/else Structure

• Output Sign

Putting it all together: Recognizing Signs

Putting it all together: Recognizing Signs

Working ExampleWorking Example

INSERT CLIP: WRAPUP2QuickTime™ and a Video decompressor are needed to see this picture.

• Design Composite Glove• Dynamic Tracking Of Primary Locations• Introduce Learning Algorithms

• Identifying Handshapes• Identifying Movement

• Translate Signing Context Into English Text

• Integrate Text To Voice Synthesizer

Future IntentionsFuture Intentions

Questions & Answers

Documents

Craig Newman Curtin University of Technology Translating Australian Sign Language into Speech