Annual Project Review

12 October 2000IST-1999-10500

Introduction

Nick LodgeITC

Presentation Structure• Introduction• Project Management Overview• Linguistic Advances for Deaf People• Progress on Internet Services• Progress on Broadcast Services• Commercial Strategies• Video of Counter Services

Project Management Overview

John GlauertUEA, Norwich, UK

Project Structure

• User Focus• Application Focus• Technology Focus

WWW High Street Broadcast

Language

Animation

Evaluation Exploitation

Application Focus

• Internet– Information and Education for Deaf People (IvD, IDGS)

• High Street– Counter Services – Post Office (UKPO)

• Television– Regulation and Standards (ITC, INT, IRT)

WWW High Street Broadcast

User Focus

• Evaluation– Close Involvement of Deaf People (RNID)

• Exploitation– Broadcasters and Service Providers (ITC, UKPO, Televirtual)

Evaluation Exploitation

Technology Focus

• Sign Language Linguistics– Use of natural sign language (IDGS, UEA, IvD, RNID)

• Animation– Increased realism in sign generation (Televirtual, INT, UEA)

Language

Animation

Workplan Structure

Project Phases

• Technology Transfer• Prototype Applications• Advanced Applications

Technology Transfer

• First 12 Months• Exchange Existing Prototypes• Establish Collaborative Working• Largely Complete

Prototype Applications

• Months 9 to 24• Enhancement of Prototypes• Extended Flexibility• Focus on Well Defined Domains• Excellent Progress

Advanced Applications

• Months 18 to 36• Increased Animation Realism• Flexible Sign Generation• Advanced Linguistic Processing• Address Wide Ranging Domains

Workpackage Progress

• Achievements in Review Period– All Deliverables achieved– Milestones achieved or rescheduled with no impact

• Progress on Deliverables– Reported here for Prototype Applications

• Planned Changes– Response to Exploitation Opportunities will be presented

Management System

• Workpackage Leadership• Consortium Meetings• Workpackage Meetings

Workpackage Leadership

• 9 Partners• 8 Workpackages• Workpackage Leadership

– Coordination of Partners– Workplan in Single Partner Tasks– Lightweight use of Microsoft Project

Consortium Meetings

• Team of Partner Leaders• Quarterly Meetings

– First day for Management and Planning

– Second day allows for Technical Discussions

Workpackage Meetings

• Meetings as required by Workpackage Leader

• Extensive use of Email• Website for Document Archive and

Exchange

Addressing the needs of Deaf people using natural sign language - Overview of linguistic processing in

ViSiCAST

Thomas Hanke, IDGS, U Hamburg

Why Sign Language?

• Natural language of most Deaf, i.e. not an artificial language

• 1‰ in Europe pre-lingually deaf, only 10% of them Deaf of Deaf

• Most of them leave school functionally illiterate

What is Sign Language?

• A language with its own syntax & lexicon

• Not a 1:1 translation from spoken language as is SSE

• Not universal, but much closer to each other than spoken languages

What is so special about Sign Language?

• Parallel use of articulators• Use of space to encode

grammatical function• Iconicity and productivity

Sign Language and Text

• No established written form for any sign language

• “phonetic” transcription systems such as HamNoSys

• Glossing: GIVE-1-3a

HamNoSys

• Flexible enough to handle at least the European sign languages, will be verified for DGS, NGT, BSL

• Further developed in the project to include nonmanual activities

• Embedded into XML: SiGML

SiGML• Three Levels:

– Glosses: mapping more or less 1:1 to the animation machines’ database of signs

– HamNoSys: phonetic description– Motion capture files: for uninterpreted

direct transmission thru SiGML

SiGML (2)• <signed_utterance>

– <sign>• <manual_activity gloss=“GIVE-BOOK-1-3a”>• <face attitude=IRONIC>

– </sign>– <sign>…

• </signed_utterance>

How does it fit in?

Web pages

EnglishSpoken lang to text

English text toSign Notation

Transport &Animation

Standard AI Machine Translation Approach

Semantic Structure

English text Sign Notation (XML)

LexiconGrammar

LexiconGrammar

To reduce complexity

• Restricted Domains– Handling objects in a kitchen– Appartment descriptions

• Semi-automatic process– User can help both with signs and

word order

Progress on Internet Services

Margriet VerlindenIvD, The Netherlands

Workpackage 2• objective

– produce tools that will allow a deaf citizen to access multimedia resources through sign language

• deliverables– Browser plug-in (month 12)– Web-page-sign (month 30)– Signing tutor (month 30)

Signing on the WWW

• What will it look like?• How does it work?• How is it created?• How far are we now?

Het weer in gebaren

Weerbericht van donderdag 17 augustus2000, 17:34 uur

Vanavond en vannacht opklaringen, in hetbinnenland mogelijk een mistbank en aan denoordwestkust een kleine kans op onweer.Minimumtemperatuur rond 13 graden. Morgengeleidelijk meer bewolking en in de avond vanhet zuiden uit enkele regen- en onweersbuien.Middagtemperatuur morgen ongeveer 22graden.

De wind: zuidwest, matig, kracht 4, aan de kusten op het IJsselmeer vrij krachtig, 5, dekomende uren af en toe krachtig, 6. Vanavondafnemend en morgen draaiend naar zuid.

daily updated

link to project info.

signing avatar

controls for signing

What will it look like?

PLAY PAUSE STOP

MOCK-UP

How does it work?text from meteo-rological institute

standard sentences

playlist of motion data

SER

VER C

LIEN

T

Internet-page with SiGML

Internet-browser +

plugin for motion player (=avatar)

How is it created?text from meteorological institute

standard sentences

playlist of motion data

Internet-page with SiGML

• model for weather forecasts• mapping from stand. senten-

ces to sign sequences (3x)• recording + post-editing of

signs (3x)• XML for signs: SiGML• user-interface (webpage+plug-

in)

• update-facility

How far are we?• basic functionality implemented• motion data is being post-edited• additional functionality in progress

DEMO

Progress on Broadcast Services

Mark WellsTelevirtual, Norwich, UK

The Problem:– Legislation (UK, soon European)

requires on-screen signing– “Open Signing” (ever present) is

obtrusive– Even “Closed Signing” uses bandwidth:

Broadcasters run a tight bandwidth economy

The Solution: Virtual Human Signing

– Can be Open or Closed– Picture Quality not compromised by

compression– Potential for User Interaction– Very Cheap in terms of bandwidth

Virtual Humans / Avatars

• Entertainment• Communications• Transactions• Guides

Virtual Humans / Avatars

• Entertainment

Virtual Humans / Avatars

• Communications

Signing Virtual Humans

Recording Signing

Recording SigningReal timeFace,Hand,Body, Tracking

Recording SigningReal timeFace,Hand,Body, Tracking - & Playback

On Television:

Transmit theMovement -not the Video

Motion Data + Calibration

Motion Data + Calibration

IHoIstCOM

Mesh

Mesh Attachment Description

Bone Set

Renderer

Avatar Codec

DSP

Compressed Motion Stream

DSP

Avatar Codec

Compressed Motion Stream

Bone Set

MPEG 2 Broadcast Stream

Scene

Transmission System

- The Demo...

• Integrated TX system for broadcast to STBs– demonstrator complete end of 2000

• Implementing virtual human s/w in STB• Optimising compression algorithm• Building multiplexer for carriage in MPEG-2 • SiGML streaming

Broadcast VH Signing:

• Low transmission rate < 30 kbit/s• Compatibility with signing on other media and

foreign deaf languages• Precise, sharp representation of signer• Many display options•Can conform to MPEG-4 standards• Future-proof:

–cost saving–allows vast no. of signed programmes–no transition from video-based to VH signing

VH on TV: The Advantages

Commercial Strategies

Nick LodgeITC

What is the Product or Service?

• Diversity of applications– Broadcast - information displays, plays, movies

– Multimedia - dictionaries, EPGs, learning tools

– Face-to-face - videoconferencing, UMTS phones

• Diversity of languages– German, Dutch, English signs– Future: ASL, Makaton, Paget Gorman

Theatre / Cinema Application

• Display allows visibility of external world and signer

What is the Product or Service?

• Diversity of applications– Broadcast - information displays, plays, movies

– Multimedia - dictionaries, EPGs, learning tools

– Face-to-face - videoconferencing, UMTS phones

• Diversity of languages– German, Dutch, English signs– Future: ASL, Makaton, Paget Gorman

What is the Product or Service?

• Increase awareness of sign languages• Promote standards for signing avatars

• Advance linguistic representation & processing of

deaf languages• Provide consultancy support

Building Brand Awareness

• Establish acceptability of virtual humans• Establish presence of ViSiCAST across

different media• Establish Visia as a character with

personality and encourage familiarity

The Asymmetric Market• Only 0.05% - 0.1% of population sign

• Millions of companies wish to sell to deaf people or

• Wish to be seen to be aware of deaf customers

Legislative Influences• Disability Discrimination Directive• UK Broadcasting Act 1996• UK Communications Act 2001

1999 2009

5

2

3

4

1

%Content ofDTT channel

WWW strategy• Give away basic web browser

• Sell SiGML authoring tool

• De facto standard

Face-to-face Product

• Stand-alone technology• Licensed speech recognition• Sell ViSiCAST signing software• Systems require s/w & h/w support

Broadcasting Production & Transmission Technology

• Body suits combining capture technologies• Digital compression systems• Multiplexers for combination with MPEG-2

services• Studio practices for handling signing data• Encouraging use of the system within DVB

The Set-top Box Strategy

• Develop special set-top box for deaf users by November 2001

• Work with STB manufacturer to develop ‘concept box’ with VH interface– speech synthesis, speech recognition

• Explore human factors with elderly users• Develop speech for blind users

Progress on Counter Services

TESSAVideo of Deliverable

ExampleThe weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight.First a shower, in the west sometimes with thunder. Furthermore now and then sunny. In the afternoon more cloudy and in the evening rain from the west.Temperature in the afternoon around 14 degrees.

The weather forecast of the KNMI, drawn up on Monday Oct. 9, and valid till midnight. (p.1)In the morning a shower. (p.7) Sometimes in the west thunder. (p.8)Sometimes sunny. (p.7)In the afternoon increasingly cloudy.(p.7)Tonight from the west rain. (p.8)In the afternoon 14 degrees. (p.11)