TP-1

Prototype of

Cyber Teaching AssistantCyber Teaching Assistant

Yoshiaki Shindo, Hiroshi Yoshiaki Shindo, Hiroshi MatsudaMatsuda

Nippon Institute of Technology, Nippon Institute of Technology, JapanJapan

Prototype of Cyber Teaching Assistant

TP-2

Nippon Institute of Technology

Constructed by Japan Information Processing Service Co. Ltd.

Teacher’s CockpitStudent’s Workstation

Computer Room

TP-3

Student's PC

Voice Speaking

CTA

File Server

ScenarioDatabase

Start CTA in student's PC with specified scenario by remote control.

Teacher's PC

RemoteControl

Local Area Network

System Outline of CTA

TP-4

Installed in each Student’s PC

Voice speaking by Voice Synthesizer

Subtitles are displayed.

Cyber Teaching Assistant Cyber Teaching Assistant ((CTACTA)

TP-5

<SPEECH> Speech Text </SPEECH> Cyber Person reads aloud the Speech Text by Voice.

<SPEECH>Hello Everyone.</SPEECH>

<SPEECH Person=“Nana” Script=“ON”>Hello Everyone.</SPEECH>

<SCRIPT Time=“50”>This is only a Caption Text.</SCRIPT>

Format

Function

Examples and Tag Option Switches

Voice SpeakingVoice Speaking

TP-6

<FACE> Lyric Word </FACE>

Cyber Person changes the Facial Expression.

<FACE>Happy</FACE>

Format

Function

Examples and Tag Option Switches

<FACE TTC=“0,10,100,5,100,0”>Angry</FACE>

<FACE Notify=“SYNC” Timing=“eStable”>Angry</FACE>

<FACE Gain=“80,30”>Surprise</FACE>

Facial ExpressionFacial Expression

TP-7

I am Happy

CPSL Engine

<TALK>

AU/TTCFacial Expression Database

FEN

Lyric Word Database

Lyric Word

CPSL Browser

<FACE>

Facial Expression Name

AU :Action UnitTTC:Time Transition Code

Lyric Word: Happy

FEN: Smile

AU: 60,120,260

Data Flow of Facial Expression

TP-8

GT : Gate TimeTT : Transition TimeST : Stable TimeRT : Release Time

EG : Expression Gain(%)RG : Remain Gain(%)

Unit of Time is 100mS

AU Motion Parts

11 Lift up the right side of eyebrow

52 Lift up the left side of eyelid

60 Lift up the both side of cheek

100 Lift up the upper lip

260 Open the mouse wider

430 Close the both eyes

Examples of FACS-AU

Gain

GT

Time

(%)

TT ST RT

RG

EG

TTC ( Time Transition Code )

Smile Angry Sad

Examples of Facial Expression

59 codes of AU are defined now.

FACS-AU is based on the FACS by P.K.Ekman’s research.

FACS-Action Unit and TTCFACS-Action Unit and TTC

TP-9

<TALK> Speech Text </TALK>

Cyber Person reads aloud the Speech Text by Voice and changes the facial expression automatically by detecting the lyric words.

<TALK>I am very happy today.</TALK>

<SPEECH> I am very happy<FACE>Smile</FACE> today. </SPEECH>

Format

Function

Examples and Tag Option Switches

Those are the same.

Option Switches of <TALK> are the same as one of <SPEECH>

Voice Speaking with Facial ExpressionVoice Speaking with Facial Expression

TP-10

<ACTION> Action Word </ACTION> Cyber Person plays specified Body Action.

Format

Function

Examples and Tag Option Switches

<ACTION TTC=“0,10,100,5,100,0”>Bend R.knee </ACTION>

<ACTION Notify=“SYNC” Timing=“eTransit”> Bend Ankles Outward </ACTION>

<ACTION Gain=“80,30”>Lift up R.arm in Front</ACTION>

Body Action by Body Action by BACS-AUBACS-AU

TP-11

AU/TTC(BACS)

CPSL Engine

MotionCaptureMatrix

Body Motion Database

BPN

Action Word Database

Action Word CPSL Browser

<ACTION> <TALK>

Body Performance Name

Action Word: Bend Ankles Outward

(BPN)

BACS-AU :Action UnitTTC:Time Transition Code

AU: 1470

Data Flow of Body Action

TP-12

BACS-AU Motion Parts of Body Skeleton ( Action Word )

1180 Bow to Front from the Waist

1241 Raise Left Arm to the Front

1351 Bend Left Wrist Inward

1442 Bend Right Knee

1470 Bend Ankles Outward

91 codes of BACS-AU are defined now.

Examples of BACS-AU

<ACTION Gain="50,50">Rotate Body to the Left</ACTION><ACTION Gain="85,85">Raise Left Arm to the Front</ACTION><ACTION>Rotate Rotate Face to the Right</ACTION><ACTION Gain="35,35>Raise Left Forearm to the Front</ACTION>

Body Action Coding System Body Action Coding System ( BACS )( BACS )

TP-13

<MOTION> Action Word </MOTION> Cyber Person plays specified Motion Capture data.

Format

Function

Examples and Tag Option Switches

<MOTION Person=“Emi” Repeat=“10”>WALKING </MOTION>

<MOTION Notify=“SYNC” Rate=“4”>BOWING </MOTION>

<MOTION>BOWING </MOTION>

<MOTION Notify=“ASYNC” Rate=“6”>BOWING </MOTION>

<MOTION Notify=“ABORT” Rate=“8”>BOWING </MOTION>

Body Action by Body Action by Motion CaptureMotion Capture

TP-14

AU/TTC(BACS)

CPSL Engine

MotionCaptureMatrix

Body Motion Database

BPN

Action Word Database

Action Word CPSL Browser

<MOTION> <TALK>

Body Performance Name

Motion Capture Filename

Action Word: WALKING

BPN: Walking

Filename: Walking.csv

Data Flow of Motion Capture

TP-15

Motion Capture Data is made and edited by BPN Encoder and data is saved in Network File Server as a CSV format file.

BPN Encoder

Motion Capture Data CSV format File

It captures the body action of human being and edits the captured data by mouse operation.

Sophisticated body action can be played.

Action Word Database

BPN Encoder

TP-16

Option Switch

Value Function

NOTIFY

SYNC Wait the end of Work.

ASYNC Passes without Waiting.

ABORT Terminate Work Forcibly

TIMING

NONE No Wait ( Same as ASYNC )

TRANSIT Wait the end of Transition Time.

STABLE Wait the end of Stable Time.

RELEASE Wait the end of TTC

WAIT Integer more than 0 Wait for specified Time

Block Notification defines the system behavior when CPSL browser encounters the </TAG> .

Unit of value is 100mS

Block Notification

TP-17

State Variables Content

Person Name of Active Person.

Tag Current Tag Name

Script ON or OFF ( Switch of Subtitles )

Notify SYNC, ASYNC or ABORT

TTC Timing SYNC Point of TTC

Wait Person’s Waiting Time

Rate Intermediate Frames of Motion

Repeat Repetition Times of Motion

Gate Gate Time of TTC

Transit Transit Time of TTC

Stable Stable Time of TTC

Release Release Time of TTC

Gain Stable Gain of AU (%)

Remain Remain Gain of AU (%)

Distance Distance of Walking

Turn Turn Angle of Rotation

Frames Frames for Specified Work

State Variables are set by Tag-Option Switches.

Value of State Variables are kept in the Block defined by <TAG> and </TAG>.

Scope of Variables

Option Switches and State Variables

TP-18

<STAGE>

<Stage> creates the background stage.

<STAGE TDM_FILE="CompEr1.tdm“

POSITION="600.0, -1300.0, -2200.0“

ROTATION="0.0, 0.0, 0.0“

SCALE="5.5, 5.5, 1.0“ >

Format

Function

Stage is 3D-CG model made by Modeling Tool.

3D-CG Modeling Tool

Creating StageCreating Stage

TP-19

Eye(Camera) Position

Background Color is converted to Transparent Pixel.

Rotate to Face the Camera Automatically.

Photograph, Picture or Image data of Screen Capture.

Axis of rotation

Rectangle of Billboard

Sideline of Rectangle and Axis of Rotation are Invisible.

Mapping

Easy way to create the stage .

Texture Mapped BillboardTexture Mapped Billboard

TP-20

<PERSON Option Switches >

<PERSON> creates the Cyber Person. More than one person can be defined. Required parameters must be specified by Option Switches.

Option Switch

Value Function

TDM_FILE Filename File name of Modeling Data.

NAME Naming in CPSL Define Name of Cyber Person.

SPEAKER Name for Speech API Select the Voice Synthesizer.

FEN_FILE Filename Select the Lyric Word Database.

BPN_FILE Filename Select the Action Word Database.

Position Values of X,Y,Z Coordinate of Located Position.

Rotation Values of X,Y,Z Angles of Rotation.

Scale Values of X,Y,Z Scaling Factor

3D-CG Modeling Tool

Creating Cyber PersonCreating Cyber Person

TP-21

<TAG> Option Switch Function

<WALK> Distance, Frames Cyber Person Walks with specified Body Action.

<SLIDE> Distance, Frames Cyber Person Slides with specified Body Action.

<JUMP> Distance, Frames Cyber Person Jumps with specified Body Action.

<ROT*> Turn, Frames Cyber Person rotates the body with specified Action.

<MUSIC> MID_FILE Play Specified Music.

<SOUND> WAV_FILE Play Specified Sound File.

<EXIT> Terminate the CPSL Browser. ( Exit Application )

<LENS> NearZ,FarZ,Fovy Setup the View Volume.

<CAMERA>

Position,

Direction,

Angle

Frames

Specify the Camera Work.

The other TAG The other TAG commandscommands

TP-22

(1)  How to use the keyboard and Mouse.

(2)  How to operate the tool bar of Word Processor.

(3) How to edit the Document

(4) How to print the document

For computer literacy course

For computer programming course

(1) How to use the Text Editor.

(2) How to edit the compile errors.

(3) How to edit the link errors.

(4) How to set the break point.

Scenario Database for CTA

TP-23

•Cyber Teaching Assistant is a self-standing 3D-CG Real-Time Animation Software installed in each student’s PC. •Cyber Teaching Assistant is a Cyber Person who plays specified Scenario saved in the Network File Server.

• Scenarios are written in CPSL ( Cyber Person Scenario Language), which is a TAG based Script Language.

•Teacher can start any CTA from Teacher’s Cockpit by Remote CTA Starter.

ConclusionsConclusions

TP-24

Scene 1. Explains the Error Message.

Scene 2. Shows the Cause of Error.

Scene 3. Show the solution to the problem.

Demonstration of Cyber Teaching Assistant

TP-25

Teacher’s Cockpit

Student’s PC Network File Server

Cyber Teaching Assistant

Scenario File

CTA StarterStart Command with Scenario File Name

CTA Remote Starter