TP-1
Prototype of
Cyber Teaching AssistantCyber Teaching Assistant
Yoshiaki Shindo, Hiroshi Yoshiaki Shindo, Hiroshi MatsudaMatsuda
Nippon Institute of Technology, Nippon Institute of Technology, JapanJapan
Prototype of Cyber Teaching Assistant
TP-2
Nippon Institute of Technology
Constructed by Japan Information Processing Service Co. Ltd.
Teacher’s CockpitStudent’s Workstation
Computer Room
TP-3
Student's PC
Voice Speaking
CTA
File Server
ScenarioDatabase
Start CTA in student's PC with specified scenario by remote control.
Teacher's PC
RemoteControl
Local Area Network
System Outline of CTA
TP-4
Installed in each Student’s PC
Voice speaking by Voice Synthesizer
Subtitles are displayed.
Cyber Teaching Assistant Cyber Teaching Assistant ((CTACTA)
TP-5
<SPEECH> Speech Text </SPEECH> Cyber Person reads aloud the Speech Text by Voice.
<SPEECH>Hello Everyone.</SPEECH>
<SPEECH Person=“Nana” Script=“ON”>Hello Everyone.</SPEECH>
<SCRIPT Time=“50”>This is only a Caption Text.</SCRIPT>
Format
Function
Examples and Tag Option Switches
Voice SpeakingVoice Speaking
TP-6
<FACE> Lyric Word </FACE>
Cyber Person changes the Facial Expression.
<FACE>Happy</FACE>
Format
Function
Examples and Tag Option Switches
<FACE TTC=“0,10,100,5,100,0”>Angry</FACE>
<FACE Notify=“SYNC” Timing=“eStable”>Angry</FACE>
<FACE Gain=“80,30”>Surprise</FACE>
Facial ExpressionFacial Expression
TP-7
I am Happy
CPSL Engine
<TALK>
AU/TTCFacial Expression Database
FEN
Lyric Word Database
Lyric Word
CPSL Browser
<FACE>
Facial Expression Name
AU :Action UnitTTC:Time Transition Code
Lyric Word: Happy
FEN: Smile
AU: 60,120,260
Data Flow of Facial Expression
TP-8
GT : Gate TimeTT : Transition TimeST : Stable TimeRT : Release Time
EG : Expression Gain(%)RG : Remain Gain(%)
Unit of Time is 100mS
AU Motion Parts
11 Lift up the right side of eyebrow
52 Lift up the left side of eyelid
60 Lift up the both side of cheek
100 Lift up the upper lip
260 Open the mouse wider
430 Close the both eyes
Examples of FACS-AU
Gain
GT
Time
(%)
TT ST RT
RG
EG
TTC ( Time Transition Code )
Smile Angry Sad
Examples of Facial Expression
59 codes of AU are defined now.
FACS-AU is based on the FACS by P.K.Ekman’s research.
FACS-Action Unit and TTCFACS-Action Unit and TTC
TP-9
<TALK> Speech Text </TALK>
Cyber Person reads aloud the Speech Text by Voice and changes the facial expression automatically by detecting the lyric words.
<TALK>I am very happy today.</TALK>
<SPEECH> I am very happy<FACE>Smile</FACE> today. </SPEECH>
Format
Function
Examples and Tag Option Switches
Those are the same.
Option Switches of <TALK> are the same as one of <SPEECH>
Voice Speaking with Facial ExpressionVoice Speaking with Facial Expression
TP-10
<ACTION> Action Word </ACTION> Cyber Person plays specified Body Action.
Format
Function
Examples and Tag Option Switches
<ACTION TTC=“0,10,100,5,100,0”>Bend R.knee </ACTION>
<ACTION Notify=“SYNC” Timing=“eTransit”> Bend Ankles Outward </ACTION>
<ACTION Gain=“80,30”>Lift up R.arm in Front</ACTION>
Body Action by Body Action by BACS-AUBACS-AU
TP-11
AU/TTC(BACS)
CPSL Engine
MotionCaptureMatrix
Body Motion Database
BPN
Action Word Database
Action Word CPSL Browser
<ACTION> <TALK>
Body Performance Name
Action Word: Bend Ankles Outward
(BPN)
BACS-AU :Action UnitTTC:Time Transition Code
AU: 1470
Data Flow of Body Action
TP-12
BACS-AU Motion Parts of Body Skeleton ( Action Word )
1180 Bow to Front from the Waist
1241 Raise Left Arm to the Front
1351 Bend Left Wrist Inward
1442 Bend Right Knee
1470 Bend Ankles Outward
91 codes of BACS-AU are defined now.
Examples of BACS-AU
<ACTION Gain="50,50">Rotate Body to the Left</ACTION><ACTION Gain="85,85">Raise Left Arm to the Front</ACTION><ACTION>Rotate Rotate Face to the Right</ACTION><ACTION Gain="35,35>Raise Left Forearm to the Front</ACTION>
Body Action Coding System Body Action Coding System ( BACS )( BACS )
TP-13
<MOTION> Action Word </MOTION> Cyber Person plays specified Motion Capture data.
Format
Function
Examples and Tag Option Switches
<MOTION Person=“Emi” Repeat=“10”>WALKING </MOTION>
<MOTION Notify=“SYNC” Rate=“4”>BOWING </MOTION>
<MOTION>BOWING </MOTION>
<MOTION Notify=“ASYNC” Rate=“6”>BOWING </MOTION>
<MOTION Notify=“ABORT” Rate=“8”>BOWING </MOTION>
Body Action by Body Action by Motion CaptureMotion Capture
TP-14
AU/TTC(BACS)
CPSL Engine
MotionCaptureMatrix
Body Motion Database
BPN
Action Word Database
Action Word CPSL Browser
<MOTION> <TALK>
Body Performance Name
Motion Capture Filename
Action Word: WALKING
BPN: Walking
Filename: Walking.csv
Data Flow of Motion Capture
TP-15
Motion Capture Data is made and edited by BPN Encoder and data is saved in Network File Server as a CSV format file.
BPN Encoder
Motion Capture Data CSV format File
It captures the body action of human being and edits the captured data by mouse operation.
Sophisticated body action can be played.
Action Word Database
BPN Encoder
TP-16
Option Switch
Value Function
NOTIFY
SYNC Wait the end of Work.
ASYNC Passes without Waiting.
ABORT Terminate Work Forcibly
TIMING
NONE No Wait ( Same as ASYNC )
TRANSIT Wait the end of Transition Time.
STABLE Wait the end of Stable Time.
RELEASE Wait the end of TTC
WAIT Integer more than 0 Wait for specified Time
Block Notification defines the system behavior when CPSL browser encounters the </TAG> .
Unit of value is 100mS
Block Notification
TP-17
State Variables Content
Person Name of Active Person.
Tag Current Tag Name
Script ON or OFF ( Switch of Subtitles )
Notify SYNC, ASYNC or ABORT
TTC Timing SYNC Point of TTC
Wait Person’s Waiting Time
Rate Intermediate Frames of Motion
Repeat Repetition Times of Motion
Gate Gate Time of TTC
Transit Transit Time of TTC
Stable Stable Time of TTC
Release Release Time of TTC
Gain Stable Gain of AU (%)
Remain Remain Gain of AU (%)
Distance Distance of Walking
Turn Turn Angle of Rotation
Frames Frames for Specified Work
State Variables are set by Tag-Option Switches.
Value of State Variables are kept in the Block defined by <TAG> and </TAG>.
Scope of Variables
Option Switches and State Variables
TP-18
<STAGE>
<Stage> creates the background stage.
<STAGE TDM_FILE="CompEr1.tdm“
POSITION="600.0, -1300.0, -2200.0“
ROTATION="0.0, 0.0, 0.0“
SCALE="5.5, 5.5, 1.0“ >
Format
Function
Stage is 3D-CG model made by Modeling Tool.
3D-CG Modeling Tool
Creating StageCreating Stage
TP-19
Eye(Camera) Position
Background Color is converted to Transparent Pixel.
Rotate to Face the Camera Automatically.
Photograph, Picture or Image data of Screen Capture.
Axis of rotation
Rectangle of Billboard
Sideline of Rectangle and Axis of Rotation are Invisible.
Mapping
Easy way to create the stage .
Texture Mapped BillboardTexture Mapped Billboard
TP-20
<PERSON Option Switches >
<PERSON> creates the Cyber Person. More than one person can be defined. Required parameters must be specified by Option Switches.
Option Switch
Value Function
TDM_FILE Filename File name of Modeling Data.
NAME Naming in CPSL Define Name of Cyber Person.
SPEAKER Name for Speech API Select the Voice Synthesizer.
FEN_FILE Filename Select the Lyric Word Database.
BPN_FILE Filename Select the Action Word Database.
Position Values of X,Y,Z Coordinate of Located Position.
Rotation Values of X,Y,Z Angles of Rotation.
Scale Values of X,Y,Z Scaling Factor
3D-CG Modeling Tool
Creating Cyber PersonCreating Cyber Person
TP-21
<TAG> Option Switch Function
<WALK> Distance, Frames Cyber Person Walks with specified Body Action.
<SLIDE> Distance, Frames Cyber Person Slides with specified Body Action.
<JUMP> Distance, Frames Cyber Person Jumps with specified Body Action.
<ROT*> Turn, Frames Cyber Person rotates the body with specified Action.
<MUSIC> MID_FILE Play Specified Music.
<SOUND> WAV_FILE Play Specified Sound File.
<EXIT> Terminate the CPSL Browser. ( Exit Application )
<LENS> NearZ,FarZ,Fovy Setup the View Volume.
<CAMERA>
Position,
Direction,
Angle
Frames
Specify the Camera Work.
The other TAG The other TAG commandscommands
TP-22
(1) How to use the keyboard and Mouse.
(2) How to operate the tool bar of Word Processor.
(3) How to edit the Document
(4) How to print the document
For computer literacy course
For computer programming course
(1) How to use the Text Editor.
(2) How to edit the compile errors.
(3) How to edit the link errors.
(4) How to set the break point.
Scenario Database for CTA
TP-23
•Cyber Teaching Assistant is a self-standing 3D-CG Real-Time Animation Software installed in each student’s PC. •Cyber Teaching Assistant is a Cyber Person who plays specified Scenario saved in the Network File Server.
• Scenarios are written in CPSL ( Cyber Person Scenario Language), which is a TAG based Script Language.
•Teacher can start any CTA from Teacher’s Cockpit by Remote CTA Starter.
ConclusionsConclusions
TP-24
Scene 1. Explains the Error Message.
Scene 2. Shows the Cause of Error.
Scene 3. Show the solution to the problem.
Demonstration of Cyber Teaching Assistant
TP-25
Teacher’s Cockpit
Student’s PC Network File Server
Cyber Teaching Assistant
Scenario File
CTA StarterStart Command with Scenario File Name
CTA Remote Starter