eNTERFACE ’10Amsterdam, July-August 2010

Hamdi Dibeklioğlu

Ilkka Kosunen

Marcos Ortega

Albert Ali Salah

Petr Zuzánek

Goal of the Project Responsive photograph frame

◦ User interaction leads to different responses

Modules of the project◦ Video segmentation module

Dictionary of responses◦ Behaviour understanding

Offline: Labelling dictionary Online: Cluster user action

◦ System logic Linking user actions to responses

System Design

External Data(Videos)

Automatic Segmentation

Segment Library

Program Logic(Learning)

Segment Selection

Visual Feature Analysis

Automatic Segmentation

Segment Library

Module 1: Offline segmentation

Module 5: Dual frame modeModule 3:System

response

Module 2: Real-time Feature Analysis

Module 4: Interface

5 video recordings (~1.5-2 min.)◦ Same individual◦ Different actions and expressions

Manual annotation of videos◦ ANVIL tool◦ Annotated by different individuals

Automatic segmentation◦ Segmentation based on actions◦ Optical flow: amount of activity over time

Module 1: Offline Segmentation

Optical flow calculation

Activity calculation based on feature tracking over the sequence

Feature detection◦ Shi-Tomasi corner detection algorithm

Feature tracking◦ Lucas-Kanade feature tracking algorithm◦ Pyramidal implementation (Bouguet)

Optical Flow Computation Movement analysis

Optical flow based segmenting To find a calm segment, just search for long

period of frames with calculated optical flow below some treshold (we used 40% of average optical flow calculated from all frames)

To find an active segment, search for frames with lot of optical flow, and then search forward and backward for the calm segments.

Smoothing Optical Flow Data

Manual vs. Automatic Segmentation

Calm Segment Example

Active Segment Example

Module 2: Real-time Feature Analysis

Face detection activates the system◦ Viola-Jones face detector

User’s behaviour can be monitored via◦ Face detection◦ Eye detection

Valenti et al., isophote-curves based eye detection◦ Optical flow energy

OpenCV Lucas-Kanade algorithm◦ Colour features◦ Facial feature analysis

The eMotion system

User Tracking Face and Eye detection: EyeAPI

Facial feature tracking Face model: 16 surface patches Face model: 16 surface patches

embedded in Bezier volumes.embedded in Bezier volumes. Piecewise Bezier Volume Deformation Piecewise Bezier Volume Deformation

(PBVD) tracker is used to trace the (PBVD) tracker is used to trace the motion of the facial features.motion of the facial features.

* R. Valenti, N. Sebe, and T. Gevers. Facial expression recognition: A fully integrated approach. In ICIAPW, pages 125–130, 2007.

Expression Classification

12 motion units12 motion units NNaive Bayesaive Bayes (NB) (NB) classifier for classifier for

categorizing expressionscategorizing expressions NB Advantage: the posterior probabilities NB Advantage: the posterior probabilities

allow a soft output of the systemallow a soft output of the system

Happiness Surprise Anger

Disgust Fear Sadness

Average Motion Units

Real-time Expression Analysis

Module 3: System Response Linking user actions and system responses An action queue is maintained

◦ Different user inputs (transitions) lead to different responses (states)

The responses (segments) are ‘unlocked’ one by one

Sleeping

Wake-up Neutral ResponseFace

detected

Period of inactivity

User input

Module 3: System ResponseBefore learning After learning

Module 4: Interface Currently two external programs are employed:

◦ SplitCam◦ eMotion

Glyphs are used to provide feedback to the user Glyph brightness is related to distance to

activation Once a glyph is activated, the same user activity

will elicit the same response Each user can have different behaviours

activating glyphs

Demo of the system

Future Work Work on the learning module Testing the segmentation parameters The dual frame mode Speeding up the system Wizard of Oz study Usability studies SEMAINE integration?