Lip Feature Extraction Using Red Exclusion Trent W. Lewis and David M.W. Powers Flinders University...

Lip Feature Extraction Using Red Exclusion

Trent W. Lewis and David M.W. Powers

Flinders University of SA

VIP2000

01/12/2000 Lip Feature Extraction Using Red Exlucsion

2

Overview

• Context

• Lip Feature Extraction– Related Work (greyscale, horizontal edges, red and hue colour spaces)

– Red Exclusion

• AVSR: Results and Issues

• Summary

01/12/2000 Lip Feature Extraction Using Red Exlucsion

3

Context

• Audio Speech Recognition (ASR)

• Psycholinguistic Research

• Audio Visual Speech Recognition (AVSR)

01/12/2000 Lip Feature Extraction Using Red Exlucsion

4

Context - ASR

• Up to 99% word accuracy

• However,– limited context– limited vocabulary– trained on individual– close microphone, cannot handle noise

01/12/2000 Lip Feature Extraction Using Red Exlucsion

5

Context - Psycholinguistic

• McGurk Effect– A[ba] + V[ga] [da]

• Viseme– visual phonemes– form complementary sets

• Demonstrates vision can assist the perception of speech AVSR

01/12/2000 Lip Feature Extraction Using Red Exlucsion

6

Context - AVSR

• Acoustic Features• Visual Features

– width

– height

– oral cavity

• Integration– Early

– Late

01/12/2000 Lip Feature Extraction Using Red Exlucsion

7

Lip Feature Extraction

• Pixel-Based Model

01/12/2000 Lip Feature Extraction Using Red Exlucsion

8

Lip Feature Extraction

• Pixel-Based Model– raw pixels or minimal processing– retain linguistically relevant data– large amounts of data, time– shift and lighting variant– normalisation and PCA

01/12/2000 Lip Feature Extraction Using Red Exlucsion

9

Lip Feature Extraction

• Pixel-Based Model– reduced input to set of hand-crafted features– width, height, average intensity, etc.– less features, time– model fitting, time– lose linguistically relevant features

01/12/2000 Lip Feature Extraction Using Red Exlucsion

10

Lip Feature Extraction

• Pixel-Based Model– feature extraction– Steps

• preprocess to enhance contrast

• locate mouth edges

• identify corners, height, and other key features

• train recogntion engine

Our Approach

01/12/2000 Lip Feature Extraction Using Red Exlucsion

11

Lip Feature Extraction

• Database

1 2 3

01/12/2000 Lip Feature Extraction Using Red Exlucsion

12

Lip Feature Extraction

• Preprocessing Techniques– Grey-scale– Horizontal Edges– Red Analysis– Hue, Saturation, and Value (HSV)– Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

13

Lip Feature Extraction

• Grey-scale– vertical position of mouth

• minimum row sum

– threshold minimum row• average of min and max of row

– search for above threshold pixels

01/12/2000 Lip Feature Extraction Using Red Exlucsion

14

Lip Feature Extraction

• Grey-scale

01/12/2000 Lip Feature Extraction Using Red Exlucsion

15

Lip Feature Extraction

• Grey-scale

01/12/2000 Lip Feature Extraction Using Red Exlucsion

16

Lip Feature Extraction

• Horizontal Edges– high horizontal edge content– 3x3, DY Prewitt operator

111

000

111

DY

01/12/2000 Lip Feature Extraction Using Red Exlucsion

17

Lip Feature Extraction

• Horizontal Edges

“Found” Corners Binary Image

01/12/2000 Lip Feature Extraction Using Red Exlucsion

18

Lip Feature Extraction

• Red Analysis– overcome bearded subjects– used for face location

limlim UG

RL

01/12/2000 Lip Feature Extraction Using Red Exlucsion

19

Lip Feature Extraction

• Red Analysis

01/12/2000 Lip Feature Extraction Using Red Exlucsion

20

Lip Feature Extraction

• HSV– disentangles illumination from colour Illumination > Hue

otherwise

whhw

hhhf o

o

,0

,)(

1)( 2

2

01/12/2000 Lip Feature Extraction Using Red Exlucsion

21

Lip Feature Extraction

• HSV

01/12/2000 Lip Feature Extraction Using Red Exlucsion

22

Lip Feature Extraction

• Red Exclusion– needed extraction method for AVSR– similar to Red Analysis

• face predominantly red

• variations occur in the blue and green colours

B

Glog

01/12/2000 Lip Feature Extraction Using Red Exlucsion

23

Lip Feature Extraction

• Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

24

Lip Feature Extraction

• Corners found using Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

25

Lip Feature Extraction

• ComparisonAlgorithm

Subject 1 female

Subject 2 male bearded

Subject 3 male thin lips

Reliable Corners

Other Features

Grey-scale

Edge

Red Analysis

HSV Red Exclusion

01/12/2000 Lip Feature Extraction Using Red Exlucsion

26

AVSR: Results and Issues

• Application for red exclusion

• Used in finding lip features– Width– Height– Key pixels

01/12/2000 Lip Feature Extraction Using Red Exlucsion

27

AVSR: Results and Issues

• Visual Speech RecognitionStatic (%) Dynamic (%)

Voicing 32.2 30.8

Viseme 54.7 51.3

Phoneme 14.7 13.6

01/12/2000 Lip Feature Extraction Using Red Exlucsion

28

AVSR: Results and Issues

• AVSR - IntegrationEarly

Static

Early

Dynamic

Late

Voice/Vis

Late Error

Voice/Vis

Phoneme20.1 18.0 29.0 19.5

01/12/2000 Lip Feature Extraction Using Red Exlucsion

29

Summary

• Vision can help ASR– AVSR

• Needed good extraction technique– Red Exclusion

• AVSR is difficult when both signals degraded

01/12/2000 Lip Feature Extraction Using Red Exlucsion

30

Questions?

Trent W. Lewis

BSc (Cognitive Science)

Flinders University

lewi0146@ist.flinders.edu.au