Automatic and Data Driven Pitch Contour Manipulation with Functional Data Analysis

Automatic and Data DrivenPitch Contour Manipulationwith Functional Data Analysis

Michele Gubian, Lou BovesRadboud University NijmegenNijmegen, The Netherlands

Francesco CangemiLaboratoire Parole et LangageUniversity of Provence, Aix-en-Provence, France

2

Outline Pitch Contour Manipulation

Context and problem

Sketch of proposed approach

Use of Functional Data Analysis (FDA)

Case study

Data preparation

Functional PCA

Functional synthesis and listening

Conclusions

3

Context Languages can express oppositions using intonation

Question/Statement opposition in Neapolitan Italian

QUESTION STATEMENT

“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)

What are the intonation cues that listeners use?

Perceptual experiments where listeners judge stimuli whose pitch (F0) contour has been manipulated

STEP 1: extract pitch contours from speech data

STEP 2: modify pitch contours

STEP 3: re-synthesize speech

4

Pitch Contour Manipulation

Use of an intonation model

Stylization

Manual changestime

F0

POSSIBLE IMPROVEMENTS Handle dynamic detail

Locally (e.g. concavity/convexity)

Long range correlation

Derive useful variation modes directly and automatically from data

5

A data driven approach

Functional

Data

Analysisx

6

Question/Statement opposition in Neapolitan Italian

DATA 2 male speakers

3 carrier sentences (read speech)

“Milena lo vuole amaro (?)” = Milena drinks it (her coffee) bitter (?)

“Valeria viene alle nove (?)” = Valeria arrives at 9 (?)

“Amelia dorme da nonna (?)” = Amelia sleeps at grandma’s (?)

2 modalities = Q / S

5 repetitions

2 x 3 x 2 x 5 - 3 discarded = 57 utterances

7

Data Preparation

Sampled F0 curves have to be turned into functions

A basis of functions (B-splines) expresses each original curve

Decide how much detail to retain (smoothing)

8

Data Preparation (2) Landmark registration

Align points in time that are deemed as having the same

meaning across the dataset

9

ClassicPrincipal Component Analysis (PCA)

age25 65

salary

xx

xxx

x

xx

xxx

xx xx xxxx

xxxx

xx x

xxx

xx

x

xx

x

x

PC1

PC2

10

Functional PCA

11

PC-based signal reconstruction

+ 1.65 x - 0.46 x

mean(t) PC1(t) PC2(t)

12

Manipulated stimuli

13

Conclusions A data driven approach is possible in the exploration of

intonation phenomena

FDA provides automatic tools to describe variation in a set

of pitch contours extracted from real utterances

provided that the relevant landmarks are annotated

The same tools allow to construct artificial contours with

desired perceptual characteristics

Smooth and global variation are applied

Variations come from a statistical analysis of data

The process is automatic

14