Upload
frederick-blakeway
View
215
Download
2
Tags:
Embed Size (px)
Citation preview
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 1
Generating narrative speech for the Virtual Storyteller Koen Meijs, Mariet Theune, Dirk Heylen* and others
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 2
Overview
• Background: The Virtual Storyteller• Analysis of human storytellers• Conversion rules and testing• Implementation• Evaluation• Conclusions and future work
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 3
The Virtual Storyteller
Automatic story
generation:• Plot creation• Natural language
generation• Storytelling
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 4
Plot creation
Characters in the story are (semi) autonomous agents, which:
• Have their own personality, goals and emotions
• Can perform planned actions to reach their goals
• Are guided by a director agent
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 5
NLG and story presentation
• Language generation using simple sentence templates
• Story presentation by an embodied, speaking agent (using Microsoft Agents as a temporary solution)
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 6
Example story settingNB: Visualisation is not part
of the system yet!
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 7
Example story text
Diana walked to the forest. Brutus walked to the plains. Diana picked up the sword. Brutus walked to the desert. Diana walked to the desert. Brutus was afraid of Diana because Brutus saw that Diana had the sword. Brutus hit Diana. Diana was afraid of Brutus because Diana saw Brutus.Diana walked to the forest. Brutus was afraid of Diana because Brutus saw that Diana had the sword. Brutus walked to the forest. Diana stabbed the villain. And she lived happily ever after!!!
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 8
Storytellers’ speech
Human storytellers engage their audience by:• General “storytelling” speech style• Different voices for characters• Expressing emotions• Different “sound effects”
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 9
Focus of this work
• General storytelling style• Use of prosody to express suspense in
stories
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 10
Analysis of human speakers
Global storytelling style, material from:• newsreader (Onno Duyvené de Wit) • children’s storyteller (Sacco van der Made)• adult storyteller (Toon Tellegen)
Analysis (using PRAAT) mainly based on children’s storyteller
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 11
Features
• Pitch• Intensity• Tempo (syllables per second)• Pause duration• Vowel length
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 12
Global storytelling style
Pitch / intensity: • Averages are
similar• Standard
deviation is much larger for storyteller
newsreader
children’s storyteller
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 13
Global storytelling style
Tempo (syllables per second): newsreader is much faster than both storytellers
Pause duration: storyteller pauses are longer (esp. between sentences)
Also: lengthening of certain adverbs/adjectives by storyteller (“A long corridor that was s o low …”)
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 14
Expressing suspense
• Sudden climax: an unexpected revelation.
E.g., opening Bluebeard’s secret chamber:“She had to get used to the darkness, and then …”
• Increasing climax: building up expectation.
Finally finding the Sleeping Beauty:“He opened the door and… there was the sleeping princess.”
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 15
Sudden climax
• “En toen…” / “And then…”• Sudden rise in pitch and intensity on “then” • Vowel lengthening in “then”
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 16
Increasing climax
• Two parts: 1 creating expectation 2 revelation• First part: increasing pitch and vowel duration • Second part: more constant, lower pitch and
intensity
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 17
Conversion rules
• Conversion from ‘neutral’ to ‘storytelling’ speech
• Rules based on analysis of human speakers• Input: paired time-value data • Output: new values for a given time domain
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 18
Example from storytelling style
• Pitch: increase the pitch of syllables carrying a sentence accent
• All pitch values inside the syllable’s time domain are multiplied by a certain factor (based on a sine function)
• Maximum increase between 40-90 Hz
→ best value to be determined experimentally
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 19
Determining constant values
• Material: speech produced by Fluency text-to-speech, manipulated using PRAAT scripts
• Five subjects compared 22 speech fragment pairs with different values for one constant
• Subjects had to indicate: – Which fragment sounded most natural or– Which had the best expression of
suspense
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 20
Results: storytelling style
Constant Range Outcome
Max. pitch increase 40 – 90 Hz 40 Hz
Intensity increase 2 - 6 Db 2 Db
Global tempo
(syllables per second)
3.0 – 3.6 sps 3.6 sps
Vowel duration increase 0 or 50% 50%
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 21
Results: sudden climax
Constant Range Outcome
Intensity rise at start of climax 6 - 10 Db 6 Db
Pitch rise at start of climax 80 – 120 Hz 80 Hz
Subsequent pitch rise 0 - 200 Hz 0 Hz
“Everybody waited in silence, and then ... there was a loud bang!”
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 22
Results: increasing climax
Constant Range Outcome
Pitch contour start at 25-50 Hz
top at 60-80 Hz
25 Hz
60 Hz
Vowel duration increase 50 - 100% 50%
“Step by step he jumped from stone to stone, slipped on the last stone and… fell into the water.”
Neutral: Pitch contour manipulated:
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 23
Pilot test of conversion rules
• 16 speech fragments:– 8 ‘neutral’ (Fluency, with no manipulation) – 8 manipulated using PRAAT according to
conversion rules, using best constant values• Eight subjects rated storytelling quality,
naturalness, and suspense on a five-point scale (subjects divided in two groups)
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 24
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 25
Pilot test results
Compared to neutral fragments, • Storytelling quality of manipulated fragments
was rated equal or better• Naturalness of manipulated fragments was
rated equal or less • Manipulated fragments were rated as having
more suspense, even if only the ‘global storytelling style’ was used
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 26
Implementation
annotated text input
partial synthesis (Fluency)
neutral prosodic
information
resynthesis (Fluency)
narrative prosodic
information
narrative speech
application of
conversion rules
Prosodic information = list of phonemes with pitch and duration values (no possible to adjust intensity)
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 27
Example annotated text
Annotation: extension of SSML. <speak>
<style type=narrative/>
<s> The beard made him look <accent extend=yes> so </accent> ugly that everybody ran away when they saw him. </s>
<s> He wanted to turn around <climax type=sudden> and then </climax> there was a loud bang. </s>
<s> Bluebeard raised the big knife, <climax type=increasing> he wanted to strike and <climax_top/> there was a knock on the door. </climax> </s>
</speak>
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 28
Example prosodic information
1: h 112
2: I: 151 50 75
3: R 75
4: l 75
5: @ 47 20 71 70 61
6: k 131
7: @ 55 80 70
8: _ 11 50 65
• Phoneme • Duration (ms)• Pitch percentage
(specifying at which point during the phoneme the pitch value should be applied)
• Pitch value
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 29
Conversion steps
• Parse XML• Look up phonemes to be manipulated• Apply function
For example, pitch for global storytelling style:
y(t).(sin((((t-t1)/(t2-t1))0,5π) + 0,25π)/n)),
where n = average pitch / 40
• Return adapted valuesNB: intensity cannot be adapted in Fluency
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 30
Evaluation of implementation
• Set-up similar to conversion rule pilot test• 16 fragments (8 neutral / narrative pairs)• 20 subjects, divided in two groups• Rating storytelling quality, naturalness, and
suspense on a 5 point scale
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 31
Mean scores
1 2 3 4 5 6 7 8
Story-telling
3,0 3,9 3,1 3,5 3,1 3,3 3,0 3,6 2,5 3,2 3,1 3,6 3,1 3,5 3,0 2,8
Natural-ness
2,6 3,7 3,3 3,2 2,6 2,8 2,6 3,3 2,5 2,3 2,5 3,2 3,1 3,5 3,1 2,9
Suspense
2,1 3,7 2,5 3,1 2,5 2,8 2,1 3,0 1,8 2,2 2,3 3,6 2,7 3,4 2,4 4,0
Significant differences (≤ 0,05) are shown in bold face. Underlining indicates near significance.
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 32
Summing up the results
• Storytelling quality of manipulated fragments: rated above average, and better than neutral fragments (but hardly significant)
• Naturalness: ratings vary; some accents were seen as misplaced (though copied from original fragment)
• Suspense of manipulated fragments rated higher than neutral fragments (some significance)
Humaine Workshop Paris Generating narrative speech for the Virtual Storyteller 33
Conclusions & future work
• Successful automatic conversion from standard text-to-speech to ‘storytelling prosody’
• Further improvement and larger-scale evaluation still needed
• Automatic derivation of features from text?