Evaluation of singing synthesis: methodology and case study with … · Evaluation of singing...

Evaluation of singing synthesis: methodology and case study

with concatenative and performative systems

Lionel Feugère1, Christophe d'Alessandro1, Samuel Delalez1,Luc Ardaillon2, Axel Roebel2

1LIMSI, CNRS, Université Paris-Saclay, 91405 Orsay, France2IRCAM, CNRS, Sorbonne Universités UPMC, 75004 Paris, France

13th Interspeech 2016, September 8th- 12th, San Francisco

Singing synthesis challenges1993 Stockolm Musical Acoustic Conference2007 Interspeech2016 Interspeech

GoalsProposing a method for evaluating singing synthesisEvaluating synthesis systems from the ChaNTeR project http://chanter.limsi.fr/

Context and Goal

Synthesis systems

Methodology

Protocol

Results

Conclusion

Outline

Case study

Segmental basis

Melodic and rhythmic control

Concatenation and/or freq.-time scaling

Case study

Text-to-Chant (TTC)Input symbolic scoreOff-line model of melodyand phoneme duration

Singing instrument (Calliphony)MusicianLive control of articulation rhythm( foot pedal) and pitch (pen-tablet)

Delalez, S. & d'Alessandro, C. (LIMSI)Ardaillon, L. & Roebel, A. (Ircam)

Segmental basis

Case study

Segmental basis

Natural Monocord-isochronDatabase for concatenation

Case study

Segmental basis

PAN SuperSVP

Le Beux, S. et al. Roebel, A.Degottex, G. et alHubber, S. et al.

RT-PSOLA

Case study

Segmental basis

PAN SuperSVP

Concatenation /freq.-time scaling

RT-PSOLA

Case study

Segmental basis

PAN SuperSVP

RT-PSOLA

Case study

Segmental basis

RT-PSOLAPAN SuperSVP

Case study

Segmental basis

Case study

Segmental basis

PAN SuperSVP

RT-PSOLA

Methodology - Types of listening tests

AB test

Task: preference bet. 2 soundsResults: mean % of preferencebetween each system

direct comparison

All sounds are compared by pair

=> Short sounds are preferableBetter for particulardimension assessment

Quality of articulationQuality of ornamentation

=> Few sounds is preferableBetter not to add references

Methodology - Types of listening tests

Absolute Category Rating

Task: opinion score (1-5)Results: mean opinion score (MOS) foreach system

indirect comparison

Each sound is assessed individually

=> Allows long soundsBetter for general quality assessment

=> Allows higher number of soundsAllows to add references

NaturalPitch/timbre/phoneme degradations

AB test

Task: preference bet. 2 soundsResults: mean % of preferencebetween each system

direct comparison

All sounds are compared by pair

=> Short sounds are preferableBetter for particulardimension assessment

Quality of articulationQuality of ornamentation

=> Few sounds is preferableBetter not to add references

Protocol

~2sec sounds

AB test 1 “Choose theitem for whichyou rate thequality of lyricsarticulation thebest”

~7sec sounds (4 bars)

25 paid subjects, active in audio/music, not involved in the project Summer Time and Autumn Leaves musics, French lyrics Synthesized by each system

Absolute Category rating Question:

Globally, how do you ratethe quality of what youhave just heard?

Response: bad (1), poor (2),fair (3), good (4), excellent (5)

AB test 2 “Choose theitem for whichyou rate thequality ofornamentation (vibrato,portamento) the best”

~2sec sounds

Results – General quality (ACR)

Diamond are MOS

REFERENCESNat = NaturalDC1 = pitch degradedDC2 = timbre degradedDC3 = phoneme degraded

SEGEMENTAL BASISCon = concatenationMi = Natural monocord-isochron

CONCATENATION / TIME-FREQ SCALINGPAN = Text-to-Chant with PANSVP = Text-to-Chant with SuperVPCal = Calliphony Singing instrument

Results – General quality (ACR)

Segmental basis

Natural Monocord-isochron Database for concatenation

Results – articulation quality (AB)

Segmental basis

<~70% preference

~60-80% preference

Results – ornamentation quality (AB)

Segmental basis

~60% preference

~60-70% preference

Conclusion

Global and analytical evaluation methods for assessing overallquality, articulation quality and ornamentation quality

Absolute category rating allows longer extracts when largenumber of systems=> better for overall musical quality

AB test allows to find differences where Absolute Category ratingdid not=> better for quality on specific dimensions

Text-to-Chant system > Singing instrument CalliphonyBut the methodology better suited for Text-to-Chant systems

Thank you for your attentionlionel.feugere@limsi.fr

Calliphony singing instrument: samuel.delalez@limsi.fr Text-to-Chant system: luc.ardaillon@ircam.fr

ChaNTeR project: http://chanter.limsi.fr/ Sound examples can be downloaded or played online (see paper)

Evaluation of singing synthesis: methodology and case study with concatenative and performative systems

Lionel Feugère, Christophe d'Alessandro, Samuel Delalez, Luc Ardaillon, Axel Roebel

Results (AB)

AB Mi-SVP Con-PAN Mi-PAN Con-Cal Mi-Cal

Con-SVP 12

68%*58%*

56%57%

15%*29%*

40%*34%*

Mi-SVP 12

20%*28%

Con-PAN 12

71%*48%

13%*31%*

35%*33%*

Mi-PAN 12

17%*37%*

Con-Cal 12

71%*55%

Percentage of preference of the column system over the line system

* = significant

yellow = less than 1/3 or more than 2/3

AB1: articulation quality

AB2: ornamentation quality

Evaluation of singing synthesis: methodology and case study with … · Evaluation of singing...

Documents

Total Synthesis of Manzamine A - Chemistrychemistry.syr.edu/totah/che686/support/3d1/ManzamineA.pdf · Total Synthesis of Manzamine A Chris Wilhelmsen Organic Synthesis Methodology

Methodology for the Enantioselective Synthesis of

One-Pot Methodology for the Synthesis of Polysubstituted ... · 2.2 - DEVELOPMENT OF NEW METHODOLOGY FOR 3-CYANOPYRIDINE SYNTHESIS 2.2.0-Aims 91 2.2.1 - Two-Component Method Development

DEVELOPMENT OF A COMPUTER-AIDED FAULT TREE SYNTHESIS METHODOLOGY

THE LOGIC OF CHEMICAL SYNTHESIS: MULTISTEP SYNTHESIS … · in Chemistry for 1990 for the “development of the theory and methodology of organic synthesis”. Chemical synthesis

Synthesis and expressive transformation of singing voice

NOVEL METHODOLOGY FOR THE SYNTHESIS OF ...wiredspace.wits.ac.za/bitstream/handle/10539/7360/MSC...NOVEL METHODOLOGY FOR THE SYNTHESIS OF XANTHONES Jeremy Morgan Naidoo Supervised by

Methodology for Polymer Supported Synthesis Towards

Digitartic: bi-manual gestural control of articulation in ... · singing voice synthesis, gestural control, syllabic synthesis, articulation, formants synthesis 1. INTRODUCTION In

A multi-layer F0 model for singing voice synthesis using a

Novel Synthesis Methodology in Digital IC Design and

The KTH rule system for singing synthesis · The KTH rule system for singing synthesis Berndtsson, G ... Another project of direct relevance to the ... It was controlled by an Eclipse

An Innovative Synthesis Methodology for Process Intorbit.dtu.dk/files/7968553/Philip@0332Lutze@0332PEC12@033222.pdf · An Innovative Synthesis Methodology for Process Intensification

1.0 Verilog Synthesis Methodology - Cornell Universitypeople.ece.cornell.edu/land/courses/ece5760/Verilog/...October 18, 2001 1 1.0 Verilog Synthesis Methodology Finbarr O’Regan

Synthesis Methodology at Tundra Seniconductors

Articulatory Synthesis of Singing Peter Birkholz Institute for Computer Science, University of Rostock Singing Synthesis Challenge 2007 at the Interspeech‘07,

VOCALISTENER2: A SINGING SYNTHESIS SYSTEM ABLE TO … · VOCALISTENER2: A SINGING SYNTHESIS SYSTEM ABLE TO MIMIC A USER S SINGING IN TERMS OF VOICE TIMBRE CHANGES AS WELL AS PITCH

A Synthesis Methodology for Application-Specific Logic-in

Singing Voice Analysis/Synthesis - Amazon S3 › che-courses › Singing+Voice...Singing Voice Analysis/Synthesis by Youngmoo Edmund Kim B.S Engineering, Swarthmore College, 1993 B.A

Singing a New Story: A Composer's Exploration of Textual Synthesis