• Home

  • Documents

  • Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for...
15
Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting Emotions from Speech Jing Han 1 , Zixing Zhang 2 , Zhao Ren 1 , Fabien Ringeval 3 , Björn Schuller 1,2 1. University of Augsburg, Germany 2. Imperial College London, UK 3. Université Grenoble Alpes, France

Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

  • Upload
    others

  • View
    5

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Special Session: Deep Learning for Computational Paralinguistics

Towards Conditional Adversarial Training for Predicting Emotions from Speech

Jing Han1, Zixing Zhang2, Zhao Ren1, Fabien Ringeval3, Björn Schuller1,2

1. University of Augsburg, Germany2. Imperial College London, UK3. Université Grenoble Alpes, France

Page 2: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Motivation

Adversarial training (also called GAN for Generative Adversarial Networks), and the variations that are now being proposed is the most interesting idea in the last 10 years in ML.

-- Yann LeCun, Director of AI Research at Facebook

April 18, 2018, ICASSP @ Calgary Jing Han

Donahue, Chris, et al. "Semantically decomposing the latent spaces of generative adversarial networks." ICLR 2018.Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks." ICLR 2017.

Page 3: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Motivation

April 18, 2018, ICASSP @ Calgary Jing Han

Isola, Phillip, et al. "Image-to-image translation with conditional adversarial networks." CVPR 2017.Antipov, Grigory, Moez Baccouche, and Jean-Luc Dugelay. "Face aging with conditional generative adversarial networks." ICIP 2017.

Page 4: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Adversarial Training

April 18, 2018, ICASSP @ Calgary Jing Han

Generator DiscriminatorLoss

Real

Fake

Real

Latent random variable z

generated samples

The Discriminator tries to distinguish between the generated (fake) and real data.

The Generator tries to turn the input noise into fake data to try to fool the discriminator.

Page 5: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Conditional Adversarial Training

April 18, 2018, ICASSP @ Calgary Jing Han

Generator DiscriminatorLoss

Real

Fake

Real

Latent random variable z

generated samples

The Condition y tries to control the modes of the data being generated, could be based on class labels or data from different modalities, etc.Condition y

Page 6: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Proposed Conditional Adversarial Training for SER

April 18, 2018, ICASSP @ Calgary Jing Han

Generator DiscriminatorLoss

Real

Fake

Real

Latent random variable z

generated samples

Condition y

Regressor

audio features

predictions

Regressor DiscriminatorLoss

Real

Fake

Real y

predictions y'

Audio features x

Page 7: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Proposed Conditional Adversarial Training for SER

April 18, 2018, ICASSP @ Calgary Jing Han

Condition yaudio features

Regressor DiscriminatorLoss

Real

Fake

Real y

Page 8: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Proposed Conditional Adversarial Training for SER

April 18, 2018, ICASSP @ Calgary Jing Han

Regressor(G) Discriminator

Loss

Real

Fake

Real y

predictions y'

Audio features x

Objective 1:

Objective 2:(wasserstein distance)

Page 9: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Database

● recordings of spontaneous collaborative and affective interactions in French

● partially used in AVEC 2015 and 2016

● 46 subjects (23 pairs) and 5 minutes per pair

● 16 training / 15 development / 15 test

● http://diuf.unifr.ch/diva/recola/index.html

April 18, 2018, ICASSP @ Calgary Jing Han

http://diuf.unifr.ch/diva/recola/index.html
Page 10: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Experiments

● feature set: mean & var of MFCC 0-12 and log energy

● structure for both NN1 and NN2: LSTM-RNN with 2 layers, 20 nodes per layer.

● measurement: CCC (concordance correlation coefficient)

April 18, 2018, ICASSP @ Calgary Jing Han

Page 11: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Experimental Results

April 18, 2018, ICASSP @ Calgary Jing Han

CCCarousal valence

dev test dev test

RNN(2 layers) .777 .718 .491 .435

RNN (4 layers) .761 .723 .487 .390

conditional adversarial training .780 .732 .501 .455

conditional adversarial training with wasserstein .797 .737 .474 .444

without adversarial

proposed

Page 12: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Experimental Results

April 18, 2018, ICASSP @ Calgary Jing Han

CCCarousal valence

dev test dev test

CCC-objective .412 .350 .242 .199

end-to-end .752 .699 .406 .311

reconstruction-error based .785 .729 .378 .360

prediction-based .774 .744 .412 .377

conditional adversarial training .780 .732 .501 .455

conditional adversarial training with wasserstein .797 .737 .474 .444proposed

state of the art

Page 13: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Experimental Results

April 18, 2018, ICASSP @ Calgary Jing Han

Page 14: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Conclusion & Perspective

Conclusion

● adversarial training for speech emotion recognition● two variants both perform better than baseline● arousal prediction benefits more from wasserstein distance than valence

Future work

● train two models based on a loss threshold rather than fixed schedule● experiments on other dataset● perform an end-to-end structure

April 18, 2018, ICASSP @ Calgary Jing Han

Page 15: Towards Conditional Adversarial Training for Predicting ... · Special Session: Deep Learning for Computational Paralinguistics Towards Conditional Adversarial Training for Predicting

Thank you for [email protected]


A (conditional) Generative Adversarial Network

A (conditional) Generative Adversarial Network

Documents
Conditional Adversarial Networks for Blind Image Deblurring

Conditional Adversarial Networks for Blind Image Deblurring

Documents
Fully Automated Image De-fencing using Conditional Generative … · 2019-08-20 · Fully Automated Image De-fencing using Conditional Generative Adversarial Networks Divyanshu Gupta

Fully Automated Image De-fencing using Conditional Generative … · 2019-08-20 · Fully Automated Image De-fencing using Conditional Generative Adversarial Networks Divyanshu Gupta

Documents
Conditional Coupled Generative Adversarial Networks for ...openaccess.thecvf.com/content_ICCV_2019/papers/... · ing zero-shot domain adaptation (ZSDA) problem, where target-domain

Conditional Coupled Generative Adversarial Networks for ...openaccess.thecvf.com/content_ICCV_2019/papers/... · ing zero-shot domain adaptation (ZSDA) problem, where target-domain

Documents
Image De-raining Using a Conditional Generative …1 Image De-raining Using a Conditional Generative Adversarial Network He Zhang, Member, IEEE, Vishwanath Sindagi, Student Member,

Image De-raining Using a Conditional Generative …1 Image De-raining Using a Conditional Generative Adversarial Network He Zhang, Member, IEEE, Vishwanath Sindagi, Student Member,

Documents
Conditional Generative Adversarial Network for Structured ......a principled way to conduct structured domain adaption for semantic segmentation, i.e., integrating GAN into the FCN

Conditional Generative Adversarial Network for Structured ......a principled way to conduct structured domain adaption for semantic segmentation, i.e., integrating GAN into the FCN

Documents
xudonmao@gmail.com itqli@cityu.edu.hk … Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks Xudong Mao Department of Computer Science City University

[email protected] [email protected] … Learning to Align Cross-Domain Images with Conditional Generative Adversarial Networks Xudong Mao Department of Computer Science City University

Documents
Image-To-Image Translation With Conditional Adversarial ...openaccess.thecvf.com/content_cvpr_2017/papers/Isola_Image-To... · Image-to-Image Translation with Conditional Adversarial

Image-To-Image Translation With Conditional Adversarial ...openaccess.thecvf.com/content_cvpr_2017/papers/Isola_Image-To... · Image-to-Image Translation with Conditional Adversarial

Documents
MC-GAN: Multi-conditional Generative Adversarial Network for Image ... · MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis Hyojin Park 1 wolfrun@snu.ac.kr

MC-GAN: Multi-conditional Generative Adversarial Network for Image ... · MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesis Hyojin Park 1 [email protected]

Documents
B(0,1,0) Adversarial A Copenaccess.thecvf.com/content_ICCV_2019/papers/... · oped[2,4,17,18,19,26],usingclasslabelsorothercharac-teristics. Conditional GANs are also used in domain

B(0,1,0) Adversarial A Copenaccess.thecvf.com/content_ICCV_2019/papers/... · oped[2,4,17,18,19,26],usingclasslabelsorothercharac-teristics. Conditional GANs are also used in domain

Documents
Multiple Generative Adversarial Networks Analysis …Multiple GANs analysis for predicting photographer’s retouching 3 Fig.1: Taken from [8]. Training a conditional GAN to map edges!photo

Multiple Generative Adversarial Networks Analysis …Multiple GANs analysis for predicting photographer’s retouching 3 Fig.1: Taken from [8]. Training a conditional GAN to map edges!photo

Documents
Introduction to Generative Adversarial Networks · Traditional MLE Approach ... convolutional generative adversarial networks. 2015. (3) Mirza, Mehdi and Osindero, Simon. Conditional

Introduction to Generative Adversarial Networks · Traditional MLE Approach ... convolutional generative adversarial networks. 2015. (3) Mirza, Mehdi and Osindero, Simon. Conditional

Documents
Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou

Image-to-Image Translation with Conditional Adversarial Networks€¦ · Image-to-Image Translation with Conditional Adversarial Networks Phillip Isola, Jun-Yan Zhu, Tinghui Zhou

Documents
Stacked Generative Adversarial NetworksStacked Generative Adversarial Networks in Sec.3.2. In Sect.3.3and3.4we will focus on our two novel loss func-tions, conditional loss and entropy

Stacked Generative Adversarial NetworksStacked Generative Adversarial Networks in Sec.3.2. In Sect.3.3and3.4we will focus on our two novel loss func-tions, conditional loss and entropy

Documents
Lecture 22: Adversarial Image, Adversarial Training, Variational Autoencoders… · 2018-11-08 · Adversarial Images Adversarial Training Autoencoder VAE GAN Conclusions Lecture

Lecture 22: Adversarial Image, Adversarial Training, Variational Autoencoders… · 2018-11-08 · Adversarial Images Adversarial Training Autoencoder VAE GAN Conclusions Lecture

Documents
Deep Generative Models: GANs and VAE · Deep convolutional generative adversarial networks Auxiliary classifier GANs Odena, A., Olah, C., & Shlens, J. (2016). Conditional image synthesis

Deep Generative Models: GANs and VAE · Deep convolutional generative adversarial networks Auxiliary classifier GANs Odena, A., Olah, C., & Shlens, J. (2016). Conditional image synthesis

Documents
Conditional generative adversarial nets for convolutional ... · Conditional generative adversarial nets for convolutional face generation Jon Gauthier Symbolic Systems Program, Natural

Conditional generative adversarial nets for convolutional ... · Conditional generative adversarial nets for convolutional face generation Jon Gauthier Symbolic Systems Program, Natural

Documents
A Conditional Wasserstein Generative Adversarial Network

A Conditional Wasserstein Generative Adversarial Network

Documents
Single Image Dehazing via Conditional Generative ...openaccess.thecvf.com/content_cvpr_2018/papers/Li_Single_Image_De... · Single Image Dehazing via Conditional Generative Adversarial

Single Image Dehazing via Conditional Generative ...openaccess.thecvf.com/content_cvpr_2018/papers/Li_Single_Image_De... · Single Image Dehazing via Conditional Generative Adversarial

Documents
Practical concerns in predicting structures...Practical concerns in predicting ... – We want to convert some input to an output – Model the conditional distribution of the output

Practical concerns in predicting structures...Practical concerns in predicting ... – We want to convert some input to an output – Model the conditional distribution of the output

Documents
Conditional Generative Adversarial Networksjbhuang/teaching/ece6554/sp... · 2017-03-02 · •Conditional GAN can serve as general mapping model X->Y •No need to define domain-specific

Conditional Generative Adversarial Networksjbhuang/teaching/ece6554/sp... · 2017-03-02 · •Conditional GAN can serve as general mapping model X->Y •No need to define domain-specific

Documents
Conditional generative adversarial nets for …cs231n.stanford.edu/reports/2015/pdfs/jgauthie_final...condition y for generation, which restricts both the generator in its output and

Conditional generative adversarial nets for …cs231n.stanford.edu/reports/2015/pdfs/jgauthie_final...condition y for generation, which restricts both the generator in its output and

Documents
DAGAN: Deep De-Aliasing Generative Adversarial Networks ... · conditional Generative Adversarial Networks-based model (DAGAN)-based model is proposed to reconstruct CS-MRI. In our

DAGAN: Deep De-Aliasing Generative Adversarial Networks ... · conditional Generative Adversarial Networks-based model (DAGAN)-based model is proposed to reconstruct CS-MRI. In our

Documents
Bayesian Conditional Generative Adverserial Networks · We propose a new GAN called Bayesian Conditional Gener- ative Adversarial Networks (BC-GANs) that use a random generator function

Bayesian Conditional Generative Adverserial Networks · We propose a new GAN called Bayesian Conditional Gener- ative Adversarial Networks (BC-GANs) that use a random generator function

Documents
Stacked Conditional Generative Adversarial Networks for ...openaccess.thecvf.com/content_cvpr_2018/papers/...which is also denoted as a shadow matte [43], to recover the image with

Stacked Conditional Generative Adversarial Networks for ...openaccess.thecvf.com/content_cvpr_2018/papers/...which is also denoted as a shadow matte [43], to recover the image with

Documents
Introduction of Generative Adversarial Network (GAN)yvchen/f106-adl/doc/171211+171214_GAN.pdf · GAN as structured learning algorithm Conditional Generation by GAN •Modifying input

Introduction of Generative Adversarial Network (GAN)yvchen/f106-adl/doc/171211+171214_GAN.pdf · GAN as structured learning algorithm Conditional Generation by GAN •Modifying input

Documents
Conditional Single-view Shape Generation for Multi-view ...b1ueber2y.me/projects/OptimizeMVS/optimizeMVS.pdf · Generative Adversarial Networks (CGAN) [27] made use ... ods [5,17,47]

Conditional Single-view Shape Generation for Multi-view ...b1ueber2y.me/projects/OptimizeMVS/optimizeMVS.pdf · Generative Adversarial Networks (CGAN) [27] made use ... ods [5,17,47]

Documents
Location Conditional Image Generation using …...Location Conditional Image Generation using Generative Adversarial Networks Mayur Saxena Department of Computer Science University

Location Conditional Image Generation using …...Location Conditional Image Generation using Generative Adversarial Networks Mayur Saxena Department of Computer Science University

Documents
Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial

Unsupervised Conditional Generation - 國立臺灣大學speech.ee.ntu.edu.tw/~tlkagk/courses/MLDS_2018/Lecture/CycleGA… · •Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial

Documents
Predicting Shot Making in Basketball Learnt from ... · Predicting Shot Making in Basketball Learnt from Adversarial Multiagent Trajectories Mark Harmon,1 Patrick Lucey,2 Diego Klabjan1

Predicting Shot Making in Basketball Learnt from ... · Predicting Shot Making in Basketball Learnt from Adversarial Multiagent Trajectories Mark Harmon,1 Patrick Lucey,2 Diego Klabjan1

Documents