• Home

  • Documents

  • Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case...
24
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised by Amélie Cordier & Mathieu Lefort Internship period: 14 th June 2017 – 11 th August 2017 Yu-Guan Hsieh Multimodal Learning September 2017 1 / 19

Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

  • Upload
    others

  • View
    11

  • Download
    0

Embed Size (px)

Citation preview

Page 1: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Soutenance de Stage de L3Multimodal Learning:A case study for Gesture and Audio-Visual SpeechRecognition

Hsieh Yu-Guan (Info 2016)Supervised by

Amélie Cordier & Mathieu LefortInternship period: 14th June 2017 – 11th August 2017

Yu-Guan Hsieh Multimodal Learning September 2017 1 / 19

Page 2: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

BEHAVIORS.AIHoomano & LIRISTensorFlow

Yu-Guan Hsieh Multimodal Learning September 2017 2 / 19

Page 3: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

Artifical Intelligence

> Robotics (embodied paradigme)> Developmental robotics> Multimodal learning> Gesture and Audio-Visual recogntion

Yu-Guan Hsieh Multimodal Learning September 2017 3 / 19

Page 4: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

Artifical Intelligence> Robotics (embodied paradigme)

> Developmental robotics> Multimodal learning> Gesture and Audio-Visual recogntion

Yu-Guan Hsieh Multimodal Learning September 2017 3 / 19

Page 5: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

Artifical Intelligence> Robotics (embodied paradigme)> Developmental robotics

> Multimodal learning> Gesture and Audio-Visual recogntion

Yu-Guan Hsieh Multimodal Learning September 2017 3 / 19

Page 6: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

Artifical Intelligence> Robotics (embodied paradigme)> Developmental robotics> Multimodal learning

> Gesture and Audio-Visual recogntion

Yu-Guan Hsieh Multimodal Learning September 2017 3 / 19

Page 7: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Introduction

Artifical Intelligence> Robotics (embodied paradigme)> Developmental robotics> Multimodal learning> Gesture and Audio-Visual recogntion

Yu-Guan Hsieh Multimodal Learning September 2017 3 / 19

Page 8: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Deep Network Architectures – Convolutional NeuralNetworks (CNNs)

Yu-Guan Hsieh Multimodal Learning September 2017 4 / 19

Page 9: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Deep Network Architectures – Convolutional NeuralNetworks (CNNs) – Convolution

Source: https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.htmlAlso see https://github.com/vdumoulin/conv_arithmetic

Yu-Guan Hsieh Multimodal Learning September 2017 5 / 19

https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html
https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html
https://github.com/vdumoulin/conv_arithmetic
Page 10: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Deep Network Architectures – Convolutional NeuralNetworks (CNNs) – Max-pooling

Source: https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html

Yu-Guan Hsieh Multimodal Learning September 2017 6 / 19

https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html
https://cambridgespark.com/content/tutorials/convolutional-neural-networks-with-keras/index.html
Page 11: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Deep Network Architectures – Autoencoder

Source:https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694

Yu-Guan Hsieh Multimodal Learning September 2017 7 / 19

https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694
Page 12: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Deep Network Architectures – Convolutional Autoencoder

Source:https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694

Yu-Guan Hsieh Multimodal Learning September 2017 8 / 19

https://hackernoon.com/autoencoders-deep-learning-bits-1-11731e200694
Page 13: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Training a Machine Learning Model

Loss function: cross-entropy, L2-distanceStochastic gradient descent (SGD)BackpropagationVaraints of SGD: AdaGrad, Adam

Yu-Guan Hsieh Multimodal Learning September 2017 9 / 19

Page 14: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Datasets – ASL Finger SpellingRGB and depthMore than 60000 images for each modality24 static signs in American Sign Language5 subjectsOnly one channel in inputResized to 83× 83 and Z-normalization

(a) (b) (c) (d)

Yu-Guan Hsieh Multimodal Learning September 2017 10 / 19

Page 15: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Datasets – AVLetters

Audio-visual26 letters from A to Z10 speakers, each letter 3 times eachAudio: 24 frames in input, 26 MFCCs for each frameVideo: 12 frames in input, z-normalized

Yu-Guan Hsieh Multimodal Learning September 2017 11 / 19

Page 16: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Results – Unsupervised Learning with CAE

Yu-Guan Hsieh Multimodal Learning September 2017 12 / 19

Page 17: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Results – Unsupervised Learning with CAE

Yu-Guan Hsieh Multimodal Learning September 2017 13 / 19

Page 18: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Results – Unsupervised Learning with CAE

Raw: Perceptron that reads raw input data.CAE features: Perceptron stacked on the middle layer of the CAE.CAE architecture: Perceptron stacked on the middle layer of the CAEbut train the whole network in a supervised way as a CNN.

Raw CAE features CAE architecture

Intensity train 69.47 % 78.87 % 91.29 %test 32.64 % 50.24 % 65.44 %

Depth train 63.64 % 79.61 % 88.80 %test 29.93 % 41.64 % 55.62 %

Yu-Guan Hsieh Multimodal Learning September 2017 14 / 19

Page 19: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Results – Shared Representation Learning

Yu-Guan Hsieh Multimodal Learning September 2017 15 / 19

Page 20: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Results – Shared Representation

Shared: Perceptron that exploits the shared representation learned bya bimodal CAE.

Raw CAE features CAE architecture Shared

Intensity train 69.47 % 78.87 % 91.29 % 85.85 %test 32.64 % 50.24 % 65.44 % 53.38 %

Depth train 63.64 % 79.61 % 88.80 % 81.83 %test 29.93 % 41.64 % 55.62 % 42.85 %

Yu-Guan Hsieh Multimodal Learning September 2017 16 / 19

Page 21: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

AVSR Knowledge Transfer

Yu-Guan Hsieh Multimodal Learning September 2017 17 / 19

Page 22: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Yu-Guan Hsieh Multimodal Learning September 2017 18 / 19

Page 23: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

AVSR Knowledge Transfer

Fine-tuned for 160 steps.For Exp1, Exp2 and Exp3, we have respectively α0 = 0.001, 0.005, 0.001and pa = 0.85, 0.85, 1.Notice that since pa = 1 for Exp3 no video data is given in input duringfine-tuning.

Trv TrvA∼T Trv

U∼Z Tev TevA∼T Tev

U∼Z

No transfer 77.67 % 100 % 0 % 40.56 % 54.48 % 0 %Exp1 81.17 % 98.28 % 21.64 % 39.44 % 47.76 % 15.22 %Exp2 40.83 % 51.07 % 5.22 % 23.89 % 30.60 % 4.35 %Exp3 19.67 % 12.23 % 45.52 % 12.22 % 2.24 % 41.34 %

Yu-Guan Hsieh Multimodal Learning September 2017 19 / 19

Page 24: Soutenance de Stage de L3 - behaviors.ai · Soutenance de Stage de L3 Multimodal Learning: A case study for Gesture and Audio-Visual Speech Recognition Hsieh Yu-Guan (Info 2016) Supervised

..........

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

.....

.....

......

.....

......

.....

.....

.

Conclusion

Datasets, hyperparametersApplications in roboticsOther approaches

Yu-Guan Hsieh Multimodal Learning September 2017 20 / 19


Fast and Furious Decision Tree Induction Soutenance de planification

Fast and Furious Decision Tree Induction Soutenance de planification

Documents
Soutenance de thèse professionnelle sur les indicateurs RSE et le management de projet

Soutenance de thèse professionnelle sur les indicateurs RSE et le management de projet

Education
Soutenance de TER Encadrant : Eric Languénou jeudi 2 juin 2005

Soutenance de TER Encadrant : Eric Languénou jeudi 2 juin 2005

Documents
Soutenance ysance

Soutenance ysance

Documents
Soutenance de thèse de Michel Héon Direction : Gilbert Paquette et Josianne Basque

Soutenance de thèse de Michel Héon Direction : Gilbert Paquette et Josianne Basque

Documents
Soutenance projet de fin d'etude IPV4-IPv6

Soutenance projet de fin d'etude IPV4-IPv6

Career
Soutenance de thèse présentée par Pierre Laforcade Laboratoire d’Informatique de

Soutenance de thèse présentée par Pierre Laforcade Laboratoire d’Informatique de

Documents
Rapport de Soutenance 2linkfly.free.fr/soutenance2.pdfRapport de Soutenance 2 1.3 Kevin´ Slytink, tres grand timide et romantique` pronant la paix dans le monde. C’est un jeuneˆ

Rapport de Soutenance 2linkfly.free.fr/soutenance2.pdfRapport de Soutenance 2 1.3 Kevin´ Slytink, tres grand timide et romantique` pronant la paix dans le monde. C’est un jeuneˆ

Documents
Soutenance de stage - e-learning & université

Soutenance de stage - e-learning & université

Education
BIENVENUE A LA CEREMONIE DE SOUTENANCE DE THESE DE MONSIEUR

BIENVENUE A LA CEREMONIE DE SOUTENANCE DE THESE DE MONSIEUR

Documents
Soutenance de stage ouvrier

Soutenance de stage ouvrier

Engineering
soutenance test

soutenance test

Documents
Soutenance Orale de Quatrième Année 24 Septembre 2010

Soutenance Orale de Quatrième Année 24 Septembre 2010

Documents
Soutenance IGH

Soutenance IGH

Documents
soutenance hicham.ppt

soutenance hicham.ppt

Documents
Rapport de Soutenance Finale - Freeeyeliner.free.fr/docs/s3.pdf · 2009-06-07 · EyeLiner : Rapport de Soutenance Finale NightWare Rapport de soutenance remis le 27 mai 2009 Marquegnies

Rapport de Soutenance Finale - Freeeyeliner.free.fr/docs/s3.pdf · 2009-06-07 · EyeLiner : Rapport de Soutenance Finale NightWare Rapport de soutenance remis le 27 mai 2009 Marquegnies

Documents
Soutenance de mémoire de Master of Science

Soutenance de mémoire de Master of Science

Education
Soutenance de thèse - ENSEAjerofell/pdf/thesis_pres_jfellus.pdfJérôme Fellus Encadrant : David Picard Directeur de thèse: Philippe-Henri Gosselin Mardi 3 octobre 2017 Soutenance

Soutenance de thèse - ENSEAjerofell/pdf/thesis_pres_jfellus.pdfJérôme Fellus Encadrant : David Picard Directeur de thèse: Philippe-Henri Gosselin Mardi 3 octobre 2017 Soutenance

Documents
Soutenance han2014v3

Soutenance han2014v3

Education
Soutenance de la thèse professionnelle - Inbound Marketing

Soutenance de la thèse professionnelle - Inbound Marketing

Education
SOUTENANCE DE STAGE Jeudi 06/02/03

SOUTENANCE DE STAGE Jeudi 06/02/03

Documents
SOUTENANCE - THESE DE DOCTORAT

SOUTENANCE - THESE DE DOCTORAT

Documents
GUIDE DE SOUTENANCE - Root - ENSCMGUIDE DE SOUTENANCE DU DOCTORANT DE L’ENSCM Version 08/10/2019 5 Il devra compléter, signer et renvoyer la procuration au Directeur de thèse qui

GUIDE DE SOUTENANCE - Root - ENSCMGUIDE DE SOUTENANCE DU DOCTORANT DE L’ENSCM Version 08/10/2019 5 Il devra compléter, signer et renvoyer la procuration au Directeur de thèse qui

Documents
Soutenance de stage V2

Soutenance de stage V2

Documents
SOUTENANCE RAPPORT DE STAGE AVRIL 2013.xlsx_1.pdf

SOUTENANCE RAPPORT DE STAGE AVRIL 2013.xlsx_1.pdf

Documents
Soutenance de stage - assistante chargée de communication

Soutenance de stage - assistante chargée de communication

Presentations & Public Speaking
Soutenance de stage de fin détudes Sophie LEMAITRE 18 octobre 2002

Soutenance de stage de fin détudes Sophie LEMAITRE 18 octobre 2002

Documents
Geoffroy de La Rochebrochard - Soutenance de mémoire - La Communication Marketing Intégrée 2012

Geoffroy de La Rochebrochard - Soutenance de mémoire - La Communication Marketing Intégrée 2012

Education
Soutenance de mémoire

Soutenance de mémoire

Data & Analytics
Paleo-Future support de soutenance

Paleo-Future support de soutenance

Documents