Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics

Improving Speech Recognitionwith Embodied Cognition

and Behaviour-based Robotics

Improving Speech Recognitionwith Embodied Cognition

and Behaviour-based Robotics

Jorge Davila-Chacon

University of Hamburg - Knowledge Technology

www.informatik.uni-hamburg.de/WTM/

Spotify ML Meetup – November 3rd 2014

MotivationMotivation

Jorge Davila-Chacon Bio-Inspired SSL for Robot ASR 2

• Why is bio-inspired SSL interesting / useful?

Neurobotic ExperimentsNeurobotic Experiments

Virtual Reality LabVirtual Reality Lab

Bauer, J., Davila-Chacon, J., Strahl, E., Wermter, S. Smoke and Mirrors — Virtual Realities for Sensor Fusion Experiments in Biomimetic Robotics. In: Multisensor Fusion and Integration for Intelligent Systems, 2012

Neurobotic ExperimentsNeurobotic Experiments

Jorge Davila-Chacon 5Bio-Inspired SSL for Robot ASR

Bio-Inspired Sound Source LocalisationBio-Inspired Sound Source Localisation

ITDs fromLow Frequencies

ILDs fromHigh Frequencies

Spatial cues allow sound source localisation:

• Interaural Time Difference (ITD)• Interaural Level Difference (ILD)

Spatial cues allow sound source localisation:

• Interaural Time Difference (ITD)• Interaural Level Difference (ILD)

Same frequency component

ITDs extracted in Medial Superior Olive (MSO)

• AVCN - Anterior Ventral Cochlear Nucleus

• AN - Auditory Nerve

• IC – Inferior Colliculus

Interaural Time DifferencesNeuroanatomy

Interaural Time DifferencesComputational Principle

ILDs extracted in Lateral Superior Olive (LSO)

• MNTB - Medial Nucleus of the Trapezoid Body

• IC – Inferior Colliculus

Interaural Level DifferencesNeuroanatomy

Output ofMSO and LSO integrated in

J. Dávila-Chacón, S. Heinrich, J. Liu, S. Wermter. Biomimetic Binaural Sound Source Localisation with Ego-Noise Cancellation. International Conference on Artificial Neural Networks, 2012.

J. Dávila-Chacón, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues for Sound Source Localisation. International Joint Conference on Neural Networks, 2013.

Simple IC outputSimple IC output

Complex IC outputComplex IC output

Static SSLStatic SSL

Dynamic SSL

Feed forwardneural network

Robotic Automatic Speech RecognitionRobotic Automatic Speech Recognition

Platforms used for ASR: iCub and Soundman

J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition with Sound Source Localisation. International Conference on Artificial Neural Networks, 2014.

Binary measure - Static ASRBinary measure - Static ASR

Continuous measure - Static ASR

● Robotics as a “sandbox” for learning ML

● Neuroscience provides clues for computational principles

● Embodiment• iCub allows computation of spatial cues

• Interaction with environment can reduce noise

● Signal processing with ANN• Spiking ANN are an effective representation of spatial cues

• Bayesian integration important for dimensionality reduction

• Softmax Neural layer robust to ego-noise and reverberation

ConclusionConclusion

Future WorkFuture Work

● Neural SSL• Integrate GPU version of MSO and LSO

• Propagation of probabilities through time

• From discrete to continuous

● Integration with vision• From supervised to unsupervised SSL

• Possible extension to sensorimotor contingencies• Vision to select between multiple sound sources

• Vision for speech segregation

Thank you for your attention.

jorgedch@gmail.com

LinkedIn: Jorge Davila Chacon

• J. Liu, D. Perez-Gonzalez, A. Rees, H. Erwin, S. Wermter. A biologically inspired spiking neural network model of the auditory midbrain for sound source localisation. Neurocomputing (2010)

• J. Davila-Chacon, S. Heinrich, J. Liu, and S. Wermter. Biomimetic binaural sound source localisation with ego-noise cancellation. International Conference on Artificial Neural Networks (2012)

• J. Bauer, J. Davila-Chacon, E. Strahl, S. Wermter. Smoke and Mirrors — Virtual Realities for Sensor Fusion Experiments in Biomimetic Robotics. Multisensor Fusion and Integration for Intelligent Systems (2012)

• J. Davila-Chacon, S. Magg, J. Liu, S. Wermter. Neural and Statistical Processing of Spatial Cues for Sound Source Localisation. International Joint Conference on Neural Networks (2013)

• J. Dávila-Chacón, J. Twiefel, J. Liu, S. Wermter. Improving Humanoid Robot Speech Recognition with Sound Source Localisation. International Conference on Artificial Neural Networks (2014)

AppendixAppendix

Best performances with clustering layer

AppendixAppendix

Best performances with clustering layer

AppendixAppendix

Bayesian IC model

AppendixAppendix

Bayesian IC model

AppendixAppendix

Levenshtein distance

Improving Speech Recognition with Embodied Cognition and Behaviour-based Robotics

Documents

Embodied Perspective on Emotion-Cognition Interactionspineda/COGS171/presentations... · 2011. 11. 16. · Embodied Cognition Theories: • High-level cognitive processes are modal

Embodied Thought — Recent Robotics (some highlights)

Embodied Cognition and Mirror Neurons: A Critical …caram/PDFs/2014_Caramazza_Anzellotti_Strnad... · 2Center for Mind/Brain Sciences, ... ential embodied theory of cognition:

Dijk 2013 embodied cognition lecture 3 small

EMBODIED COGNITION - Technische Universität Mü · PDF fileBrain, Mind and Cognition Essay EMBODIED COGNITION by Lawrence Shapiro When I looked at the beginning of the table of contents

Humana_Mente 15 Agency_From Embodied Cognition to Free Will

Embodied Cognition in Robots and Human Evolution

Embodied music cognition - ERA Home

Situated Embodied Cognition - PersCom

Material-Based Imagination: Embodied Cognition in Animated ...groups.csail.mit.edu/icelab/sites/default/files/... · Material-Based Imagination: Embodied Cognition in Animated Images

An Embodied Cognition Approach for Understanding Role … · An Embodied Cognition Approach for Understanding Role-playing ... refers to immersive story, whereas Harviainen (2003)

Different Methods of Embodied Cognition in Pedagogy and

Cognition, Mental Representations and Embodied Emotions

Implications of Embodied Cognition & Enactivism for Instructional Design

Cognitive Robotics, Embodied Cognition and Human-Robot ... · Cognitive Robotics, Embodied Cognition and Human-Robot Interaction 5a. CONTRACT NUMBER 5b. GRANT NUMBER 5c. PROGRAM ELEMENT

Embodied Cognition Shapiro

Embodied Cognition 2013

Applying embodied cognition approaches to L2 learning

Metaphor; Creativity; Embodied Cognition; Brain and Mind; UnconsciousSeitz06

Embodied music cognition and mediation technology