• Home

  • Documents

  • New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and...
94
Inference and learning for sensory data – New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning Cluster of Excellence Hearing4all, Dept for Medical Physics and Acoustics, Carl-von-Ossietzky University Oldenburg, Germany Oldenburg, March 6, 2015

New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Embed Size (px)

Citation preview

Page 1: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Inference and learning for sensory data–

New priors, non-linear features, and the challenge of masking and occlusion

Jörg LückeMachine Learning

Cluster of Excellence Hearing4all, Dept for Medical Physics and Acoustics, Carl-von-Ossietzky University Oldenburg, Germany

Oldenburg, March 6, 2015

Page 2: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sensory Inference

... ......

Page 3: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sensory Inference

unconscious inferenceHelmholtz, HB physiol. Optik, 1867

......

Page 4: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sensory Inference: Example

Page 5: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sensory Inference: Example

Inference Task:

Which piano keyswere pressed?

Build an artificial system that solves the task.

Page 6: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

We re-express data point :

Sensory Inference: Example

Page 7: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

For piano data, we would choose:

etc.

We re-express data point :

Sensory Inference: Example

Page 8: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

etc.

For piano data, we would choose:

We re-express data point :

Sensory Inference: Example

Page 9: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

etc.

For piano data, we would choose:Real data causes/components:

Estimates of causes/components:

We re-express data point :

Sensory Inference: Example

Page 10: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

etc.

For piano data, we would choose:

Estimates of causes/components:

Real data causes/components:

We re-express data point :

Sensory Inference: Example

Page 11: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

etc.

For piano data, we would choose:

Estimates of causes/components:

Real data causes/components:

We re-express data point :

Sensory Inference: Example

Page 12: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

etc.

For piano data, we would choose:

Estimates of causes/components:

Real data causes/components:

We re-express data point :

Sensory Inference: Example

Page 13: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

We re-express data point :

etc.

For piano data, we would choose:

Estimates of causes/components:

Sensory Inference: Example

Page 14: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

We re-express data point :

For piano data, we would choose:

Estimates of causes/components:

Probabilistic generative model (e.g., SC):

... dictionary

Sensory Inference: Example

Page 15: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

We re-express data point :

For piano data, we would choose:

Estimates of causes/components:

Probabilistic generative model (e.g., SC):

... dictionary

Sensory Inference: Example

Page 16: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

We re-express data point :

For a keyboard instrument:

Estimates of causes/components:

Probabilistic generative model (e.g., SC):

... Solution: learn dictionary from data

?

Sensory Inference: Example

Page 17: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Non-linear components

generatingsystem

combination of causes model assumptions for dictionary

street scene

Page 18: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Non-linear components

model assumptions forChange model parameters until:

real data model data ~~Measure for this similarity:

Data Likelihood

dictionary

Page 19: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Non-linear components

model assumptions for

Dayan & Zemel, 1996;Lücke & Sahani, 2007, 2008;Lücke, 2009; Lücke et al. etc.

Change model parameters until:

real data model data ~~Measure for this similarity:

Data Likelihood

E-step

M-step

E.g., Expectation Maximization (EM) framework.Dempster, 1977; Neal & Hinton, 1998.

dictionary

Page 20: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

(00110)

Page 21: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

(00110)

Page 22: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

(00110)

(00110)

Gaussian prior

PCA / Factor Analysis(vgl. principal axis transform)

Sparse prior

Sparse coding / (ICA)

Olshausen & Field '96… Sheikh et al., '14 ...

Page 23: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

Page 24: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

Olshausen & Field, Nature 1996

Lee, Battle, Raina, Ng, NIPS 2006;Bornschein, Henniges, Lücke, PLOS Comp Biology 2013

Page 25: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

...

Page 26: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

linearity assumption not realistic

...

Page 27: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Dictionary Examples

...

Page 28: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

...

Dictionary Examples

PCA – standard approach, Pearson, 1901FA – standard approach, e.g., Gorsuch, '83ICA – has own conference, Comon, 1994SC – Olshausen & Field, Nature, 1996NMF – Lee & Seung, Nature, 1999etc.

Page 29: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

...

Dictionary Examples

crucial forcompressed sensing

Page 30: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

...

Dictionary Examples

Dai et al., NIPS 2013Bornschein et al., PLOS CB 2013Shelton et al., NIPS 2012Puertas, Bronschein, Lücke, NIPS 2010Lücke, Sahani, J Mach Learn Res 2008...Roweis, Eurospeech 2003Roweis, NIPS 2002Varga & Moore, ICASSP 1990

Page 31: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Cochleagram Dictionaries

...

Roweis, Eurospeech 2003Roweis, NIPS 2002Varga, Moore, ICASSP 1990

Page 32: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Computational Challenges

Page 33: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Henniges et al., 2010

Example: Binary Prior

Maximal Causes Analysis (MCA):

Page 34: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Henniges et al., 2010

Example: Binary Prior

Optimization using Expectation Maximization (EM)

Page 35: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Henniges et al., 2010

Example: Binary Prior

Optimization using Expectation Maximization (EM)

Expectation values scale exponentially.

Page 36: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Henniges et al., 2010

Example: Binary PriorExpectation values scale exponentially.

Idea: Truncate the sums

Page 37: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Example: Binary PriorExpectation values scale exponentially.

Such a truncation of sums is equivalent to a variational approximation:

variational distribution (not factored)

Page 38: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Binary Sparse Coding (BSC):

Henniges et al., 2010

Example: Binary PriorExpectation values scale exponentially.

Idea: Truncate the sums

Expectation Truncation:

Page 39: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

exact:

truncated:

MAP:

Laplace:

mean-field:

Relation to Other Approximations

Page 40: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

ET

latent space

data point

Page 41: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

ET

preselect subset

evaluate all states in

latent space

data point

Page 42: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

optimal caseET

preselect subset

evaluate all states in

Page 43: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

optimal caseETdeterministic

Page 44: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

Exact:

MAP:

Page 45: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation TruncationLücke, Eggert, JMLR 2010

Exact:

ET:

MAP:

Page 46: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation Truncation (ET)

preselect subset evaluate states in

discriminative approaches generative approaches

Page 47: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Expectation Truncation (ET)

preselect subset evaluate states in

discriminative approaches generative approaches

Lücke, Eggert, JMLR 2010Henniges et al., LVA 2010Puertas et al., NIPS 2010Shelton et al., NIPS 2011Exarchakis et al., LVA 2012Dai, Lücke, CVPR 2012aDai, Lücke, CVPR 2012bShelton et al., NIPS 2012Bornschein et al., PLOS CB 2013Sheikh et al., JMLR 2014Henniges et al., JMLR 2014

Page 48: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

multiple modescorrelations

Expectation Truncation (ET)Lücke, Eggert, JMLR 2010

example papersvariational approximation

max a-posteriori (MAP)

Gaussian

mean-field

Olshausen, Field, 1996;A. Ng et al.

Opper et al.; Seeger, 2008

Titsias et al., 2011;Goodfellow … Bengio, 2012;

no

yes

no

no

no

yes

Page 49: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

multiple modescorrelations

Expectation Truncation (ET)Lücke, Eggert, JMLR 2010

example papersvariational approximation

max a-posteriori (MAP)

Gaussian

mean-field

expectation truncation

Olshausen, Field, 1996;A. Ng et al.

Opper et al.; Seeger, 2008

Titsias et al., 2011;Goodfellow … Bengio, 2012;

Sheikh, Shelton, Lücke, 2012;Puertas … '10; Dai&Lücke, '12;

no

yes

no

yes

no

no

yes

yes

Page 50: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

multiple modescorrelations

Expectation Truncation (ET)Lücke, Eggert, JMLR 2010

broad posterior distributionsvariational approximation

max a-posteriori (MAP)

Gaussian

mean-field

expectation truncation

no

yes

no

yes

no

no

yes

yes

no

yes

yes

no

Page 51: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

exact EM

Expectation TruncationLücke, Eggert,JMLR 2010;

Page 52: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

ET-EM

Lücke, Eggert,JMLR 2010;

Expectation Truncation

Expectation Truncation

ET parametrizes accuracy

Page 53: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

ET-EM

Lücke, Eggert,JMLR 2010;

Expectation Truncation

ET allows for optimizing prior parametersPuertas et al., NIPS 2010;Henniges et al., LVA/ICA 2010;Lücke, Eggert, JMLR 2010;

Shelton et al., NIPS 2011Dai & Lücke, CVPR 2012 a & bShelton et al., NIPS 2012

Page 54: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

- Problem: local likelihood optima

- Problem: E-step comput. intractable

- Problem: no closed-form M-steps

simulated annealing

general sol. for non-linear modelsDai, Lücke, TPAMI 2014Lücke et al., NIPS 2009Lücke, Sahani, JMLR 2008

Dai et al.,, NIPS 2013Shelton et al., NIPS 2012Shelton et al., NIPS 2011Puertas et al., NIPS 2010Lücke, Eggert, JMLR 2010Lücke et al., NIPS 2009 Lücke, Sahani, JMLR 2008

up to 4000 cores or GOLD (16 GPUs)

- multiple modes non-linear models- correlations advanced linear} {

Applicable to large-scale since 2010.

MCA generative model:

Page 55: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

The Bars Test, Földiák, 1990

Bars Test

Page 56: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

The Bars Test, Földiák, 1990

Lücke and Sahani, JMLR 2008Lücke and Eggert, JMLR 2010

Bars Test

Page 57: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Encoding of Visual Scenes

LGN

Page 58: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sparse Coding

Olshausen & FieldNature 1996

Nobel Prize in Physiology 1981 Hubel & Wiesel

Non-linear Sparse Coding

Puertas et al. 2010; Bornschein et. al., 2013

Page 59: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Sparse Coding Non-linear Sparse Coding

Puertas et al. 2010; Bornschein et. al., 2013

Page 60: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

non-linear model(MCA)

macaque

Bornschein, Henniges, Lücke, PLOS Comp Biology 2013

Page 61: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Encoding of Visual Scenes(“Viterbi” for sparse coding)

globular fields

max sum

Bornschein, Henniges, Lücke, PLOS Comp Biology 2013

Page 62: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Encoding of Acoustic Scenes

Apply non-linear model:

Page 63: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Encoding of Acoustic Scenes

Currently ongoing work (Univ. Oldenburg / UC Berkeley / Univ. Cambridge / TU Berlin)

generative fields (selection of H=1000 fields)

Page 64: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Encoding of Acoustic Scenes

STRFs learned by the model Ferret A1 recordings

Currently ongoing work (Univ. Oldenburg / UC Berkeley / Univ. Cambridge / TU Berlin)

Both estimated using (regularized) reverse correlation.

Page 65: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Example: Spike-and-Slab Sparse Coding (GSC)

Titsias, Lazaro-Gredilla, NIPS '11Sheikh, Lücke, LVA '12Goodfellow et al., ICML '12Shelton et al., NIPS '12… and more

Page 66: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Example: Gaussian Sparse Coding (GSC)

GSC is state-of-the-art in denoising.

… but denoising is just one tasks.

Lücke, Sheikh, LVA 2012;Sheikh, Shelton, Lücke, JMLR 2014.

Page 67: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Maximal Causes Analysis (MCA)is state-of-the-art

Source:Hoyer, J Mach Learning Res, 2004Spratling, J Mach Learning Res, 2006Lücke/Sahani, J Mach Learning Res, 2008

Lücke and Sahani, JMLR, 2008

MaximalCauses

Page 68: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Selection of Statistical Models

noisy-OR

occlusion

exclusion

mixtures

linear SC

GraphicalModel

Mask Feature

Z. Dai

J. Bornschein

C. Keck,S. Sheikh,C. Savin

M. Henniges

Lücke et al.,NIPS 2009

A.-S. Sheikh,M. HennigesJ. Shelton

Page 69: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Example App: Inpainting

80% lost pixelsOriginal image reconstruction

Shelton et al.,NIPS 2012

spike-and-slab prior

Page 70: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Example: Structured Noise Removal

Page 71: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

More characters (9 chars)Unusual character set (Klingon)Irregular placement (randomly placed, rotated)Occluded by spilled ink

9chars KlingonRotated,

random placed Occluded

ori

gin

al

Example: Structured Noise Removal

Page 72: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

More characters (9 chars)Unusual character set (Klingon)Irregular placement (randomly placed, rotated)Occluded by spilled ink

9chars KlingonRotated,

random placed Occluded

ori

gin

al

Example: Structured Noise Removal

Dai & Lücke, CVPR 2012, oral presentation, Google award.Dai & Lücke, IEEE Trans. on Pattern Analysis and Machine Intelligence, 2014.

Page 73: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Thanks to:

Non-linear Probabilistic Models for RepresentationalRecognition and Unsupervised Learning in VisionDeutsche Forschungsgemeinschaft

Bernstein Focus Neurotechnology Frankfurt Bundesministerium für Bildung und Forschung

LOROCHonda Research Institute Europe

Cluster of Excellence Hearing4allDeutsche Forschungsgemeinschaft

Page 74: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Thanks to:

Project ResearchersUniv. Oldenburg, TU Berlin, Univ. Frankfurt

Christian Keck

Jörg Bornschein

Marc Henniges

Dennis Forster

Abdul Saboor Sheikh

Jacquelyn Shelton

Jörg Lücke

Zhenwen Dai

Georgios Exarchakis

Page 75: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Thanks to:

Maneesh SahaniGatsby Unit, UCL, UK

Pietro BerkesEnthought, Ltd., UKBrandeis Univ., USA

Julian EggertHonda-RI, Europe

Richard TurnerCristina Savin (now IST Austria)Cambridge University, UK

Bruno OlshausenNicol HarperBerkeley University, US

Jörg LückeUniversity of Oldenburg

Marc-Thilo FiggeUniversität Jena

Roland MemisevicJörg BornscheinUniversity of Montreal, CA

Page 76: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Thank you.

Page 77: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Use Natural Image Patches as Input

Cortical Circuits

Page 78: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Cortical Circuits

Numerical simulations of stochastic andnon-linear differential equations.

Page 79: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Cortical Circuits

Lücke, Neural Computation, 2009.

Page 80: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Application NMF

The model can be constraint by allowing only for positive and positive .

We obtain a generative version of NMF (with binary latents).

Application to MNIST data.

Page 81: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

sum

Acoustic Data

Page 82: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

sum

Acoustic Data

Page 83: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

sum

max

Acoustic Data

Page 84: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

sum

max

log-maxapproximation

Acoustic Data

Page 85: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Max of the two individuallog-spectrograms.

Log-max approximation (Moore, 1983)

Log-spectrogramof a mixture oftwo sound sources.

Source: Roweis, 2004

Acoustic Data

Page 86: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Acoustic Data

Page 87: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Acoustic Data

Bornschein et al., PLOS CB 2013Shelton et al., NIPS 2012Puertas, Bronschein, Lücke, NIPS 2010Lücke, Sahani, J Mach Learn Res 2008...Roweis, Eurospeech 2003Roweis, NIPS 2002Varga, Moore, ICASSP 1990

Page 88: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Acoustic Data

The Bars Test.Földiák, 1990

hand-written digits(e.g., MNIST) Natural image patches

10s-100s hidden10s hidden 100s-1000s hidden

Page 89: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

MicroscopyImage Analysis

Deep Learning Architecturesfor Pattern RecognitionKeck, Savin, LückePLOS Comp Bio, 2012

feed-forward

weights

Other projects

Page 90: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Deep Learning Architecturesfor Pattern RecognitionKeck, Savin, Lücke PLOS Comp Bio, 2012

weights

Deep Learning Architecturesfor Pattern RecognitionLücke, Neural Comp 2009Lücke, ICANN 2005-2007Lücke, Neural Networks 2004Lücke, Malsburg, Neural Comp 2004...

Other projects

Page 91: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

NeurosciencePhysics

signal processing;computer hearing

computationalneuroscience

probability theory / applied mathematics / computer science

generative modelslearning

approximate inference

Page 92: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

probability theory / applied mathematics / computer science

signal processing;computer hearing

generative models

computationalneuroscience

learningapproximate inference

artificial intelligent systems

new applications

Page 93: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

probability theory / applied mathematics / computer science

signal processing;computer hearing

generative models

computationalneuroscience

learningapproximate inference

Hearing4allarea A

IndividualisierteHörakustik

algorithms for hearing instruments

HörTech

Hearing4allarea B

personalizinghearing devices

Zentrum fürNeurosensorik

Page 94: New priors, non-linear features, and the challenge of ... · New priors, non-linear features, and the challenge of masking and occlusion Jörg Lücke Machine Learning ... Apply non-linear

Jörg Lücke

Obtain basisfunctions:

The Bars Test, Földiák, 1990

Source:Hochreiter &Schmidhuber,Neural Comp,1999

Linear CausesLinear generative models:

...


Non Linear Systems

Non Linear Systems

Documents
A supervised non-linear dimensionality reduction approach ...ccc.inaoep.mx/~ariel/2013/A supervised non-linear dimensionality... · A supervised non-linear dimensionality reduction

A supervised non-linear dimensionality reduction approach ...ccc.inaoep.mx/~ariel/2013/A supervised non-linear dimensionality... · A supervised non-linear dimensionality reduction

Documents
Simultaneous Linear, and Non-linear Congruences · Linear CongruencesSimultaneous Linear CongruencesSimultaneous Non-linear CongruencesChinese Remainder Theorem - An Extension Theorem

Simultaneous Linear, and Non-linear Congruences · Linear CongruencesSimultaneous Linear CongruencesSimultaneous Non-linear CongruencesChinese Remainder Theorem - An Extension Theorem

Documents
Comparative Analysis of Linear and Non-Linear Programming ... · linear and non-linear optimization routines. The results of the linear and the non-linear approaches are compared,

Comparative Analysis of Linear and Non-Linear Programming ... · linear and non-linear optimization routines. The results of the linear and the non-linear approaches are compared,

Documents
Non Linear Vibrations

Non Linear Vibrations

Documents
Non linear powerpoint

Non linear powerpoint

Entertainment & Humor
Chapter 4 Non-Linear Oscillations and Chaos Non-Linear … · 2019. 9. 13. · Non-Linear Oscillations and Chaos Non-Linear Differential Equations Up to now we have considered differential

Chapter 4 Non-Linear Oscillations and Chaos Non-Linear … · 2019. 9. 13. · Non-Linear Oscillations and Chaos Non-Linear Differential Equations Up to now we have considered differential

Documents
Regresi Linear dan Non Linear

Regresi Linear dan Non Linear

Documents
Linear & Non-Linear Planning

Linear & Non-Linear Planning

Documents
Non Linear

Non Linear

Documents
Linear and non linear equation

Linear and non linear equation

Technology
Non-Linear Regression - Penn State College of Engineeringrtc12/CSE586/lectures/regrNonlinear... · 2013-04-01 · Non-linear regression Linear regression: Non-Linear regression: where

Non-Linear Regression - Penn State College of Engineeringrtc12/CSE586/lectures/regrNonlinear... · 2013-04-01 · Non-linear regression Linear regression: Non-Linear regression: where

Documents
Leveraging Appearance Priors in Non-Rigid Registration, with ...people.eecs.berkeley.edu/.../2015-IROS-TPS-RPM-w-appearance-prior… · Leveraging Appearance Priors in Non-Rigid Registration,

Leveraging Appearance Priors in Non-Rigid Registration, with ...people.eecs.berkeley.edu/.../2015-IROS-TPS-RPM-w-appearance-prior… · Leveraging Appearance Priors in Non-Rigid Registration,

Documents
CONVERTING NON-LINEAR GRAPHS INTO LINEAR …emp.byui.edu/CullenJ/Chem 106/graphing non-linear functions.pdfCONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS Linear graphs have several

CONVERTING NON-LINEAR GRAPHS INTO LINEAR …emp.byui.edu/CullenJ/Chem 106/graphing non-linear functions.pdfCONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS Linear graphs have several

Documents
Hyper-g Priors for GLMs - arXiv · Hyper-g Priors for Generalized Linear Models Daniel Sabanés Bové Leonhard Heldy Version: ... polynomial modelling in Section 5. Section 6 discusses

Hyper-g Priors for GLMs - arXiv · Hyper-g Priors for Generalized Linear Models Daniel Sabanés Bové Leonhard Heldy Version: ... polynomial modelling in Section 5. Section 6 discusses

Documents
Linear and Non-Linear Data Structures and Non-Linear Data Structures ... E.g. – array, linked list. Non-Linear data structure: ... Array- Representation of a stack:

Linear and Non-Linear Data Structures and Non-Linear Data Structures ... E.g. – array, linked list. Non-Linear data structure: ... Array- Representation of a stack:

Documents
Data Dependent Priors for Stable Learning - … · Background PAC-Bayes Analysis Linear Classifiers Data distribution dependent priors Data Dependent Priors for Stable Learning John

Data Dependent Priors for Stable Learning - … · Background PAC-Bayes Analysis Linear Classifiers Data distribution dependent priors Data Dependent Priors for Stable Learning John

Documents
Linear and Non-Linear ICA-BSS

Linear and Non-Linear ICA-BSS

Documents
Energy deposition ElectronsLattice Perturbation Strong non -linear regime Small linear regime Frenkel-pair creation linear cascades Non-linear cascades

Energy deposition ElectronsLattice Perturbation Strong non -linear regime Small linear regime Frenkel-pair creation linear cascades Non-linear cascades

Documents
CONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS 106/graphing non... · CONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS Linear graphs have several important attributes. First, it is

CONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS 106/graphing non... · CONVERTING NON-LINEAR GRAPHS INTO LINEAR GRAPHS Linear graphs have several important attributes. First, it is

Documents
Non-Adversarial Video Synthesis with Learned Priors ...openaccess.thecvf.com/content_CVPR_2020/supplemental/...Non-Adversarial Video Synthesis with Learned Priors (Supplementary Material)

Non-Adversarial Video Synthesis with Learned Priors ...openaccess.thecvf.com/content_CVPR_2020/supplemental/...Non-Adversarial Video Synthesis with Learned Priors (Supplementary Material)

Documents
Non Linear 2

Non Linear 2

Documents
Concepts linear vs Non-linear Regression

Concepts linear vs Non-linear Regression

Documents
Rank Priors for Continuous Non-Linear Dimensionality Reduction · 3 Continuous Dimensionality Reduction via Rank Priors We introducea novel method for probabilistic non-linearma nifold

Rank Priors for Continuous Non-Linear Dimensionality Reduction · 3 Continuous Dimensionality Reduction via Rank Priors We introducea novel method for probabilistic non-linearma nifold

Documents
Linear non linear

Linear non linear

Design
mpmouzzonwebpage.weebly.commpmouzzonwebpage.weebly.com/uploads/5/8/7/1/... · Non-Linear Examples Non-Linear Graphs Non-Linear Tables Non-Linear Sequences 2, 4, 8, 16, 32, . 1, 4,

mpmouzzonwebpage.weebly.commpmouzzonwebpage.weebly.com/uploads/5/8/7/1/... · Non-Linear Examples Non-Linear Graphs Non-Linear Tables Non-Linear Sequences 2, 4, 8, 16, 32, . 1, 4,

Documents
Non-linear Paper

Non-linear Paper

Documents
NON-LINEAR SYSTEMS, - WSEAS · NON-LINEAR SYSTEMS, NANOTECHNOLOGY . Proceedings of the 14th International Conference on Non -Linear Analysis, Non-Linear Systems and Chaos (NOLASC

NON-LINEAR SYSTEMS, - WSEAS · NON-LINEAR SYSTEMS, NANOTECHNOLOGY . Proceedings of the 14th International Conference on Non -Linear Analysis, Non-Linear Systems and Chaos (NOLASC

Documents
Non linear kinetics

Non linear kinetics

Education
Linear Non Linear FEM

Linear Non Linear FEM

Documents