Gaussian Process Regression for Dummies

Greg CoxRichard Shiffrin

Continuous response measures

The problem

What do we do if we do not know the functional form?

Rasmussen & Williams, Gaussian Processes for Machine Learninghttp://www.gaussianprocesses.org/

Linear regression

baxy

bxy

Bayesian linear regression

Gaussian processes

A Gaussian process is a collection of random variables, any subset of which is jointly normally distributed.

Normal regression:assume functional form mean and covariance among data

Gaussian process regression:assume form of mean and covariance among data functional form

*** ,~| yNyybaxy

*** ,~|,~ yy NyyKNy

Covariance kernel

How much does knowledge of one point tell us about another point?

Returning to linear regression

Mean = Function of parametersCovariance = Uncertainty about parameters + Observation noise

Takeaways from linear regression

• Rather than work in “parameter space”, we can bypass it by just working in “data space”

• This allows us to worry only about how different data points relate to one another without needing to specify the parameters of the data generating process

• The posterior predictive distribution encapsulates our uncertainty about the data generating process

• The choice of covariance kernel—which says how different observations inform one another—implies certain properties of the data generating process

Posterior predictive distribution

So far, we have computed the posterior predictive via the parameters (e.g., b) of the data generating process. But, a Gaussian process may have an infinite number of parameters (q). How can we compute the posterior predictive in this case?

The covariance kernel to the rescue! Let’s say we don’t know the data generating process, but we assume all observations are drawn from the same Gaussian process (i.e., are multivariate normal) and have an idea about how observations can mutually inform one another, the covariance kernel k(x, x’). Then...

New data values f*(x*), given observed data f(x):

But these are all multivariate normal!

dpp xfxxfxfxxxf *** ,||,,

),(),(),(),(,),(),(~|

),(),(

),(),(,~

*1***1*

***

*

XXKXXKXXKXXKXXKXXKN

XXKXXK

XXKXXKN

xfxfxf

0xf

xf

*

*

Building a function

A hierarchical Bayesian approach

Spivey, Grosjean, & Knoblich, 2005

The GP model

Model structure

The GP model

Results

Results

Inflection points can indicate important changes in cognitive processing

Summary

• Gaussian process models offer a useful and extensible way of dealing with behavioral trajectories

• Able to model entire spectrum of dynamics

• Can be embedded in a generative model to infer attractors and inflection points

• Allow for deeper inferences about underlying cognitive processes