IEEE TRANSACTIONS ON PATTERN ANALYSIS AND …memisevr/pubs/AEenergy.pdf · Hanna Kamyshanska, Roland Memisevic Abstract—Autoencoders are popular feature learning models, that are

0162-8828 (c) 2013 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. Seehttp://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citationinformation: DOI 10.1109/TPAMI.2014.2362140, IEEE Transactions on Pattern Analysis and Machine Intelligence

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE (TPAMI), VOL. ??, NO. ??, JANUARY 2014 1

The Potential Energy of an AutoencoderHanna Kamyshanska, Roland Memisevic

Abstract—Autoencoders are popular feature learning models, that are conceptually simple, easy to train and allow for efficientinference and training. Recent work has shown how certain autoencoders can be associated with an energy landscape, akin tonegative log-probability in a probabilistic model, which measures how well the autoencoder can represent regions in the inputspace. The energy landscape has been commonly inferred heuristically, by using a training criterion that relates the autoencoderto a probabilistic model such as a Restricted Boltzmann Machine (RBM). In this paper we show how most common autoencodersare naturally associated with an energy function, independent of the training procedure, and that the energy landscape can beinferred analytically by integrating the reconstruction function of the autoencoder. For autoencoders with sigmoid hidden units,the energy function is identical to the free energy of an RBM, which helps shed light onto the relationship between these twotypes of model. We also show that the autoencoder energy function allows us to explain common regularization procedures, suchas contractive training, from the perspective of dynamical systems. As a practical application of the energy function, a generativeclassifier based on class-specific autoencoders is presented.

Index Terms—Autoencoders, representation learning, unsupervised learning, generative classification

F

1 INTRODUCTION

THE difficulty of many classification tasks dependsstrongly on the representation of the data, more

specifically on the choice of features used to representthe inputs. Representation learning, or feature learn-ing, tries to simplify classification tasks by learninga good representation automatically from the inputdata (eg., [1]). Many feature learning methods, like theRestricted Boltzmann Machine (RBM), factor analysis,and many others, are probabilistic models, and train-ing them amounts to maximum likelihood estimationor an approximation thereof.

Since many common models, including the RBM,define probabilities only up to an intractable nor-malizing constant, approximate maximum likelihoodlearning typically amounts to optimizing the modelonly locally, near the training data ([2]). It can bemotivated by the observation that many applicationsrely on the learned representations (in the form ofhidden unit activations) rather than the probabilitythat the model assigns to the data.

The idea of learning a “landscape” of unnormal-ized log-probability has been generalized to includearbitrary real-valued functions defined over the data-space, which can be trained to yield small values nearthe data and large values far away from the data ([3]).These models are commonly referred to as “energy-based models” in the literature ([3]).

Some of the highly successful examples of non-probabilistic feature learning models are autoencoder

• H. Kamyshanska is with Frankfurt Institute for Advanced Studies,Ruth-Moufang-Str. 1, 60438, Frankfurt am Main, Germany.E-mail: [email protected]

• R. Memisevic is with University of Montreal, CP 6128, succ Centre-Ville, Montreal H3C 3J7, Canada.E-mail: [email protected]

networks. Autoencoders have been shown to yieldstate-of-the-art performance in a variety of tasks rang-ing from object recognition and learning invariantrepresentations to syntactic modeling of text [4], [5],[6], [7], [8], [9], [10]. Learning amounts to minimizingreconstruction error using back-prop.

Autoencoders have been known to be similar inspirit to RBMs in that they learn a good model nearthe training data only. In fact, if noise is added tothe input data during training, minimizing squarederror in an autoencoder with sigmoid activations isrelated to performing score matching [7], [8], a commonapproximation to maximum likelihood learning inthe RBM [11]. Score matching itself can be shownto be related to contractive divergence training ofthe RBM [12]. The relation between RBMs and au-toencoders can be extended to other training crite-ria. For example, [13] suggest training autoencodersusing a “contraction penalty” that encourages latentrepresentations to be locally flat, and [14] show thatsuch a regularization penalty applied to real-valuedobservations allows us to interpret the autoencoderreconstruction function as an estimate of the gradientof the data log probability.

However, beyond the special cases of sigmoid hid-den units, or special training criteria that relate au-toencoders to RBMs, it has not been clear in general,whether all autoencoders come with an associatedenergy function, and if so what the form of this energyfunction is. The restriction of activation function isin particular at odds with the growing interest inunconventional activation functions, like quadratic orrectified linear units, which were shown to workmuch better than sigmoid units in many tasks.




1.1 Autoencoder as dynamical system

In this work, we show how an energy function maybe derived for the autoencoder by interpreting it asa dynamical system. The view of the autoencoder asa dynamical system was proposed originally by [15],who also demonstrated how de-noising as a learningcriterion follows naturally from this perspective.

The dynamical systems perspective allows us toobtain an analytical expression for the energy functionof the autoencoder by integration [16]. This makes itpossible to assign energies to autoencoders with manydifferent types of activation functions and outputs. Weshow that the energy function exists irrespective ofwhether (or how) the model was trained and thatits form is not related in any way to any trainingprocedure.

We also show that for autoencoders with sigmoidhidden units and binary or real-valued observations,the energy function is identical to the free energy ofthe corresponding RBM, showing that autoencodersand RBMs may be viewed as two different ways toderive training criteria for forming the same type ofanalytically defined energy landscape.

An application of the autoencoder energy surfaceis generative classification, using autoencoders as theclass-specific models. We show how such a classifieryields competitive performance in a variety of tasksin Section 4.2.

2 AUTOENCODER POTENTIAL ENERGIES

Autoencoders are feed forward neural networks usedto learn representations. They map input data to ahidden representation

h(Wx+ bh

)(1)

using an encoder function h(.), from which the datais reconstructed using a linear decoder

r(x) = Wrech(Wx+ bh

)+ br (2)

For the reconstruction matrix Wrec we shall assumethat Wrec = WT in the following (“tied weights”).This is common in practice, because it reduces thenumber of parameters and because related probabilis-tic models, like the RBM, are based on tied weights,too.

For training, one typically minimizes the averagesquared reconstruction error for a set of training cases:

1

N

∑

i

‖r(xi)− xi‖22 (3)

When the number of hidden units is small, autoen-coders learn to perform dimensionality reduction. Inpractice, it is more common to learn sparse represen-tations by using a large number of hidden units andadding a regularization penalty, such as a “contractionterm” [13], to Eq. 3. Alternatively, it is common to

reformulate Eq. 3 to reconstruct data from corruptedinputs instead of from the original data during train-ing. The resulting model is known as the de-noisingautoencoder [17].

A wide variety of models can be learned, depend-ing on the activation function, number of hiddenunits and nature of the regularization during training.Autoencoders defined using Eq. 2 with tied weightsand logistic sigmoid non-linearity h(a) = σ(a) =(1 + exp(−a)

)−1 are closely related to RBMs [7], [8],[14].

For binary data, a sigmoid non-linearity is typicallyused also in the decoder:

r(x) = σ(Wrech

(Wx+ bh

)+ br

)(4)

which makes it possible to interpret the outputs asBernoulli probabilities. For training, the reconstruc-tion error (Eq. 3) is then usually replaced by a cross-entropy loss.

While binary output RBMs make it possible toassign an unnormalized log-probability to the databy integrating over binary hidden units, the analogof an energy function for the autoencoder has beenproposed only for real-valued observations. The rea-son is that the relationship with score matching breaksdown in the binary case [14]. As we shall show,the perspective of dynamical systems allows us toattribute the missing link to the lack of symmetry.We also show how we can regain symmetry andthereby obtain a confidence score for binary outputautoencoders by applying a log-odds transformationon the outputs of the autoencoder.

2.1 Reconstruction as energy minimizationSince the autoencoder maps an observation x ∈ Rnto a reconstruction r(x) ∈ Rn it naturally defines adynamical system [15]. The function r(x) − x is avector field that represents the local displacement thatx undergoes as a result of applying the autoencoderreconstruction r(x). Repeatedly applying the recon-struction function to an initial point x, possibly witha small “inference rate” ε, will trace out a non-lineartrajectory x(t) in the data-space.

If the number of hidden units is smaller than thenumber of data dimensions, then the set of fixedpoints of the dynamical system will be approximatelya low-dimensional manifold in the data-space [15].For overcomplete hiddens it can be a more complexstructure. [18], for example, show that for an autoen-coder that was trained with a specific denoising orcontraction criteria, the reconstruction function willbe approximately proportional to the derivative of thelog probability of x:

r(x)− x = λ∂ logP (x)

∂(x)+ o(λ), λ→ 0 (5)

Running the autoencoder by following the trajectoryprescribed by the vector field and starting point x(0)




may also be viewed in analogy to running a Gibbssampler in an RBM, where the fixed points play therole of a maximum probability “ridge” and where thesamples are deterministic not stochastic.

An important observation now is that some vectorfields are gradient fields, which means they can bewritten as the derivative of a scalar field [19]. In thatcase, running the dynamical system can be thoughtof as performing gradient descent in the scalar field.In analogy to physics, the scalar field may then bethought of as a “potential energy” and the vector fieldas a corresponding “force”.

Evaluating the potential energy for the autoencodercan be useful since it allows us to assess how muchthe autoencoder “likes” a given input x (up to anormalizing constant which would be the same forany two inputs). The potential energy can thereforeplay an analogous role to the free energy in an RBM[2], [7]. (In fact, it is identical to the free energy ofcertain RBMs for sigmoid activation functions as weshall show).

A simple condition for a vector field to be a gradientfield is given by the integrability criterion, which is aspecial case of Poincare‘s Lemma [20]: For some open,simple connected set U , a continuously differentiablefunction F : U → Rn defines a gradient field if andonly if

∂Fj(x)

∂xi=∂Fi(x)

∂xj, ∀i, j = 1..n (6)

In other words, integrability follows from the symme-try of the partial derivatives.

2.2 Computing the energy surface

Consider an autoencoder with shared weight matrix,W , and vectors of hidden/observable biases definedby bh and br, respectively, as well a continuouslydifferentiable activation function (e.g. sigmoid, hyper-bolic tangent, linear). The integrability criterion holds,since the reconstruction error satisfies

∂(rm(x)− xm)

∂xn=

∑

j

Wmj∂h(Wx+ bh)

∂(Wx+ bh)Wnj − δmn

=∂(rn(x)− xn)

∂xm(7)

where δmn denotes the Kronecker delta.One way to find the potential energy, whose deriva-

tive is r(x)−x, is to integrate the vector field (computeits antiderivative).1 For the autoencoder

r(x) = WTh(Wx+ bh

)+ br (8)

1. After computing the energy function, it is easy to check, inhindsight, whether the vector field defined by the autoencoder isin fact the gradient field of that energy function: We just need tocompute the derivative of the energy, and check if it is equal tor(x)−x. For example, one may check the correctness of Eqs. 12 or16 by differentiating w.r.t. x.

we have

E(x) =

∫(r(x)− x) dx

=

∫ (WTh

(Wx+ bh

)+ br − x

)dx

= WT

∫h(Wx+ bh

)dx+

∫(br − x) dx

(9)

Define the auxiliary variable u = Wx + bh (whichrepresents the total input of the hidden units). Nowusing

du

dx= WT ⇔ dx = W−Tdu (10)

we can write

E(x) =

∫WTW−Th(u) du+ br

Tx− 1

2‖x‖22 + const

=

∫h(u) du+ br

Tx− 1

2‖x‖22 + const

=

∫h(u) du− 1

2‖x− br‖22 +

1

2‖br‖22 + const

=

∫h(u) du− 1

2‖x− br‖22 + const (11)

where the last equation uses the fact that br does notdepend on x.

If h(u) is an elementwise activation function, thenthe final integral is simply the sum over the an-tiderivatives of the hidden unit activation functionsapplied to x. We can thus compute the energy usingthe following recipe:

1) Compute the net inputs to the hidden units:

u = Wx+ bh

2) Compute hidden unit activations using the an-tiderivative H(u) of h(u) as the activation func-tion.

3) Sum up the resulting activations across all hid-den units.

4) Subtract 12‖x − br‖22. The result is the desired

energy (up to integration constant).

2.3 Energy functions for autoencoders with com-mon activation functions

In the following, we derive the energy functions asso-ciated with some widely used activation functions. Afew example activation functions and their antideriva-tives are shown in Figure 1. We only consider autoen-coders with real-valued observations in the following.We discuss the modifications necessary to obtain theenergy functions for the corresponding models withsigmoid output units in Section 2.5.




−3 −2 −1 0 1 2 3−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

σ(x)∫σ(x) dx

−3 −2 −1 0 1 2 3−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

tanh(x)∫tanh(x) dx

−3 −2 −1 0 1 2 3−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

x∫x dx

−3 −2 −1 0 1 2 3−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

relu(x)∫relu(x) dx

Fig. 1. Some hidden unit activation functions and their integrals.

2.3.1 SigmoidThe antiderivative of the sigmoid, h(u) = (1 +exp(−u))−1, is given by the “softplus” functionH(u) = log(1 + exp(u)). It follows that the energyfunction for an autoencoder with sigmoid hiddenstakes the form

Eσ(x) =

∫(1 + exp(−u))−1 du− 1

2‖x− br‖22 + const

=∑

k

log(1 + exp(WT·kx+ bhk))− 1


(12)

Thus, the energy is identical to the free energy in abinary-Gaussian RBM [21]. The analog of the unkownnormalizing constant in the RBM is the constant ofintegration for the autoencoder.

2.3.2 Hyperbolic tangentAnother widely used activation function in neuralnetworks is the hyperbolic tangent:

h(u) = tanh(u) =exp(u)− exp(−u)

exp(u) + exp(−u)(13)

The corresponding energy function may be written

Etanh(x) =

∫u

exp(u)− exp(−u)exp(u) + exp(−u)

du−1


=∑k

log

(exp(uk) + exp(−uk)

2

)−

1


=∑k

log (cosh(uk))−1

2‖x− br‖22 + const (14)

=∑k

(uk + softplus(−2uk))−1

2‖x− br‖22 + const (15)

The energy function involves a hyperbolic cosine.Equation 15 is a numerically stable version of 14,whose derivation we show in appendix A.

2.3.3 Linear activationThe antiderivative of the linear function h(u) = u,is ‖u‖22/2. Hence, for the linear autoencoder, whose

weight vectors will span the PCA solution [22], wehave:

Elinear(x) =

∫u du− 1


=1

2‖u‖22 −

1


=1

2‖Wx+ bh‖22 −

1


(16)

The energy is simply the norm of the latent repre-sentation. It is interesting to note that, if we disregardbiases and assume WTW = I (the PCA solution), thenElinear(x) turns into the negative squared reconstruc-tion error. Interestingly, this is exactly how one wouldassign confidence scores to PCA models, such as aPCA based generative classfier.

In Eq. 11 we make use of the invertibility of WT

to derive the energy function in its general form.However, differentiating Elinear(x) verifies that forlinear hidden units, Eq. 16 will be the correct energyfunction regardless of shape or invertibility of WT :

∂Elinear(x)

∂x= WTWx+WTbh + br − x= WT(Wx+ bh) + br − x= r(x)− x

(17)

2.3.4 Square activationThe square activation, h(u) = u2, is increasinglycommon, and it can be shown to be useful for learn-ing video and motion representations [23], [24]. Theenergy function for the square activation is:

Esq(x) =

∫u2 du− 1


=∑

k

u3k3− 1


(18)

2.3.5 Rectified linear (ReLU)Another increasingly common activation function isthe rectified linear activation:

relu(u) =

0 if u < 0,

u else(19)




It can be approximated in the RBM using a stochasticvariant called the “noisy rectified linear” activation[25]. Integrating piecewise shows that the antideriva-tive of relu(x) is a “half-square” (see also Figure 1):Hence,

Erelu(x) =∑

k

(sign(uk) + 1)u2k2− 1


2.3.6 Modulus activation

The piecewise antiderivative of the modulus (absolutevalue) function, |u|, is given by sign(u)u2

2 , whichyields the energy function

Eabs(x) =∑

k

sign(uk)u2k2− 1

2‖x− br‖22 + const (20)

2.3.7 Synchrony autoencoder

The synchrony autoencoder (SAE) was introducedby [26] as an alternative to the square activation forextracting motion features from videos or image pairs.It is based on hidden units whose activation functionis a sigmoid applied to squared linear projections ofthe concatenation of frames in the video. By integrat-ing the sigmoid-of-square activation, one obtains thecorresponding energy function

ESAE(x) =∑

k

log(exp(u2k) + 1)− ‖x‖22 + const (21)

2.3.8 Maximum activation and Kmeans clustering

By using the maximum activation function and unitnorm data and weight vectors, we obtain an autoen-coder that is equivalent to Kmeans clustering. Kmeansminimizes the sum of squared distances betweenobservations xj and cluster centers wi:

k∑

i=1

∑

j:xj∈Ci

‖xj −wi‖22 (22)

where Ci is the set of points for which wi is the closestcluster center ([27]). If cluster centers are normalized,we have

‖xj −wk‖22 = −2xTj wk + Ω (23)

with Ω = ‖xj‖22 + 1 constant. Define the activationfunction

hi(x) =

1 if 〈x,wi〉 = maxj〈x,wj〉,0 else

(24)

that chooses the nearest cluster center, as well as aselection function s(x) which returns the index ofthe non-zero hidden unit. The reconstruction in anautoencoder with tied weights can now be writtenW Th(x) = ws(x) which is the nearest cluster center,

whose training criterion is the standard Kmeans objec-tive (Eq. 22). The energy function for this autoencoderis

Emax(x) =

∫(x−ws(x)) dx = ‖x−ws(x)‖22 + const

which is simply the squared distance from the nearestcluster center. Note that the energy function is dif-ferentiable almost everywhere. As in the case of thelinear autoencoder and PCA, the energy Emax is theusual confidence score of the Kmeans model.

2.4 Example of a 2-d energy surfaceAn example of a two-dimensional vector field and thecorresponding energy surface is shown in Figure 2.It shows the result of training a contractive autoen-coder with 100 hiddens and sigmoid activation ontwo-dimensional data randomly distributed around acircle. The data, learned vector field and associatedenergy function are shown in Figures 2(a), (b), (c) and(g), respectively. The figure also shows the reconstruc-tions of points drawn from a regular grid (Fig. 2(d))and the corresponding squared reconstruction error(Fig. (e) and (g)), which illustrates why reconstructionerror is not a suitable energy function: It has localminima at both the local maxima and the local minimaof the energy function and thus it cannot distinguishbetween “good” and “bad” stationary points of theenergy. The unsuitability of squared error as an energysurface is discussed also in [18].

2.5 Binary dataIn the case of sigmoid outputs activations, the inte-grability criterion (Eq. 7) does not hold, because ofthe lack of symmetry of the derivatives. However,it is possible to regain integrability and obtain anenergy function by monotonically transforming thevector space as follows: Use the inverse of the logisticsigmoid (the “log-odds” transformation)

ξ(x) = log( x

1− x)

(25)

and define the new vector field

B(x) = ξ(r(x))− ξ(x)

= ξ(σ(WTh

(Wx+ bh

)+ br

))− log

( x

1− x)

= WTh(Wx+ bh

)+ br − log

( x

1− x)

(26)

The vector field B(x) has the same fixed points asr(x)− x, because invertibility of ξ(x) implies

r(x) = x ⇔ ξ(r(x)) = ξ(x) (27)

Thus running the original and transformed autoen-coder will converge to the same representations of x.

Although the log-odds transformation is defined(and invertible) only for points x in the interior of




AE vector field Energy surfaceTraining set

Squared error slice

Energy surface slice

Squared error surfaceReconstruction

Fig. 2. Example of a learned vector field and the associated energy function. See the text for details.

the hypercube x ∈ (0, 1)n (in other words, not forbinary x), any fixed point of the original autoencoder,r(x), will reside in the interior, too. The reason isthat (even for a binary input vector x) the sigmoidoutputs of the autoencoder will yield a reconstructioninside the hypercube, as a sigmoid unit can neverattain the values 0 or 1. As we show below, the energyassociated with the transformed autoencoder, is well-defined also for binary data.

By integrating B(x), we obtain the energy

E(x) =

∫h(u) du+ br

Tx (28)

−∑

j

(log(1− xj

)+ xj log

( xj1− xj

))+ const

As compared to the real-valued autoencoder (c.f.,Eq. 11) this energy function contains a bias term that islinear rather than quadratic, as well as the additionalentropy term

−∑

j

(log(1− xj

)+ xj log

( xj1− xj

))

= −∑

j

(xj log xj + (1− xj) log

(1− xj

)) (29)

Using the convention 0 log(0) = 0 [28], which is basedon the fact that limx→0 x log(x) = limx→0−x

−1

x−2 = 0(l’Hopitals rule), this term vanishes for binary data.

2.5.1 Binary-binary autoencoderBased on the discussion in the previous section, theenergy function for an autoencoder with sigmoid

hidden and output units, evaluated at strictly binarydata, is given by:

Eσ(x) =∑

k

log(1 + exp(WT·kx+ bhk)) + br

Tx+ const

(30)

The energy is identical to the free energy of a binary-binary RBM [2].

Note that in general, hidden unit activation func-tions do not need to be sigmoid and enter the energycomputation using the recipe described in Section 2.2.Furthermore, on data that is not binary but that takeson values between 0 and 1, the entropy-terms in Eq. 28are not equal to 0 and need to be included in theenergy computation.

3 THE LAPLACIAN AND DIVERGENCE OF ANAUTOENCODER

We now discuss how the existence of the energyfunction makes it possible to apply notions fromvector calculus, such as the divergence and Laplacian,to the autoencoder. We then show how these notionslend themselves naturally to defining regularizationcriteria similar to the contractive regularizer proposedby [13], suggesting to re-interpret this criterion fromthe perspective of dynamical systems.

The divergence of a vector field is a scalar field thatmeasures the outward flux of the vector field at eachpoint [19]. A negative divergence at a point x signifiesthat the point is a “sink” of the dynamics, a positivedivergence that it is a “source”. The divergence of




vector field F can be defined as the sum of the first-order derivatives:

divF (x) =∑

i

∂Fi(x)

∂xi(31)

In a conservative vector field the divergence is equalto the Laplacian of the energy function, which can bedefined as the sum of (non-mixed) second derivatives[19]:

(divF (x) =

)∆E(x) =

∑

i

∂2E

∂x2i(32)

To discuss these notions in the context of the au-toencoder we first write the autoencoder dynamicscompactly as

∂E(x)

∂x= r(x)− x = WTh(Wx)− x (33)

Here and in the following we absorb the biases intoinputs and hidden units to avoid clutter.

The Hessian of the energy function can now bewritten:

∂2E(x)

∂x2= WTdiag

(h′(x)

)W − ID (34)

where h′ denotes the first derivative of the hiddenactivation function applied element-wise, and D is thedimensionality of the data.

It follows that the Laplacian of the energy function,or equivalently the divergence of the autoencodervector field, can be written:

∆E(x) = tr(WTdiag(h′(Wx))W − ID

)(35)

= tr(WTdiag(h′(Wx))W

)−D (36)

= tr(diag(h′(Wx))WWT

)−D (37)

=∑

k

h′(wTk x)‖wk‖22 −D (38)

Training an autoencoder by minimizing reconstruc-tion error may be thought of as encouraging trainingdata points to be fixed points of the dynamics, as wediscussed. However, this by itself is not a suitabletraining criterion, especially for overcomplete autoen-coders, since they can learn the identity mapping,which would turn all points into fixed points.

From the perspective of reconstruction dynamics, anatural way to regularize the autoencoder is to turntraining points into a sink of the dynamics (thus limit-ing the total amount of energy). In this case, trainingpoints will not only be encouraged to act as a fixedpoints but also to be the center of a basin of attraction.Intuitively, traversing points that act as sinks amountsto loosing potential energy, so if the energy surfaceis bounded, the dynamics have to converge. Anotherperspective is that the only way to allow for perfectreconstruction of the data by overfitting is to learn aperfectly flat energy surface. But at points that aresinks, the curvature of the energy surface will be

negative, making it impossible for the energy surfaceto be flat everywhere.

Formally, a point xwill be a sink of the autoencoderdynamics, if the Laplacian ∆E(x), is negative, orequivalently if

∑

k

h′(wTk x)‖wk‖22 < D (39)

As we show below, contractive regularization [13]may be viewed as a way to satisfy this criterion. In thefollowing, we shall call the autoencoder absorbing atdata point x if that data point satisfies Eq. 39. Oneway (though obviously not the best way) to makeall points absorbing would be to restrict weights tobe sufficiently small (for example using very strongweight decay). For saturating activation functions theautoencoder can be absorbing locally even for largeweights, by having data-points saturate hidden units(in particular, those with large weight vectors).

The contractive regularizer [13], which was orig-inally motivated by encouraging the derivatives ofhidden units wrt. the inputs to be small, is definedas the Frobenius norm of the Jacobian:

∑

k

(h′(wT

k x))2‖wk‖22 (40)

Up to the squaring of the derivative in Eq. 40 (whichis strictly positive for most common autoencoders, sothe square will have only a limited effect and notchange signs), Eq. 40 is identical to the left-hand sideof Eq. 39.

This shows that contractive regularization as de-fined in [13] may be viewed as a way to encouragethe Laplacian of the autoencoder to be small nearthe data (although an interesting research directioncould be regularizing an autoencoder by enforcingEq. 39 directly rather than using Eq. 40). A similarpoint can be made for denoising autoencoders, whichthemselves may be viewed as a way to approximatea contractive regularizer [18]. Eq. 39 also shows un-der what conditions contractive regularization willsucceed in making the autoencoder absorbing at alltraining examples: the contraction penality (in theform of the left-hand side of Eq. 39) has to be smallerthan D, the dimensionality of the data.

As we discussed in Section 2, an autoencoder isguaranteed to have a well-defined energy function ifit has tied weights. It is interesting to note that for anautoencoder whose weights are not tied, contractiveregularization will encourage the vector field to beconservative. The reason is that encouraging the firstderivative to be small and the second derivative to benegative will tend to bound the energy surface nearthe training data. Training a contractive or denoisingautoencoder may thus also be viewed as a way tooptimize the Lyapunov stability of the autoencoderdynamics near the data.




In the following we show how several existingmethods are naturally absorbing according the abovedefinition.

3.1 PCA is absorbing everywhereThe PCA reconstruction is given by

∂E(x)

∂x= r(x)− x = WTWx− x (41)

It follows that the PCA Laplacian is given by:

∆E(x) = tr(WTW

)−D = tr

(WWT

)−D = d−D

(42)The Laplacian is constant (independent of the data),and it is negative as long as the dimensionality of thehidden layer is smaller than the data dimensionality.Intuitively, this means that there is a projection (ontoa subspace).

3.2 Kmeans clustering is absorbing everywhereBy using the selection function s(x) which returns theindex of the nearest cluster center (cf., Section 2.3.8)we can write the Kmeans clustering reconstructionfunction as

∂E(x)

∂x= r(x)− x = ws(x) − x (43)

Note that r(x) is differentiable almost everywhere(more specifically, it is differentiable everywherewithin the Voronoi cell defined by a cluster center).Since within each cell, ws(x) is constant (and itsderivative is zero), the Laplacian takes the form

∆E(x) = tr(− ID

)= −D (44)

This shows that Kmeans clustering is globally absorb-ing, independent of the number and the position ofcluster centers.

3.3 CD training as optimizing the LaplacianContrastive divergence (CD) training is a popularapproximation to maximum likelihood learning forRBMs. It is based on approximating the intractablederivative of the log-normalizing constant of thedata-log probability by using local samples from themodel [2], [29]. In practice this amounts to applyingan update rule which increases unnormalized log-probability at the training examples and decreasesit near the data. Nearness is defined in practice byapplying a Gaussian perturbation around a one-stepreconstruction of the training-example.

A common way to define the one-step reconstruc-tion is the mean-field approach, which amounts toinferring hiddens from the data and subsequently re-inferring the data from the hidden units, which isequivalent to applying the sigmoid autoencoder as-sociated with the RBM. As we showed in Section 2.3,the energy function of this autoencoder is identical

to the RBM free energy. Like contractive training ofthe autoencoder, CD training has the effect of turningthe training examples into local optima of the energyfunction. The close relation between RBMs and au-toencoders, and their way of locally optimizing theenergy surface, may suggest alternative approaches totraining RBMs, such as replacing the sampling-basednegative phase with a contractive regularizer.

4 CLASSIFICATION WITH CLASS-SPECIFICAUTOENCODERS

One application of the ability to assign an energy, orconfidence score, to data-points is to turn a set ofclass-specific autoencoders into a generative classifier.To this end, one may train one autoencoder per classand combine their confidence scores on a new datasample into a classification decision. One advantageof generative classifiers is that they make it possibleto add classes at test-time without having to re-trainthe parameters of the already trained class-specificmodels. The most common choice of class-specificmodel are directed probabilistic models, which aretypcially combined into a classification decision usingBayes’ rule. But it is possible to use undirected models(and, in fact, autoencoders as we discuss below) as theclass-specific models instead, as long as these modelsare combined properly to obtain a well-calibratedclassification decision [2], [30].

Figure 3 shows energies that autoencoders withvarious different activation functions assign to testcases from classes 6 and 9 of the MNISTsmall digitdataset ([31]), after training on the training cases fromeach class (one model per class). A similar plot specificto RBMs was presented in [2] which, not surprisingly,looks similar to the plot for the autoencoder with sig-moid activation functions. Here, we used contractiveautoencoders for training (see Section 4.2 for moredetails on the learning). The figure shows that allmodels yield fairly well separated confidence scores,when the apropriate anti-derivatives are used.

Like for RBMs, scores are relative, not absolute,however, due to the lack of the normalizing constant.This means that we can compare the scores thatan autoencoder assigns to multiple data-points, butwe cannot compare the scores that multiple differentautoencoders assign to a single data-point.

One solution is to aggregate class-specific scoreswithin a discriminative classifier, as shown for Re-stricted Boltzmann Machines by [30]. The “gatedsoftmax”-model proposed in that work contains aparameter tensor holding all class-specific RBM pa-rameters. Instead of using Bayes rule to turn the class-specific probabilities into a classification decision, thatmodel uses the parameter tensor to define class-probabilities directly. [30] show how, in practice, thisamounts to computing the RBM free energy for eachclass, and combining these with a softmax function.




0.45 0.50 0.55 0.60 0.65 0.70 0.75

E6(x)

0.50

0.55

0.60

0.65

0.70

0.75

0.80

E9(x

)

sigmoid activation

class 6class 9

0.719 0.720 0.721 0.722 0.723 0.724 0.725

E6(x)

0.711

0.712

0.713

0.714

0.715

0.716

0.717

0.718

0.719

E9(x

)

tanh activation

class 6class 9

0.0 0.2 0.4 0.6 0.8 1.0

E6(x)

0.0

0.2

0.4

0.6

0.8

1.0

E9(x

)

linear activation (real input)

class 6class 9

0.60 0.65 0.70 0.75 0.80 0.85 0.90

E6(x)

0.70

0.75

0.80

0.85

0.90

0.95

1.00

1.05

E9(x

)

linear activation (binary input)

class 6class 9

−0.05 0.00 0.05 0.10 0.15 0.20

R6(x)

0.00

0.02

0.04

0.06

0.08

R9(x

)

sigmoid activation (squared errors)

class 6class 9

Fig. 3. Examples of class-dependent contractive autoencoder scores distributions and squared reconstructionerrors of digits 6 and 9 from the MNISTsmall data set, learned using different activation functions.

Closely related models include the discriminativeRBM [32], and the discriminative/generative RBMdescribed in [33].

It is straightforward to apply this approach to au-toencoders using their energy function as confidencesscore. More specifically, denoting the unnormalizedenergies that the class-specific autoencoders assign tothe data as Ei(x), i = 1, . . . ,K, we can define theconditional distribution over classes yi, for example,using multiclass logistic regression as

P (yi|x) =exp(Ei(x) + Ci)∑j exp(Ej(x) + Cj)

(45)

where Ci is the bias term for class yi. We shall refer tothis model as class-specific autoencoders (CSAE) in thefollowing.

It is interesting to note that each Ci may be viewedas the normalizing constant of the ith autoencoder,which, since it cannot be determined from the inputdata, needs to be trained from the labeled data.

Eq. 45 may therefore be viewed as a “contrastive”objective function that compares class yi against allother classes in order to determine the missing nor-malizing constants of the individual autoencoders.This is reminiscent of noise-contrastive estimation[34], with the difference that the contrastive signal isprovided by other classes not by noise. Optimizing thelog of Eq. 45 is straightforward using gradient basedoptimization. Like in the gated softmax model [30],after training the K class specific autoencoders, wemay perform discriminative finetuning of the wholemodel (including all parameters of the K involvedautoencoders and the normalizing constants), by op-timizing Eq. 45 with respect to these parameters. Toobtain their derivatives, one can back-propagate thelogistic regression cost.

4.1 Parameter FactorizationWe showed in Section 2 that an energy function isguaranteed to exist only for autoencoders with asingle hidden layer. We shall now discuss a simplebut effective approach to training multilayer modelswithout sacrificing the ability to compute scores, usingfactorization of a parameter tensor [35], [36].

Note that we may add hidden layers with a linearactivation function, since we could in principle absorbtheir weights into the other layers. To define a mul-tilayer autoencoder classification model, we suggestadding linear layers on the bottom and, by symmetry,its transpose on the top. At first sight, it seems thatnothing is gained by this proceduce. But since we willbe training one autoencoder per class, we may now tiethe weights of the outer-most layers across classes, toperform class-independent preprocessing of the data.Thus, even though the linear layers do not add anyinformation when training each autoencoder on datafrom its class, training them with tied weights does.

In many classification tasks such class-specific pre-processing makes sense, because similar features mayappear in several classes, so there is no need tolearn them separately. Class-specific autoencoderswith shared bottom-layer weights can also be inter-preted as standard autoencoders whose set of weightmatrices is factorized, as proposed by [30] for thegated softmax model based on RBMs.

Fig. 4 shows an illustration of a factored autoen-coder. The model parameters are filter matrices W x

and W h which are shared among all classes, aswell as matrices W f ,Bh and Br which consist ofstacked class-dependent feature-weights and bias vec-tors. Using one-hot encoded labels, t, the hidden unitactivations and the reconstructions can be written

hk =∑

f

(∑

i

Wxifxi)(

∑

j

tjWfjf )Wh

fk +∑

j

tjBhjk (46)

rm =∑

f

(∑

k

WhTkf hk)(

∑

j

tjWfjf )WxT

fm +∑

j

tjBrjm (47)

Multiplying by t, we “mask” the matrices W f , Bh

and Br for each input sample individually, so thatonly the rows relevant to the current training caseremain.

One can interpret this model as a combination ofK class-dependent autoencoders with tied parametersets W j , bjh, b

jr|j = 1..m, where each weight ma-

trix W j is factorized into a product of two class-independent (shared) matrices W x and W h and one




class-dependent vector wf :

W jik =

∑

f

WxifW

fjfW

hfk (48)

The first encoder-layer (W x) learns the class-independent features, the second layer (W f

j· ) learns,how dominant these features are in current classand weights them accordingly. Finally, the third layer(W h) learns how to overlay the weighted features toget the hidden representation. All these layers havelinear activations except for the last one. Reconstruc-tions and decoder weights take the form

rji (x) =∑

k

W jTik σ(hjk) + bjr (49)

W jik

T=

∑

f

WhTfk W

fjfW

xTif (50)

For training in the presence of tied weights, we canupdate all K autoencoders on data across all classesand use the labels in order to determine for eachtraining case which top-level weights to train. Sharedparameters are updated on all training cases. Thus,although the lower layers learn to perform class inde-pendent pre-processing, the whole model, includingpre-preprocessing weights, will be trained using asingle, discriminative objective.

Fig. 4. Factored autoencoder with tied weights. Top:a single factored autoencoder; bottom: encoder-partof the combination of several autoencoders. Dashedlines represent the class-dependent weights, solidlines class-independent weights.

4.2 Classification experimentsWe tested the class-specific autoencoder model (fac-tored and plain) on the regular MNIST dataset [37]

and on the deep learning benchmark set [31]. The lat-ter consists of eight datasets, most of which are varia-tions of MNIST. One of the eight datasets, MNISTs-mall, is a subset of the ordinary MNIST datasetwith only 10000 training samples. All datasets containgray-scale images of 28×28 pixels with values in therange [0, 1]. We used the Python Theano library [38]for most of our experiments. An implementation ofthe model is available at: http://fias.uni-frankfurt.de/∼kamyshanska/aescoring code.html

In our experiments, we regularized autoencoderswith differentiable activation function using contrac-tion penalty [13], and those with non-differentiable ac-tivation function using denoising regularization [17].The factored models were all regularized using adenoising criterion, because the computation of con-traction penalties is not feasible for multilayer net-works [13].

4.3 Class-specific Pretraining

Although it would be possible to train the log ofEq. 45 starting with a random initialization for allthe parameters, [30] report better performance bypretraining all class-specific models separately on datafrom their own class. We found the same to be truehere (see also Section 4.5 for a detailed evaluation).

In our evaluations, we train one set of autoencodersfor each number of hiddens (and of factors for thefactored model) for pretraining, because at this stageit is not possible to decide which setting will performthe best after discriminative learning. The set of au-toencoders per setting (ie., per number of hiddens andof factors) consists of one model per class. We trainedeach autoencoder separately for the unfactored model,and we trained the autoencoders across all classessimultaneously for the factored model since someparameters are shared between classes, as explainedabove. We used gradient descent for optimization. Insome of our experiments we found that normalizingfilters to have equal norm after each gradient step canhelp stabilize the learning.

For contractive autoencoders the contractionpenalty varied between 0.2, 0.5 and 1.0; for denoisingautoencoders the corruption level varied between 0.2and 0.5. Since it is infeasible to perform discriminativefinetuning on a very large number of pretrainedcandidate models, we chose various hyperparameters,such as the number of training epochs, by visualinspection in the pretraining phase. Although apartial search over hyperparameters involving anouter loop for finetuning may slightly improveoverall performance, we found the performance to begenerally robust to small changes in the pretraining,as long as the pretrained filters looked reasonable.Thus, we do not expect better pretraining to be ableto yield a large potential performance improvement.We included the validation sets in the training data

http://fias.uni-frankfurt.de/~kamyshanska/aescoring_code.html

http://fias.uni-frankfurt.de/~kamyshanska/aescoring_code.html




used for pretraining to improve the quality of thelearned filters (even though this may slightly distortthe validation decisions, potentially resulting insuboptimal performance).

4.4 Supervised Finetuning

For discriminative finetuning, we validated over thefollowing hyperparameters: number of hiddens, numberof factors (for the factored model), learning rate, weightdecay. As weight decay, we used an L2 penalty onthe model parameters. In most cases, we tested 100,200, 300 and 500 hiddens and factors. For all datasets, in both the pretraining and finetuning phases,we used minibatches of size 100, and momentum of0.9 for training. We trained all models for 100 epochsusing gradient descent (but we found that the bestvalidation cost was often reached during the first 10epochs).

Finally, after finding the best settings accordingto the validation data, we finetuned the model onthe complete training set (train and validation), andevaluated the resulting model on the test data.

4.5 Classification With and Without Pretraining

We first compared the classification performance ofthe pretrained and randomly initialized CSAE modelson all data sets for linear and sigmoid activations aswell as for sigmoid activations with feature sharing.Pretrained models almost always lead to better resultson the test data as seen in Figure 6. We also observedthat often, the pre-trained filters are already close tothe optimum of the discriminative objective, so thatjust a few epochs of finetuning suffice. Furthermore,finetuning often tends to adjust mainly the scale,rather than the structural details, of the pretrained fil-ters. An illustration of this effect is shown in Figure 5,which shows that filters look similar before and afterfinetuning. In the example, pretrained filters (left) takeon values between −4.72 and 4.67 whereas finetunedfilters (right) between −0.64 and 0.63.

Fig. 5. Initial filters learned by a contractive autoen-coder trained on the ”0“– digits from MNIST, beforesupervised finetuning (left), and after supervised fine-tuning (right).

To get a better understanding of the performanceimprovement due to finetuning, we compare the test-and the train-errors for autoencoders with linear hid-den units in Table 1. It shows that the reason for thelower error rates is data dependent, in that in can bedue to the optimization of the training objective (toppart of the table), or better generalization (bottom partof the table). The fact that pretraining in some cases

Fig. 6. Test error rates for the CSAE model withand without discriminative pretraining. Top: ordinarycontractive class-specific AE with sigmoid hidden units.Middle: factored denoising AE with sigmoid hiddenunits. Bottom: ordinary contractive class-specific AEwith linear hidden units.

simply helps the optimization, may be related to thefact that it may encourage orthonormalization of theweights, as recently discussed in [39].

Overall, our experiments show clearly that unsu-pervised pretraining generally leads to better resultsand is preferable over random initialization.

DATA SET WITH PRETR. WITHOUT PRETR.TRAIN/TEST TRAIN/TEST

RECTANGLES 0.08 / 0.84 0.41 / 3.96MNISTRAND 4.16 / 17.96 20.74 / 21.5

MNISTSMALL 0.04 / 3.91 0.02 / 4.29MNISTROT 4.12 / 16.19 0.14 / 16.56MNISTIMG 0.94 / 22.37 0.18 / 29.03MNISTROTIMG 8.45 / 56.15 0.54 / 61.85

TABLE 1Error rates on train- and test-data sets with and

without pretraining, using the linear activation function.Best results shown in bold font.

4.6 Comparison of Activation Functions

We tested the CSAE classification approach on theMNIST data set for a variety of autoencoders: con-tractive autoencoder with sigmoid and linear acti-vation; denoising autoencoder with modulus acti-vation; factored denoising autoencoder with hyper-bolic, modulus, squared, and rectifier activation. Ta-ble 2 shows test error rates for these versions of the




CSAE:RBM 3.39 LINEAR 1.99DRBM 1.81 SQUARED (FACTORED) 2.5HDRBM 1.28 MODULUS (FACTORED) 2.04

RELU (FACTORED) 2.35MODULUS 1.76TANH (FACTORED) 2.02SIGM 1.27

TABLE 2Test-error rates on MNIST for the CSAE Model based

on autoencoders with different activation functions.

CSAE model. It also shows error rates for relatedgenerative/discriminative models that are based onRBMs [32], including the discriminative RBM (DRBM)and hybrid DRBM (HDRBM). In this experiment, thesigmoid activation yields the lowest error rates; themodulus function also performed well. Somewhatsurprising is the comparably good performance of thelinear activation, which is the simplest model.

We then evaluated the best performing activationfunctions on the deep learning benchmark set [31].The results are shown in Table 3. Again, models withsigmoid activation performed best, with the linearmodel performing comparably on the RECTANGLES,MNISTsmall and MNISTbackImg tasks. We foundthe MNISTrand data set in this experiment not tohave enough samples per class to learn meaningfulclass-dependent filters (by inspection). We thereforeinitialized the final model for this data set using filtersfrom the MNISTsmall data.

DATA SET LINEAR MODULUS SIGM SIGM(FACT.)

RECTANGLES 0.84 4.82 2.72 0.84RECTANGLESIMG 25.3 24.9 21.45 22.76CONVEXSHAPES 36.34 23.18 21.52 22.12MNISTSMALL 3.91 5.17 3.18 3.62MNISTROT 16.19 13.22 13.11 14.46MNISTRAND 17.96 17.08 19.52 12.64MNISTBACKIMG 22.37 23.38 21.87 22.77MNISTROTIMG 56.15 52.88 54.79 47.14

TABLE 3Test-error rates on deep learning benchmark for the

basic CSAE model.

Fig. 8. Test error rates of factored vs unfactored model.

Some learned filters are shown in Figure 7. Thefigure shows features learned on rotated digits and

digits with background images using linear, modu-lus and sigmoid activation. Interestingly, the filterslearned by the different models look qualitativelyvery different. The modulus autoencoder seems tolearn ”too much“ on MNISTbackImg, while the linearautoencoder captures mostly global structure, such asthe shape of the digit ”2“. On more difficult taskshowever, the more local structure seems to be crucialas reflected in the performance table. Figure 7 alsoshows the shared features from the factored modelswith sigmoid, modulus and rectifier activations onMNIST. The overall qualitative differences across themodels persists.

Another important observation from Table 3 is thatfactored models are not generally better or worse. Itis task-dependent which kind of model to choose.This is also illustrated in Figure 8, which comparesthe factored versus unfactored variants of the CSAEclassifier.

4.7 Comparison of Model VariationsFinally, we trained and compared a variety of varia-tions of the CSAE model:

(hinge): Pretrain the filters, then minimize hinge-loss instead of log-loss (negative log of Eq. 45) forfinetuning.

(learn normalization): Train an autoencoder foreach class separately, then learn only the normal-izing constants by maximizing the conditional log-likelihood (Eq. 45).

(energies): Train an autoencoder for each classseparately, then compute for each sample x ∈ Rn

a vector of energies (E1(x), · · · , Em(x)), setting theunknown integration constants to zero, and train alinear classifier on labeled energy vectors instead ofusing the original data [2]. This model performs bothnormalization and scaling of the pretrained filters.

In our experiments, the log-loss consistently out-performed the hinge loss in all tasks. Methods en-ergies and learn normalization are very fast due tothe small amount of trainable parameters, but theyshow weaker performance, as shown in Table 4. Forcomparison, the table also lists the performances ofthe basic CSAE approach without pre-training.

Two lessons to learn from our experiments are that(i) generative pre-training of each autoencoder ondata from its own class is crucial to achieve goodperformance, (ii) it is not sufficient to adjust merelynormalizing and scaling constants, since backpropa-gating to the filters themselves significantly improvesoverall performance.

4.8 Comparisons with Other ModelsTable 5 shows the classification error rates of CSAEin comparison to various similar models from theliterature. We followed the same validation proce-dure as discussed above. In these experiments, to




Fig. 7. Filters learned using autoencoders with different activation functions.

DATA SET ENERGIES NORMALIZE NO PRETR. CSAEONLY

MNISTSMALL 3.39 3.66 5.53 3.38MNISTROTATION 14.32 20.85 19.95 13.77CONVEX 44.36 33.9 24.02 21.52

TABLE 4Classification results for variations of the model.

DATA SVM RBM DEEP GSM CSAERBF SAA3

RECTANGLES 2.15 4.71 2.14 0.56 0.84RECTANGLESIM 24.04 23.69 24.05 22.51 21.45CONVEXSHAPES 19.13 19.92 18.41 17.08 21.52MNISTSMALL 3.03 3.94 3.46 3.70 2.61MNISTROT 11.11 14.69 10.30 11.75 11.25MNISTIMG 22.61 16.15 23.00 22.07 20.15MNISTRAND 14.58 9.80 11.28 10.48 12.64MNISTROTIMG 55.18 52.21 51.93 55.16 47.14

TABLE 5Test-error rates on the deep learning benchmark data

set. CSAE results use validation over factored andunfactored models with sigmoid hidden units.

RBF-kernel SVM and RBM results taken from [40];deep net and GSM results from [30].

be consistent with [30], we furthermore normalizedfilters to have constant norm during the optimization.For the GSM model, we report the best performanceof factored vs. unfactored on the test data, whichmay introduce a bias in favor of that model. Someexample images with corresponding filters learned bythe ordinary and factored CSAE model are shown inFigure 9.

5 CONCLUSION

We showed how we may assign an unnormalizedenergy surface to an autoencoder by interpreting itas a dynamical system. Unlike previous approachesto defining an autoencoder energy, the dynamical sys-tems perspective is not restricted to sigmoid activationfunctions, which make the autoencoder resemble anRBM, and it is independent of the training criterion.

We also show how multiple class-specific autoen-coders can be turned into a generative classifier thatyields competitive performance in difficult bench-mark tasks. Class-specific dynamical systems mayoffer an appealing alternative perspective onto clas-sification than the commonly used linear (eg. logisticregression-) layer atop a deep neural network. If a

class is represented not just by a weight vector, butby a dynamic sub-network such as an autoencoder, itis easy to model highly complex intra-class variabilityusing a comparably small amount of computationalresources.

Fig. 9. Example images and filters learned by theCSAE model. (a): Examples of RECTANGLESimgdata; (b)-(c): learned horizontal vs vertical filters. (d):MNISTrotImg (factored model); (e): RECTANGLES(factored model).

APPENDIX

A numerically stable computation of log (cosh(uk))may be derived as follows:

log (cosh(uk)) = log

(exp(uk) + exp(−uk)

2

)

= log (exp(uk) + exp(−uk))− log 2

∝ log (exp(uk)(1 + exp(−2uk)))

= log(exp(uk)) + log(1 + exp(−2uk))

= uk + softplus(−2uk)(51)

REFERENCES[1] Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. Un-

supervised feature learning and deep learning: A review andnew perspectives. CoRR, abs/1206.5538, 2012.

[2] G. E. Hinton. Training products of experts by minimizingcontrastive divergence. Neural Computation, 14(8):1771–1800,2002.




[3] Y. LeCun, S. Chopra, R. Hadsell, M. A. Ranzato, and F. J.Huang. A tutorial on energy-based learning. In PredictingStructured Data. MIT Press, 2006.

[4] Q. Le, M.A. Ranzato, R. Monga, M. Devin, K. Chen, G. Cor-rado, J. Dean, and A. Ng. Building high-level features usinglarge scale unsupervised learning. In International Conferenceon Machine Learning (ICML), 2012.

[5] R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D.Manning. Semi-Supervised Recursive Autoencoders for Pre-dicting Sentiment Distributions. In Conference on EmpiricalMethods in Natural Language Processing (EMNLP), 2011.

[6] J. T. Rolfe and Y. LeCun. Discriminative recurrent sparse auto-encoders. In International Conference on Learning Representations(ICLR), 2013.

[7] K. Swersky, D. Buchman, B.M. Marlin, and N. de Freitas. Onautoencoders and score matching for energy based models. InInternational Conference on Machine Learning (ICML), 2011.

[8] Pascal Vincent. A connection between score matching anddenoising autoencoders. Neural Computation, 23(7):1661–1674,July 2011.

[9] R. Memisevic. Gradient-based learning of higher-order imagefeatures. In the International Conference on Computer Vision(ICCV), 2011.

[10] W.Y. Zou, S. Zhu, A. Ng, and K. Yu. Deep learning of invariantfeatures via simulated fixations in video. In Advances in NeuralInformation Processing Systems (NIPS), 2012.

[11] A. Hyvarinen. Estimation of non-normalized statistical modelsby score matching. Journal of Machine Learning Research, 6:695–709, December 2005.

[12] Aapo Hyvarinen. Connections between score matching, con-trastive divergence, and pseudolikelihood for continuous-valued variables. Neural Networks, IEEE Transactions on,18(5):1529–1531, 2007.

[13] S. Rifai, P. Vincent, X. Muller, X. Glorot, and Y. Bengio.Contractive auto-encoders: Explicit invariance during featureextraction. In International Conference on Machine Learning(ICML), 2011.

[14] G. Alain and Y. Bengio. What regularized auto-encoderslearn from the data generating distribution. In InternationalConference on Learning Representations (ICLR), 2013.

[15] H.S. Seung. Learning continuous attractors in recurrent net-works. Advances in neural information processing systems (NIPS),10:654–660, 1998.

[16] Hanna Kamyshanska and Roland Memisevic. On autoencoderscoring. In Proceedings of the 30th International Conference onMachine Learning (ICML-13), pages 720–728, 2013.

[17] P. Vincent, H. Larochelle, Y. Bengio, and P. A. Manzagol.Extracting and composing robust features with denoisingautoencoders. In International Conference on Machine Learning(ICML), 2008.

[18] G. Alain, Y. Bengio, and S. Rifai. Regularized auto-encodersestimate local statistics. arXiv preprint arXiv:1211.4246, 2012.

[19] John M Lee. Introduction to smooth manifolds, 2001.[20] RM Santilli. Foundations of theoretical mechanics II. Birkhof-

fian generalization of Hamiltonian mechanics. 1982.[21] M. Welling, M. Rosen-Zvi, and G. Hinton. Exponential family

harmoniums with an application to information retrieval. Ad-vances in neural information processing systems (NIPS), 17, 2005.

[22] P. Baldi and K. Hornik. Neural networks and principalcomponent analysis: Learning from examples without localminima. Neural networks, 2(1):53–58, 1989.

[23] E.H. Adelson and J.R. Bergen. Spatiotemporal energy modelsfor the perception of motion. J. Opt. Soc. Am. A, 2(2):284–299,1985.

[24] Roland Memisevic. Learning to relate images. IEEE Transac-tions on Pattern Analysis and Machine Intelligence, 35(8):1829–1846, 2013.

[25] Vinod Nair and Geoffrey E. Hinton. Rectified linear unitsimprove restricted boltzmann machines. In Johannes Frnkranzand Thorsten Joachims, editors, ICML, 2010.

[26] Kishore Reddy Konda, Roland Memisevic, and VincentMichalski. The role of spatio-temporal synchrony in theencoding of motion. arXiv preprint arXiv:1306.3162, 2013.

[27] Christopher M. Bishop. Pattern Recognition and Machine Learn-ing (Information Science and Statistics). Springer-Verlag NewYork, Inc., Secaucus, NJ, USA, 2006.

[28] TM Cover and J Thomas. Elements of Information Theory. NewYork: John Wiley & Sons, Inc, 1991.

[29] Geoffrey Hinton and Ruslan Salakhutdinov. Reducing thedimensionality of data with neural networks. Science,313(5786):504 – 507, 2006.

[30] R. Memisevic, C. Zach, G. Hinton, and M. Pollefeys. Gatedsoftmax classification. Advances in Neural Information ProcessingSystems (NIPS), 23, 2011.

[31] H. Larochelle, D. Erhan, A. Courville, J. Bergstra, and Y. Ben-gio. An empirical evaluation of deep architectures on prob-lems with many factors of variation. In International Conferenceon Machine Learning (ICML), 2007.

[32] H. Larochelle and Y. Bengio. Classification using discrimina-tive restricted Boltzmann machines. In International Conferenceon Machine Learning (ICML), 2008.

[33] Tanya Schmah, Geoffrey E Hinton, Steven L Small, StephenStrother, and Richard S Zemel. Generative versus discrimi-native training of rbms for classification of fmri images. InAdvances in neural information processing systems, pages 1409–1416, 2008.

[34] M. U. Gutmann and A. Hyvarinen. Noise-contrastive estima-tion of unnormalized statistical models, with applications tonatural image statistics. Journal of Machine Learning Research,13:307–361, March 2012.

[35] Graham Taylor and Geoffrey Hinton. Factored conditionalrestricted Boltzmann machines for modeling motion style.In Proceedings of the 26th International Conference on MachineLearning, 2009.

[36] Roland Memisevic and Geoffrey E Hinton. Learning torepresent spatial transformations with factored higher-orderBoltzmann machines. Neural Computation, 22(6):1473–92, 2010.

[37] Yann LeCun and Corinna Cortes. The MNIST database ofhandwritten digits, 1998.

[38] J. Bergstra, O. Breuleux, F. Bastien, P. Lamblin, R. Pascanu,G. Desjardins, J. Turian, D. Warde-Farley, and Y. Bengio.Theano: a CPU and GPU math expression compiler. In Pythonfor Scientic Computing Conference (SciPy), 2010.

[39] Andrew M Saxe, James L McClelland, and Surya Ganguli.Exact solutions to the nonlinear dynamics of learning in deeplinear neural networks. arXiv preprint arXiv:1312.6120, 2013.

[40] P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P.A. Man-zagol. Stacked denoising autoencoders: Learning useful repre-sentations in a deep network with a local denoising criterion.Journal of Machine Learning Research, 11:3371–3408, 2010.

Hanna Kamyshanska studied Mathemat-ics at the National Technical University ofUkraine, Kiew, and received the Diploma inComputer Science from the Goethe Univer-sity of Frankfurt, Germany, in 2013. She iscurrently a PhD candidate in ComputationalNeuroscience in Matthias Kaschube’s Groupat Frankfurt Institute for Advanced Studies.Her research interests are in computer visionand in the models for visual cortical develop-ment in biological systems.

Roland Memisevic received the PhD inComputer Science from the University ofToronto in 2008. Subsequently, he held po-sitions as a research scientist at PNYLabLLC in Princeton, as post-doctoral fellow atthe University of Toronto and ETH Zurich,and as a junior professor at the Universityof Frankfurt, Germany. In 2012, he joinedthe University of Montreal, Canada, as anassistant professor in Computer Science. Hisresearch interests are in machine learning

and computer vision.

Documents

IEEE TRANSACTIONS ON PATTERN ANALYSIS AND …memisevr/pubs/AEenergy.pdf · Hanna Kamyshanska, Roland Memisevic Abstract—Autoencoders are popular feature learning models, that are