Unsupervised Joint Alignment and Clustering using Bayesian Nonparametrics

Marwan A. Mattar Allen R. HansonDepartment of Computer Science

University of Massachusetts Amherst, USAhttp://vis-www.cs.umass.edu/

Erik G. Learned-Miller

Abstract

Joint alignment of a collection of functions is theprocess of independently transforming the func-tions so that they appear more similar to eachother. Typically, such unsupervised alignment al-gorithms fail when presented with complex datasets arising from multiple modalities or make re-strictive assumptions about the form of the func-tions or transformations, limiting their general-ity. We present a transformed Bayesian infinitemixture model that can simultaneously align andcluster a data set. Our model and associatedlearning scheme offer two key advantages: theoptimal number of clusters is determined in adata-driven fashion through the use of a Dirichletprocess prior, and it can accommodate any trans-formation function parameterized by a continu-ous parameter vector. As a result, it is applica-ble to a wide range of data types, and transfor-mation functions. We present positive results onsynthetic two-dimensional data, on a set of one-dimensional curves, and on various image datasets, showing large improvements over previouswork. We discuss several variations of the modeland conclude with directions for future work.

1 Introduction

Joint alignment is the process in which data points aretransformed to appear more similar to each other, basedon a criterion of joint similarity. The purpose of alignmentis typically to remove unwanted variability in a data set,by allowing the transformations that reduce that variability.This process is widely applicable in a variety of domains.For example, removing temporal variability in event re-lated potentials allows psychologists to better localize brainresponses [32], removing bias in magnetic resonance im-ages [21] provides doctors with cleaner images for their

Figure 1: Joint alignment and clustering: given 100 un-labeled images (top), without any other information, ouralgorithm (§ 3) chooses to represent the data with two clus-ters, aligns the images andclustersthem as shown (bot-tom). Our clustering accuracy is94%, compared to54%with K-means using two clusters (using the minimum erroracross 200 random restarts). Our model is not limited toaffine transformations or images.

analyses, and removing (affine) spatial variability in im-ages of objects can improve the performance of joint com-pression [9] and recognition [15] algorithms. Specifically,it has been found that using an aligned version of the La-beled Faces in the Wild [16] data set significantly increasesrecognition performance [5], even for algorithms that ex-plicitly handle misalignments. Aside from bringing datainto correspondence, the process of alignment can be usedfor other scenarios. For example, if the data are similarup to known transformations, joint alignment can removethis variability and, in the process, recover the underlyinglatent data [23]. Also, the resulting transformations fromalignment have been used to build classifiers using a singletraining example [21] and learn sprites in videos [17].

1.1 Previous Work

Typically what distinguishes joint alignment algorithms arethe assumptions they make about the data to be aligned

(1) Original (2) Congealing (3) Clustering (4) Alignment and Clustering

Figure 2: Illustrative example. (1) shows a data set of 2Dpoints. The set of allowable transformations is rotationsaround the origin. (2) shows the result of the congealingalgorithm which transforms points to minimize the sum ofthe marginal entropies. This independence assumption inthe entropy computation causes the points to be squeezedinto axis aligned groups. (3) highlights that clustering alonewith an infinite mixture model may result in a larger num-ber of clusters. (4) shows the result of the model presentedin this paper. It discovers two clusters and aligns the pointsin each cluster correctly. This result is very close to theideal one, which would have created tighter clusters.

and the transformations they can incur along with the levelof supervision needed. Supervision takes several formsand can range from manually selecting landmarks to bealigned [5] to providing examples of data transformations[28]. In this paper we focus on unsupervised joint align-ment which is helpful in scenarios where supervision is notpractical or available. Several such algorithms exist.

In the curve domain, thecontinuous profile model[23] usesa variant of the hidden Markov model to locally transformeach observation, while a mixture of regression model ap-pended with global scale and translation transformationscan simultaneously align and cluster [12]. Mattaret al. [25]adapted thecongealingframework [21] to one dimensionalcurves. Congealing is an alignment framework that makesfew assumptions about the data and allows the use of con-tinuous transformations. It is a gradient-descent optimiza-tion procedure that searches for the transformations param-eters that maximize the probability of the data under a ker-nel density estimate. Maximizing the likelihood is achievedby minimizing the entropy of the transformed data. It wasinitially applied to binary images of digits, but has sincealso been extended to grayscale images of complex ob-jects [15] and 3D brain volumes [33]. Additionally, sev-eral congealing variants [31, 30, 6] have been presentedthat can improve its performance on binary images of dig-its and simple grayscale images of faces. Also in the imagedomain, the transformed mixture of Gaussians [11] and thework of Lui et al. [24] are used to align and cluster.

One of the attractive properties of congealing is a clearseparation between the transformation operator and opti-mization procedure. This has allowed congealing to beapplied to a wide range of data types and transformationfunctions [1, 15, 21, 22, 25, 33]. Its main drawback is itsinability to handle complex data sets that may contain mul-tiple modes (i.e. images of the digits1 and7). While con-

gealing’s use of an entropy-based objective function can intheory allow it to align multiple modes, in practice the inde-pendence assumption (temporally for curves and spatiallyfor images) can cause it to collapse modes (see Figure 2for an illustration). Additionally, its method for regulariz-ing parameters to avoid excessive transformations isad hocand does not prevent it from annihilating the data (shrink-ing to size zero) in some scenarios.

1.2 Our Approach

The problem we address here is joint alignment of a dataset that may contain multiple groups or clusters. Previousnonparametric alignment algorithms (e.g. congealing [21])typically fail to acknowledge the multi-modality of the dataset resulting in poor performance on complex data sets. Weaddress this by simultaneously aligning and clustering [11,12, 24] the data set. As we will show (and illustrated inFigure 2), solving both alignment and clustering togetheroffers many advantages over clustering the data set first andthen aligning the points in each cluster.

To this end, we developed a nonparametric1 Bayesian jointalignment and clustering model that is a generalization ofthe standard Bayesian infinite mixture model. Our modelpossesses many of the favorable characteristics of congeal-ing, while overcoming its drawbacks. More specifically, it:

• Explicitly clusters the data which provides a mech-anism for handling complex data sets. Furthermore,the use of a Dirichlet process prior enables learningthe number of clusters in a data-driven fashion.

• Can use any generic transformation function param-eterized by a vector. This decouples our model fromthe specific transformations which allows us to plug indifferent functions for different data types.

• Enables the encoding of prior beliefs regarding the de-gree of variability in the data set, as well as regularizesthe transformation parameters in a principled way bytreating them as random variables.

We first present a Bayesian joint alignment model (§ 2) thatassumes a unimodal data set (i.e. only one cluster). Thismodel is a special case of our proposed joint alignment andclustering model that we introduce in§ 3. We then discussseveral variations of our model in§ 4 and conclude in§ 5with directions for future work.

1.3 Problem Definition

We are provided with a data setx = {xi}Ni=1 of N itemsand a transformation function,xi = τ(yi, ρi) parameter-

1Here, we use the term nonparametric to imply that the num-ber of model parameters can grow (a property of the infinite mix-tures), and not that the distributions are not parametric.

ized byρi. Our objective is to recover the set of transfor-mation parameters{ρi}Ni=1, such that the aligned data set{yi = τ(xi, ρ

−1i )}

Ni=1 is more coherent. In the process,

we also learn a clustering assignment{zi}Ni=1 of the datapoints. Hereρ−1i is defined as the parameter vector gener-ating the inverse of the transformation that would be gener-ated by the parameter vectorρ (i.e.xi = τ(τ(xi, ρ

−1i ), ρi)).

2 Bayesian Joint Alignment

The Bayesian alignment (BA) model assumes a unimodaldata set (in§ 3 this assumption is relaxed). Consequentlythere is a single set of parameters (θ andρ) that generate theentire data set (see Figure 3). Under this model, every ob-served data item,xi, is generated by transforming a canon-ical data item,yi, with transformation,ρi. More formally,xi = τ(yi, ρi), whereyi ∼ FD(θ) andρi ∼ FT (ϕ). Theauxiliary variableyi is not shown in the graphical modelfor simplicity. Given the Bayesian setting, the parametersθ andϕ are random variables, with their respective priordistributions,HD(λ) andHT (α).2

The model does not assume that there exists a single per-fect canonical example that explains all the data, but usesa parametric distributionFD(θ) to generate a slightly dif-ferent canonical example,yi, for each data item,xi. Thisenables it to explain variability in the data set that may notbe captured with the transformation function alone. Themodel treats the transformation function as a black-box op-eration, making it applicable to a wide range of data types(e.g. curves, images, and 3D MRI scans), as long as anappropriate transformation function is specified.

For both this model and the full joint alignment and clus-tering model introduced in the next section we use expo-nential family distributions forFD(θ) andFT (ϕ) and theirrespective conjugate priors forHT (α) andHD(λ). Thisallows us to use Rao-Blackwellized sampling schemes [4]by analytically integrating out the model parameters andcaching sufficient statistics for efficient likelihood compu-tations. Furthermore, the hyperparameters now play intu-itive roles where they act as a pseudo data set and are easierto set or learn from data.

2.1 Learning

Given a data set{xi}Ni=1 we wish to learn the parametersof this model ({ρi}Ni=1, θ, ϕ). We use a Rao-BlackwellizedGibbs sampler that integrates out the model parameters,θ andϕ, and only samples the hidden variables,{ρi}Ni=1.Such samplers typically speed-up convergence. The intu-ition is that the model parameters are implicitly updatedwith the sampling of every transformation parameter in-stead of once per Gibbs iteration. The resulting Gibbs sam-

2Here we assume that the hyperparametersα andλ are fixed,but they can be learned or sampled if necessary.

Figure 3: Graphical representation for our proposedBayesian alignment model (§ 2).

pler only iterates over the transformation parameters:

∀i=1:N ρ(t)i ∼ p(ρi|x,ρ

(t)−i, α, λ)

∝ p(ρi, xi|x−i,ρ(t)−i, α, λ)

= p(xi|ρi,x−i,ρ(t)−i, λ)p(ρi|ρ

(t)−i, α)

= p(yi|y(t)−i , λ)p(ρi|ρ

(t)−i, α),

whereyi = τ(xi, ρ−1i ). The t superscript in the above

equations refers to the Gibbs iteration number.

Samplingρi is complicated by the fact thatp(yi|y(t)−i , λ) de-

pends on the transformation function. Previous alignmentresearch [21, 24], has shown that the gradient of an align-ment objective function with respect to the transformationsprovides a strong indicator for how alignment should pro-ceed. One option would be Hamiltonian Monte Carlo sam-pling [26] which uses the gradient as a drift factor to influ-ence sampling. However, instead of relying on direct sam-pling techniques, we use approximations based on the pos-terior mode [13]. Such an approach is more direct since itis expected that the distribution will be tightly concentratedaround the mode. Thus, at each iteration the transformationparameter is updated as follows:

ρi = argmaxρi

p(yi|y(t)−i , λ)p(ρi|ρ

(t)−i, α).

Interestingly, the same learning scheme can be derived us-ing the incremental variant [27] of hard-EM.

2.2 Model Characteristics

The objective function optimized in our model containstwo key terms, a data term,p(x|ρ, θ), and a transforma-tion term,p(ρ|ϕ). The latter acts as a regularizer to pe-nalize large transformations and prevent the data from be-ing annihilated. One advantage of our model is that largetransformations are penalized in a principled fashion. Morespecifically, the cost of a transformation,ρi is based on the

Figure 4: Top row. Means before alignment. Bottom row.Means after alignment with BA. The averages of pixelwiseentropies are as follows. Before: 0.3 (top), with congeal-ing: 0.23 (not shown), and with BA: 0.21 (bottom).

learnedparameterϕ which depends on the transformationsof all the other data items,ρ

−i, and the hyperparameters,α. Learningϕ from the data is a more effective means forassigning costs than handpicking them.

The model has several other favorable qualities. It is ef-ficient, can operate on large data sets while maintaining alow memory footprint, allows continuous transformations,regularizes transformations in a principled way, is applica-ble to a large variety of data types, and its hyperparametersare intuitive to set. Its main drawback is the assumptionof a unimodal data set, which we remedy in§ 3. We firstevaluate this model on digit and curve alignment.

2.3 Experiments

Digits. We selected50 images of every digit from theMNIST data set and performed alignment on each digitclass independently. The mean images before and afteralignment are presented in Figure 4. We allowed7 affineimage transformations: scaling, shearing, rotating andtranslation.FD(θ) is the product of independent Bernoullidistributions, one for each pixel location, andFT (ϕ) is a7−D zero mean diagonal Gaussian. For comparison, wealso ran the congealing algorithm (see Figure 4).

Curves. We generated85 curve data sets in a manner sim-ilar to curve congealing [25], where we took five originalcurves from the UCR repository [18] and for each one gen-erated 17 data sets, each containing 50 random variationsof the original curve. We used the same transformationfunction in curve congealing [25], which allows non-lineartime warping (4 parameters), non-linear amplitude scaling(8 parameters), linear amplitude scaling (1 parameter), andamplitude translation (1 parameter).FD(θ) was set to a di-agonal Gaussian distribution (i.e. we treat the raw curvesas a random vector), andFT (ϕ) was a14−D zero meandiagonal Gaussian. Again, we compared against the curvecongealing algorithm. We computed a standard deviationscore by summing the standard deviation at each time stepof the final alignment produced by both algorithms. Fig-ure 5 shows a scatter plot of these scores obtained by con-gealing and BA for all 85 data sets, as well as sample align-ment results on two difficult cases. As the figure shows, thecurve data sets can be quiet complex.

0 0.10

0.1

with Bayesian Alignment

with

Con

geal

ing

Avg. std across time Original with Bayesian Alignment

Original with Congealing with Bayesian Alignment

Figure 5: Top row. Left: scatter plot of the standard de-viation score (see text) of congealing and the Bayesianalignment algorithm across the85 synthetic curve data sets.Middle: An example of a difficult data set. Right: Thealignment result of the difficult data set. The bottom rowshows an example where BA outperformed congealing.

Discussion.On the digits data sets, BA performed at leastas well as congealing for every digit class and on averageperformed better. On the curves data sets, BA does sub-stantially better than congealing in many cases, but in somecases congealing does slightly better. In all the experi-ments, both congealing and BA converged. BA’s advantageis largely due to its explicit regularization of transforma-tions which enables it to perform a maximization at eachiteration. Congealing’s lack of such regularization requiresit to take small steps at each iteration making it more sus-ceptible to local optima. Furthermore, congealing typicallyrequires five times the number of iterations to converge.

3 Clustering with Dirichlet Processes

We now extend the BA model introduced in the previoussection to explicitly cluster the data points. This providesa mechanism for handling complex data sets that may con-tain multiple groups.

The major drawback of the BA model is that a single pairof data and transformation parameters (θ andϕ, respec-tively) generate the entire data set. One natural extensionto this generative process is to assume that we have severalsuch parameter pairs (finite but unknown a priori) and eachdata point samples its parameter pair. By virtue of pointssampling the same parameter pair, they are assigned to thesame group or cluster. A Dirichlet process (DP) providesprecisely this construction and serves as the prior for thedata and transformation parameter pairs.

A DP essentially provides a distribution over distributions,or, more formally, a distribution on random probability

measures. It is parameterized by a base measure and a con-centration parameter. A draw from a DP generates a finiteset of samples from the base measure (the concentration pa-rameter controls the number of samples). A key advantageof DP’s is that the number of unique parameters (i.e. clus-ters) can grow and adapt to each data set depending on itssize and characteristics. Under this new probability model,data points are generated in the following way:

1. Sample from the DP,G ∼ DP (γ,Hα ×Hλ). γ is theconcentration parameter, andHα andHλ are the basemeasures forFT (ϕ) andFD(θ) respectively.

2. For each data point,xi, sample a data and transforma-tion parameter pair,(θi, ϕi) ∼ G.

3. Sample a transformation and canonical data item fromtheir distributions,yi ∼ FD(θi) andρi ∼ FT (ϕi).

4. Transform the canonical data item to generate the ob-served sample,xi = τ(yi, ρi).

Figure 6 depicts the generative process as described above(distributional form, right) and in the more traditionalgraphical representation with the cluster random variable,z, and mixture weights,π, made explicit (left).

Our model can thus be seen as an extension of the stan-dard Bayesian infinite mixture model where we introducedan additional latent variable,ρi, for each data point to rep-resent its transformation. Several existing alignment mod-els [11, 12, 21, 23] can be viewed as similar extensionsto other standard generative models. Sometimes the trans-formations are applied to other model parameters insteadof data points as in the case of transformed Dirichlet pro-cesses (TDP) [29]. TDP is an extension ofhierarchicalDirichlet processeswhere global mixture components aretransformed before being reused in each group. The chal-lenge in introducing additional latent variables is in design-ing efficient learning schemes that can accommodate thisincrease in model complexity.

3.1 Learning

We consider two different learning schemes for this model.The first is a blocked, Rao-Blackwellized Gibbs sampler,where we sample both the cluster assignmentzi, and trans-formation parametersρi, simultaneously:

(z(t)i , ρ

(t)i ) ∼ p(zi, ρi|z

(t)−i,ρ

(t)−i,x, γ, α, λ)

∝ p(zi|z(t)−i, γ)p(ρi|ρ

(t)−i, α)p(yi|y

(t)−i , λ).

As with the BA model, we approximatep(ρi|ρ

(t)−i, α)p(yi|y

(t)−i , λ) with a point estimate based

on its mode. Consequently this learning scheme is a directgeneralization of the one derived for the BA model. Note

Figure 6: Graphical representation for our proposed non-parametric Bayesian joint alignment and clustering model(left) and its corresponding distributional form (right).

thatp(zi|z(t)−i, γ) is the cluster predictive distribution based

on the Chinese restaurant process (CRP) [2].

While this sampler is effective (it produced the positive re-sult in Figure 1) it scales linearly with the number of clus-ters and computing the most likely transformation for acluster is an expensive operation. We designed an alter-native sampling scheme that does not require the expensivemode computation and whose running time is independentof the number of clusters.

Thesecondsampler further integrates out the transforma-tion parameter, and only samples the cluster assignment.We now derive an implementation for this sampler.

∀i=1:N z(t)i ∼ p(zi | z

(t)−i,x, γ, α, λ)

∝ p(zi, xi | z(t)−i,x−i, γ, α, λ)

= p(zi | z(t)−i, γ)p(xi | z

(t),x−i, α, λ).

p(xi | z,x−i, α, λ)

=

∫

θ

∫

ϕ

∫

ρi

p(xi, ρi,θ,ϕ | z,x−i, α, λ) dρi dϕ dθ

=

∫

θ

∫

ϕ

(∫

ρi

p(xi, ρi | zi,θ,ϕ, α, λ) dρi

)

· · ·

p(θ,ϕ | z−i,x−i, α, λ) dϕ dθ

(1)≈

∫

ρi

p(xi, ρi | zi, θ̂, ϕ̂, α, λ) dρi,

s.t. (θ̂, ϕ̂) = argmaxθ,ϕ

p(θ,ϕ | z−i,x−i, α, λ)

=

∫

ρi

p(ρi | ϕ̂, zi, α)p(xi | ρi, zi, θ̂, λ) dρi

=

∫

ρi

p(ρi | ϕ̂zi , α)p(xi | ρi, θ̂zi , λ) dρi

(2)≈

∑Ll=1 wi · p(xi | ρ̂i

l, θ̂zi , λ)∑L

l=1 wi

s.t. {ρ̂il}Ll=1 ∼ q(ρ), wi =

p(ρ̂il | ϕ̂zi , α)

q(ρ̂il)

(1) approximates the posterior distribution of the parame-ters by its mode. The mode is computed using incremen-tal hard-EM. Furthermore, the mode can be computed foreach cluster’s parameters independently. For every otherdata pointj, perform an EM update:

E: ρ̂j = argmaxρj

p(ρj |xj , θ̂zj , ϕ̂zj )

= argmaxρj

p(ρj | ϕ̂zj )p(xj | ρj , θ̂zj )

M: θ̂zj = argmaxθ

p(θ, | {xk, ρk | zk = zj}, λ)

ϕ̂zj = argmaxϕ

p(ϕ | {ρk | zk = zj}, α)

(2) uses importance sampling in order to reduce the numberof data transformations that need to be performed. Comput-ing p(xi | ρ̂i

l, θ̂zi , λ) requires transforming the data point,which is the most computationally expensive single oper-ation for this sampler. Thus it would be wise to reuse thesamples,{ρ̂i

l}Ll=1, across different clusters. We achievethis through importance sampling, which proceeds by sam-pling a set of transformation parameters from a proposaldistribution,q(ρ) and using those samples for all the clus-ters by reweighting them differently for each cluster. Thisis a large computational saving since the number of datatransformation operations performed in a single iterationofthis sampler is now independent of the number of clusters.Furthermore, the quality of approximation is controlled bythe number of samples,L, generated.

To further increase the efficiency of the sampler, weapproximate the maximization in the E-step by reusingthe samples and selecting the one that maximizesp(ρj |xj , θ̂zj , ϕ̂zj ). This avoids the direct maximizationoperation in the E-step which can be expensive. While notadopted in this work, further computational gains might beachieved at the expense of memory by storing and reusingsamples (i.e. transformed data points) across iterations andreweighting them accordingly.

Thus our sampler iterates over every point in the data set,samples a cluster assignment and then updatesθ̂ andϕ̂ forthe sampled cluster. It also updates its own transformationparameter,̂ρi in the process.

Summary. We presented two samplers for our joint align-ment and clustering model. Both samplers work well inpractice, but the second is more efficient. For both sam-plers, every iteration begins by randomly permuting the or-der of the points and the DP concentration parameter is re-sampled using auxiliary variable methods [10]. As in the

BA model, we cache the sufficient statistics for every clus-ter which can be updated efficiently as points are reassignedto clusters to allow for efficient likelihood and mode com-putation.

3.2 Incorporating Labelled Examples

The model presented in the previous section was used with-out any supervision. Supervision here refers to the ground-truth labels for some of the data points or the correct num-ber of clusters. However, there are many scenarios wherethis information is available and would be advantageous toincorporate.

It is straightforward to modify the joint alignment and clus-tering model to accommodate such labelled examples. Letsassume we have positive examples for each cluster as wellas a large data set of unlabeled examples. Before attempt-ing to align and cluster the unlabeled examples, we wouldinitialize several clusters and assign the positive examplesto their respective clusters. By assigning these examples totheir clusters and updating the sufficient statistics accord-ingly, the cluster parameters have incorporated the positiveexamples. Depending on the strength of the priors (i.e. thehyperparameters) and the number of positive examples percluster it may be necessary to add the positive examplesseveral times. The stronger the prior, the more times thepositive examples need to be replicated. Note that replicat-ing the positive examples does not increase memory usagesince we only store the sufficient statistics for each cluster.

If the labelled portion contains positive examples for allthe clusters, then setting the concentration parameter of theDP to0 would prevent additional, potentially unnecessary,clusters from being created.

3.3 Experiment: Alignment and Clustering of Digits

We evaluated our unsupervised and semi-supervised mod-els on two challenging data sets. The first contains100 im-ages of the digits “4” and “9”, which are the two most simi-lar and confusing digit classes (the performance of KMeanson this data set is close to random guessing). The sec-ond contains the 200 images of all 10 digit classes usedby Liu et al. [24].3 For the second data set we used theHistogram of Oriented Gradients (HOG) feature represen-tation [7] used by Liuet al. to enable a fair comparison.

For both digit data sets we compared several algorithms us-ing the same two metrics reported by Luiet al.: alignment

3Liu et al. also evaluated their model on 6 Caltech-256 cate-gories and the CEAS face data set. For both data sets they ran-domly selected 20 images from each category. We found that thedifficulty of a data set varied greatly from one sample to another,so we reached out to the authors. Unfortunately, they were onlyable to provide us with the digits data set which we do use. Thedigits data set was the most difficult of the three.

AlgorithmDigits 4 and 9 (Fig 1) All 10 digits (Fig 7)

Alignment Clustering Alignment Clustering

KMeans 4.18 (1.57)± 0.031 54.0% 4.88 (1.61)± 0.033 62.5%Infinite mixture model [10] 3.64 (1.34)± 0.036 86.0%, 4 4.87 (1.64)± 0.037 69.5%, 13Congealing [21] 2.11 (0.93)± 0.019 83.0% 3.51 (1.34)± 0.029 70.5%TIC [11] − − 6.00 (1.1) 35.5%Unsupervised SAC [24] − − 3.80 (0.9) 56.5%Semi-supervised SAC [24] − − not reported 73.7%Unsupervised JAC [§ 3.1] 1.44 (0.69)± 0.014 94.0%, 2 2.38 (1.12)± 0.027 87.0%, 12Semi-supervised JAC [§ 3.2] 1.58 (0.79)± 0.016 94.0% 2.71 (1.25)± 0.028 82.5%

Table 1: Joint alignment and clustering of images. The left subtable refers to the first digit data set comprising100images containing the digits “4” and “9” (Figure 1), while the right subtable refers to the second data set comprising200images containing all 10 digits (the same data set used by Liuet al. [24], Figure 7). The alignment score columns containthree metrics that adhere to the following template: mean (standard deviation)± standard error. The number followingthe clustering accuracy in the “Infinite mixture model” and “Unsupervised JAC” rows is the number of clusters that themodel discovered (i.e. chose to represent the data with). Onboth data sets, our models significantly outperforms previousnonparametric alignment [21], joint alignment and clustering [11, 24], and nonparametric Bayesian clustering [10] models.

scoremeasures the distance between pairs of aligned im-ages assigned to the same cluster (we report the mean andstandard deviation of all the distances, and the standard er-ror4), andclustering accuracyis the Rand index with re-spect to the correct labels.

Table 1 summarizes the results on the models we evaluated:

• KMeans: we clustered the digits into the correct num-ber of ground-truth classes (2 for the first data set, and10 for the second) using the best of 200 KMeans runs.

• Infinite mixture model: removing the transforma-tion/alignment component of our model reduces it to astandard Bayesian infinite mixture model. We ran thismodel to evaluate the advantage of joint alignment andclustering.

• Congealing: we ran congealing on all the imagessimultaneously and after alignment converged, clus-tered the aligned images using KMeans (with the cor-rect number of ground-truth clusters). This allows usto evaluate the advantages of simultaneous alignmentand clustering over alignment followed by clustering.

• TIC, USAC and SSAC results are listed exactly as re-ported by Liuet al.

• Unsupervised JAC refers to our full nonparametricBayesian alignment and clustering model (§ 3.1).

• Semi-supervised JAC refers to the semi-supervisedvariant of our alignment and clustering model (§ 3.2).We used asinglepositive example for each digit andset the DP concentration parameter to0.

4The standard error here is defined as the sample standard de-viation divided by the square root of the number of pairs.

Note that the alignment scores for KMeans and the infinitemixture model are not relevant since no alignment takesplace in either of these two algorithms. They are only in-cluded to offer a reference for the alignment score when thedata is not transformed.

As the results show, our models outperform previous workwith respect to both alignment and clustering quality. Wemake three observations about these results:

1. Our unsupervised model outperformed the unsuper-vised model of Liuet al. by 30.5%, and our semi-supervised model outperformed their semi-supervisedmodel by8.8%. This is in addition to the significantimprovement in alignment quality.

2. Our unsupervised model improved upon the standardinfinite mixture model in terms of alignment quality,clustering accuracy, and correctness of the discoverednumber of clusters.

3. The number of clusters discovered by our unsuper-vised model is quite accurate. For the first data set themodel discovered the correct number of clusters (seeFigure 1), and for the second it needed two additionalclusters (see Figure 7).

These positive results validate our joint alignment and clus-tering models and associated learning schemes. Further-more, it provides evidence for the advantage of solvingboth alignment and clustering problems simultaneously in-stead of independently.

3.4 Experiment: Alignment and Clustering of Curves

We now present joint alignment and clustering results ona challenging curve data set of ECG heart data [19] that is

Figure 7: Unsupervised joint alignment and clustering of 200 images of all 10 digits. (Top) All 200 images provided to ourmodel. (Bottom) The 12 clusters discovered and their alignments.

helpful in identifying the heart condition of patients. Thisdata set contains46 curves. 24 represent a normal heart-beat, and22 represent an abnormal heartbeat. We ran bothcongealing and our nonparametric Bayesian joint align-ment and clustering model. In both cases we excludedthe non-linear scaling in amplitude transformation since theamplitudes of the curves are helpful in classifying whetherthe curve is normal or abnormal.

Our model discovered5 clusters in the data set resultingin a clustering accuracy of84.8%. Inspecting the clustersdiscovered by our model in Figure 8 highlights the fact thatalthough the data set represents two groups (normal andabnormal), the curves do not naturally fall into two clus-ters and more are needed to explain the data appropriately.Figure 8 also displays the result of congealing the curves.Clustering the congealed curves into2 clusters using thebest of200 KMeans runs results in a clustering accuracy of71.7%. Clustering the congealed curves into5 clusters in asimilar manner results in a clustering accuracy of76.1%.

The large improvement in clustering accuracy over con-gealing in addition to a much cleaner alignment result (Fig-ure 8) highlights the importance of explicit clustering whenpresented with a complex data set. Furthermore it show-cases our models ability to perform equally well on bothimage and curve data sets.

4 Discussions

In this section we discuss the adaptation of our joint align-ment and clustering model to both online (when the data ar-rives at intervals) and distributed (when multiple processorsare available) settings. Both of these adaptations are appli-cable to the unsupervised and semi-supervised settings.

4.1 Online Learning

There are several scenarios where online alignment andclustering may be helpful. Consider for instance a verylarge data set that cannot fit in memory or the case wherethe data set is not available up front but arrives over an ex-tended period of time (such as in a tracking application).

An advantage of our model that has not yet been raised is itsability to easily adapt to an online setting where only a por-tion of the data set is available in the beginning. This is dueto our use of conjugate priors and distributions in the expo-nential family which enable us to efficiently summarize anentire cluster through its sufficient statistics. Consequently,we can align/cluster the initial portion of the data set andsave out the sufficient statistics for every cluster after eachiteration (for both the data and transformations). Then asnew data arrives, we can load in the sufficient statistics anduse them to guide the alignment and clustering of the new

50 100 150 200 250−3

−2

−1

0

1

2

3

4Original Data Set

50 100 150 200 250−3

−2

−1

0

1

2

3

4with Congealing

50 100 150 200 250−3

−2

−1

0

1

2

3

4with JAC, cluster 1 of 5

50 100 150 200 250−3

−2

−1

0

1

2

3

4with JAC, cluster 2 of 5

50 100 150 200 250−3

−2

−1

0

1

2

3

4with JAC, cluster 3 of 5

50 100 150 200 250−3

−2

−1

0

1

2

3

4with JAC, cluster 4 of 5

50 100 150 200 250−3

−2

−1

0

1

2

3

4with JAC, cluster 5 of 5

Figure 8: Joint alignment and clustering of ECG heart data.The first row displays the original data set (left) and theresult of congealing (right). The last three rows display the5 clusters discovered by our model.

data in lieu of the original data set which can now be dis-carded.

Given a sufficiently large initial data set, the alignment ofanew data point using the procedure described above wouldbe nearly identical to the result had that data point beenincluded in the original set. This is true since the additionof a single point to an already large data set would have anegligible effect on the sufficient statistics. This process isalso applicable to the Bayesian alignment model.

4.2 Distributed Learning

We now describe how to adapt our sampling scheme to adistributed setting using the MapReduce [8] framework.This facilitates scaling our model to large data sets inthe presence of many processors. The key difference be-

tween the MapReduce implementation and the one de-scribed in§ 3.1 is that the cluster parameters are updatedonce per sampling iteration instead of after each point’s re-assignment (i.e. using a standard sampler instead of a Rao-Blackwellized sampler).

A MapReduce framework involves two key steps, Map andReduce. For our model the mapper would handle updatingthe transformation parameter and clustering assignment ofa single data point, while the reducer would handle updat-ing the parameters of a single cluster. More specifically, theinput to each Map operation would be a data point alongwith a snapshot of the model parameters (the set of suf-ficient statistics that summarize the data set). The Mapwould output the updated cluster assignment and transfor-mation parameter for that data point. The input to the Re-duce step would then be all the data points that were as-signed to a specific cluster (i.e. we would have a Reduceoperation for every cluster created). The Reducer wouldthen update the cluster parameters. Thus each sampling it-eration is composed of a Map and Reduce stage.

5 Conclusion

We presented a nonparametric Bayesian joint alignmentand clustering model that has been successfully applied tocurve and image data sets. The model outperforms con-gealing and yields impressive gains in clustering accuracyover infinite mixture models. These results highlight theadvantage of solving both alignment and clustering taskssimultaneously.

A strength of our model is the separation of the transforma-tion function and sampling scheme, which makes it appli-cable to a wide range of data types, feature representations,and transformation functions. In this paper we presentedresults on three data types (2D points, 1D curves, and im-ages), three transformation functions (point rotations, non-linear curve transformations and affine image transforma-tions), and two feature representations (identity and HOG).

In the future we foresee our model applied to a wide ar-ray of problems. Since curves are a natural representationfor object boundaries [18], one of our goals is to apply ourmodel to shape matching. We also intend to explore al-ternative parameter learning schemes based on variationalinference [3, 20, 14].

Acknowledgements

Special thanks to Gary Huang for many entertaining andhelpful discussions. We would also like to thank SameerSingh and the anonymous reviewers for helpful feedback.This work was supported in part by the National ScienceFoundation under CAREER award IIS-0546666 and GrantIIS-0916555.

References

[1] P. Ahammad, C. Harmon, A. Hammonds, S. Sastry, andG. Rubin. Joint nonparametric alignment for analyzing spa-tial gene expression patterns of Drosophila imaginal discs.In IEEE Conference on Computer Vision and Pattern Recog-nition, 2005.

[2] D. Blackwell and J. MacQueen. Ferguson distributions viaPólya urn schemes.The Annals of Statistics, 1:353–355,1973.

[3] D. M. Blei and M. I. Jordan. Variational inference forDirichlet process mixtures.Journal of Bayesian Analysis,1(1):121–144, 2006.

[4] G. Cassella and C. P. Robert. Rao-Blackwellization of sam-pling schemes.Biometrika, 83(1):81–94, 1996.

[5] T. F. Cootes, G. J. Edwards, and C. J. Taylor. Active appear-ance models.IEEE Transactions on Pattern Analysis andMachine Intelligence, 2003.

[6] M. Cox, S. Lucey, J. Cohn, and S. Sridharan. Least squarescongealing for unsupervised alignment of images. InIEEEConference on Computer Vision and Pattern Recognition,2008.

[7] N. Dalal and B. Triggs. Histograms of oriented gradients forhuman detection. InIEEE Conference on Computer Visionand Pattern Recognition, 2005.

[8] J. Dean and S. Ghemawat. MapReduce: Simplified dataprocessing on large clusters. InSymposium on OperatingSystem Design and Implementation, 2004.

[9] M. Elad, R. Goldenberg, and R. Kimmel. Low bit-rate com-pression of facial images.IEEE Transactions on ImageAnalysis, 9(16):2379–2383, 2007.

[10] M. D. Escobar and M. West. Bayesian density estimationand inference using mixtures.Journal of the American Sta-tistical Association, 90(430):577–588, June 1995.

[11] B. J. Frey and N. Jojic. Transformation-invariant clusteringand dimensionality reduction using EM.IEEE Transactionson Pattern Analysis and Machine Intelligence, 25(1):1–17,Jan. 2003.

[12] S. Gaffney and P. Smyth. Joint probabilistic curve clusteringand alignment. InNeural Information Processing Systems,2004.

[13] A. Gelman, J. B. Carlin, H. S. Stern, and D. B. Rubin.Bayesian Data Analysis. Chapman and Hall, CRC Press,2003.

[14] R. Gomes, M. Welling, and P. Perona. Incremental learningof nonparametric Bayesian mixture models. InIEEE Con-ference on Computer Vision and Pattern Recognition, 2008.

[15] G. B. Huang, V. Jain, and E. G. Learned-Miller. Unsuper-vised joint alignment of complex images. InIEEE Interna-tional Conference on Computer Vision, 2007.

[16] G. B. Huang, M. A. Mattar, T. Berg, and E. G. Learned-Miller. Labeled Faces in the Wild: A database for studyingface recognition in unconstrained environments. InIEEEECCV Workshop on Faces in Real-Life Images, 2008.

[17] N. Jojic and B. J. Frey. Learning flexible sprites in videolayers. InIEEE Conference on Computer Vision and PatternRecognition, 2001.

[18] E. Keogh, X. Xi, L. Wei, and C. A. Ratanamahatana. UCRTime Series Repository, Feb. 2008.

[19] S. Kim and P. Smyth. Segmental hidden Markov modelswith random effects for waveform modeling.Journal of Ma-chine Learning Research, 7:945–969, 2006.

[20] K. Kurihara, M. Welling, and Y. W. Teh. Collapsed vari-ational Dirichlet process mixture models. InInternationalJoint Conference on Artificial Intelligence, 2007.

[21] E. G. Learned-Miller. Data-driven image models throughcontinuous joint alignment.IEEE Transactions on PatternAnalysis and Machine Intelligence, 28(2):236–250, Feb.2006.

[22] E. G. Learned-Miller and P. Ahammad. Joint MRI bias re-moval using entropy minimization across images. InNeuralInformation Processing Systems, 2004.

[23] J. Listgarten, R. M. Neal, S. T. Roweis, and A. Emili. Mul-tiple alignment of continuous time series. InNeural Infor-mation Processing Systems, 2004.

[24] X. Liu, Y. Tong, and F. W. Wheeler. Simultaneous alignmentand clustering for an image ensemble. InIEEE InternationalConference on Computer Vision, 2009.

[25] M. A. Mattar, M. G. Ross, and E. G. Learned-Miller. Non-parametric curve alignment. InIEEE International Confer-ence on Acoustics, Speech and Signal Processing, 2009.

[26] R. M. Neal. MCMC using Hamiltonian dynamics. InHand-book of Markov Chain Monte Carlo. Chapman and Hall,CRC Press, May 2011.

[27] R. M. Neal and G. E. Hinton. A view of the EM algo-rithm that justifies incremental, sparse, and other variants.In Learning in Graphical Models, pages 355–368, 1998.

[28] A. Rahimi and B. Recht. Learning to transform time serieswith a few examples.IEEE Transactions on Pattern Analy-sis and Machine Intelligence, 29(10):1759–1775, Oct. 2007.

[29] E. B. Sudderth, A. Torralba, W. T. Freeman, and A. S. Will-sky. Describing visual scenes using transformed Dirichletprocesses. InNeural Information Processing Systems, 2005.

[30] A. Vedaldi, G. Guidi, and S. Soatto. Joint alignment up to(lossy) transformations. InIEEE Conference on ComputerVision and Pattern Recognition, 2008.

[31] A. Vedaldi and S. Soatto. A complexity-distortion approachto joint pattern alignment. InNeural Information ProcessingSystems, 2006.

[32] K. Wang, H. Begleiter, and B. Projesz. Warp-averaging event-related potentials.Clinical Neurophysiol-ogy, (112):1917–1924, 2001.

[33] L. Zollei, E. G. Learned-Miller, E. Grimson, and W. M.Wells. Efficient population registration of 3D data. InIEEEICCV Workshop on Computer Vision for Biomedical ImageApplications: Current Techniques and Future Trends, 2005.