Learning Mixtures of Smooth Product Distributions:Identifiability and Algorithm

Nikos Kargas Nicholas D. SidiropoulosUniversity of Minnesota University of Virginia

Abstract

We study the problem of learning a mix-ture model of non-parametric product dis-tributions. The problem of learning a mix-ture model is that of finding the componentdistributions along with the mixing weightsusing observed samples generated from themixture. The problem is well-studied in theparametric setting, i.e., when the componentdistributions are members of a parametricfamily – such as Gaussian distributions. Inthis work, we focus on multivariate mixturesof non-parametric product distributions andpropose a two-stage approach which recoversthe component distributions of the mixtureunder a smoothness condition. Our approachbuilds upon the identifiability properties ofthe canonical polyadic (low-rank) decompo-sition of tensors, in tandem with Fourier andShannon-Nyquist sampling staples from sig-nal processing. We demonstrate the effective-ness of the approach on synthetic and realdatasets.

1 Introduction

Learning mixture models is a fundamental problem instatistics and machine learning having numerous ap-plications such as density estimation and clustering.In this work, we consider the special case of mixturemodels whose component distributions factor into theproduct of the associated marginals. An example isa mixture of axis-aligned Gaussian distributions, animportant class of Gaussian Mixture Models (GMMs).Consider a scenario where different diagnostic tests areapplied to patients, and test results are assumed to be

Proceedings of the 22nd International Conference on Ar-tificial Intelligence and Statistics (AISTATS) 2019, Naha,Okinawa, Japan. PMLR: Volume 89. Copyright 2019 bythe author(s).

independent conditioned on the binary disease statusof the patient which is the latent variable. The jointProbability Density Function (PDF) of the tests canbe expressed as a weighted sum of two components,and each component factors into the product of uni-variate marginals. Fitting a mixture model to an unla-beled dataset allows us to cluster the patients into twogroups by determining the value of the latent variableusing the Maximum a Posteriori (MAP) principle.

Most of the existing literature in this area has fo-cused on the fully-parametric setting, where the mix-ture components are members of a parametric fam-ily, such as Gaussian distributions. The most popu-lar algorithm for learning a parametric mixture modelis Expectation Maximization (EM) (Dempster et al.,1977). Recently, methods based on tensor decomposi-tion and particularly the Canonical Polyadic Decom-position (CPD) have gained popularity as an alterna-tive to EM for learning various latent variable mod-els (Anandkumar et al., 2014). What makes the CPDa powerful tool for data analysis is its identifiabilityproperties, as the CPD of a tensor is unique under rel-atively mild rank conditions (Sidiropoulos et al., 2017).

In this work we propose a two-stage approach basedon tensor decomposition for recovering the conditionaldensities of mixtures of smooth product distributions.We show that when the unknown conditional densi-ties are approximately band-limited it is possible touniquely identify and recover them from partially ob-served data. The key idea is to jointly factorize his-togram estimates of lower-dimensional PDFs that canbe easily and reliably estimated from observed sam-ples. The conditional densities can then be recoveredusing an interpolation procedure. We formulate theproblem as a coupled tensor factorization and proposean alternating-optimization algorithm. We demon-strate the effectiveness of the approach on both syn-thetic and real data.

Notation: Bold, lowercase, x, and uppercase letters,X, denote vectors and matrices respectively. Bold, un-derlined, uppercase letters, X, denote N -way (N ≥ 3)tensors. We use the notation x[i], X[i, j], X[i, j, k] to

Learning Mixtures of Smooth Product Distributions

refer to specific elements of a vector, matrix and ten-sor respectively. We denote the vector obtained byvertically stacking the columns of the tensor X into avector by vec(X). Additionally, diag(x) ∈ RM×M de-notes the diagonal matrix with the elements of vectorx ∈ RM on its diagonal. The set of integers {1, . . . , N}is denoted as [N ]. Uppercase, X, and lowercase let-ters, x, denote scalar random variables and realizationsthereof, respectively.

2 Background

2.1 Canonical Polyadic Decomposition

In this section, we briefly introduce basic conceptsrelated to tensor decomposition. An N -way tensorX ∈ RI1×I2×···×IN is a multidimensional array whoseelements are indexed by N indices. A polyadic decom-position expresses X as a sum of rank-1 terms

X =

R∑r=1

A1[:, r] ◦A2[:, r] ◦ · · · ◦AN [:, r], (1)

where An ∈ RIn×R, 1 ≤ r ≤ R, An[:, r] denotes ther-th column of matrix An and ◦ denotes the outerproduct. If the number of rank-1 terms is minimal,then Equation (1) is called the CPD of X and R iscalled the rank of X. Without loss of generality, wecan restrict the columns of {An}Nn=1 to have unit normand have the following equivalent expression

X =

R∑r=1

λ[r]A1[:, r] ◦A2[:, r] ◦ · · · ◦AN [:, r], (2)

where ‖An[:, r]‖p = 1 for a certain p ≥ 1, ∀ n, r,

and λ = [λ[1], . . . ,λ[R]]T

‘absorbs’ the norms ofcolumns. For convenience, we use the shorthand nota-tion X = [[λ,A1, . . . ,AN ]]R. We can express the CPDof a tensor in a matricized form. With � denoting theKhatri-Rao (columnwise Kronecker) matrix product,it can be shown that the mode-n matrix unfolding ofX is given by

X(n) =

N�k=1k 6=n

Ak

diag(λ)ATn , (3)

whereN�k=1k 6=n

Ak = AN � · · · �An+1 �An−1 � · · · �A1.

The CPD can be expressed in a vectorized form as

vec(X) =

(N�n=1

An

)λ. (4)

It is clear that the rank-1 terms can be arbitrarily per-muted without affecting the decomposition. We saythat a CPD of a tensor is unique when it is only sub-ject to this trivial indeterminacy.

2.2 Learning Problem

Let X = {Xn}Nn=1 denote a set of N random variables.We say that a PDF fX is a mixture of R componentdistributions if it can be expressed as a weighted sumof R multivariate distributions

fX (x1, . . . , xN ) =

R∑r=1

wrfX|H(x1, . . . , xN |r), (5)

where fX|H are conditional PDFs and {wr}Rr=1 are

non-negative numbers such that∑Rr=1 wr = 1, called

mixing weights. When each conditional PDF factorsinto the product of its marginal densities we have that

fX (x1, . . . , xN ) =

R∑r=1

wr

N∏n=1

fXn|H(xn|r), (6)

which can be seen as a continuous extension of theCPD model of Equation (2). A sample from the mix-ture model is generated by first drawing a componentr according to w and then independently drawing sam-ples for every variable {Xn}Nn=1 from the conditionalPDFs fXn|H(·|r). The problem of learning the mix-ture is that of finding the conditional PDFs as well asthe mixing weights given observed samples.

2.3 Related Work

Mixture models have numerous applications in statis-tics and machine learning including clustering anddensity estimation to name a few (McLachlan andPeel, 2000). A common assumption made in multi-variate mixture models is a parametric form of theconditional PDFs. For example, when the conditionalPDFs are assumed to be Gaussian, the goal is to re-cover the mean vectors and covariance matrices defin-ing each multivariate Gaussian component and themixing weights. Other common choices include cate-gorical, exponential, Laplace or Poisson distributions.The most popular algorithm for learning the param-eters of the mixture is the EM algorithm (Dempsteret al., 1977) which maximizes the likelihood functionwith respect to the parameters. EM-based methodshave been also considered for learning mixture mod-els of non-parametric distributions1 by parameterizingthe unknown conditional PDFs using kernel densityestimators (Benaglia et al., 2009; Levine et al., 2011),which lack however theoretical guarantees.

Tensor decomposition methods can be used as an al-ternative to EM for learning various latent variablemodels (Anandkumar et al., 2014). High-order mo-ments of several probabilistic models can be expressed

1The term non-parametric is used to describe the casein which no assumptions are made about the form of theconditional densities.

Nikos Kargas, Nicholas D. Sidiropoulos

using low-rank CPDs. Decomposing these tensors re-veals the true parameters of the probabilistic models.In the absence of noise and model mismatch, algebraicalgorithms can be applied to compute the CPD un-der certain conditions, see (Sidiropoulos et al., 2017)and references therein, and (Hsu and Kakade, 2013)for the application to GMMs. Tensor decompositionapproaches have been proposed for learning mixturemodels but are mostly restricted to Gaussian or cate-gorical distributions (Hsu and Kakade, 2013; Jain andOh, 2014; Gottesman et al., 2018). In practice, mainlydue to sampling noise the result of these algorithmsmay not be satisfactory and EM can be used for re-finement (Zhang et al., 2014; Ruffini et al., 2017). Inthe case of non-parametric mixtures of product dis-tributions, identifiability of the components has beenestablished in (Allman et al., 2009). The authorshave shown that it is possible to identify the condi-tional PDFs given the true joint PDF, if the condi-tional PDFs of each Xn across different mixture com-ponents are linearly independent i.e., the continuousfactor “matrices” have linearly independent columns.However, the exact true joint PDF is never given –only samples drawn from it are available in practice,and elements may be missing from any given sample.Furthermore, (Allman et al., 2009) did not provide anestimation procedure, which limits the practical ap-peal of an interesting theoretical contribution.

In this work, we focus on mixtures of product distribu-tions of continuous variables and do not specify a para-metric form of the conditional density functions. Weshow that it is possible to recover mixtures of smoothproduct distributions from observed samples. The keyidea is to first transform the problem to that of learn-ing a mixture of categorical distributions by decom-posing lower-dimensional and (possibly coarsely) dis-cretized joint PDFs. Given that the conditional PDFsare (approximately) band-limited (smooth), they canbe recovered from the discretized PDFs under certainconditions.

3 Approach

Our approach consists of two stages. We express theproblem as a tensor factorization problem and showthat if N ≥ 3, we can recover points of the unknownconditional CDFs. Under a smoothness condition,these points can be used to recover the true condi-tional PDFs using an interpolation procedure.

3.1 Problem Formulation

We assume that we are given M N -dimensionalsamples {xm}Mm=1 that have been generated froma mixture of product distributions as in Equa-

tion (6). We discretize each random variable Xn

by partitioning its support into I uniform inter-vals {∆i

n =(di−1n , din

)}1≤i≤I . Specifically, we

consider a discretization of the PDF and definethe probability tensor (histogram) X[i1, . . . , iN ] ,Pr(X1 ∈ ∆i1

n , . . . , XN ∈ ∆iNn

)given by

X[i1, . . . , iN ] =

R∑r=1

wr

N∏n=1

∫∆inn

fXn|H(xn|r)dxn

=

R∑r=1

wr

N∏n=1

Pr(Xn ∈ ∆in

n

∣∣H = r). (7)

Let An[in, r] , Pr(Xn ∈ ∆in

n

∣∣H = r), λ[r] , wr.Note that X is an N -way tensor and admits a CPDwith non-negative factor matrices {An}Nn=1 and rankR, i.e., X = [[λ,A1, . . . ,AN ]]R. From equation (7) it isclear that the discretized conditional PDFs are identi-fiable and can be recovered by decomposing the truejoint discretized probability tensor, if N ≥ 3 and R issmall enough, by virtue of the uniqueness propertiesof CPD (Sidiropoulos et al., 2017).

In practice we do not observe X but we have to dealwith perturbed versions. Based on the observed sam-ples, we can compute an approximation of the proba-bility tensor X by counting how many samples fall intoeach bin and normalizing the tensor by dividing withthe total number of samples. The size of the proba-bility tensor grows exponentially with the number ofvariables and therefore the estimate will be highly in-accurate even when the number of discretization inter-vals is small. More importantly, datasets often containmissing data and therefore its impossible to constructsuch tensor. On the other hand, it may be possibleto estimate low-dimensional discretized joint PDFs ofsubsets of the random variables which correspond tolow-order tensors. For example, in the clustering ex-ample given in the introduction some patients may betested on a subset of the available tests. Finally, themodel of Equation (7) is just an approximation of ouroriginal model, as our ultimate goal is to recover thetrue conditional PDFs. To address the aforementionedchallenges we have to answer the following two ques-tions

1. Is it possible to learn the mixing weights and dis-cretized conditional PDFs from missing/limiteddata?

2. Is it possible to recover non-parametric condi-tional PDFs from their discretized counterparts?

Regarding the first question, it has been recentlyshown that a joint Probability Mass Function (PMF)of a set of random variables can be recovered from

Learning Mixtures of Smooth Product Distributions

lower-dimensional joint PMFs if the joint PMF haslow enough rank (Kargas et al., 2018). This resultallows us to recover the discretized conditional PDFsfrom low-dimensional histograms but cannot be ex-tended to the continuous setting in general because ofthe loss of information induced from the discretizationstep. We further discuss and provide conditions underwhich we can overcome these issues.

3.2 Identifiability using Lower-dimensionalStatistics

In this section we provide insights regarding thefirst issue. It turns out that realizations of sub-sets of only three random variables are sufficient torecover Pr

(Xn ∈ ∆in

n

∣∣H = r) and {wr}Rr=1. Un-der the mixture model (6), a histogram of anysubset of three random variables Xj , Xk, X` de-noted as Xjk`, with Xjk`[ij , ik, i`] = Pr(Xj ∈∆ijj , Xk ∈ ∆ik

k , X` ∈ ∆i`` ) can be written as

Xjk`[ij , ik, i`] =∑Rr=1 λ[r]Aj [ij , r]Ak[ik, r]A`[i`, r],

which is a third-order tensor of rank R. A fun-damental result on the uniqueness of tensor de-composition of third-order tensors was given byin (Kruskal, 1977). The result states that if Xadmits a decomposition X = [[λ,A1,A2,A3]]R, withkA1

+ kA2+ kA3

≥ 2R+ 2 then rank(X) = R and thedecomposition of X is unique. Here, kA denotes theKruskal rank of the matrix A which is equal to thelargest integer such that every subset of kA columnsare linearly independent. When the rank is smalland the decomposition is exact, the parameters of theCPD model can be computed exactly via GeneralizedEigenvalue Decomposition (GEVD) and related alge-braic algorithms (Leurgans et al., 1993; Domanov andLathauwer, 2014; Sidiropoulos et al., 2017).

Theorem 1 (Leurgans et al., 1993) Let X bea tensor that admits a polyadic decompositionX = [[λ,A1,A2,A3]]R, A1 ∈ RI1×R, A2 ∈ RI2×R,A3 ∈ RI3×R, λ ∈ RR and suppose that A1, A2 arefull column rank and kA3 ≥ 2. Then rank(X) = R,the decomposition of X is unique and can be found al-gebraically.

More relaxed uniqueness conditions from the field ofalgebraic geometry have been proven in recent years.

Theorem 2 (Chiantini and Ottaviani, 2012) Let Xbe a tensor that admits a polyadic decomposition X =[[λ,A1,A2,A3]], where A1 ∈ RI1×F , A2 ∈ RI2×F ,A3 ∈ RI3×F , I1 ≤ I2 ≤ I3. Let α, β be the largest in-tegers such that 2α ≤ I1 and 2β ≤ I2. If F ≤ 2α+β−2

then the decomposition of X is essentially unique al-most surely.

Theorem 2 is a generic uniqueness result i.e, all non-identifiable parameters form a set of Lebesgue mea-sure zero. To see how the above theorems can beapplied in our setup, consider the joint decomposi-tion of the probability tensors Xjk`. Let S1, S2, andS3 denote disjoint ordered subsets of [N ], with car-dinality c1 = |S1|, c2 = |S2|, and c3 = |S3|, re-spectively. Let Y be the c1 × c2 × c3 block tensorwhose (j, k, `)-th block is the tensor Xjk`, j ∈ S1,k ∈ S2, ` ∈ S3. It is clear that the tensor Yadmits a CPD Y = [[λ, A1, A2, A3]]R where A1 =

[ATS1(1), · · · ,A

TS1(c1)]

T , A2 = [ATS2(1), · · · ,A

TS2(c2)]

T ,

A3 = [ATS3(1), · · · ,A

TS3(c3)]

T . By considering the jointdecomposition of lower-dimensional discretized PDFs,we have constructed a single virtual non-negativeCPD model and therefore uniqueness properties hold.For example, by setting S1 = {1, . . . , bN−1

2 c − 1},S2 = {bN−1

2 c, . . . , N − 1}, S3 = {N} we have that

Y(1) =

AbN−1

2 c...

AN−1

� A1

...AbN−1

2 c−1

diag(λ)AT

N .

According to Theorem 1, the CPD can be computedexactly if R ≤ (bN−1

2 c − 1)I. Similarly, it is easy

to verify that by setting c1 = c2 = bN3 cI, i.e., α =

blog2(bN3 cI)c, the CPD of Y is generically unique for

R ≤ 22(α−1) according to Theorem 2. The later in-

equality is implied by R ≤ (bN3 cI+1)2

16 which shows thatthe bound is quadratic in N and I.

Remark 1: The previous discussion suggests thatfiner discretization can lead to improved identifiabil-ity results. The number of hidden components may bearbitrarily large and we may still be able to identify thediscretized conditional PDFs by increasing the dimen-sions of the sub-tensors i.e., the discretization intervalsof the random variables. The caveat is that one willneed many more samples to reliably estimate these his-tograms. Ideally, one would like to have the minimumnumber of intervals that can guarantee identifiabilityof the conditional PDFs.

Remark 2: The factor matrices can be recoveredby decomposing the lower-order probability tensorsof dimension N ≥ 3. It is important to note thathistograms of subsets of two variables correspond toNon-negative Matrix Factorization (NMF) which isnot identifiable unless additional conditions such assparsity are assumed for the latent factors (Fu et al.,2018). Therefore, second-order distributions are notsufficient for recovering dense latent factor matrices.

3.3 Recovery of the Conditional PDFs

In the previous section we have shown that givenlower-dimensional discretized PDFs, we can uniquely

Nikos Kargas, Nicholas D. Sidiropoulos

identify and recover discretized versions of the condi-tional PDFs via joint tensor decomposition. Recov-ering the true conditional PDFs from the discretizedcounterparts can be viewed as a signal reconstructionproblem. We know that this is not possible unless thesignals have some smoothness properties. We will usethe following result.

Proposition 1 A PDF that is (approximately) band-limited with cutoff frequency ωc can be recovered fromuniform samples of the associated CDF taken π

ωcapart.

Proof : Assume that the PDF fX is band-limitedwith cutoff frequency ωc i.e., its Fourier transformF(ω) = 0, ∀ |ω| ≥ ωc. Let FX denote the CDF of fX ,FX(x) =

∫ x−∞ fX(t)dt. We can express the integration

as a convolution of the PDF with a unit step func-tion, i.e., FX(x) =

∫∞−∞ fX(t)u(x − t)dτ . The Fourier

transform of a convolution is the point-wise productof Fourier transforms. Therefore, we can express theFourier transform G(ω) of the CDF as

G(ω) = πδ(ω)F(0) +F(ω)

jω, (8)

where δ(·) is the Dirac delta. From Equation (8), itis clear that the CDF obeys the same band-limit asthe PDF . From Shannon’s sampling theorem we havethat

FX(x) =

∞∑n=−∞

FX(nT ) sinc

(x− nTT

), (9)

where T = πωc

. The PDF can then be determined bydifferentiation, which amounts to linear interpolationof the CDF samples using the derivative of the sinckernel. Note that for exact reconstruction of fX aninfinite number of data points are needed. In signalprocessing practice we always deal with finite supportsignals which are only approximately band-limited; thepoint is that the bandlimited assumption is accurateenough to afford high-quality signal reconstruction. Inour present context, a good example is the Gaussiandistribution: even though it is of infinite extent, it isnot strictly bandlimited (as its Fourier transform is an-other Gaussian); but it is approximately bandlimited,and that is good enough for our purposes, as we willsee shortly.

In section 3.2, we saw how lower-dimensional his-tograms can be used to obtain estimates of the dis-cretized conditional PDFs. Now, consider the condi-tional PDF of the n-th variable given the r-th compo-nent. The corresponding column of factor matrix An

is

An[:, r] = [FXn|H(d1n|r)− FXn|H(d0

n|r), . . . ,1− FXn|H(dI−1

n |r)]T .

-20 0 200

0.02

0.04

0.06

0.08

-2 0 20

0.5

1PDF

CDF

-20 0 200

0.5

1CDF

Samples

Recovered CDF

-20 0 200

0.02

0.04

0.06

0.08PDF

Recovered PDF

c 0.8

Figure 1: Illustration of the key idea on a univariateGaussian mixture. The CDF can be recovered fromits samples if Ts ≤ π

0.8 .

0 5000 10000 1500010

-3

10-2

10-1

100

101

Figure 2: KL divergence between the true mixture ofGaussians and different approximations.

Since, FXn|H(d0n|r) = 0, we can compute FXn|H(din|r),

∀i ∈ [I−1], n ∈ [N ]. We also know that FXn|H(xn|r) =1, ∀xn ≥ dIn. Therefore, we can recover the conditionalCDFs using the interpolation formula

FX|H(xn|r) =

L∑k=−L

FXn|H(kT |r) sinc

(x− kTT

), (10)

where T = din − di−1n and L a large integer. The con-

ditional PDF fXn|H can then be recovered via differ-entiation.

3.4 Toy example

An example to illustrate the idea is shown in Figure 1.Assume that the PDF of a random variable is a mix-ture of two Gaussian distributions with means µ1 =−6, µ2 = 10 and standard deviations σ1 = σ2 = 5. Itis clear from Figure 1 that F(ω) ≈ 0 for |ω| ≥ ωc = 0.8and therefore the PDF is approximately band-limited.The CDF has the same band-limit, thus, it can be

Learning Mixtures of Smooth Product Distributions

recovered from points being T = πωc≈ 4 apart. In

this example we have used only 10 discretization in-tervals as they suffice to capture 99% of the data. Weuse the finite sum formula of Equation (10) to recoverthe CDF and then we recover the PDF by differentiat-ing the CDF. The recovered PDF essentially coincideswith the true PDF given a few exact estimates of theCDF as shown in Figure 1.

Figure 2 shows the approximation error for differentmethods when we do not have exact points of the CDFbut estimate them from randomly drawn samples. Weknow that a histogram converges to the true PDF asthe number of samples grows and the bin width goes to0 at appropriate rate. However, when the conditionalPDF is smooth, the interpolation procedure using afew discretization intervals leads to a lower approxi-mation error compared to plain histogram estimatesas illustrated in the figure.

4 Algorithm

In this section we develop an algorithm for recoveringthe latent factors of the CPD model given the his-togram estimates of lower-dimensional PDFs (Alg. 1).We define the following optimization problem

min.{An}Nn=1,λ

N∑j=1

N∑k>j

N∑`>k

D(Xjk`, [[λ,Aj ,Ak,A`]]R

)s.t. λ ≥ 0,1Tλ = 1

An ≥ 0, n = 1 . . . N

1TAn = 1T , n = 1 . . . N

(11)

where D(·, ·) is a suitable metric. The Frobenious normand Kullback-Leibler (KL) divergence are consideredin this work. For probability tensors X,Y we define

DKL(X,Y) ,∑i1,i2,i3

X[i1, i2, i3] logX[i1, i2, i3]

Y[i1, i2, i3]

DFRO(X,Y) ,∑i1,i2,i3

(X[i1, i2, i3]−Y[i1, i2, i3]

)2.

Optimization problem (11) is non-convex and NP-hard in its general form. Nevertheless, sensible ap-proximation algorithms can be derived, based on well-appreciated nonconvex optimization tools. The idea isto cyclically update the variables while keeping all butone fixed. By fixing all other variables and optimizingwith respect to Aj we have

min.Aj∈C

∑k 6=j

∑l 6=jl>k

D(X

(1)jk`, (A` �Ak)diag(λ)AT

j

), (12)

where C = {A | A ≥ 0,1TA = 1T }. Problem (12) isconvex and can be solved efficiently using Exponen-tiated Gradient (EG) (Kivinen and Warmuth, 1997)

Algorithm 1 Proposed Algorithm

Input: A dataset D ∈ RM×N

1: Estimate Xjk` ∀j, k, ` ∈ [N ], ` > k > j from data.

2: Initialize {An}Nn=1 and λ.3: repeat4: for all n ∈ [N ] do5: Solve opt. problem (12) via mirror descent.6: end for7: Solve opt. problem (14) via mirror descent.8: until convergence criterion is satisfied9: for all n ∈ [N ] do

10: Recover fXn|H by differentiation using Eq. (10)11: end for

– which is a special case of mirror descent (Beck andTeboulle, 2003). At each iteration τ of mirror descentwe update Aτ

j by solving

Aτj = arg min

Aj∈C〈 ∇f

(Aτ−1j

),Aj〉+

1

ητBΦ

(Aj ,A

τ−1j

)where BΦ(A, A) = Φ(A)− Φ(A)− 〈 A− A,∇Φ(A)〉is a Bregman divergence. Setting Φ to be the nega-tive entropy Φ(A) =

∑i,j A(i, j) log A(i, j), the up-

date becomes

Aτj = Aτ−1

j ~ exp(−ητ∇f

(Aτ−1j

)), (13)

where ~ is the Hadamard product, followed by column

normalization Aτj [:, r] =

Aj [:,r]1TA[:,r]

. The optimization

problem with respect to λ is the following

min.λ∈C

∑j,k,`

D(vec(Xjk`), (A` �Ak �Aj)λ

). (14)

The update rules for λ are similar

λτ = λτ−1 ~ exp(−ητ∇f

(λτ−1

)). (15)

The step ητ can be computed by the Armijo rule (Bert-sekas, 1999).

5 Experiments

5.1 Synthetic Data

In this section, we employ synthetic data simula-tions to showcase the effectiveness of the proposedalgorithm. Experiments are conducted on syntheticdatasets {xm}Mm=1 of varying sample sizes, generatedfrom R component distributions. We set the numberof variables to N = 10, and vary the number of com-ponents R ∈ {5, 10}. We run the algorithms using5 different random initializations and for each algo-rithm keep the model that yields the lowest cost. We

Nikos Kargas, Nicholas D. Sidiropoulos

0 5000 10000

0

0.2

0.4

0.6

0.8

1

0 5000 10000

0

0.5

1

1.5

2

Figure 3: KL divergence (Gaussian).

0 5000 10000

0.92

0.94

0.96

0.98

1

0 5000 10000

0.9

0.92

0.94

0.96

0.98

1

Figure 4: Clustering accuracy (Gaussian).

evaluate the performance of the algorithms by calcu-lating the KL divergence between the true and learnedmodel, which is approximated using Monte Carlo in-tegration. Specifically, we generate {xm′}M

′

m′=1 testpoints, M ′ = 1000 drawn from the mixture and ap-proximate the KL divergence between the true andlearned model by

DKL

(fX , fX

)≈ 1

M ′

M ′∑m′=1

log fX (xm′)/fX (xm′).

We also compute the clustering accuracy on the testpoints as follows. Each data point xm′ is first assignedto the component yielding the highest posterior prob-ability cm = arg maxc fH|X (c|xm). Due to the labelpermutation ambiguity, the obtained components arealigned with the true components using the Hungar-ian algorithm (Kuhn, 1955). The clustering accuracyis then calculated as the ratio of wrongly labeled datapoints over the total number of data points.For eachscenario, we repeat 10 Monte Carlo simulations and re-port the average results. We explore the following set-tings for the conditional PDFs: (1) Gaussian (2) GMMwith two components (3) Gamma and (4) Laplace.The mixing weights are drawn from a Dirichlet dis-tribution ω ∼ Dir(α1, . . . , αr) with αr = 10 ∀r. Weemphasize that our approach does not use any knowl-edge of the parametric form of the conditional PDFs;it only assumes smoothness.

Gaussian Conditional Densities: In the first ex-periment we assume that each conditional PDF is aGaussian. For cluster r and random variable Xn weset fXn|H(xn|r) = N (µnr, σ

2nr). Mean and variance

are drawn from uniform distributions, µnr ∼ U(−5, 5),

0 5000 10000

0

0.5

1

1.5

0 5000 10000

0

0.5

1

1.5

2

2.5

Figure 5: KL divergence (GMM).

0 5000 10000

0.4

0.6

0.8

0 5000 10000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Figure 6: Clustering Accuracy (GMM).

σ2nr ∼ U(1, 2). We compare the performance of our al-

gorithms to that of EM (EM GMM). Figure 3 showsthe KL divergence between the true and the learnedmodel for various dataset sizes and different numberof components. We see that the performance of ourmethods converges to that of EM despite the fact thatwe do not assume a particular model for the condi-tional densities. Interestingly, our approach performsbetter in terms of clustering accuracy as shown in Fig-ure 4. We can see that although the joint distributionlearned by EM is closer to the true in terms of the KLdivergence, EM may fail to identify the true parame-ters of every component.

GMM Conditional Densities: In the sec-ond experiment we assume that each conditionalPDF is itself a mixture model of two univariateGaussian distributions. More specifically, we set

fXn|H(xn|r) = 12N

(µ

(1)nr , σ

(1)2nr

)+ 1

2N(µ

(2)nr , σ

(2)2nr

).

Means and variances are drawn from uniform distribu-tions µ

(1)nr ∼ U(0, 7), σ

(1)2nr ∼ U(1, 4), µ

(2)nr ∼ U(−7, 0),

σ(2)2nr ∼ U(1, 4). Our method is able to learn the

mixture model in contrast to the EM GMM which ex-hibits poor performance, due to the model mismatch,as shown in Figures 5, 6.

Gamma Conditional Densities: Another exam-ple of a smooth distribution is the shifted Gammadistribution. We set fXn|H(xn|r) = 1

βαΓ(α) (x −µnr)

α−1 exp(−x−µnrβ ) with α = 5, µnr ∼ U(−5, 0),

βnr ∼ U(0.1, 0.5). As the number of samples growsour method exhibits better performance, significantlyoutperforming EM GMM as shown in Figures 7, 8.

Laplace Conditional Densities: In the last

Learning Mixtures of Smooth Product Distributions

0 5000 10000

0

0.5

1

1.5

2

0 5000 10000

0

1

2

3

4

Figure 7: KL divergence (Gamma).

0 5000 10000

0.92

0.94

0.96

0.98

1

0 5000 10000

0.85

0.9

0.95

1

Figure 8: Clustering accuracy (Gamma).

simulated experiment we assume that each condi-tional PDF is a Laplace distribution with meanµnr and standard deviation σnr i.e., fXn|H(xn|r) =

1√2σnr

exp(√

2|xn−µnr|σnr

). A Laplace distribution in

contrast to the previous cases is not smooth (at itsmean). Parameters are drawn from uniform distribu-tions, µnf ∼ U(−5, 5), σ2

nf ∼ U(5, 10). We comparethe performance of our methods to that of the EMGMM and an EM algorithm for a Laplace mixturemodel (EM LMM). The proposed method approachesthe performance of EM LMM and exhibits better per-formance in terms of KL and clustering accuracy com-pared to the EM GMM for higher number of data sam-ples, as shown in Figures 9, 10.

5.2 Real Data

Finally, we conduct several real-data experiments totest the ability of the algorithms to cluster data. Weselected 7 datasets with continuous variables suitablefor classification or regression tasks from the UCIrepository. For each labeled dataset we hide the la-bel and treat it as the latent component. For datasetsthat contained a continuous variable as a response, wediscretized the response into R uniform intervals andtreated it as the latent component. For each datasetwe repeated 10 Monte Carlo simulations by randomlysplitting the dataset into three sets; 70% was used as atraining set, 10% as a validation set and 20% as a testset. The validation set was used to select the numberof discretization intervals which was either 5 or 10. Wecompare our methods against the EM GMM with di-agonal covariance, EM GMM with full-covariance andthe K-means algorithm in terms of clustering accuracy.

0 5000 10000

0

0.5

1

1.5

2

0 5000 10000

0

1

2

3

4

Figure 9: KL divergence (Laplace).

0 5000 10000

0.7

0.8

0.9

0 5000 10000

0.4

0.6

0.8

Figure 10: Clustering accuracy (Laplace).

Abalone

(R=2)

Power Plant

(R=2)

Wilt

(R=2)

HTRU

(R=2)

Seeds

(R=3)

Statlog

(R=3)

Wireless loc.

(R=4)

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

Figure 11: Clustering accuracy on real datasets.

Note that although the conditional independence as-sumption may not actually hold in practice, almost allthe algorithms give satisfactory results in the testeddatasets. The proposed algorithms perform well, out-performing the baselines in 5 out of 7 datasets whileperforming reasonably well in the remaining.

6 Discussion and Conclusion

We have proposed a two-stage approach based ontensor decomposition and signal processing tools forrecovering the conditional densities of mixtures ofsmooth product distributions. Our method does notassume a parametric form for the unknown conditionalPDFs. We have formulated the problem as a cou-pled tensor factorization and proposed an alternating-optimization algorithm. Experiments on syntheticdata have shown that when the underlying conditionalPDFs are indeed smooth our method can recover themwith high accuracy. Results on real data have shownsatisfactory performance on data clustering tasks.

Nikos Kargas, Nicholas D. Sidiropoulos

References

E. S. Allman, C. Matias, J. A. Rhodes, et al. Identifia-bility of parameters in latent structure models withmany observed variables. The Annals of Statistics,37(6A):3099–3132, 2009.

A. Anandkumar, R. Ge, D. Hsu, S. M. Kakade, andM. Telgarsky. Tensor decompositions for learninglatent variable models. Journal of Machine LearningResearch, 15(1):2773–2832, Aug. 2014.

A. Beck and M. Teboulle. Mirror descent and nonlinearprojected subgradient methods for convex optimiza-tion. Operations Research Letters, 31(3):167–175,2003.

T. Benaglia, D. Chauveau, and D. R. Hunter. AnEM-like algorithm for semi-and nonparametric esti-mation in multivariate mixtures. Journal of Com-putational and Graphical Statistics, 18(2):505–526,2009.

D. P. Bertsekas. Nonlinear programming. Athena sci-entific Belmont, 1999.

L. Chiantini and G. Ottaviani. On generic identifia-bility of 3-tensors of small rank. SIAM Journal onMatrix Analysis and Applications, 33(3):1018–1037,2012.

A. P. Dempster, N. M. Laird, and D. B. Rubin. Max-imum likelihood from incomplete data via the EMalgorithm. Journal of the royal statistical society.Series B (methodological), pages 1–38, 1977.

I. Domanov and L. D. Lathauwer. Canonical polyadicdecomposition of third-order tensors: reduction togeneralized eigenvalue decomposition. SIAM Jour-nal on Matrix Analysis and Applications, 35(2):636–660, 2014.

X. Fu, K. Huang, and N. D. Sidiropoulos. On identi-fiability of nonnegative matrix factorization. IEEESignal Processing Letters, 25(3):328–332, Mar. 2018.

O. Gottesman, W. Pan, and F. Doshi-Velez. Weightedtensor decomposition for learning latent variableswith partial data. In Proceedings of the Twenty-First International Conference on Artificial Intelli-gence and Statistics, volume 84, pages 1664–1672,Apr. 2018.

D. Hsu and S. M. Kakade. Learning mixtures of spheri-cal gaussians: moment methods and spectral decom-positions. In Proceedings of the 4th conference onInnovations in Theoretical Computer Science, pages11–20, 2013.

P. Jain and S. Oh. Learning mixtures of discreteproduct distributions using spectral decompositions.In Conference on Learning Theory, pages 824–856,2014.

N. Kargas, N. D. Sidiropoulos, and X. Fu. Ten-sors, learning, and kolmogorov extension for finite-alphabet random vectors. IEEE Transactions onSignal Processing, 66(18):4854–4868, Sept 2018.

J. Kivinen and M. K. Warmuth. Exponentiated gra-dient versus gradient descent for linear predictors.Information and Computation, 132(1):1–63, 1997.

J. B. Kruskal. Three-way arrays: rank and unique-ness of trilinear decompositions, with application toarithmetic complexity and statistics. Linear algebraand its applications, 18(2):95–138, 1977.

H. W. Kuhn. The Hungarian method for the assign-ment problem. Naval research logistics quarterly, 2(1-2):83–97, 1955.

S. Leurgans, R. Ross, and R. Abel. A decomposi-tion for three-way arrays. SIAM Journal on MatrixAnalysis and Applications, 14(4):1064–1083, 1993.

M. Levine, D. R. Hunter, and D. Chauveau. Maxi-mum smoothed likelihood for multivariate mixtures.Biometrika, pages 403–416, 2011.

G. J. McLachlan and D. Peel. Finite mixture models.Wiley Series in Probability and Statistics, 2000.

M. Ruffini, R. Gavalda, and E. Limon. Clustering pa-tients with tensor decomposition. In Machine Learn-ing for Healthcare Conference, pages 126–146, 2017.

N. D. Sidiropoulos, L. De Lathauwer, X. Fu, K. Huang,E. E. Papalexakis, and C. Faloutsos. Tensor decom-position for signal processing and machine learn-ing. IEEE Transactions on Signal Processing, 65(13):3551–3582, July 2017.

Y. Zhang, X. Chen, D. Zhou, and M. I. Jordan. Spec-tral methods meet EM: A provably optimal algo-rithm for crowdsourcing. In Advances in neuralinformation processing systems, pages 1260–1268,2014.