Density Corrected Sparse RecoveryWhen R.I.P. Condition Is Broken

Ming Lin, Zhengzhong Lan, Alexander G. HauptmannCarnegie Mellon University

Pittsburgh, PA, USA

AbstractThe Restricted Isometric Property (R.I.P.) is a veryimportant condition for recovering sparse vectorsfrom high dimensional space. Traditional methodsoften rely on R.I.P or its relaxed variants. How-ever, in real applications, features are often cor-related to each other, which makes these assump-tions too strong to be useful. In this paper, westudy the sparse recovery problem in which the fea-ture matrix is strictly non-R.I.P. . We prove thatwhen features exhibit cluster structures, which of-ten happens in real applications, we are able to re-cover the sparse vector consistently. The consis-tency comes from our proposed density correctionalgorithm, which removes the variance of estimatedcluster centers using cluster density. The proposedalgorithm converges geometrically, achieves nearlyoptimal recovery bound O(s2 log(d)) where s isthe sparsity and d is the nominal dimension.

1 IntroductionIn high dimensional statistics, an important problem is to re-cover a sparse vector from a small number of observationswhose number is usually much smaller than the nominal di-mension of the observation matrix. [Candes et al., 2006]proves that sparse recovery is possible as long as the fea-ture matrix satisfies the Restricted Isometric Property (R.I.P.)condition. Informally, the R.I.P. condition requires any setof columns of the feature matrix with size less than thesparsity to be linearly independent. More likely than not,the R.I.P. condition is too strong to be useful in real worldapplications. Many R.I.P.-like relaxations have been pro-posed in the past decades [Van De Geer and Bhlmann, 2009;Foucart, 2012]. These alternatives usually have better con-stants, or more general forms of the restricted isometric in-equalities. See [Van De Geer and Bhlmann, 2009] for a nicesurvey.

Although there is a vast literature studying relaxed R.I.P.-like conditions, their core requirement is that the feature ma-trix must be well-conditioned on the sparse subspace. In otherwords, their largest and smallest singular values must be closeto unit. For many real applications, such assumptions are stilltoo strong to be useful. Considering a simple case where we

duplicate some columns of the feature matrix, these R.I.P.-like conditions are no longer true because we can find a subsetof columns whose smallest singular value is exactly zero.

In this paper we consider a prevalent non-R.I.P. setting inwhich the features form cluster structures, as can be seen inmany machine learning [Lehiste, 1976] and computer visionproblems [Lan et al., 2013; Lowe, 2004]. Due to the fact thatmany features extractors are similar to each others and theyreflect the characteristics of the same image, vision featuresare often correlated and have cluster structures. This corre-lation is even stronger in those systems that have thousandsto millions of features [Lan et al., 2013; Gan et al., 2015a;2015b].

Under this setting, instead of requiring the observed fea-ture matrix to be R.I.P., we only have the same requirementon the cluster center matrix, which is much easier to satisfy.A trivial solution to perform sparse recovery under this set-ting is by first clustering the features and then applying theconventional sparse recovery methods on the clustered fea-ture matrix. However, due to the random perturbation, theestimated cluster center is the biased version of the real clus-ter centers. Unlike instance clustering where we can im-prove the estimation accuracy by having more samples, thebias in feature clustering process cannot be asymptoticallyremoved even if infinite training instances were given. Thisinconsistency happens in all conventional methods, includingconvex/non-convex sparse regularizers , greedy methods andtheir stochastic or adaptive variants [Agarwal et al., 2012;Shalev-Shwartz and Tewari, 2011; Ghadimi and Lan, 2013;Ji Liu, 2013; Jin et al., 2013; Zhaoran Wang, 2013; Lin etal., 2014b; Lin and Xiao, 2014; Yang et al., 2014; Lin et al.,2014a].

In this paper, under cluster assumption, we develop a con-sistent sparse recovery method, called density correction. Thekey idea of density correction is that although we cannot elim-inate the bias in feature clustering, we are able to estimate thevariance of this bias via the cluster density. We can correct thebias in the gradient of the loss function with cluster densityin the sparse recovery process. When combining the densitycorrection with hard iterative thresholding, we obtain a con-sistent sparse estimator.

The remaining paper is organized as following. In section 2we briefly review several closely related works. We introduceour notations and backgrounds of our method in section 3.

Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence (IJCAI 2015)

3664

Section 4 gives the details of our algorithm. In section 5 weprove the geometrical convergence rate and the consistencyof our method. We conclude our paper with discussions insection 6.

2 Related WorkIn this section, we briefly review several closely relatedworks. Our review is by no means to be comprehensive butto capture the backgrounds of our approaches.

Sparse recovery methods can be roughly grouped into threecategories: convex regularizer, non-convex regularizer andgreedy methods. The convex regularizer mainly bases on`1-norm LASSO problems [Shah, 2012; Xu et al., 2010;Tibshirani, 1996]. The non-convex regularizer is proved to bemore accurate than convex methods in several cases [Zhao-ran Wang, 2013; Xiang et al., 2013; Loh and Wainwright,2013; Gong et al., 2013; Zhang, 2012]. Recently, greedymethods are rediscovered in the sparse community due to itssimplicity and efficiency [Xiaotong Yuan, 2014; Ji Liu, 2013;Blumensath, 2012; Foucart, 2011]. However, all these stud-ies are based on R.I.P.-like assumptions therefore fail to coverthe non-R.I.P. problems discussed in this paper.

The gradient correction technique dates back to the cor-rected least square regression [Markovsky and Van Huffel,2007]. The same technique is considered in sparse regres-sion under feature corruption settings [Chen et al., 2013;Loh and Wainwright, 2012]. These methods require that thefeature matrix to be R.I.P. when noise is small. In our setting,the observed feature matrix becomes non-R.I.P. when the per-turbation is small. Also, traditional methods require the noiselevel to be known while our method not.

3 Notations and BackgroundsThe sparse recovery problem aims to recover a sparse vectorw∗ ∈ Rd from feature matrix X ∈ Rn×d and label y ∈Rn. The sparsity of w∗ is denoted by s and its support setis denoted as S∗ . We assume that y is generated by a linearmodel

y = Xw∗ + ξ , (1)where ξ is additive subgaussian noise with noise level

‖ 1√nXTξ‖∞ ≤ ξ .

We denote Xs the submatrix of X whose columns are in-dexed by set S. we denote [w]F the truncation of w whoseelements are zero outside F . For simplicity, let ∇F `(w) ,[∇`(w)]F be the truncated gradient. ∇s`(w) is the truncatedgradient with largest s elements in absolute value. In this pa-per we assume that both S∗ and X are unknown. We canonly observe feature matrix X whose columns are randomlysampled from a clustering model with cluster center X . Thatis,

X = [X1, X1, X2, X2, X2 · · · Xd] + ε , XP + ε , (2)

where P is called duplication matrix. The random noise ε issubgaussian satisfying

E{ 1

nεTε} = D([σ1, σ2, · · · , σd]) .

D(·) is the diagonal function. In the i-th cluster, there are kifeatures. We denote σ = maxi σi .

The feature cluster model defined in Eq. (2) coincides withmany real world applications. For example, in computer vi-sion, we usually need to encode low level features by clus-tering, which is called “feature encoding” in literature. Somefeatures are built on other features. For example, semanticfeatures are built from low-level features hence are correlatedto low-level features. In those scenarios, for set S with size s,the submatrix XS is usually ill-conditioned.

The conventional sparse recovery methods assume the fea-ture matrix X to satisfy the R.I.P. [Candes and Plan, 2011] orits variants [Trzasko and Manduca, 2009]. Roughly speakingit requires XS to be well-conditioned for |S| ≤ s. Then wecan estimate w∗ via the following optimization problem:

P1 w(1)∗ = arg min

w`(w) ,

1

2n‖Xw − y‖2

s.t. ‖w‖0 ≤∑i∈S∗

ki .

To recover w∗ ,w∗ ≈ Pw

(1)∗ .

However in Eq. (2), when ε→ 0, the R.I.P.-like conditionclearly does not hold hence conventional methods P1 are nolonger applicable. To address this issue, a naive approach is tocluster the feature matrix and then apply P1 on the clusteredfeatures. Denote X the clustered feature matrix. The naiveapproach can be formulated as

P2 w(2)∗ = arg min

wˆ(w) ,

1

2n

∥∥∥XTw − y∥∥∥2

s.t. ‖w‖0 ≤ s .

After clustering, X is expected to be R.I.P. if X is R.I.P. . Amain drawback of P2 is that w

(2)∗ is not a consistent estima-

tor. The inconsistency of w(2)∗ comes from the uncertainty of

X . Recall that X is an estimation of X by matrix columnclustering, from concentration inequality, it is easy to see thatwith high probability,∥∥∥Xi − Xi

∥∥∥ ≤ O(σi√ki

) . (3)

Therefore, X is a biased estimation of X . This bias cannot beeliminated by more training instances since the right side ofEq. (3) does not depend on n. As a consequence, P2 whichis based on X is biased. Even if we already know the supportset S∗ , w

(2)∗ is still biased due to the uncertainty of X . This

is known as the least square inconsistency when we are giventhe support set S∗ . In short, we have the following proposi-tion. We omit the proof since it is trivial once we realize theuncertainty in Eq. (3).Proposition 1. No matter how many training instances weregiven, P1 and P2 cannot estimate w∗ consistently even if weknow the support set S∗ and the duplication matrix P .

Proposition 1 shows that the conventional sparse recoverymethods are far from optimal because they are all inconsis-tent even in the ideal case. Note that their suboptimality does

3665

not inherit from the way they solve the problem P1 or P2.We can use any algorithm to solve the above two problems.For example, directly optimize the `0 norm constraint bygreedy methods [Tropp and Gilbert, 2007; Blumensath andDavies, 2008; Xiaotong Yuan, 2014], or use convex [Tibshi-rani, 1996] or non-convex relaxation methods [Zhang, 2012;Zhaoran Wang, 2013; Lin et al., 2014a]. All these methodscannot eliminate the uncertainty in X . By proposition 1, noconsistent estimation is possible by these methods.

A direct corollary of proposition 1 is that it is impossible toconsistently estimate X by any other method. Based on Eq.(3), even when we know the duplication matrix P , the bestpossible accuracy in estimating Xi is O(σi/

√ki). This up-

per bound is based on the concentration of mean value of kirandom variables. Because ki is finite, it is impossible to im-prove this estimation unless we find a better way to estimatethe mean of random variables.

In next section, we will show that by using the proposeddensity correction, we can design a consistent sparse estima-tor under model Eq. (2) even if we cannot eliminate the un-certainty in X .

4 Density Corrected Sparse RecoveryFrom Eq. (3), it is impossible to eliminate the uncertainty

in X . To motivate an alternative approach, we examine thegradient of loss function in P2,

n∇wˆ(w) = XTXw − XTy .

Denote X = X + θ, we have

n∇wˆ(w) =(X + θ)T(X + θ)w − (X + θ)Ty

=XTXw + θTθw︸ ︷︷ ︸E1

+ (θTX + XTθ)w − (X + θ)Ty︸ ︷︷ ︸E2

.

Therefore, the bias in the gradient comes from E1 and E2.Since the perturbation ε is zero mean subgaussian randomvariable, θ is also zero mean subgaussian. When n is largeenough, E2/n will converge to XTy/n thus is consistent.However, E1 will not converge to zero. This inspires us toeliminate E1 without eliminating θ itself. In other words,although it is impossible to eliminate the bias term θ, we canstill get a consistent estimator if we are able to estimate thevariance of this bias term. Based on this intuition, we proposeour Density Corrected Sparse Recovery (DCSR) in Algorithm1.

Algorithm 1 is based on Hard Iterative Thresholding (HIT)[Blumensath and Davies, 2009]. It first clusters features intor groups. r could be potentially much larger than the sparsitys. Although we can use any clustering algorithm to clusterfeatures, we prove that the pairwise tree clustering is suffi-cient to recover the cluster center X with the optimal biasO(σi/

√ki) up to constants. For the i-th cluster, its cluster

density is stored in νi. Actually, νi is an unbiased estimationof the variance of perturbation term ξ in model Eq. (2). Thenthe algorithm does a gradient descent with step size η. The

Algorithm 1 Density Corrected Sparse Recovery (DCSR)1: Input: X is the feature matrix . r is the number of

columns of X . s is the sparsity of w∗ . T is the totaliteration number. γ is the step size in gradient descent.

2: Use pairwise tree clustering to cluster columns of X intor groups. The distance between i-th and j-th column isdefined by

d(i, j) , ‖Xi −Xj‖ , (4)Denote the index set of i-th cluster as Ci , i = 1, 2, · · · r.

3: Denote Xi the cluster center and νi the density ofCi. For|Ci| = ki, Xi and νi is computed by

Xi =1

ki

∑j∈Ci

Xj . (5)

νi =1

nki(ki − 1)

∑j∈Ci

∥∥∥Xj − Xi

∥∥∥2 . (6)

4: ν = [ν1, ν2, · · · , νr]T.5: w0 = 0 .6: for t = 1 to T do7: Gradient descent with density correction:

wt =wt−1 − η∇wˆ(wt−1) + ηD(ν)wt−1 (7)

=wt−1 − η1

nXTXwt−1 (8)

+ η1

nXTy + ηD(ν)wt−1 . (9)

8: Hard iterative sparse thresholding: let St be the set ofindexes of elements in wt of s largest amplitude ,

{wt}i =

{{wt}i i ∈ St

0 otherwise.

9: end for10: Output: Cluster index Ci and sparse vector wT .

range of η is given in the next section. The key difference isthat the gradient is corrected by the cluster density νi in Eq.(7). Eq. (7) can be reformulated as a gradient descent step forthe following objective function

P3 minwt−1

ˆ(wt−1)− 1

2‖D−1/2(ν)wt−1‖2 . (10)

This is similar to Elastic-Net [Zou and Hastie, 2005]. InElastic-Net, an `2-norm regularizer is added in the sparseregression objective function to ensure a faster convergencerate. While in density correction, we subtract an `2-norm reg-ularizer adaptively to remove the bias in the gradient.

After gradient descent, the algorithm truncates wt greed-ily. Note that the density correction is a principled method.We can apply density correction in various sparse regressionmethods such as LASSO and adaptive non-convex regular-izer. We choose HIT because it is recently proven to be ef-ficient in sparse recovery [Xiaotong Yuan, 2014]. In eachiteration, the intermediate solution wt is always s sparse.

3666

5 Theoretical AnalysisIn this section, we give theoretical guarantees for Algorithm1. Our analysis relies on the following matrix concentrationfacts [Tropp, 2012].Lemma 1. Let ε ∈ Rn×d be a subgaussian random matrixwith covariance

E{ 1

nεTε} = D([σ2

1 , σ22 , · · · , σ2

d]) .

The eigenvalue of 1nAA

T is bounded by [µ, β]. Then withprobability at least 1− δ, denote A = A+ ε,

λmax{1

nATA} ≤β λmin{

1

nATA} ≥ µ ,

where c is at least 1/4,

∆ ,

√(d+ 1) log(2d/δ)

nc(β + max

i{σ2

i })

β ,(β + maxi{σ2

i }) + ∆

µ ,(µ−maxi{σ2

i })−∆ .

We assume the feature cluster center matrix X to be τ -distinguishable:Definition 1. Feature matrix X is τ -distinguishable, if for apositive constant τ ,

mini6=j

1

n‖Xi − Xj‖ ≥ τ .

Clearly, feature clustering is possible if and only if X is τ -distinguishable for τ large enough. In the following theorem,we prove that the pairwise clustering step correctly groupsfeatures in X if X is sufficiently τ -distinguishable.Theorem 1. When X is at least τ -distinguishable with

τ ≥2 maxt{σ2

t }(2 + 2

√log(2kt/δ)

nc

+

√2 log(2kt(d− kt)/δ)

nc) ,

then with probability at least 1− 2rδ, ∀i ∈ {1, 2, · · · , r}, thecluster center Xi is an inconsistent estimator of Xi ,

1

n

∥∥∥Xi − Xi

∥∥∥2 ≤ 1

ki(

1

ki

∑t∈Ci

σ2t +

√2 log(2/δ)

ncmaxt∈Ci

σ2t ) .

(11)

Proof. First we prove that the columns in X can be separatedwhen noise ε is not too large. For Xt and its cluster index setCt, we require

maxi,j∈Ct

1

n‖Xi −Xj‖2 ≤ min

i∈Ct,k/∈Ct

1

n‖Xi −Xk‖2 . (12)

With probability at least 1− 2rδ,

maxi,j∈Ct

1

n‖Xi −Xj‖2 = max

i,j∈Ct

1

n‖εi − εj‖2

≤2 maxt∈Ci

{σ2t }(1 + 2

√log(2kt/δ)

nc) .

And

mini∈Ct,k/∈Ct

1

n‖Xi −Xk‖2

= mini∈Ct,k/∈Ct

1

n

∥∥Xt − Xk + εi − εk∥∥2

≥ mini∈Ct,k∈Cq 6=Ct

1

n

∥∥Xt − Xq

∥∥2 − maxi∈Ct,k/∈Ct

1

n‖εi − εk‖2

≥γ − 2 maxt{σ2

t }(1 +

√2 log(2kt(d− kt)/δ)

nc) .

Therefore, Eq. (12) is satisfied when

γ ≥2 maxt{σ2

t }(2 + 2

√log(2kt/δ)

nc+√

2 log(2kt(d− kt)/δ)nc

) .

From concentration,

1

n

∥∥∥Xi − Xi

∥∥∥2 =1

n

∥∥∥∥∥ 1

ki

ki∑i=1

εi,:

∥∥∥∥∥2

≤ 1

ki(

1

ki

∑t∈Ci

σ2t +

√2 log(2/δ)

ncmaxt∈Ci

σ2t ) .

Theorem 1 claims that when different features in X is dis-similar to each other, we can correctly cluster the feature ma-trix X . However, the estimated cluster center is not consis-tent. The squared bias of the cluster center estimation is onorder of O(σ2

i /ki +σ2/√n). Even n is infinity, this bias will

never converge to zero. Theorem 1 shows that it is impossi-ble to consistently estimate X , no matter how many traininginstances are given.

As conventional sparse analysis, we assume that X satis-fies R.I.P. condition. Note that our definition is slightly differ-ent from conventional one in order to simplify the notation.Definition 2 (δs-RIP). A matrix X is δs-R.I.P. if there is aconstant 0 < δs < 1, for any s sparse vector w whose sup-port set is F ,

(1− δs)‖w‖ ≤ ‖1

nXF

TXw‖ ≤ (1 + δs)‖w‖ .

Remark 1 If X is δs-R.I.P. , it must be τ -distinguishable.To see this, take any two 1-sparse vector ei, ej where ei is theunit vector along the i-th coordinate. When s ≥ 2, we have

1

n‖Xi − Xj‖ = ‖ 1

nX(ei − ej)‖ ≥ 2

√1− δs .

Therefore X is at least 2√

1− δs distinguishable. The con-verse is not true. Clearly XF can be well separated but is lowrank. Here we want to emphasize that we should treat theconstants δs and τ independently. This is because althoughX is at least 2

√1− δs distinguishable, its optimal τ may be

significantly larger than 2√

1− δs.Following Definition 2, it is easy to check the next lemma.

We omit the proof to save space.

3667

Lemma 2. Assume that u, v are s sparse vectors and X isδ2s-R.I.P. . Denote set F = supp(u)∪ supp(v), then for anypositive η ≤ 1/(1− δ2s),

‖u−η 1

nXF

TXu−(v−η 1

nXF

TXv)‖ ≤ ρ‖u−v‖ . (13)

where ρ = 1− η(1− δ2s).

Lemma 2 claims that the gradient descent operator is re-stricted isometric when X is R.I.P. . Lemma 2 can be ex-tended to a more generalized version of R.I.P. condition, asshown in [Xiaotong Yuan, 2014]. Lemma 2 is critical in theconvergence analysis because when ρ < 1, Eq. (13) is a con-traction map which indicates a geometrical convergence rate.

With the help of the above discussion, we give the conver-gence rate of Algorithm 1.

Theorem 2. Under the same assumptions in Theorem 1 andX is δ3s-R.I.P. . Algorithm 1 converges to w∗ geometrically,

‖wt − w∗‖ ≤ (‖w∗‖+ b)ρt + b ,

where

σ = maxi{ 1

k2i

∑t∈Ci

σ2i }

∆3s =(‖w∗‖(σ + 1 + δ3s) + σ)

√(3s+ 1)3s log(6d/δ)

nc

+ ‖w∗‖√

6s log(6d/δ)

ncσ

ρ =2(1− η(1− δ3s −∆3s))

b =2η

1− ρ(

3s√nξ + ∆3s‖w∗‖) ,

provide that 0 ≤ ρ < 1 .

Theorem 2 claims that the convergence rate of Algorithm1 is linear. The ρ controls the exponential convergence ratewhich depends on the step size and R.I.P. condition. ∆3s isthe bias term due to clustering which converges to zero atspeed O(1/

√n). b is the estimation error due to the noise

in label y and the uncertainty in the clustering step. We seethat when the noise level ξ is zero and n is sufficiently large,the estimation error b converges to zero, which indicates theconsistency of Algorithm 1.

The novelty of Theorem 2 is its consistency. It is easy toverify that without density correction, sparse recovery meth-ods based on P1 and P2 can only estimate w∗ up to ac-curacy O(σi/

√ki), no matter how many training instances

are given. By density correction, we improve the accuracyto O(σi/

√nki). This is significantly better than traditional

methods when n is large enough. However, this power doesnot come free. In R.I.P. sparse recovery, the sampling com-plexity is O(s log d). From Theorem 2, in non-R.I.P. sparserecovery, our sampling complexity is O(s2 log d), which isslightly larger than the R.I.P. one. This extra s in the samplingcomplexity comes from the uncertainty of clustering step. Inour proof, we prove that with a fix support set we can boundthe gradient after density correction and this bound holds forany set no larger than s. This step brings us an extra s in

sampling complexity. At the time when we write this paper,we are not aware of any published results showing a sharperbound.

We give a sketch of our proof of Theorem 2. First we showthat in Eq. (7) the gradient descent after correction is consis-tent.Lemma 3. For any w supported on set F , |F | ≤ s, withprobability at least 1-δ,

‖∇Fˆ(w)− [D(ν)]Fw −∇F

¯(w)‖ ≤ ∆s‖w‖

where

∆s = ‖w‖√

2s log(6d/δ)

ncmax

i{ 1

k2i

∑t∈Ci

σ2i }+

‖w‖√

(s+ 1)s log(6d/δ)

nc(max

i{ 1

k2i

∑t∈Ci

σ2i }+ 1 + δs)

+

√(s+ 1)s log(6d/δ)

ncmax

i{ 1

k2i

∑t∈Ci

σ2i } . (14)

Proof. For a fix set F , similar to standard online learninganalysis,

‖∇Fˆ(w)− [D(ν)]Fw −∇F

¯(w)‖

≤‖ 1

n(XF

TXF − XFTXF )− E{D(ν)}︸ ︷︷ ︸

E1

+ E{D(ν)} −D(ν)︸ ︷︷ ︸E2

)‖2‖w‖+ ‖ 1

n(XF

Ty − XFTy)︸ ︷︷ ︸

E3

‖ .

From matrix concentration Lemma 1, with probability at least1− δ,

‖E1‖2 ≤√

(s+ 1) log(2s/δ)

nc(max

i{ 1

k2i

∑t∈Ci

σ2i }+ 1 + δs)

‖E2‖2 ≤√

2 log(2s/δ)

ncmax

i{ 1

k2i

∑t∈Ci

σ2i }

‖E3‖ ≤√

(s+ 1) log(2s/δ)

ncmax

i{ 1

k2i

∑t∈Ci

σ2i } .

Now we consider the all possible set |F | ≤ s in d dimen-sional space. Clearly there are no more than

(ds

)≤ (ed/s)s

such sets. We have

log(2s(ed

s)s/δ) ≤ s log(6d/δ) .

Combining all the above together, the proof is done.

In Lemma 3, ∆s is the bias of the corrected gradient∇F

ˆ(w) + [D(ν)]Fw. ∆s converges to zero at speedO(1/

√n). The complete proof of Theorem 1 follows im-

mediately. Our proof is based on the proof given in [Yuan etal., 2013, Theorem 2 part b]. It is worth to mention that theirmodel cannot deal with non-R.I.P. feature matrix.

3668

proof of Theorem 2. Denote F = S(t) ∪ S(t−1) ∪ S∗ . So|F | ≤ 3s. We try to bound estimation accuracy of the 3ssparse vector u = [wt]F . We have

‖u− w∗‖ = ‖[wt]F − w∗‖≤‖wt−1 − η(∇F

ˆ(wt−1)− [D(ν)]Fwt−1)

− (w∗ − η∇F¯(w∗))‖+ η‖∇F

¯(w∗)‖≤‖wt−1 − η∇F

¯(wt−1)− (w∗ − η∇F¯(w∗))‖

+ η‖∇F¯(w∗)‖

+ η‖∇Fˆ(wt−1)− η[D(ν)]Fwt−1 −∇F

¯(wt−1)‖≤(1− η(1− δ3s))‖wt−1 − w∗‖+ η‖∇F

¯(w∗)‖+ η‖∇F

ˆ(wt−1)− [D(ν)]Fwt−1 −∇F¯(wt−1)‖

≤(1− η(1− δ3s))‖wt−1 − w∗‖+ η‖∇F

¯(w∗)‖+ η∆3s‖wt−1‖≤(1− η(1− δ3s) + η∆)‖wt−1 − w∗‖

+ η‖∇3s¯(w∗)‖+ η∆3s‖w∗‖ .

Fro R.I.P. condition,

‖u− w∗‖≤(1− η(1− δ3s))‖wt−1 − w∗‖+ η‖∇F

¯(w∗)‖+ η‖∇F

ˆ(wt−1)− [D(ν)]Fwt−1 −∇F¯(wt−1)‖

(1)

≤ (1− η(1− δ3s))‖wt−1 − w∗‖+ η‖∇F

¯(w∗)‖+ η∆3s‖wt−1‖≤(1− η(1− δ3s) + η∆)‖wt−1 − w∗‖

+ η‖∇3s¯(w∗)‖+ η∆3s‖w∗‖ .

(1) comes from Lemma 3. Because St is the largest s ele-ments in wt, we have

‖wt −w∗‖ ≤ 2‖ut −w∗‖ .

Then we get the recursive inequality about wt. Therefore wehave

‖wt − w∗‖ ≤ (‖w∗‖+ b)ρt + b ,

where

ρ =2(1− η(1− δ3s −∆3s))

b =2η

(1− ρ)(‖∇3s

¯(w∗)‖+ ∆3s‖w∗‖) .

To ensure the convergence, we require ρ < 1.Since y is generated by model Eq. (1),

‖∇3s¯(w∗)‖ = ‖[ 1

nXTξ]3s‖ ≤

3s√nξ .

6 ConclusionWe propose a density correction sparse recovery algorithmfor non-R.I.P. problems. We first show that the conventionalmethods cannot consistently recover the sparse vector due tofeature correlation and cluster center uncertainty. Then we

propose the density correction that combines with hard iter-ative thresholding to recover the sparse vector consistently.The proposed algorithm has geometrical convergence rate. Itadaptively removes the variance of the bias term in the clus-tered feature matrix.

The density correction has potentially more applicationsother than sparse recovery. It shows that directly learning onthe clustered features may not be optimal. The cluster centeris usually biased therefore the estimation is usually incon-sistent. The density correction improves the estimation to beconsistent without estimating the bias term itself, which couldbe rather difficult in many applications. Instead it exploits thecluster structure to estimate the second order statistics: thevariance of the bias term. Considering the fact that cluster-ing is widely used in machine learning, the density correctionmight be applied in various circumstances.

We mainly focus on least square loss function in this paper.For general convex loss function, it is much more difficult toderive the density correction. A key difficulty is that the ex-pectation of the bias term in the gradient is no longer the clus-ter density. We need more sophisticated statistics to estimatethe variance of this bias term. For example, recent devel-opment in convex total least square [Malioutov and Slavov,2014] extends the total least square estimation to general con-vex loss functions. We open this topic for future research.

AcknowledgementThis paper was partially supported by the Intelligence Ad-vanced Research Projects Activity (IARPA) via Depart-ment of Interior National Business Center contract numberD11PC20068 and by the National Science Foundation un-der Grant No. IIS-1251187. The U.S. Government is autho-rized to reproduce and distribute reprints for Governmentalpurposes notwithstanding any copyright annotation thereon.Disclaimer: The views and conclusions contained herein arethose of the authors and should not be interpreted as neces-sarily representing the official policies or endorsements, ei-ther expressed or implied, of IARPA, DOI/NBC, the NationalScience Foundation or the U.S. Government.

References[Agarwal et al., 2012] Alekh Agarwal, Sahand Negahban, and

Martin J Wainwright. Stochastic optimization and sparse statis-tical recovery: An optimal algorithm for high dimensions. arXivpreprint arXiv:1207.4421, 2012.

[Blumensath and Davies, 2008] Thomas Blumensath and Mike EDavies. Iterative thresholding for sparse approximations. Journalof Fourier Analysis and Applications, 14(5-6):629–654, 2008.

[Blumensath and Davies, 2009] Thomas Blumensath and Mike EDavies. Iterative hard thresholding for compressed sensing. Ap-plied and Computational Harmonic Analysis, 27(3):265–274,2009.

[Blumensath, 2012] Thomas Blumensath. Accelerated iterativehard thresholding. Signal Processing, 92(3):752–756, 2012.

[Candes and Plan, 2011] Emmanuel J Candes and Yaniv Plan. Aprobabilistic and RIPless theory of compressed sensing. IEEETransactions on Information Theory, 57(11):7235–7254, 2011.

3669

[Candes et al., 2006] Emmanuel J Candes, Justin Romberg, andTerence Tao. Robust uncertainty principles: Exact signal recon-struction from highly incomplete frequency information. IEEETransactions on Information Theory, 52(2):489–509, 2006.

[Chen et al., 2013] Yudong Chen, Constantine Caramanis, and ShieMannor. Robust sparse regression under adversarial corruption.In ICML, 2013.

[Foucart, 2011] Simon Foucart. Hard thresholding pursuit: an al-gorithm for compressive sensing. SIAM Journal on NumericalAnalysis, 49(6):2543–2563, 2011.

[Foucart, 2012] Simon Foucart. Sparse recovery algorithms: suf-ficient conditions in terms of restricted isometry constants. InApproximation Theory XIII: San Antonio 2010, pages 65–77.Springer, 2012.

[Gan et al., 2015a] Chuang Gan, Ming Lin, Yi Yang, YuetingZhuang, and Alexander G Hauptmann. Exploring semantic inter-class relationships (SIR) for zero-shot action recognition. InAAAI, 2015.

[Gan et al., 2015b] Chuang Gan, Naiyan Wang, Yi Yang, Dit-YanYeung, and Alexander G Hauptmann. Devnet: A deep event net-work for multimedia event detection and evidence recounting. InCVPR, 2015.

[Ghadimi and Lan, 2013] Saeed Ghadimi and Guanghui Lan. Op-timal stochastic approximation algorithms for strongly convexstochastic composite optimization, II: shrinking procedures andoptimal algorithms. SIAM Journal on Optimization, 23(4):2061–2089, 2013.

[Gong et al., 2013] Pinghua Gong, Changshui Zhang, ZhaosongLu, Jianhua Z. Huang, and Jieping Ye. A general iterative shrink-age and thresholding algorithm for non-convex regularized opti-mization problems. In ICML, 2013.

[Ji Liu, 2013] Jieping Ye Ji Liu, Ryohei Fujimaki. Forward-backward greedy algorithms for general convex smooth functionsover a cardinality constraint. arXiv:1401.0086, 2013.

[Jin et al., 2013] Rong Jin, Tianbao Yang, and Shenghuo Zhu. Anew analysis of compressive sensing by stochastic proximal gra-dient descent. arXiv preprint arXiv:1304.4680, 2013.

[Lan et al., 2013] Zhen-Zhong Lan, Lu Jiang, Shoou-I Yu,Shourabh Rawat, Yang Cai, Chenqiang Gao, Shicheng Xu, Hao-quan Shen, Xuanchong Li, Yipei Wang, et al. Cmu-informediaat trecvid 2013 multimedia event detection. In TRECVID 2013Workshop, volume 1, page 5, 2013.

[Lehiste, 1976] Ilse Lehiste. Suprasegmental features of speech.Contemporary issues in experimental phonetics, 225:239, 1976.

[Lin and Xiao, 2014] Qihang Lin and Lin Xiao. An adaptive accel-erated proximal gradient method and its homotopy continuationfor sparse optimization. In ICML, 2014.

[Lin et al., 2014a] Ming Lin, Rong Jin, and Changshui Zhang. Ef-ficient sparse recovery via adaptive non-convex regularizers withoracle property. In UAI, 2014.

[Lin et al., 2014b] Qihang Lin, Xi Chen, and Javier Pea. A spar-sity preserving stochastic gradient methods for sparse regres-sion. Computational Optimization and Applications, pages 1–28,2014.

[Loh and Wainwright, 2012] Po-Ling Loh and Martin J Wain-wright. High-dimensional regression with noisy and missingdata: Provable guarantees with nonconvexity. The Annals ofStatistics, 40(3):1637–1664, 2012.

[Loh and Wainwright, 2013] Po-Ling Loh and Martin J Wain-wright. Regularized m-estimators with nonconvexity: Statis-tical and algorithmic theory for local optima. arXiv preprintarXiv:1305.2436, 2013.

[Lowe, 2004] David G Lowe. Distinctive image features fromscale-invariant keypoints. IJCV, 60(2):91–110, 2004.

[Malioutov and Slavov, 2014] Dmitry Malioutov and NikolaiSlavov. Convex total least squares. arXiv:1406.0189 [cs, q-bio,stat], June 2014. arXiv: 1406.0189.

[Markovsky and Van Huffel, 2007] Ivan Markovsky and SabineVan Huffel. Overview of total least-squares methods. Signalprocessing, 87(10):2283–2302, 2007.

[Shah, 2012] Rajen Shah. The lasso: Variable selection, predictionand estimation. Technical report, 2012.

[Shalev-Shwartz and Tewari, 2011] Shai Shalev-Shwartz and Am-buj Tewari. Stochastic methods for l1-regularized loss minimiza-tion. JMLR, 12:1865–1892, 2011.

[Tibshirani, 1996] R. Tibshirani. Regression shrinkage and selec-tion via the lasso. Journal of the Royal Statistical Society. SeriesB (Methodological), pages 267–288, 1996.

[Tropp and Gilbert, 2007] Joel A Tropp and Anna C Gilbert. Signalrecovery from random measurements via orthogonal matchingpursuit. IEEE Transactions on Information Theory, 53(12):4655–4666, 2007.

[Tropp, 2012] Joel A Tropp. User-friendly tail bounds for sums ofrandom matrices. Foundations of Computational Mathematics,12(4):389–434, 2012.

[Trzasko and Manduca, 2009] Joshua Trzasko and Armando Man-duca. Relaxed conditions for sparse signal recovery with generalconcave priors. TSP, 57(11):4347–4354, 2009.

[Van De Geer and Bhlmann, 2009] Sara A Van De Geer and PeterBhlmann. On the conditions used to prove oracle results for thelasso. Electronic Journal of Statistics, 3:1360–1392, 2009.

[Xiang et al., 2013] Shuo Xiang, Xiaotong Shen, and Jieping Ye.Efficient sparse group feature selection via nonconvex optimiza-tion. In ICML, 2013.

[Xiaotong Yuan, 2014] Tong Zhang Xiaotong Yuan, Ping Li. Gra-dient hard thresholding pursuit for sparsity-constrained optimiza-tion. In ICML, 2014.

[Xu et al., 2010] Huan Xu, Constantine Caramanis, and Shie Man-nor. Robust regression and lasso. IEEE Transactions on Infor-mation Theory, 56(7):3561–3574, 2010.

[Yang et al., 2014] Eunho Yang, Aurelie Lozano, and PradeepRavikumar. Elementary estimators for high-dimensional linearregression. In ICML, 2014.

[Yuan et al., 2013] Xiao-Tong Yuan, Ping Li, and Tong Zhang.Gradient hard thresholding pursuit for sparsity-constrained op-timization. arXiv:1311.5750 [cs, stat], November 2013. arXiv:1311.5750.

[Zhang, 2012] Tong Zhang. Multi-stage convex relaxation for fea-ture selection. Bernoulli, 2012.

[Zhaoran Wang, 2013] Tong Zhang Zhaoran Wang, Han Liu. Opti-mal computational and statistical rates of convergence for sparsenonconvex learning problems. arXiv:1306.4960, 2013.

[Zou and Hastie, 2005] Hui Zou and Trevor Hastie. Regularizationand variable selection via the elastic net. Journal of the Royal Sta-tistical Society: Series B (Statistical Methodology), 67(2):301–320, 2005.

3670