ROBUST 2018

25.1.2018

Robust Principal Component Analysis

Tom MaskJoint work with Christian Kmmerle and Felix Krahmer

Tom Mask Robust Principal Component Analysis

Principal Component Analysis (PCA)

Standard PCA

X L? E?

= +

observed low-rank error

Standard PCA with outliers

= ? + ?

Tom Mask Robust Principal Component Analysis

Principal Component Analysis (PCA)

Standard PCA

X L? E?

= +

observed low-rank error

Standard PCA with outliers

= +

Standard PCA with outliers

= ? + ?

Tom Mask Robust Principal Component Analysis

Principal Component Analysis (PCA)

Standard PCA

X L? E?

= +

observed low-rank error

Standard PCA with outliers

= +

Standard PCA with outliers

= ? + ?

Tom Mask Robust Principal Component Analysis

Principal Component Analysis (PCA)

Standard PCA

X L? E?

= +

observed low-rank error

Standard PCA with outliers

= ? + ?

Tom Mask Robust Principal Component Analysis

Robust PCA

Robust PCA:

X L? S? E?

= + +

observed low-rank sparse error

observed background foreground noise

Tom Mask Robust Principal Component Analysis

Robust PCA

Robust PCA:

X L? S? E?

= + +

observed low-rank sparse error

Application: video surveillance

= + + E

observed background foreground noise

Tom Mask Robust Principal Component Analysis

Robust PCA

Robust PCA:

X L? S? E?

= + +

observed low-rank sparse error

Application: video surveillance

= + + E

observed background foreground noise

Tom Mask Robust Principal Component Analysis

Robust PCA

Robust PCA:

X L? S? E?

= + +

observed low-rank sparse error

Application: video surveillance

= + + E

observed background foreground noise

Tom Mask Robust Principal Component Analysis

original.aviMedia File (video/avi)

low_rank.aviMedia File (video/avi)

sparse.aviMedia File (video/avi)

Robust PCA

Problem: Given matrix X Rn1n2 , find a low-rank matrix L? and a sparsematrix S? such that

X = L? + S?df: n1n2 (n1 +n2)r + s

where r = rankL? and s = (S?)vec0Is the problem solveable?

Natural optimization formulation:

(L, S) := argminL,S

rankL+Svec0

s.t. X = L+S

It is NP-hard.

The seminal paper of Cands et al. (2011) showed that (given certain assumptions)the separation is possible via a convex program

(L, S) := argminL,S

rankL+Svec1

s.t. X = L+S

Tom Mask Robust Principal Component Analysis

Robust PCA

Problem: Given matrix X Rn1n2 , find a low-rank matrix L? and a sparsematrix S? such that

X = L? + S?df: n1n2 (n1 +n2)r + s

where r = rankL? and s = (S?)vec0Is the problem solveable?

Natural optimization formulation:

(L, S) := argminL,S

rankL+Svec0

s.t. X = L+S

It is NP-hard.

The seminal paper of Cands et al. (2011) showed that (given certain assumptions)the separation is possible via a convex program

(L, S) := argminL,S

rankL+Svec1

s.t. X = L+S

Tom Mask Robust Principal Component Analysis

Robust PCA

Problem: Given matrix X Rn1n2 , find a low-rank matrix L? and a sparsematrix S? such that

X = L? + S?df: n1n2 (n1 +n2)r + s

where r = rankL? and s = (S?)vec0Is the problem solveable?

Natural optimization formulation:

(L, S) := argminL,S

rankL+Svec0

s.t. X = L+S

It is NP-hard.

The seminal paper of Cands et al. (2011) showed that (given certain assumptions)the separation is possible via a convex program

(L, S) := argminL,S

rankL+Svec1

s.t. X = L+S

Tom Mask Robust Principal Component Analysis

Robust PCA

Issues with the convex approach:

relatively slow

mediocre performance (especially in practical applications, where theassumptions are typically broken)

new algorithms emerged in the past few years, most of them non-convex

We also propose a non-convex algorithm...

Tom Mask Robust Principal Component Analysis

Robust PCA

Issues with the convex approach:

relatively slow

mediocre performance (especially in practical applications, where theassumptions are typically broken)

new algorithms emerged in the past few years, most of them non-convex

We also propose a non-convex algorithm...

Tom Mask Robust Principal Component Analysis

Proposed algorithm

Derivation of the algorithm in a nutshell:

1. Smooth the non-convex objective to become differentiable.

2. Express the derivative in a special form.

3. Solve the system of non-linear equations arising from the first orderoptimality conditions via a fixed point scheme.

The resulting algorithm is an instance of Iteratively Reweighted Least Squares(IRLS) method.

Where is the novelty?

special form of the derivative a local quadratic convergence rate a competitive performance

Tom Mask Robust Principal Component Analysis

Proposed algorithm

Derivation of the algorithm in a nutshell:

1. Smooth the non-convex objective to become differentiable.

2. Express the derivative in a special form.

3. Solve the system of non-linear equations arising from the first orderoptimality conditions via a fixed point scheme.

The resulting algorithm is an instance of Iteratively Reweighted Least Squares(IRLS) method.

Where is the novelty?

special form of the derivative a local quadratic convergence rate a competitive performance

Tom Mask Robust Principal Component Analysis

Proposed algorithm

Our algorithm is not

1. fastest the algorithm of Yi et al. (2016) attains the time complexity ofthe standard PCA and thus is hard to beat

2. statistically most accurate the algorithm of Oh et al. (2016) is hard tobeat

3. numerically most accurate

actually, it is

Tom Mask Robust Principal Component Analysis

Proposed algorithm

Our algorithm is not

1. fastest the algorithm of Yi et al. (2016) attains the time complexity ofthe standard PCA and thus is hard to beat

2. statistically most accurate the algorithm of Oh et al. (2016) is hard tobeat

3. numerically most accurate actually, it is

Tom Mask Robust Principal Component Analysis

Small Comparison

Take X = AB> with A,B R200r and replace s random entries by randomcorruptions. Algorithm succeeds if LAB>F/AB>F < 0.01.

Tom Mask Robust Principal Component Analysis

Small Comparison

Take X = AB> with A,B R200r and replace s random entries by randomcorruptions. Algorithm succeeds if LAB>F/AB>F < 0.01.

Tom Mask Robust Principal Component Analysis

Small Comparison

Take X = AB> with A,B R200r and replace s random entries by randomcorruptions. Algorithm succeeds if LAB>F/AB>F < 0.01.

Tom Mask Robust Principal Component Analysis

Small Comparison

Take X = AB> with A,B R200r and replace s random entries by randomcorruptions. Algorithm succeeds if LAB>F/AB>F < 0.01.

Tom Mask Robust Principal Component Analysis

Conclusions

A new algorithm for Robust PCA proposed.

the only competitive algorithm among the IRLS class

the only algortihm with super-linear convergence rate

the only algorithm uniformly outperforming the convex approach (disclaimer:subjective)

The local quadratic convergence rate proved.

Tom Mask Robust Principal Component Analysis

References

Cands, E. J., Li, X., Ma, Y., & Wright, J. (2011). Robust principal componentanalysis?. Journal of the ACM (JACM), 58(3), 11.

Oh, T. H., Matsushita, Y., Kweon, I., & Wipf, D. (2016). A pseudo-bayesianalgorithm for robust PCA. In Advances in Neural Information ProcessingSystems (pp. 1390-1398).

Yi, X., Park, D., Chen, Y., & Caramanis, C. (2016). Fast algorithms for robustPCA via gradient descent. In Advances in neural information processingsystems (pp. 4152-4160).

Tom Mask Robust Principal Component Analysis