Recent Advances in Algorithmic

Heavy -Tailed Statistics

Sam Hopkins , UC Berkeley (→ MIT)Based on joint work with Yeshwanth Cherapanamjeri , Tarun Kathvria ,Prasad Raghavendra , and Nilesh Tripuraneni .

Measuring success probability in estimation ( confidence intervals)" rate

"

--

Given X . . . . Xu- Po , find ⑤ sit . 110--041 Mdg's failurewith prob . x t - s no '

ambient ratesamples

dimension

For PoEPTclass of distributions

Our game : large class P leg . all distributions with

0111 bold . moments) , but similar guarantees as if

P contains Gaussian s.

Example : Estimating the mean in kElltnexi- Ex H

e-EnXi

, . . . ,Xn -X , Cov IX ) t I +myGaussian : Ht -2Xi - EXILE flattens up . I - s

Drop the Gaussian assumption ?

Is there an estimator right . . . . . Xn) s .t .

this-nut Offal- FEI ) up . I- s ?

Example : Estimating the mean in k

D= I : median of means , truncated mean,. .-

Lugosi -Mendelson ' 18 : estimator based on high -dimensional

quantile 's / median of means (exp . time,"""dits )

H.

'

18 : algorithmic approach to high - dimensional quartilesTheorem : for every 8 > 2-

n

exists As 1h . . .Xn) sit .

Huis-MH e Offa +FLI ) w . p .I -8

, poly In , d , log' 1st

time

AgendaI. Survey of recent ( algorithmic) developments

2. Heavy - tailed covariance estimation

• Algorithm of fcherapanamjeri - H .

- Kathrin - Raghavendra- Tripuraneni)

• 2 key techniques for concentration t convex prog -

- bounded differences Kini Hop -- Yg:ix. Emil

try

- Sos Bernstein { Hemings -- %%s" 'Emil

[Lugosi-Mendelson ‘19, ‘19, ‘19, ‘20, ‘20, Mendelson-Zhivotovskiy ‘20, Minsker ‘15, ‘18, ‘20, Hsu-Sabato ‘16, Lerasale-Oliveira ‘12, Joly-Lugosi-Oliveira ‘17, ... ]

Matching Gaussian Confidence w/ only 0111 moments

• Mean estimation , lz - median of means , trimmed mean

• Mean estimation,Holl

• Covariance estimation

• regression• regularized regression /sparse recovery

B) r i.

Matching Gaussian Confidence w/ only 0111 moments

in polynomial time

Mean estimation,la covariance , H' Hop

authors time authors suboptimal by

H. Ofnd)

's Minster FE

(herapanamjeri - -

s,CIIKRT ( log Ys)

" "

Flammarion - BartlettOln th'd)

similar for linear regressionDeepersin - Leave ,Lei -Luh -Venkat -Zhang Oln'd) ( Hsv -Sabato )

[Depersin-Lecue ‘19, Diakonikolas-Kane-Pensia ‘20, H.-Li-Zhang ‘20,...]

Connections to Robust Statistics

*

I;¥ → Y . . . . Yn→ - n"" "" "

# An• related techniques - new notions of high -dimensional quantile

4 algorithms to compute them• simultaneously robust and high - confidence alqos

AgendaI. Survey of recent ( algorithmic) developments

2. Heavy - tailed covariance estimation

• Algorithm of fcherapanamjeri - H .

- Kathrin - Raghavendra- Tripuraneni)

• 2 key techniques for concentration t convex prog -

- bounded differences- Sos Bernstein

Setup : Xi.. . .

,Xn - X on Rd

,covariance EHR

""'

for simplicity , TRI = Old) , HI Hop -- od )

Theorem : can find Is set.HIS-E Hop a-Offal . Hog Ysl"" t FEI)

Wp . I- S in time poly In ,d , log' Is) if X is certifiably 12,81-hypercontractive.

• sub -optimal by ( log' Is )""- can remove in

:

• exp . time (Mendelson -Zhivatovsky)• poly In ,dits ) time (C # KRT)

⑧ best previous :ois, """ "er, / "''m: " -

-

Theorem : can find Is set.HIS-Ello ,. a-Offal . Hog Ysl"" t FEI)

Wp . I- S in time poly In ,d , log' Is) if X is certifiably 12,81-hypercontractive.

( 2,8) - hypercontractility : Echr)'E OCEAN't

"

Certifiable :O ( ElXiii)"- Efx ,vY= E picul

'

• products of heavy -tailed dis tins

• mixtures thereof

• affine transformations thereof

⑨. . .

The High -Dimensional Median -of -Means ParadigmB,- Z , = FIB,XiXit① Buckets ×n} :

tBlog lls - Flog 'ls= ftp.?gYig② Quantile

Z ."

spectral r- center"

: •

Zz

Hr,Kziivvt) - (Minister •

..

•

for 60% of Zi 's . Zs

II Zi- Mllop Er for 60% ofzi's

The High -Dimensional Median -of -Means Paradigm"

spectral r- center"

: Z ,

•

Hv,Kziivvt) - (Mini) Itr Zz

•

'

for 60% of Zi 's . •

Zs

• -2 is a spectral fat Fin center of Z , . . -Ziogyg w-p- l- S

(Lugosi -Mendelson , Mendelson - Zhiva?

•M.M'

spectral r - centers → HM -M '

Hope LrJust need to find a spectral r-center !

Our Strategy : Man , → . . .⇒M

• Attempt to certify that Mi is a spectralr center

• Success → output Mi

•• Failure → witness V, update Min = Mi INT

→•

Mitt•

• Zi 'sMic. •

[CFB '

19 , CHKRT'24

If GIVEN E and Zi, .. .

,Zioglls ,

can you check that

I (Zi,wt) - ( I

,WTI l E r

for 60% of Zi's and all unit v ?

If GIVEN E and Zi, .. .

,Zioglls ,

Optimization Approach can you check that

Variables bi . . - biogys , Vi . . - Nd Kzinti - Himmler

may"b ;

for 60% of Zi's and all unit v ?

b.V

s 't . bi c- {oil's , Hulk l ,

bikzisvvtl-SM.mil/xbir

If GIVEN E and Zi, .. . ,Ziog4s ,

Optimization Approach can you check that

Variables bi . . - biogys , H . . - Nd -- v Kzinti) - Himmler

May"b ;

for 60% of Zi's and all unit v ?

b.V

s 't . bi C- {oil} , Hull =/ ,• Z .

Et•

Zs

bikzisvvtl-SM.mil/xbir • •

M Zz

If GIVEN E and Zi, .. .

,Zioglls ,

Optimization Approach can you check that

Variables bi . . - biogys , Vi . . - Nd Kzinti) - Himmler

Max"b ;

for 60% of Zi's and all unit v ?

b.v•• Know : E 0.6 - log

' Is forSt . bi c- {oil's , Hulk l , F- I'Intf,bikini) - GM

,wt) ) > bi . r WP . I -8

• Convex relaxation : t 0.610948 , for r -- dog 'lsl""ft t Ft,

W - p . I - S

AgendaI. Survey of recent ( algorithmic) developments

2. Heavy - tailed covariance estimation

• Algorithm of fcherapanamjeri - H .

- Kathrin - Raghavendra- Tripuraneni)

• 2 key techniques for concentration t convex prog -

- bounded differences- Sos Bernstein

Optimization Approach If GIVEN E and Z. .. .

.Zions ,

can you check thatVariables bi . . -biogys , h . . - Nd

Kzinti) - SM ,will er

Max"

bi

b,v for 60% of Zi's and all unit v ?

St . bi c- {oil's , Hulk l,

bi#innit - Gezi,vvt) ) " bi ' r

Any" reasonable

"

convex relaxation will satisfy bounded dias .

~ ~•

E'

( OPT ( Z . . -

y Zqogyg ) - O PT ( Zi . .. .

.E.gig ) ( E l

t '.

- * 9 :• &GR .g

Bi n bi O E Bi E Iµ

. -3

New Goal : convex relaxation of zi-ienpi.MEmax

"

bi

b,V

s 't . bi c- {oil} , Hulk I,

bikini) - Ke,vvt) ) x bir

St . z.IE#.,O.,gPTEO.6log'Is , assuming onlycertifiable 12,8) -hypercontractility

We use Sum of Squares Semidefinite Program-

hierarchy of convex relaxations for anypolynomial optimization problem

max"

bi

b.vg.iii.biko =3"

'

proxy variables for bi,bib ; , vivjbk

2 ✓

St . bit {oil} , Hull =/ , Bi. . .

I

bKZi x bi ' r

① max"

bi

b,V

s 't . bi c- {on} , Huh =/ ,E Mb.at/trEbiKtiivvTI-sqwtDbiKZisvvTI-52.vvTHxbir± Max t's's Kei -e.until

"

WH -- I

Suffices to bound

a.EE?,so4Hn9IiEiIzi-e.uutiy

Suffices to bound

a.E.io?.o4mnaix.SUM of ii.d . polynomials in V

Suffices to bound

# Himmat zie.mil

( svmotfiidpolynomiab in v

) Max [( (Zi -E) ① ( Zi -E ) , Y )YeyzdlxdlYE Sos

Try ''

,Yao

Suffices to bound

HE?.gs/Min9IiiEizi-e.uui-iy( svmotfiidpolynomiab in v

¥ Max [(Ei -2) ① ( Zi -e ) , Y )YeyzdlxdlHE Sos

= EH Elzi -Elahi -E) H- Sos

② EH -2 Hi -Elahi -sell!Sos Bernstein : control via HE-2 Hi -250 Hi -E5 Hs

.se- -

degree 8 polynomial in X

⇒ control via 12,81 certifiable

hypercontractivity :

HEX 8/1so,I 041

Yields covariance estimation algo . with r -- 041091"" ft + FIH

Open : remove (log'

1st" " ?

Open : optimal heavy- tailed linear regression ?

Applications of• bold . Gifts for convex programs ?

• Sos Bernstein ?

EM'

p①DB$