EstimatingSymmetricDistributionPropertiesMaximumLikelihoodStrikesBack!

Jayadev AcharyaCornell

Hirakendu DasYahoo

Alon OrlitskyUCSD

Ananda SureshGoogle

ShannonChannelMarch5,2018

Basedon:

AUnifiedMaximumLikelihoodApproachforEstimatingSymmetricPropertiesofDiscreteDistributions,InternationalConferenceonMachineLearning(ICML),2017

http://proceedings.mlr.press/v70/acharya17a.html

Slidesavailableat:https://people.ece.cornell.edu/acharya/talks/shannon-18acharya.pdf

Outline1. Motivation2. Methods3. Proofs4. Futuredirections

Chapter1:Motivation

Distributionproperties𝒫:acollectionofdiscretedistributions• 𝒫 = Δ$ :alldistributionsover 𝑘 = {1,… , 𝑘}• Δ+:distributionsover[6]

Property𝑓:𝒫 → ℝ

• 𝑝 3 =?• Isitfair?Is𝑝 𝑖 = 1/6 forall𝑖?

Propertyestimation

𝑝 unknown distributionin𝒫

Givenindependentsamples𝑋:; = 𝑋:, 𝑋<,… , 𝑋; ∼ 𝑝

Estimate𝑓 𝑝

Samplecomplexity 𝑆 𝑓,𝒫, 𝜀, 𝛿

Minimum𝑛 necessaryto

Estimate𝑓 𝑝 ± 𝜀

Witherrorprobability< 𝛿 (usuallyconstant)

Symmetricproperties𝑓 symmetric ifunchangedunderinputpermutations

Entropy𝐻 𝑝 ≜ ∑ 𝑝(𝑥) log :M(N)N

SupportsizeS 𝑝 ≜ ∑ 𝕀 M N QRS

Manyothers:Renyi entropy,supportcoverage,…

Coinswithbias0.4,andwithbias0.6havesameentropy!

Entropy

𝐻 𝑝 ≜ −U𝑝(𝑥) log 𝑝(𝑥)𝒙

• Mostpopularmeasureofrandomness• Centralquantityininformationtheory[Shannon’48]

Howmanysamplestoestimate𝐻(𝑝) to±𝜀?

Longlineofwork:[Empirical,Miller-Maddow,Jackknifed,Coverageadjusted,BUB(paninski’03),…]

Estimatingentropy• Randomnessofneuralspiketrains• Featureselectionindecisiontrees• Graphicalmodels(Chow-Liu)

Traditionalsettingforpropertyestimation:𝒫 = Δ$:distributionson[𝑘] (small𝑘)Obtainmanysamples(large𝑛)

Genetics,neuralspikes,text,computervision,ecology:𝒌 large,possiblyinfinite,perhapsunknown

Estimatingtheunseen:Corbet’sbutterflies

2yearstrappingbutterfliesinMalaypeninsula:

AskedFisher:

howmanynewspeciesifhegoesfortwomoreyears?

Frequency 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Species 118 74 44 24 29 22 20 19 20 15 12 14 6 12 6

Estimatingtheunseen: formulation𝑝: unknown discretedistribution

𝑆X 𝑝 ≜ 𝔼[#𝐝𝐢𝐬𝐭𝐢𝐧𝐜𝐭𝐬𝐲𝐦𝐛𝐨𝐥𝐬in𝑚ind. samples ∼ 𝑝]

𝑆X 𝑝 =U 1 − 1 − 𝑝(𝑥) X

N

Normalizedcoverage:pq MX

Howmanysamplestoestimatepq MX

to±𝜀?

Estimatingtheunseen:applications• Estimatingvocabularysize• Ecologicaldiversity• Microbialdiversityonskin

Wellstudied[GoodToulmin,Efron Thisted]Forconstant𝜀:Requires 𝒎

𝟐samples

Morerecently,[Zou ValiantValiantChan…’16,OrlitskySureshWu’16]:

Requires 𝒎𝐥𝐨𝐠𝒎

samples

Chapter2:Methods

Plug-inestimation

Using𝑋:;,findanestimate�̂� of𝑝

Estimate𝑓 𝑝 by𝑓 �̂�

Howtoestimate𝑝?

Sequencemaximumlikelihood (SML)

𝑝Nvwxy ≜ argmax

M 𝑝(𝑥;)= arg max

M∏ 𝑝(𝑥})}

𝑋:~ = ℎ, ℎ, 𝑡𝑝�X� = argmax𝑝< ℎ · 𝑝 𝑡

𝑝�X� ℎ = <~, 𝑝�X� 𝑡 = :

~

Sameastheempirical-frequency distribution

Multiplicity𝑁N - #times𝑥 appearsin𝑋:;

𝑝wxy 𝑥 =𝑁N𝑛

SMLforentropy

SamplecomplexityofSMLtoestimate𝐻 𝑝 over𝚫$ :

𝑆�X� 𝐻, 𝚫$, 𝜀 = Θ𝑘𝜀

Intheasymptotic𝑛 → ∞,SMLisoptimal

Samplecomplexityofentropy

Samplecomplexityof𝐻(𝑝):

𝑆 𝐻, 𝚫$, 𝜀 = Θ𝑘

𝜀 ⋅ log 𝑘

[…,Paninski’03,ValiantValiant’11,HanJiaoVenkatWeissman’15,WuYang’15]

[ValiantValiant’11]:plug-in,sub-optimal in𝜀

[HanJiaoWeissman ’18]:optimalin𝜀 bytweakingVV’11

Optimalestimators

Generalrecipe:

1. Approximate𝐻(𝑝) withapolynomialin𝑝2. Estimatethepolynomial

Differentnon-plug-in estimatorforeachproperty

Sophisticatedapproximationtheoryresults

PriorworkForseveralimportantproperties

Optimalisalogarithmic factor betterthanempirical

Property 𝒫 SML Optimal References

𝐻 𝒑 𝚫$𝑘𝜀

𝑘𝜀 ⋅ log 𝑘

ValiantValiant ‘11,HanJiaoVenkatWeissman ‘15,WuYang‘15

PriorworkForseveralimportantproperties

Optimalisalogarithmic factor betterthanempirical

Property 𝒫 SML Optimal References

𝐻 𝒑 𝚫$𝑘𝜀

𝑘𝜀 ⋅ log 𝑘

ValiantValiant ‘11,HanJiaoVenkatWeissman ‘15,WuYang‘15

𝑆X 𝒑𝑚

𝚫� 𝑚𝑚

log 𝑚log

1𝜀

Orlitsky SureshWu’16

PriorworkForseveralimportantproperties

Optimalisalogarithmic factor betterthanempirical

Property 𝒫 SML Optimal References

𝐻 𝒑 𝚫$𝑘𝜀

𝑘𝜀 ⋅ log 𝑘

ValiantValiant ‘11,HanJiaoVenkatWeissman ‘15,WuYang‘15

𝑆X 𝒑𝑚

𝚫� 𝑚𝑚

log 𝑚log

1𝜀

Orlitsky SureshWu’16

𝑆 𝒑𝑘

𝚫$ 𝑘 log1𝜀

𝑘log 𝑘

log<1𝜀

WuYang’16

∥ 𝒑 − 𝑢 ∥: 𝚫$𝑘𝜀<

𝑘𝜀< ⋅ log 𝑘

Han JiaoWeissman ‘16

Ourresults [AcharyaDasOrlitsky Suresh’17]Unified,simple,sample-optimalapproachforallaboveproblems

• Plug-inestimator• Maximumlikelihoodprinciple:

sequencemaximumlikelihood(SML)profilemaximumlikelihood(PML)

Chapter3:PML

ProfilesProfileisthe multi-setofmultiplicities

Φ 𝑋:; ≜ {𝑁N: 𝑥 ∈ 𝑋:;}

Φ(ℎ, ℎ, 𝑡) = Φ(𝑡, ℎ, 𝑡) = {1,2}

Φ(𝛼, 𝛾, 𝛽, 𝛾) = {1,1,2}

ProbabilitymultisetSymmetricpropertiesdeterminedby

• Probabilitymultiset:{𝑝(1), 𝑝(2),… }

𝑝 ℎ = 0.4 ⟹ {0.4,0.6}

𝑝 ℎ = 0.6 ⟹ {0.4,0.6}

Profilesaresufficientstatisticforsymmetricproperties

ℎ, ℎ, 𝑡,OR𝑡, ℎ, 𝑡 ⟹ sameestimate

EstimatingprobabilitymultisetOrlitsky Santhanam Viswanathan Zhang ‘04:

“Onmodelingprofilesinsteadofvalues”,UAI

Moreextensively:

OSVZ:“Onestimatingaprobabilitymultiset”,online

Profilemaximumlikelihood(PML)Profile probability

𝑝 Φ = U 𝑝(𝑥;)Nv:� Nv ��

Distributionmaximizing theprofileprobability

𝑝�MX� = argmax

M∈𝒫𝑝(Φ)

PMLexample𝑋:~ = ℎ, ℎ, 𝑡Φ ℎ, ℎ, 𝑡 = 1,2

𝑝 Φ = 1,2

= 𝑝 𝑠, 𝑠, 𝑑 + 𝑝 𝑠, 𝑑, 𝑠 + 𝑝 𝑑, 𝑠, 𝑠

= 3 ⋅ 𝑝 𝑠, 𝑠, 𝑑

= 3 ⋅ ∑ 𝑝< 𝑥 𝑝(𝑦)N��

Symmetricpolynomial

SMLof{1,2}

𝑝 Φ = 1,2 = 3 U 𝑝< 𝑥 𝑝(𝑦)N��

𝑝�X� ℎ = <~, 𝑝�X� 𝑡 = :

~

𝑝�X� 1, 2 = 323

< 13 +

13

< 23 =

1827 =

23

PMLof{1,2}

𝑝 Φ = 1,2 = 3 U 𝑝< 𝑥 𝑝(𝑦)N��

U 𝑝< 𝑥 𝑝(𝑦)N��

=U𝑝< 𝑥 (1 − 𝑝 𝑥 )N

=U𝑝 𝑥 ⋅ 𝑝(𝑥)(1 − 𝑝 𝑥 )N

≤14

𝑝MX� 1, 2 =34

PMLof{1,1,2}Φ(𝛼, 𝛾, 𝛽, 𝛾) = {1,1, 2}

𝑝MX� 1,1,2 = 𝑈[5]

PMLcanpredictexistenceofunseensymbols

Maximize:

U 𝑝 𝑥 <𝑝 𝑦 𝑝 𝑧N���¤

,

subjectto:

U𝑝(𝑥)N

= 1,𝑝 𝑥 ≥ 0

Uniform[500],700samples

700samples

U[500],700x,12experiments

PMLplug-inToestimatesymmetric𝑓(𝑝):• Find𝑝MX� Φ(𝑋:;)• Output𝑓(𝑝MX�)

Advantages:• Notuningparameters• Notfunctionspecific

Rootedinthemaximumlikelihoodprinciple

Ingredient1:GoodnessofML

GeneralMLpluginestimation

𝒫: collectionofdistributionsoveranabstract domain𝒵

𝑓:𝒫 → ℝ anyproperty

Given𝑧 ∈ 𝒵 estimate𝑓

MLestimator:

Determine𝑝¤§¨ ≜ arg maxM∈𝒫

𝑝 𝑧

Output𝑓(𝑝¤§¨)HowgoodisMLE?

CompetitivenessofMLplugin

Theorem:Suppose𝑓©: 𝒵 → ℝ issuchthat∀𝑝 ∈ 𝒫,

Pr¬∼M

𝑓© 𝑍 − 𝑓 𝑝 > 𝜀 < 𝛿,

thenMLEpluginerrorboundedby

Pr¬∼M

𝑓 𝑝¤§¨ − 𝑓 𝑝 > 2 ⋅ 𝜀 < 𝛿 ⋅ |𝒵|.

Competitive withthebest𝑓©

CompetitivenessofMLEplugin- proofConsiderany𝑝 ∈ 𝒫

𝒵°± ≜ {𝑧 ∈ 𝒵: 𝑝 𝑧 ≥ 𝛿}

• 𝑧 ∈ 𝒵°±:• 𝑓© 𝑧 − 𝑓(𝑝) ≤ 𝜀 (byconditioninTheorem)

• 𝑝¤§¨ 𝑧 ≥ 𝑝 𝑧 ≥ 𝛿,hence 𝑓© 𝑧 − 𝑓(𝑝¤§¨) ≤ 𝜀• Triangleinequality: 𝑓(𝑝¤§¨) − 𝑓 𝑝 ≤ 2𝜀

• 𝑧 ∈ 𝒵²± ≜ {𝑧: 𝑝 𝑧 < 𝛿}

• Pr 𝑓 𝑝¤§¨³ − 𝑓 𝑝 > 2𝜀 ≤ Pr 𝒵²± < 𝛿 ⋅ 𝒵

Ingredient2:Errorprobabilities

PMLperformancebound

Theorem:If𝑛 = 𝑆 𝑓, 𝒫, 𝜀, 𝛿 ,then

𝑆MX� 𝑓, 𝒫, 2 ⋅ 𝜀, Φ; ⋅ 𝛿 ≤ 𝑛

|Φ;|:numberofprofilesoflength𝑛

Profileoflengthn:partitionofn

{3},{1,2}, {1,1,1}➞ 3,2+1,1+1+1

|Φ;| = partition#of𝑛

Hardy-Ramanujan: |Φ;| < 𝑒~ ;

PMLperformance:Try1Theorem:𝑛 = 𝑆 𝑓, 𝒫, 𝜀, 1/3 ⇒ 𝑆MX� 𝑓, 𝒫, 2𝜀, 1/3 ≤ 𝑂(𝑛<).

Proof:• Boosterrorprobability:• Take𝑛 ⋅ ℓ independentsamples,anddividetheminℓ parts• Estimate𝑓 foreachofthesamples• Takethemedianoftheestimates⟹ 𝛿 < exp(−ℓ)

• Φ;ℓ < exp(3 𝑛ℓ .º)

ErrorprobabilityofPMLmostexp −ℓ + 3 𝑛ℓ .º

Whenℓ > 𝑛,theerrordominates#profiles

PMLperformance:Try2Recall

𝑆 𝐻, Δ$, 𝜀, 1/3 = Θ𝑘

𝜀 ⋅ log 𝑘Withtwicethesampleserrordropsexponentially

𝑆 𝐻, Δ$, 𝜀, 𝑒»;¼.½ = Θ $

¾⋅y¿À $

• Modified estimatorswithsmallboundeddifferences• StrongerguaranteesfromMcDiarmid’s inequalityMosttechnicalpartofthepaperSimilarresultsforotherproperties

Combiningeverything

Fasterrorforpropertieswestudy:

If 𝒏 = 𝑺 𝒇,𝒫, 𝜺, 𝟏/𝟑 ,then 𝑺 𝒇,𝒫, 𝜺, 𝒆»𝟒 𝒏 ≤ 𝟒𝒏

MLplug-inresult:

If 𝑺 𝒇,𝒫, 𝜺, 𝒆»𝟒 𝒏 ≤ 𝟒𝒏,then 𝑺𝒑𝒎𝒍 𝒇,𝒫, 𝟐𝜺, 𝒆» 𝒏 ≤ 𝟒𝒏

Combining,wearedone!

ComputingPMLdistribution• EMalgorithm[OrlitskyPanSajamaSanthanamViswanathanZhang‘04,’13]

• ApproximatePMLviaBethePermanents[Vontobel’14]• ExtensionsofMarkovChains[Vatedka Vontobel’16]• Approximationviarelaxation[JiaoPavlichinWeissman ‘17]:

Chapter5:Directions

ApproximatePML• Perhaps findingexactPMLishard• CanshowthatapproximatingPMLenough

Question:Computeexp(−𝑛Ê) approximatePMLforany𝛽 < 1

Eventhisisoptimal(forlarge𝑘)

Higherdimensions• EstimateKLdivergencebetweendiscretedistributionsgivensamples(underassumptionsofcourse)

[BuZou LiangVeeravalli ‘16,HanJiaoWeissman ’16]

[Acharya’18]:PMLisoptimalforKLdivergenceestimation- Higherorderpartitions

IndependentProofTechniquesMLEgoodwhensomethingelseisgood

• MLperformanceindependentofotherresults?

OtherIsPMLoptimalforevery symmetricproperty?

Canwedosomethingforcontinuousdistributions?

Summary• Symmetricpropertyestimation• PMLplug-inapproach• Universal,simpletostate• Independentofparticularproperties

InFisher’swords…

OfcoursenobodyhasbeenabletoprovethatMLEisbestunderallcircumstances.MLEcomputedwithalltheinformationavailablemayturnouttobeinconsistent.Throwingawayasubstantialpartoftheinformationmayrenderthemconsistent.

R.A.Fisher

ThankYou!