beyond objectivity and subjectivity; a discussion paper

beyond objective and subjective statisticsa discussion

Christian P. Robert

Universite Paris-Dauphine (CEREMADE)& University of Warwick (Dept. of Statistics)

http://xianblog.wordpress.com,seriesblog.netRoyal Statistical Society, April 2017

the elephant in the room...

Statistical analysis invariably starts

with the unquestioned premise of therandom nature of the data

which differs from the assumption of aprobabilistic model generating thedata [not much discussed therein]

& connects with repeatability ofobservations, which is almost alwayswrong

How do we address this blatantly wrong start?!

[perspective that led Keynes to abandon statistics and move toeconomics!]

the elephant in the room...

Statistical analysis invariably starts

with the unquestioned premise of therandom nature of the data

which differs from the assumption of aprobabilistic model generating thedata [not much discussed therein]

& connects with repeatability ofobservations, which is almost alwayswrong

How do we address this blatantly wrong start?!

[perspective that led Keynes to abandon statistics and move toeconomics!]

...and the tortoise in the next room

Is focus on wrong issue

arguing between ourselves about the best way to solve thewrong problem,

while users seek approximate solutions with some modicum ofefficiency

i.e., satisfied with imprecise inference

E.g., how many statistical problems “solved” Amazon in aday, compared with uncovering new fundamental particles???

...and the tortoise in the next room

Is focus on wrong issue

arguing between ourselves about the best way to solve thewrong problem,

while users seek approximate solutions with some modicum ofefficiency

i.e., satisfied with imprecise inference

E.g., how many statistical problems “solved” Amazon in aday, compared with uncovering new fundamental particles???

a forensic illustration

Vote on 10 April by US National Commission on Forensic Scienceon how forensic analysts should testify about evidence:

Analysts must

explain

how they examined evidence

what statistical analyses theychose

inherent uncertainties in theirmeasurements

[R. Mejia, Nature, 4 April 2017]

a forensic illustration

Vote on 10 April by US National Commission on Forensic Scienceon how forensic analysts should testify about evidence:

Analysts must

never claim with certainty thatanything on a crime scene islinked to a suspect

quantify the probability thatobserved similarities occurred bychance

[R. Mejia, Nature, 4 April 2017]

the ultimate issue with statistics

“...the ultimateinaccessibility of a realitythat is truly independent ofobservers is a basic humancondition.” A. Gelman& C. Hennig

Except for the most [basic] scientificsettings, there is not reality behindstatistical models [Box], hence aninaccessible consensus is the rule

”Keynesians who focus on more subjective factors...”

Read in John Maynard Keynes’s A Treatise on Probability (1921):

“...where general statistics areavailable, the numericalprobability which might bederived from them is inapplicablebecause of the presence ofadditional knowledge withregards to the particular case...”

”Keynesians who focus on more subjective factors...”

Read in John Maynard Keynes’s A Treatise on Probability (1921):

“...until a prima facie case hasbeen established for the existenceof a stable probable frequency,we have but a flimsy basis forany statistical induction...”

models, models, models

missing: trial-and-error way ofbuilding a statistical modeland/or analysis,while subjective inputs fromoperator(s) and regulators foundat all stages of constructionand should be spelled out ratherthan ignored (or rejected)

99% of the Universe

discussion on foundations (§5) thorough [objective judgement!] but

does not expand on issue of “default” or all-inclusivestatistical solutions

used through “point-and-shoot” software by innumeratepractitioners

while under impression of conducting a statistical analysis

false feeling also occurs in treatment of statistical expertise bymedia, courts, and scientific journals

[We are the 1%!!!]

99% of the Universe






[We are the 1%!!!]

99% of the Universe






[We are the 1%!!!]

Relativity

Pertains to awareness of multiple perspectives but also to stability

Fundamental dependence of inferential output on statisticalframework(s)

difference in outcomes perfectly acceptable

inner assessment of model feasible by producing pseudo data

comparison of frameworks only to the extent of predictivecharacteristics

Relativity

Pertains to awareness of multiple perspectives but also to stability

Bayesian analysis [both objective and subjective] well-suited tothis purpose / rescued by relativity

move to quantum theory where objectivity is impossible dueto the presence of the observer/experimenter [??]

reproducibility fraught with danger: production of similarresults depends on rigid framework, relates to fact thatstatistics not experimental science

rise of the machines

relevance of machinelearning for [model-free]learning

dismissal ofmachine-learning perspectivedisappointing

given robustness [againstmodels] of machine learningprediction

...if missing in uncertaintyassessment

the five pillars of statistical wisdom

frequentism, subjective about the Universe and its probabilitydistribution, and about the choice of an inference procedure,with falsifiability only achievable by replicating data viasimulation, but this only rejects poor models

[agreeing with Davies, 2014]


frequentism, subjective about the Universe and its probabilitydistribution, and about the choice of an inference procedure,with falsifiability only achievable by replicating data viasimulation, but this only rejects poor models

[agreeing with Davies, 2014]

subjective Bayesianism is quite objective and open about itssubjectivity or relativity, working in parallel universes that donot need to intersect, provided they produce such universes(discussion seems to swerve towards a common prior principlethat I do not understand!)


subjective Bayesianism is quite objective and open about itssubjectivity or relativity, working in parallel universes that donot need to intersect, provided they produce such universes(discussion seems to swerve towards a common prior principlethat I do not understand!)

very subjective concept of objective Bayes when it reduces toJaynes’s! and missing the point of “objective” Bayesprinciples which is not to be unique but rather define a genericprinciple of derivation of a prior from the likelihood functionwithout a particular justification


very subjective concept of objective Bayes when it reduces toJaynes’s! and missing the point of “objective” Bayesprinciples which is not to be unique but rather define a genericprinciple of derivation of a prior from the likelihood functionwithout a particular justification

I do not feel the gap between the above and the falsificationistBayesian perspective expressed in Section 5.5, hence soundsvery subjective to me. Unless the school reduces to Gelmanand Shalizi (2013)?!

embracing uncertainty with a subjective hug

basic realism and uncertain nature of data call for an absence ofhard decisions like tests and model choices, but rather fordescriptive performances of the suggested procedures, acceptingimperfection and variability in the answer produced locally

“Its not so hard to move awayfrom hypothesis testing andtoward a Bayesian approach ofembracing variation andaccepting uncertainty.” A.Gelman

conclusion

exposes the need to spell out the various inputs leading to astatistical analysis

reinforces the call for model awareness:

critical stance on all modelling inputs, including priors!,disbelief that any model is true

potential if realistic outcome would be to impose not onlyproduction of all conscious choices but also through theposting of (true or pseudo-) data and of relevant code for allpublications involving a statistical analysis

conclusion

proposal too idealistic in that most users (and most makers)of statistics cannot or would not spell out their assumptionsand choices, being unaware of or unapologetic about those

central difficulty with statistics as a service discipline, namelythat almost anyone anywhere can produce an estimate or ap-value without ever being proven wrong

how epistemological argument here going to profit statisticalmethodology

add layers of warning that the probability behind the model isnot connected with the phenomenon

a debate to be continued, hopefully

See you at O’Bayes17:International Workshop on Objective Bayes Methodology

held in Austin, Texas, Su, December 10 through We, December13, 2017

[https://sites.google.com/site/obayes2017/]