Everything You Always Wanted To Know About Limits*

Everything You Always Wanted To Know About

Limits*Roger Barlow

Manchester UniversityYETI06

*But were afraid to ask

Roger Barlow: YETI06 Everything you wanted to know about limits

Slide 2

SummaryPrediction

confronts data & sees

small/zero signal

Frequentist

probability and

Confidence

Level languageBayesian

Probability

(Health

Warning)Gaussian

ln L= -½

Zero events

Few events:

Confidence belt The horrendous

case of large

backgrounds

Extension to

several

parameters

Likelihood


Slide 3

Model predictionsInput model and parameters

Low energyLagrangian

Feynman Rules for Feynman

diagrams

Cross Sectionsand Branching

RatiosExperiment duration,

luminosity,Efficiency etc

Number of events

Monte Carloprograms

Cuts designed tobring out signal

Data


Slide 4

What happens if there’s nothing there?

Even if your analysis finds no events, this is still useful information about the way the universe is built

Want to say more than: “We looked for X, we didn’t see it.”

Need statistics – which can’t prove anything.

“We show that X probably has a mass greater than../a coupling smaller than…”


Slide 5

Probability(1): Frequentist

Define Probability of X as P(X)=Limit N∞ N(X)/N

Examples: coins, dice, cards For continuous x extend to Probability

DensityP(x to x+dx)=p(x)dx

Examples: • Measuring continuous quantities (p(x)

often Gaussian)• Parton momentum fractions (proton pdfs)


Slide 6

Digression: likelihood

Probability distribution of random variable x often depends on some parameter a.

Joint function p(x,a)Considered as p(x)|a this is the pdf.

Normalised: ∫p(x)dx=1Considered as p(a)|x this is the Likelihood L(a)Not ‘likelihood of a’ but ‘likelihood that a

would give x’Not normalised. Indeed, must never be

integrated.


Slide 7

Limitation of Frequentist Probability

Have to say“The statement ‘It will rain tomorrow.’ is

probably true.”Can then even quantify (meteorology).

Can’t say“It will probably rain tomorrow.”

There is only one tomorrow. P is either 1 or 0


Slide 8

Interpreting physics results: Mt =173±2 GeV/c2

Can’t say ‘Mt has a 68% probability of lying between 171

and 175 GeV/c2’Have to say‘The statement “Mt lies between 171 and 175

GeV/c2”has a 68% probability of being true’i.e. if you always say a value lies within its error

bars, you will be right 68% of the time.Say “Mt lies between 171 and 175 GeV/c2” with

68% Confidence. Or 169-177 with 95% confidence. Or…


Slide 9

Interpreting null resultYour analysis searches for events. Sees none.Use Poisson formula: P(n; )=e-n/n!Small could well give 0 events =0.5 gives P(0)=61% =1.0 gives P(0)=37% =2.3 gives P(0)=10% =3.0 gives P(0)=5%If you always say ‘ 3.0’ you will be right (at least)

95% of the time. 3.0 – with 95% confidence (a.k.a 5% significance.)

‘If is actually 3, or more, the probability of a fluctuation as far as zero is only 5%, or less.’

given by model parameters. Limit on translates to limit on mass, coupling, ,branching ratio or whatever


Slide 10

Probability(2): Bayesian

P(X) expresses by degree of belief in XCan calibrate against cards, dice, etc.Extend to probability density p(x) as

beforeNo restrictions on X or x. Rain, MT, MH,

whateverInterpret physics results using Bayes’

Theorem:pposterior(a|data) p(data|a) x pprior(a)


Slide 11

Bayes at work

= x

P(0 events|)

(Likelihood)

Prior: uniformPosterior P()

3 P() d= 0.95

0

Same as Frequentist limit - Happy coincidence

Zero events seen


Slide 12

Bayes at work again

= x

P(0 events|) Prior: uniform in ln Posterior P()

3 P() d >> 0.95

0

Is that uniform prior really credible?

Upper limit totally different!


Slide 13

Bayes: the bad news• The prior affects the posterior. It is your choice. That

makes the measurement subjective. This is BAD. (We’re physicists, dammit!)

• A Uniform Prior does not get you out of this.• SUSY ‘parameter space’ is not a ‘phase space’• Attempts to invent universally-agreed priors

(‘Objective’ and/or ‘Reference’ Priors) have not worked

Better news: If there is a lot of data then the prejudicial effects of the choice of prior can be small.

• This should ALWAYS be checked for (‘robustness under choice of prior’.)


Slide 14

Frequentist versus Bayesian?

Statisticians do a lot of work with Bayesian statistics and there are a lot of useful ideas. But they are careful about checking for robustness under choice of prior.

Beware snake-oil merchants in the physics community who will sell you Bayesian statistics (new – cool – easy – intuitive) and don’t bother about robustness.

Use Frequentist methods when you can and Bayesian when you can’t (and check for robustness.) But ALWAYS be aware which you are using.


Slide 15

A Gaussian MeasurementNo problems

p(x)=exp[-(x-)2/2 2]/√2x: symmetric

x is within ± of with 68% probability is within ± of x at 68% confidencex is above -1.645 with 95% probability is below x+1.645 at 95% confidenceChoice of confidence level and arrangement

Can read regions off log likelihood plot asL(a)=exp[-(x-)2/2 2]/√2

Ln L -(-x)2/2 2

68% region corresponds to fall of ½ from peak


Slide 16

A Poisson measurement

You detect 5 events. Best value 5. But what about the errors?

1. 5±√5=5±2.24 Assumes e-n/n! is Gaussian in n. True only for large - and 5 is small

2. Find points where log likelihood falls by ½.

Assumes e-n/n! is Gaussian in .Gives upper error of 2.58, lower error of 1.92


Slide 17

3: Doing it properly: Confidence belt (Neyman

interval)Use e-n/n! For any true the

probability that (n, ) is within the belt is 68% (or more) by construction

For any n, lies in [-, +] at 68% confidence

Get upper error 3.38, lower error 2.16

n

-

+

68%

16%16%

Technique works for any CL, and single or double sided


Slide 18

Consumer guide

ln L =- ½ is a standard and easy to use. Fine for everyday use. (Though for a simple count the Neyman limit is quite easy)

For 90% 1-sided (upper) limit use ln L =-0.82 (1.28 ) For 95% use ln L =-1.35 (1.645 ) Just plot the likelihood and read off the

value. Then translate back to model parameters


Slide 19

Frequency method: the big problem

Observe 5 events. Expected background of 0.9 events.Data = signal + background

Say with 68% confidence: data in range 2.84 to 8.38So say with 68% confidence: signal in range 1.94 to

7.48Suppose expected* background 4.9. Or 6.9. Or 10.9 ?“We say at 68% confidence that the number of signal

events lies between -8.06 and -2.52”This is technically correct. We are allowed to be wrong

32% of the time. But stupid. We know that the background happens to have a downward fluctuation but have no way of incorporating that knowledge

*We assume that the background is calculated correctly


Slide 20

Strategy 1: Bayes

Prior is uniform for positive , zero for negative . No problem.

Get requirement (for n observed, known background b, 90% upper limit)

0.1=nexp(-+-b) (++b)r/r!

nexp(-b) br/r!Known as “the old PDG formula” or

“Helene’s formula” or “that heap of crap”


Slide 21

Strategy 2: Feldman-Cousins

Also called* ‘the Unified Approach’Real physicists wait to see their result and then

decide whether to quote an upper limit or a range.This ‘flip-flopping’ invalidates the method.They provide a procedure that incorporates it

automatically, and always gives non-stupid results.Critics say (1) can lead to experiments quoting a

range when they’re not claiming a discovery (2) is computationally intensive and (3) For zero observed events, the higher the background estimate the better (i.e. lower) the limit on signal

* By Feldman and Cousins, principally


Slide 22

Strategy 3: CLs

As used by LEP Higgs working groupGeneralisation of Helene formulaSome quantity Q. Could be number of

events, or something more cleverCLb=P(Q or less|b) CLs+b=P(Q or less|s+b)

CLs=CLs+b/CLb

Used as confidence level. Optimise strategy using it and quote results


Slide 23

2(+) parameters Fix b, find 68% confidence

range for a, using ln L=-½

Fix a, find 68% range for bCombination (square) has

0.682=46%

a

b

L(a,b)

ln L=-½ circle has 39% ConfidenceDefine regions through contours of log L – Confidence

content given by 2ln L= for which P(n)=CL Caution! Cannot redefine a as b+c+d, claim 3

parameters and cut with P(3) instead of P(1)


Slide 24

SummaryPrediction

confronts data & sees

small/zero signal

Frequentist

probability and

Confidence

Level languageBayesian

Probability

(Health

Warning)Gaussian

ln L= -½

Zero events

Few events:

Confidence belt The horrendous

case of large

backgrounds

Extension to

several

parameters

Likelihood


Slide 25

Remember!

Zero events = 95% CL upper limit of 3 events

If it’s more involved, plot the likelihood function and use ln L=-½ for 68% central, etc

Be suspicious of anything you don’t understand

If you’re integrating the likelihood you are a Bayesian. I hope you know what you’re doing.


Slide 26

Further Reading

• Workshop on Confidence Limits, CERN yellow report 2000-005

• Proc. Conf. Advanced Statistical Techniques in Particle Physics, Durham, IPPP02/39

• Proc. PHYSTAT03 – SLAC-R-703• Proc PHYSTAT05, Oxford -

forthcoming

Documents

Everything You Always Wanted To Know About Limits*