Version 2.20July 1, 2009

Search Procedures in High Energy Physics

Luc Demortier 1

Laboratory of Experimental High-Energy Physics

The Rockefeller University

Abstract

The usual procedure for searching for new phenomena in high energy physicsinvolves a frequentist hypothesis test followed by the construction of an intervalfor the parameter of interest. This procedure has a couple of well-known flaws:the effect of the test on subsequent inference is ignored, and in some circum-stances the size of the reported interval does not properly reflect experimentalconditions. Furthermore, proper treatment of nuisance parameters in a frequen-tist context is nearly always troublesome. For these reasons there has recentlybeen considerable interest in applying the ideas of Bayesian reference analysis tothis problem. We describe ongoing work to calculate reference priors for searchexperiments, via both numerical and analytical methods. We also show howintrinsic interval estimation can provide a very elegant solution to the testingproblem in search procedures.

Prepared for O-Bayes09, the Seventh International Workshop on Objective Bayes Method-ology, Philadelphia, USA, June 5–9, 2009.

1luc.demortier@rockefeller.edu

2 CONTENTS

Contents

1 Introduction and Motivation 3

2 Reference Priors with Partial Information 62.1 The Single-Count Model . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 Method 1 Applied to the Single-Count Model . . . . . . . . . . . . . . 72.3 Method 2 Applied to the Single-Count Model . . . . . . . . . . . . . . 92.4 Marginal Priors and Posteriors . . . . . . . . . . . . . . . . . . . . . . . 112.5 Generalizations of the Single-Count Model . . . . . . . . . . . . . . . . 12

3 Upper Limits 14

4 Coverage 14

5 Intrinsic intervals 155.1 Conditional approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175.2 Marginal approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

6 Reference Analysis Search Procedures 18

7 Open Questions 22

Acknowledgments 22

Figures 23

References 35

3

1 Introduction and Motivation

One of the main goals of experimental high energy physics is to discover new particlesthat may shed light on such basic questions as the origin of mass, the architecture offundamental forces, the origin of the universe, and the nature of dark matter and darkenergy. The principal tools for these investigations are the large particle acceleratorcenters currently operating at Fermilab (near Chicago, IL), at CERN (near Geneva,Switzerland), and in a few other places. Accelerated particles are collided with eachother at very high energies and very high rates, and state-of-the-art detectors are usedto search the collision products for evidence of new particles. The great complexityof these experiments has motivated physicists to take a closer look at the statisticalaspects of the procedures they use to search for new phenomena and claim discoveries.

In a typical search the new particle of interest, or “signal”, is produced (if it exists)in a very small fraction of collisions; however, its signature in the detecting apparatusis often mimicked by more common particles. Although this background can almostnever be completely eliminated, it can be minimized by selecting collision events withappropriate characteristics. The background contamination µ that remains after eventselection must then be quantified and subtracted from the observed event rate in orderto obtain the signal rate. The latter is proportional to a quantity of intrinsic physicsinterest, the so-called signal cross section σ; the proportionality factor is a product ofaccelerator dependent parameters and event selection efficiencies collectively referredto as the effective luminosity ε. In general the process by which collision events areselected for further study is very stringent and rejects the vast majority of collisions.Thus the final observed event rate obeys Poisson statistics to high accuracy, and thePoisson mean can be written as εσ + µ. In the remainder of this paper we will assumethat after event selection we are left with a data sample of n events, and that someinformation is available about the true values of ε and µ.

For historical reasons physicists tend to favor frequentist approaches to statisticalproblems, and their standard search procedure takes the following form:

1. Choose three confidence levels α1, α2, and α3.

2. Find a test statistic T that is sensitive to the presence of signal.

3. Compute the p value corresponding to the observed value of T .

4. If p ≤ 1 − α1, claim discovery and compute an α2 confidence level two-sidedinterval on the signal cross section σ.

5. If p > 1 − α1, make no claim, and compute an α3 confidence level upper limiton σ.

Common confidence level choices are 1 − α1 = 2.8 × 10−7, α2 = 0.68, and α3 = 0.95.The two-sided interval calculated at step 4 provides an estimate of the magnitudeof the observed effect. When no discovery is claimed, the upper limit at step 5 isan upper bound on values of the signal cross section σ that the experiment cannotconvincingly distinguish from zero. In a sense, this bound serves as a measure of the

4 1 INTRODUCTION AND MOTIVATION

severity of the test to which the background-only model was subjected [14]: the smallerthe upper limit, the smaller a deviation from background the test is able to detect, andthe stronger the case is in favor of the background-only model when the test failedto reject it. The upper limit is also an indicator to future experiments of what theirsensitivity should be if they want to probe the same phenomenon more deeply.

Statisticians are often surprised by the stringency of the p value test in the aboveprocedure: in a standard normal distribution, a one-sided tail probability of 2.8× 10−7

corresponds approximately to a 5σ deviation from the mean. This high discoverythreshold was suggested more than 40 years ago by the physicist A. Rosenfeld [16],who was concerned by the so-called “look-elsewhere” effect, whereby the probability ofa large background fluctuation increases with the number of independent tests that areperformed. He estimated the number of histograms that are examined every year byhigh energy physicists, and with a few simple assumptions concluded that one shouldexpect several spurious discovery claims at the 4σ level per year. This led him torecommend a 5σ threshold.

In a paper widely cited by experimental physicists, Feldman and Cousins (them-selves experimental physicists) criticized the above search procedure for yielding in-tervals that do not have exact frequentist coverage, and in fact undercover, becausethe p value test is not incorporated in the frequentist reference ensemble of these inter-vals [11]. In other words, the decision to report an upper limit or a two-sided interval isbased on the result of the test, but the intervals are constructed as if that decision hadbeen made before looking at the data. The authors called this practice “flip-flopping”.A second problem they noted is that in some cases the above procedure yields anempty interval, a result that, even though correct from a frequentist point of view,is unsatisfying from a scientific one. The solution they proposed was to replace thestandard search procedure by a Neyman interval construction with a likelihood ratioordering rule. In addition to never producing an empty interval 2, this rule has theproperty that the resulting interval smoothly transitions from one-sided to two-sidedas the observed effect increases in strength. Thus the decision to claim discovery at agiven significance level depends on whether or not the corresponding likelihood ratiointerval contains the zero-signal value σ = 0, and the choice between reporting anupper limit or a two-sided interval is taken out of the hands of the user and becomesautomatic.

Although this method has become quite popular, it does seem to remove the degreesof freedom that the standard procedure offers in the choice of the confidence levels α1,α2, and α3. The Feldman-Cousins construction requires these three α’s to be equal,resulting either in intervals that are too wide to be informative or in tests that are toolenient to be trustworthy. To address this issue, the authors propose to report morethan one interval for each measurement. One could for example report three intervals,each corresponding to one of the confidence levels of the standard procedure. This doesnot solve all problems however. If the α1 confidence interval includes the null valueσ = 0, one cannot claim a discovery, but there is no guarantee that the α3 confidenceinterval will be an upper limit, as it should be in order to serve the same necessary

2Although intervals constructed with a likelihood-ratio ordering rule are never empty, they some-times have very low Bayesian credibility; see [9] for an example.

5

purpose as in the standard procedure. Furthermore, if a consumer down the line adoptsthe policy of using the α2 interval every time a discovery is claimed and the α3 intervalotherwise, then he or she falls back into the undercoverage problem of the standardprocedure. Thus, while the Feldman-Cousins procedure is useful for characterizing theresult of a measurement, it does not allow one to follow through with a choice of intervalonce a decision has been made regarding discovery, essentially rendering that decisiontoothless. One of the authors argues that this last step, the act of deciding aboutdiscovery and drawing further inferences, should be handled via a subjective Bayesianapproach [7], but it is not entirely clear how a set of Feldman-Cousins intervals canserve as input to such a procedure.

Some statisticians may wonder why the flip-flopping problem wasn’t solved byadopting a conditional frequentist approach: in principle one could construct post-test intervals conditionally on the result of the test, simply by appropriately restrictingthe sample space and renormalizing the relevant probability densities [9]. The answer,unfortunately, is that most physicists are not very familiar with conditional frequen-tism, and all the possible ramifications of such an approach have not been investigatedyet. In addition, this approach does not necessarily solve the empty-interval problem.

The Feldman-Cousins construction has another well known problem [12]: when theobserved number of events is smaller than the expected background, the upper limit onthe signal decreases rapidly as the expected background increases. This would argueagainst efforts at optimizing the composition of the dataset, since a larger backgroundcontamination will ensure a more stringent upper limit. The authors respond that whengiven two experimental results, one should compare expected upper limits in additionto the observed ones, the former being a measure of the sensitivity of an experiment.It is interesting that Bayesian methods do not have this problem because they obeythe likelihood principle. For example, when no events are observed and when thebackground µ and effective luminosity ε are known exactly, then the Bayesian upperlimit on σ is independent of µ if the σ prior is flat.

Finally, as with all frequentist methods, there is the problem of nuisance parameters.In principle it can be handled via Neyman’s construction, but as the number of suchparameters increases this approach quickly becomes unmanageable in practice. Simplersolutions involve profiling or marginalizing the likelihood with respect to the nuisanceparameters, but the coverage of the resulting intervals is not guaranteed and must beinvestigated on a case-by-case basis.

The above discussion suggests that an objective Bayesian approach may providea solution to the problems encountered. We propose here to follow the methodol-ogy known as reference analysis and advocated by J. Bernardo, J. Berger, and theircollaborators [5, and references therein]. This approach has several advantages thatshould resonate with the concerns of high energy physicists: the inferences it yieldsabout physics parameters are transformation invariant; it has good frequentist cover-age properties; it is very general; it is computationally tractable; it avoids the famousmarginalization paradoxes discovered in the 1970’s; and it can (and should) be embed-ded in a subjective Bayesian framework.

The remainder of this paper is organized as follows. In the next section we describebriefly the calculation of reference priors when partial information is available about

6 2 REFERENCE PRIORS WITH PARTIAL INFORMATION

the nuisance parameters, a common situation in high energy physics. We apply thistechnology to the so-called single-count model, basically a Poisson pdf with mean µ+εσas described earlier in this introduction. We examine the marginal reference prior andposterior for this model in section 2.4. Ongoing work on numerical calculations dealingwith various generalizations of the single-count model is described in section 2.5. Westudy reference posterior upper limits in section 3 and the frequentist coverage ofreference posterior intervals in section 4. Intrinsic intervals for the single-count modelare introduced in section 5 and are used to define a Bayesian search procedure insection 6. Finally, some open questions are listed in section 7.

2 Reference Priors with Partial Information

A good description of the reference prior algorithm for any number of nuisance andinterest parameters is provided in Ref. [5]. In contrast with the general situation de-scribed in that reference, most nuisance parameters encountered in high energy physicscome with non-trivial prior information, in the form of auxiliary measurements, MonteCarlo simulations, and/or theoretical beliefs. The incorporation of this partial infor-mation in the construction of reference priors is described in Ref. [17]. Suppose that φlabels the nuisance parameter(s) and θ the parameter of interest. There are two waysto proceed:

Method 1: Assume that we are given a marginal prior π(φ) for the nuisance parame-ters; compute the conditional reference prior πR(θ |φ) for the interest parametergiven a fixed value of φ; the full prior is then π(θ, φ) = πR(θ |φ) π(φ);

Method 2: Assume that we are given a conditional prior π(φ | θ) for the nuisance pa-rameter given the interest parameter; marginalize the probability model p(x|θ, φ)with respect to φ in order to obtain p(x|θ) =

∫p(x|θ, φ) π(φ|θ) dφ, and com-

pute the reference prior πR(θ) for the marginalized model; the full prior is thenπ(θ, φ) = πR(θ) π(φ | θ).

In many high energy physics measurements there are physics reasons for assumingthat the nuisance parameter is independent of the parameter of interest. Informationabout a detector energy scale for example, is typically determined separately from themeasurement of interest, say of a particle mass, and is therefore considered to be a prioriindependent from one’s information about that particle mass. When an experimenteris willing to make this assumption, he or she can declare that π(φ | θ) = π(φ) and useMethod 2. When this assumption does not seem fully justified, and it is too difficultto elicit the θ dependence of π(φ | θ), then it will seem preferable to use Method 1,which only requires knowledge of the marginal prior π(φ). When one is unsure ofwhich method to use, one should use both, and treat the results as part of a testof robustness. An important practical advantage of Method 1 is that the conditionalreference prior is computed once and for all, for a given model, and can be used with anysubjective prior for the nuisance parameters. In contrast, for Method 2 the referenceprior must be recomputed every time the subjective priors change.

2.1 The Single-Count Model 7

2.1 The Single-Count Model

As indicated earlier, the basic observable in high energy physics is an event count N(a number of collisions passing a predefined selection procedure) that obeys Poissonstatistics. In a very common measurement model the expectation value of N has theform ε σ + µ, where σ is the cross section of a physics signal process, which we detectwith an effective luminosity ε, and µ is a background contamination. Thus, σ is theparameter of interest, whereas ε and µ are nuisance parameters for which we usuallyhave partial information. For physical reasons none of these three parameters can benegative. The likelihood for this model is given by

p(n|σ, ε, µ) =(ε σ + µ)n

n!e−ε σ−µ with 0 ≤ σ < ∞ and 0 < ε, µ < ∞. (2.1)

Information about ε and µ usually comes from a variety of sources, such as auxiliarymeasurements, Monte Carlo simulations, theoretical calculations, and evidence-basedbeliefs (for example, some sources of background contributing to µ may be deemed smallenough to ignore, and some physics effects on ε may be believed to be well enoughreproduced by the simulation to be reliable “within a factor of 2”). It is thereforenatural to represent that information by a subjective prior π(ε, µ). The problem weare facing is that of finding a prior for σ, about which either little is known or onewishes to pretend that this is so.

2.2 Method 1 Applied to the Single-Count Model

In method 1 [17, section 2.3], we construct the conditional reference prior πR(σ | ε, µ).The first step consists in calculating Jeffreys’ prior for σ while holding ε and µ fixed:

πJ(σ | ε, µ) ∝

E[− ∂2

∂σ2ln p(n |σ, ε, µ)

] 12

∝ ε√ε σ + µ

. (2.2)

This prior is improper with respect to σ however, so that an additional step, knownas the “compact support argument,” is required. One starts by choosing a nestedsequence Λ1 ⊂ Λ2 ⊂ · · · of compact subsets of the parameter space Λ for (σ, ε, µ), suchthat ∪iΛi = Λ and the integral Ki(ε, µ) of πJ(σ | ε, µ) over Ωi ≡ σ : (σ, ε, µ) ∈ Λi isfinite. The conditional reference prior for σ on Ωi is then:

πR,i(σ | ε, µ) =πJ(σ | ε, µ)

Ki(ε, µ)IΩi

(σ), (2.3)

where IΩi(σ) is the indicator function of σ in Ωi. To obtain the conditional reference

prior on the whole parameter space, one chooses a fixed point (σ0, ε0, µ0) and takes thelimit of the ratio:

πR(σ | ε, µ) = limi→∞

πR,i(σ | ε, µ)

πR,i(σ0 | ε0, µ0). (2.4)

The theory of reference priors does not currently provide guidelines for choosing thecompact sets Λi, other than to require that the resulting posterior be proper. In most

8 2 REFERENCE PRIORS WITH PARTIAL INFORMATION

cases this choice makes no difference and one is free to base it on considerations ofsimplicity and convenience. However, we have found that some care is required withthe single-count model. Indeed, suppose we choose:

Λi =

(σ, ε, µ) : σ ∈ [0, ui], ε ∈ [0, vi], µ ∈ [0, wi]

, (2.5)

where ui, vi, and wi are increasing sequences of positive constants. If we usethese sets, applying eqs. (2.3) and (2.4) to the prior (2.2) yields:

πR,i(σ | ε, µ) =1

Ki(ε, µ)

ε√ε σ + µ

I[0,ui](σ), (2.6)

where:

Ki(ε, µ) ≡∫

Ωi

ε√ε σ + µ

dσ = 2[√

ε ui + µ−√µ], (2.7)

and therefore:

πR(σ | ε, µ) ∝√

ε

ε σ + µ. (2.8)

Although this prior is still improper with respect to σ, its dependence on ε is differentfrom that of the conditional Jeffreys’ prior (2.2). This demonstrates the potentialimportance of the compact subset argument. The prior (2.8) has a serious problemhowever. Suppose that the marginal prior for ε and µ can be factorized as follows:

π(ε, µ) =e−ε√π ε/2

π(µ). (2.9)

The full posterior density for (σ, ε, µ) is then:

p(σ, ε, µ |n) ∝ (ε σ + µ)n

n!e−ε σ−µ

√ε

ε σ + µ

e−ε√π ε/2

π(µ). (2.10)

It is easy to see that this posterior is improper. Indeed, integrating the right-hand sideof the above expression over σ yields:

Γ(n + 12)

Γ(n + 1)

[1− P (n +

1

2, µ)

] √2

π

e−ε

επ(µ), (2.11)

with P (a, x) the incomplete Gamma function. The ε dependence of this expression ise−ε/ε, which is not integrable over the range of ε. The cause of this problem is thechoice of compact sets (2.5).

Fortunately it is not difficult to find a sequence of compact sets that lead to a properposterior. A hint is provided by the fact that the Jeffreys prior (2.2) yields a “density”ε dσ/

√εσ + µ that is invariant under scale transformations ε → cε, σ → σ/c, where c is

constant. Surprisingly, this property is not shared by the prior (2.8), even though thelatter also depends on σ through the product εσ. This suggests that the compact sets

2.3 Method 2 Applied to the Single-Count Model 9

should be constructed in such a way that they respect the scale invariance of Jeffreys’prior. [2] Accordingly, we set:

Λi =

(σ, ε, µ) : σ ∈ [0, ui/ε], ε ∈ [1/vi, vi], µ ∈ [0, wi]

, (2.12)

where ui, vi, and wi are as before. Again using eqs. (2.2), (2.3), and (2.4), we now find:

πR1(σ | ε, µ) ∝ ε√εσ + µ

, (2.13)

which is identical to Jeffreys’ prior for this problem and yields well-behaved posteriors.For future use, the subscript R1 on the left-hand side indicates that this reference priorwas obtained with Method 1.

2.3 Method 2 Applied to the Single-Count Model

In contrast with Method 1, Method 2 requires from the start that we specify a subjectiveprior for the effective integrated luminosity ε and the background contamination µ.Furthermore, this specification must be done conditionally on the signal rate σ, so thatwe need an expression for π(ε, µ |σ). Here we will assume that ε and µ are independentof σ and that their prior factorizes as a product of two gamma densities:

π(ε, µ |σ) = π(ε, µ) =a(aε)x−1/2 e−aε

Γ(x + 1/2)

b(bµ)y−1/2 e−bµ

Γ(y + 1/2), (2.14)

where a, b, x, and y are known constants. There are two ways of interpreting this prior.The first one is appropriate when information about ε and µ comes from one or morenon-experimental sources, such as simulation studies and theoretical calculations, andtakes the form of a central value plus an uncertainty. Since the ε and µ componentsof the prior are each modeled by a two-parameter density, one can fix the parameters(x, a, y, b) by identifying the means of the component distributions with the centralvalues (ε, µ) of the measurements, and their coefficients of variation with the relativeuncertainties (δε, δµ):

ε =x + 1

2

a, δε =

1√x + 1

2

, or x =1

δε2− 1

2, a =

1

ε δε2; (2.15)

µ =y + 1

2

b, δµ =

1√y + 1

2

, or y =1

δµ2− 1

2, b =

1

µ δµ2. (2.16)

It will then be necessary to check the robustness of the final analysis results to rea-sonable changes in this procedure. For example, one may want to replace the gammadistribution by a log-normal or truncated Gaussian one, and identify the central valueof the measurement with the mode or median instead of the mean.

The second interpretation of prior (2.14) follows from the analysis of two indepen-dent, auxiliary Poisson measurements, in which the observed number of events is x for

10 2 REFERENCE PRIORS WITH PARTIAL INFORMATION

the effective luminosity and y for the background. The expected numbers of events inthese auxiliary measurements are aε and bµ, respectively. For a Poisson likelihood withmean aε the standard reference prior coincides with Jeffreys’ prior and is proportionalto 1/

√ε. Given a measurement x, the posterior will then be a gamma distribution

with shape parameter x + 1/2 and scale parameter 1/a. A similar result holds forthe background measurement. In this manner the prior (2.14) is obtained as a jointreference posterior from two auxiliary measurements.

Although interesting, we take the second interpretation with a grain of salt. Thisis because it implies that the data we have at our disposal is actually (n, x, y) ratherthan just n, and that our model is:

p(n, x, y |σ, ε, µ) =(µ + εσ)n e−µ−εσ

n!

(aε)x e−aε

x!

(bµ)y e−bµ

y!, (2.17)

instead of (2.1). However, the reference priors for these two models are not the same.Our position here is to assume that prior information is available about ε and µ, butwe do not look too deeply into how it was obtained.

The next step in the application of Method 2 is to marginalize the probabilitymodel (2.1) with respect to ε and µ:

p(n |σ) =

∫∫p(n |σ, ε, µ) π(ε, µ |σ) dε dµ, (2.18)

=

∫∫(εσ + µ)n

n!e−εσ−µ a(aε)x−1/2

Γ(x + 1/2)e−aε b(bµ)y−1/2

Γ(y + 1/2)e−bµ dε dµ, (2.19)

=

[a

a + σ

]x+ 12

[b

b + 1

]y+ 12

n∑k=0

unk

[σ

a + σ

]k

, (2.20)

where

unk =

(x− 1

2+ k

k

) (y − 1

2+ n− k

n− k

) [1

b + 1

]n−k

, (2.21)

and we used generalized binomial coefficients:(v

w

)≡ Γ(v + 1)

Γ(w + 1) Γ(v − w + 1). (2.22)

The marginal pdf p(n |σ) is shown as a function of σ for several values of n in Fig. 1.Aside from being the starting point of the calculation of Method 2 reference priors,this quantity will also be used to study the behavior of reference posteriors undermeasurement replications (section 4), and to compute intrinsic intervals (section 5.2).

Finally, the reference prior algorithm must be applied to the marginalized modelp(n |σ). Since this model involves a single, continuous parameter, the reference priorcoincides with Jeffreys’ prior; it can be written as:

πR2(σ) ∝

√√√√E

[d

dσln p(n |σ)

]2∝

√√√√ ∞∑n=0

[(x + 1/2) S0

n − (a/σ) S1n

]2

(a + σ)x+5/2 S0n

, (2.23)

2.4 Marginal Priors and Posteriors 11

with

Smn ≡

n∑k=0

km unk

[σ

a + σ

]k

for m = 0, 1. (2.24)

We will use the notation πR2(σ) to refer to the marginal reference prior for σ obtainedwith Method 2.

2.4 Marginal Priors and Posteriors

One way to compare the Method 1 and Method 2 reference priors calculated in theprevious sections is to examine their σ marginals, since σ is the parameter of interest.For this purpose we will use the nuisance prior introduced in equation (2.14). As thestarting point of Method 2 is the marginalized model (2.20), its marginals are alreadyknown: the prior is eq. (2.23), and the posterior is simply

πR2(σ |n) ∝ p(n |σ) πR2(σ). (2.25)

The normalization of the latter must be obtained numerically. Examples of the Method 2prior and posterior are shown in Figures 3 and 6 respectively.

Method 1 requires more work. The marginal cross section prior is:

πR1(σ) ∝∫ ∞

0

dε

∫ ∞

0

dµε√

εσ + µ

a(aε)x−1/2 e−aε

Γ(x + 12)

b(bµ)y−1/2 e−bµ

Γ(y + 12)

. (2.26)

To calculate this integral we first perform the parameter substitution

(ε, µ) → (ν, µ) where ν ≡ ε/µ. (2.27)

The differential transforms according to dε dµ = µ dν dµ and the integral becomes:

πR1(σ) ∝Γ(x + y + 3

2)

Γ(x + 12) Γ(y + 1

2)

∫ ∞

0

dν1√

νσ + 1

(aν)x+1/2 by+1/2

(aν + b)x+y+3/2. (2.28)

Recognizing a Gauss hypergeometric function [1] in this expression, we have:

πR1(σ) ∝ 2F1

(1

2, x +

3

2; x + y + 2; 1− b

aσ

). (2.29)

This prior is improper; indeed, using standard properties of the hypergeometric func-tion, it can be rewritten as

πR1(σ) ∝√

a

b σ2F1

(1

2, y +

1

2; x + y + 2; 1− a

b σ

). (2.30)

Thus, as σ →∞, πR1(σ) decreases only as 1/√

σ. Equations (2.29) and (2.30) indicatethat a and b are simple scaling parameters for the marginal prior, whereas x and y areshape parameters. Some examples of this prior are plotted in Figure 2 and comparedwith Method 2 priors in Figure 4.

12 2 REFERENCE PRIORS WITH PARTIAL INFORMATION

The Method 1 marginal posterior for the cross section σ is given by:

πR1(σ |n) ∝∫ ∞

0

dε

∫ ∞

0

dµε (εσ + µ)n− 1

2 e−εσ−µ

n!

a(aε)x− 12 e−aε

Γ(x + 12)

b(bµ)y− 12 e−bµ

Γ(y + 12)

. (2.31)

Performing the parameter substitution (2.27), we obtain here:

πR1(σ |n) ∝Γ(n + x + y + 3

2)

n! Γ(x + 12) Γ(y + 1

2)

by+ 12

∫ ∞

0

dν(νσ + 1)n− 1

2 (aν)x+ 12

(aν + σν + b + 1)x+y+n+ 32

. (2.32)

This can also be expressed with the help of a hypergeometric function:

πR1(σ |n) = K2F1(

12− n, x + 3

2; x + y + 2; 1−bσ/a

1+σ/a)

(1 + σ/a)x+ 32

, (2.33)

where K is a normalization constant. To compute it, we first integrate the right-handside of (2.31) over σ; the integral over ε is then trivial and that over µ yields anincomplete beta function. The final result is:

K =

(x + y + n + 1

2

n− 12

)x + 1

2

a

by+ 12

(1 + b)y+n

1

I b1+b

(y + 12, n + 1

2), (2.34)

where

Iz(u, v) ≡ Γ(u + v)

Γ(u) Γ(v)

∫ z

0

tu−1 (1− t)v−1 dt. (2.35)

Examples of the marginal Method-1 posterior are shown in Figure 5. Note the inter-esting feature that for n = 0 the order of the curves labeled y = 0, y = 1, and y = 10is inverted at small σ with respect to the case n > 0 (left-hand side plots).

2.5 Generalizations of the Single-Count Model

A straightforward and common generalization of the single-count model is the multi-binmodel (e.g. histograms), based on the likelihood:

p(~n |σ,~ε, ~µ) =M∏i=1

(µi + εi σ)ni

ni!e−µi−εi σ. (2.36)

To obtain the Method 1 reference prior for this model, we first calculate Jeffreys’ prior:

πJ(σ |~ε, ~µ) =

√√√√ M∑i=1

ε2i

µi + εi σ. (2.37)

Since it is improper, we need to apply the compact support argument described insection 2.2. We note that Jeffreys’ prior density πJ(σ |~ε, ~µ) dσ is invariant under the

2.5 Generalizations of the Single-Count Model 13

transformation that maps εi to cεi and σ to σ/c, with c a constant. Hence we constructcompact sets that respect this scale invariance:

Λ` =

(σ,~ε, ~µ) : σ ∈ [0, u`/ε+], εi ∈ [1/vi`, vi`], µi ∈ [0, wi`], i = 1, . . . ,M

, (2.38)

where ε+ ≡∑M

i=1 εi, and u`, vi`, and wi` are increasing sequences of constantswith respect to the index `. The remainder of the calculation is similar to that for thesingle-count model and concludes that here too the Method 1 reference prior equalsJeffreys’ prior. Note that we could have used a different sequence of compact sets here,one that generalizes (2.5) instead of (2.12). Had we done this, we would have obtaineda different form for the Method 1 reference prior; we have not checked whether thisalternate form leads to improper posteriors, as in the case of the single-count model.Our choice of (2.38) is solely motivated by consistency with (2.12).

We have not attempted to obtain analytical expressions for the marginal prior andposterior of the multi-bin model.

Further generalizations include cases where the µi and εi are correlated across bins,as well as unbinned likelihoods. To handle these more complicated situations we aredeveloping numerical algorithms in collaboration with Harrison Prosper and SupriyaJain [10]. For Method 1 the algorithm is as follows:

1 Set ~no to the array of observed event numbers.

2 For i = 1, . . . , I:

3 Generate (σi,~εi, ~µi) ∼ p(~no |σ,~ε, ~µ) π(~ε, ~µ).

4 For j = 1, . . . , J :

5 Generate ~nj ∼ p(~n |σi,~εi, ~µi).

6 Calculate d2[− ln p(~nj |σi,~εi, ~µi)]/dσ2i by numerical differentiation.

7 Average the J values of d2[− ln p(~n |σi,~εi, ~µi)]/dσ2i obtained

at line 6, and take the square root. This yields a numerical

approximation to the conditional Jeffreys’ prior π(σi |~εi, ~µi).

8 Histogram the σi values generated at line 3, weighing them by

π(σi |~εi, ~µi)/p(~no |σi,~εi, ~µi). This yields πR1(σ), the σ-marginal prior.

9 Histogram the σi values generated at line 3, weighing them by

π(σi |~εi, ~µi). This yields πR1(σ |~no), the σ-marginal posterior.

The generation step at line 3 is done via a Markov chain Monte Carlo procedure.The particular choice of parent distribution for the generated (σ,~ε, ~µ) triplets is mo-tivated by the desire to obtain weights with reasonably small variance at steps 8 and9. However, we discovered that the sampling density p(~n0 |σ,~ε, ~µ) π(~ε, ~µ) is not alwaysproper with respect to (σ,~ε, ~µ). When M = 1 for example (single-count model), thedistribution is improper if x ≤ 1/2. Propriety can then be restored by multiplying thesampling density by ε. Another feature of the above algorithm is that it does not im-plement the compact support argument. In the cases that we examined, this argument

14 4 COVERAGE

made no difference, but this may not be true for all problems our code seeks to solve.Unfortunately the current lack of guidelines in the choice of compact sets limits ourability to address this issue in the code.

The algorithm for Method 2 is much simpler, since all it does is calculate Jeffreys’prior on a marginalized likelihood provided by the user. In addition, it does not requirea compact support argument. We will not discuss it further in this paper.

In the following sections we consider only the single-count model.

3 Upper Limits

A common way to summarize posterior distributions is via the computation of intervals.Figures 7 and 8 show some 68% and 95% credibility level upper limits and centralintervals for the signal cross section σ as a function of the observed number of eventsNobs. These were obtained with the Method 1 reference posterior (2.33). As Nobs

increases, the interval boundaries become straight lines. There is no difference betweenMethod 1 and Method 2 on the scale of these figures. Figure 9 provides a closer lookat the difference between the two methods when Nobs is small. The difference increaseswith the credibility level, but goes down to zero as Nobs becomes large. This latterbehavior is expected since the likelihood dominates inferences at large Nobs.

Figure 10 illustrates the variation of Method 1 posterior upper limits as a functionof the prior mean background µ, when the prior relative uncertainty (i.e. the coefficientof variation) δµ on the background is kept constant. These quantities are related tothe prior parameters y and b according to eq. (2.16). In these plots the prior effectiveluminosity ε is assigned a mean of 1 and a relative uncertainty identical to that on thebackground (ε = 1, δε = δµ). These plots show that once the mean background exceedsthe observed number of events, the upper limits vary little. This is to be contrastedwith the physically undesirable behavior of Feldman-Cousins limits, which decreasequickly with the mean background, even when no events are observed (see for exampleFigure 2 in [12]).

Another interesting feature of Fig. 10 is that for n = 0 the upper limit increaseswith the mean background, whereas for n > 0 it decreases. This behavior is directlyrelated to the corresponding inversion of the y-labeled posterior densities in Fig. 5,mentioned at the end of section 2.4. It is due to the form of the reference prior for σ.

A comparison of the left and right panels of Figure 10 also demonstrates how upperlimits increase with the uncertainties on background and effective luminosity.

4 Coverage

A well known property of subjective Bayesian intervals is that they cover exactly, whenthe coverage (a function of the parameters) is averaged over the prior. Similarly, fre-quentist intervals have exact credibility when the latter (a function of the observeddata) is averaged over the prior-predictive distribution. This is a straightforward con-sequence of the law of total probability [15].

15

For a Bayesian, averaging over subjective priors is very natural since such priorsprovide proper, data-independent measures over parameter space. The same is notnecessarily true for objective priors however. Thus, to calibrate an inference abouta parameter with an objective prior, one option for Bayesians is to check how theinference behaves under replication of the measurement, i.e. its pointwise coverage, asopposed to average coverage. For measurements that combine subjective and objectivepriors, the logical extension of these procedures is to average over the subjective priorsand check pointwise coverage with respect to the remaining parameters. Reference [8]writes:

With the rapid advances in computational techniques for Bayesian statisticsthat exploit the increased computing power now available, researchers areable to adopt more realistic, and usually more complex, models. However,it is then less likely that the statistician will be able to properly elicitprior beliefs about all aspects of the model. Moreover, many parametersmay not have a direct interpretation. This suggests that there is a needto develop general robust methods for prior specification that incorporateboth subjective and nonsubjective components. In this case, the matchingproperty could be recast as being the approximate equality of the posteriorprobability of a suitable set and the corresponding frequentist probabilityaveraged over the parameter space with respect to any continuous priorthat preserves the subjective element of the specified prior. [pg. 27]

In our case, adopting this procedure means that we will calculate the coverage withrespect to the marginalized data pdf p(n |σ); this is equation (2.20) for the case of thesingle-count model.

We start by calculating posterior upper limits and intervals on σ for an observednumber of events n going from 0 to 200. The sum of p(n |σ) over this range of n valuesequals 1 within a fraction of a percent, for σ as high as 25 and for the x, y, a, b valuesused in the figures. For a given true value of σ, the coverage of a set of intervals isdefined as the sum of p(n |σ) over all n values for which σ is bracketed by the cor-responding interval. Figures 11, 12, and 13 show the coverage of the Method 1 andMethod 2 intervals plotted in Figures 7 and 8. Due to the large number of discontinu-ities in the coverage curves, the coverage was evaluated at 1001 equidistant values ofσ ∈ [0, 25] and plotted as a set of unconnected points. As expected, central intervalsundercover near σ = 0 since they always exclude that value, and upper limits overcoverthere since they always include it. The coverage of a 68% upper limit or interval settlesmuch faster around its nominal level than the coverage of a 95% upper limit or interval.

5 Intrinsic intervals

Intrinsic intervals are regions of parameter space with lowest posterior loss, for a specialtype of loss function [6]. In brief, suppose we have one observation x from a modelp(x | θ) and we are interested in the true value of θ. Let `(θ0, θ) be the loss suffered ifthe value θ0 is used as a proxy for the unknown true value of θ in some application of

16 5 INTRINSIC INTERVALS

our measurement result. The posterior expected loss from using θ0 is then:

l(θ0 |x) = Eθ |x

[`(θ0, θ)

]=

∫dθ `(θ0, θ) π(θ |x). (5.1)

The idea is to use the θ value with lowest expected loss as a point estimate of θ, andregions of lowest expected loss as interval estimates.

Reference analysis proposes to use intrinsic discrepancy to define the loss function.In its general form, the intrinsic discrepancy loss function is:

δθ0, θ = min

∫dx p(x | θ0) ln

p(x | θ0)

p(x | θ),

∫dx p(x | θ) ln

p(x | θ)p(x | θ0)

. (5.2)

If nuisance parameters ν are present, the definition of δθ0, θ includes an additionalminimization:

δθ0, (θ, ν) = infν0

min

∫dx p(x | θ0, ν0) ln

p(x | θ0, ν0)

p(x | θ, ν),

∫dx p(x | θ, ν) ln

p(x | θ, ν)

p(x | θ0, ν0)

. (5.3)

When compared with other loss functions, intrinsic loss has the advantage of being in-variant under one-to-one transformations of the parameter(s), under one-to-one trans-formations of the data, and under reduction of the data by sufficiency.

Unfortunately Ref. [6] does not discuss situations like ours, where (1) the parameterof interest is not identifiable via the model (only a specific combination of interest andnuisance parameters is identifiable), and (2) the nuisance parameters are constrained bysubjective priors. If we were to minimize the intrinsic discrepancy loss over the nuisanceparameters (ν0) we would obtain zero, because any difference between p(x | θ0, ν0) andp(x | θ, ν) in the log-likelihood ratios of eq. (5.3) can always be exactly cancelled by anappropriate change in ν0.

In principle there are two ways of solving this problem, which we shall label “con-ditional” and “marginal”, respectively.

1. In the conditional approach the loss `((θ, ν), (θ0, ν)) is evaluated from the fullmodel p(x | θ, ν) by assuming that the nuisance parameters ν are exactly known.The expected loss is then obtained by averaging `((θ, ν), (θ0, ν)) over the posteriorπ(θ, ν |x).

2. In the marginal approach the loss `(θ, θ0) is evaluated from the marginal modelp(x | θ) =

∫dν p(x | θ, ν) π(ν) and is therefore independent of nuisance parame-

ters. The expected loss is then obtained by integrating the loss function over themarginal posterior π(θ |x) =

∫dν π(θ, ν |x).

A possible objection against the conditional approach is that we may not want the losssuffered from using the wrong value of θ to depend on nuisance parameters that areconceptually independent of θ, and that are, furthermore, unknown.

We now apply these two approaches to our single-count Poisson model with gammapriors.

5.1 Conditional approach 17

5.1 Conditional approach

Here we work with a simple Poisson model:

p(n |σ, µ, ε) =(µ + εσ)n e−µ−εσ

n!, (5.4)

where ε and µ are assumed known. The intrinsic loss is:

δσ0, σ = minκσ0 |σ, κσ |σ0

, (5.5)

where:

κσ0 |σ =∞∑

n=0

(µ + εσ)n e−µ−εσ

n!ln

(µ + εσ)n e−µ−εσ

(µ + εσ0)n e−µ−εσ0, (5.6)

=∞∑

n=0

(µ + εσ)n e−µ−εσ

n!

[n ln

µ + εσ

µ + εσ0

+ ε (σ0 − σ)

], (5.7)

= (µ + εσ) lnµ + εσ

µ + εσ0

+ ε (σ0 − σ). (5.8)

To compute the minimum (5.5), we need to find out for what values of σ we haveκσ0 |σ < κσ |σ0, or:

(µ + εσ) lnµ + εσ

µ + εσ0

+ ε (σ0 − σ) < (µ + εσ0) lnµ + εσ0

µ + εσ+ ε (σ − σ0), (5.9)

or

g(σ) ≡ [2µ + ε(σ + σ0)] lnµ + εσ

µ + εσ0

+ 2ε (σ0 − σ) < 0. (5.10)

To figure this out, note that:

g′(σ) = ε lnµ + εσ

µ + εσ0

+ε2 (σ0 − σ)

µ + εσ, (5.11)

g′′(σ) = −ε3 (σ0 − σ)

(µ + εσ)2. (5.12)

Hence when σ < σ0, g′′(σ) < 0 so that g′(σ) is decreasing. Combined with the factthat g′(σ0) = 0, this implies that g′(σ) > 0. Thus, g(σ) increases for σ < σ0, and sinceit is zero at σ0, it must be negative for σ < σ0. Therefore, δσ0, σ = κσ0 |σ forσ < σ0. By symmetry we have δσ0, σ = δσ, σ0 = κσ |σ0 for σ > σ0. Weconclude that

δσ0, σ |µ, ε = ε |σ0 − σ| −[µ + ε min(σ, σ0)

] ∣∣∣∣ln µ + εσ

µ + εσ0

∣∣∣∣ , (5.13)

where we reintroduced the nuisance parameters µ and ε in the argument list of δ. Theposterior expected intrinsic loss is then

d(σ0 |n) =

∫dσ

∫dµ

∫dε δσ0, σ |µ, ε π(σ, µ, ε |n). (5.14)

Because of the previously mentioned objection we do not pursue this approach here.

18 6 REFERENCE ANALYSIS SEARCH PROCEDURES

5.2 Marginal approach

Here we use the marginal model p(n |σ) of equation (2.20). The intrinsic loss is

δσ0, σ = min

∞∑

n=0

p(n |σ0) lnp(n |σ0)

p(n |σ),

∞∑n=0

p(n |σ) lnp(n |σ)

p(n |σ0)

, (5.15)

and the posterior expected intrinsic loss is

d(σ0 |n) =

∫dσ δσ0, σ π(σ |n), (5.16)

where π(σ |n) is the marginal posterior, obtained via either Method 1 or Method 2.Neither of the above expressions can be further simplified, but they can be codednumerically.

Figure 14 illustrates the construction of minimal reference posterior expected intrin-sic loss intervals 3 for the single-count model. The reference posterior used is that fromMethod 1 and is shown by the dashed lines. The solid lines indicate the expected lossas a function of σ. Starting with the point of lowest expected loss, one adds σ valueswith increasing expected loss until the credibility of all the included points equals thedesired level. For this figure we set both the background and effective luminosity to1.0± 0.2. When the number Nobs of observed events equals 1, the left plot shows thatthe 95, 99, and 99.9% intrinsic intervals are upper limits, whereas they are two-sidedwhen Nobs = 10 (right plot).

The left-hand plot of Figure 15 shows the boundaries of the 95% credibility intrinsicintervals as a function of Nobs. For small Nobs the intervals are upper limits, and theybecome two-sided as Nobs increases. At large Nobs the boundaries follow straight lines.The corresponding frequentist coverage is shown in the right-hand plot. Since intrinsicintervals transition smoothly from one-sided to two-sided, their coverage avoids theextremes of 100% and 0%. Excursions from the nominal level decrease as σ increases.

6 Reference Analysis Search Procedures

We now return to the search procedure discussed in the introduction. As we sawthere, one of the difficulties in the frequentist approach is the interaction between thehypothesis test and the subsequent interval construction. This problem is absent inthe Bayesian approach, which is based on credibility rather than coverage. Using toolsfrom reference analysis, the search procedure becomes then:

1. Choose three credibility levels α1, α2, and α3.

2. Compute the marginal reference posterior for the parameter of interest, say σ.

3. Compute an α1-credibility intrinsic interval S for σ.

3This terminology is a mouthful; we will try to abbreviate it in the following.

19

4. If interval S does not include the value σ = 0, claim discovery and compute anα2-credibility intrinsic interval on σ.

5. If interval S includes σ = 0, make no claim and compute an α3-credibility upperlimit on σ.

The motivation for the post-test interval calculations (steps 4 and 5) is the same as inthe original procedure. However, in the Bayesian reference analysis framework thesecalculations avoid the problems that plagued the frequentist approach: flip-flopping,empty intervals, upper limit sensitivity, and treatment of nuisance parameters.

For an example consider Figure 14. Suppose we expect a background of 1.0±0.2 andan effective luminosity of 1.0± 0.2, and set α1 = 0.1%. If we then observe 1 event, weconclude from the left-hand plot that our observation is consistent with background.On the other hand, if we observe 10 events, the right-hand plot shows that this issignificant at the 0.1% level, and we can proceed to the calculation of narrower, moreinformative two-sided intervals on σ.

Physicists often like to quote a significance. Within a given class of tests, signifi-cance can be defined in general as the discovery threshold of the most stringent test forwhich the observed data rejects the null hypothesis. The classical formulation of thisconcept is the p value. Although the latter is quite popular in high energy physics, itcannot be justified from a strictly Bayesian point of view. To remedy this shortcoming,Refs. [3, 4, 5] propose another measure of significance, the Bayes Reference Criterion(BRC). This is essentially the reference posterior expected intrinsic loss evaluated atthe null hypothesis H0 : σ = 0. Using eq. (5.16) with the Method-1 reference posterior,this yields:

BRC ≡ d(0 |n) =

∫ ∞

0

dσ πR1(σ |n) min

κσ | 0, κ0 |σ

, (6.1)

where

κσ1 |σ2 ≡∞∑

n=0

p(n |σ2) lnp(n |σ2)

p(n |σ1), (6.2)

and p(n |σ) is given by eq. (2.20). The BRC can be interpreted as the minimum pos-terior expected log-likelihood ratio in favor of the model that generated the data,and it therefore provides its own calibration of the evidence against the null hy-pothesis: BRC=ln(10) ≈ 2.3 is mild evidence, BRC=ln(100) ≈ 4.6 is strong, andBRC=ln(1000) ≈ 6.9 is decisive. In contrast with p values, the BRC does not needadjusting for dimensionality and sample size. Returning to the example of Fig. 14,we obtain BRC=6.12 for Nobs = 10 (right-hand plot). This represents strong, butsomewhat less than decisive evidence against the background-only hypothesis. It isinteresting to compare this result with the standard p value used by physicists, whichgives the probability for a background of 1.0± 0.2 events to fluctuate up to 10. Usinga prior-predictive approach to the treatment of the background uncertainty, the resultis pPP = 3.966 × 10−7. One can convert this p value into a “number of σ’s”, or a Zvalue, i.e. the distance a normal variate is from zero when the probability outside ±Z

20 6 REFERENCE ANALYSIS SEARCH PROCEDURES

equals 2 p:

p =

∫ +∞

Z

e−x2/2

√2 π

dx, or Z(p) =√

2 erf−1(1− 2p). (6.3)

The above value of pPP corresponds to ZPP = 4.94, just short of the 5σ discoverythreshold favored by physicists.

There are other Bayesian measures of significance. Following Lindley [13], one canask for the credibility γ of the widest highest-posterior-density (HPD) interval thatdoes not contain the value σ = 0. A significance can then be defined as 1 − γ. Usingthe right-hand plot of Figure 14 as example, we find that the posterior density at σ = 0equals 9.72× 10−7; the interval of σ values with higher posterior density is ]0.0, 47.41[and has a posterior probability γ = 1 − 3.37 × 10−6. The Z value corresponding to1− γ is ZHPD = 4.50. Another possibility is to use intrinsic intervals instead of HPDones. The set of σ values with expected loss lower than our observed BRC of 6.12 isthe interval ]0.0, 33.0[, which has a posterior credibility of 99.973%. The significanceis therefore 0.027%, corresponding to ZBRC = 3.46. That these various measures ofsignificance give rather different results for our example should perhaps not be toosurprising, since they simply answer different questions:

• BRC: What is the reference posterior expected intrinsic loss at σ = 0?

• ZBRC : What is the posterior probability for a larger expected loss than the oneat σ = 0?

• ZHPD: What is the posterior probability for a smaller posterior density than theone at σ = 0?

• ZPP : What is the prior-predictive probability for the σ = 0 hypothesis to yielddata at least as extreme as observed?

The behavior of BRC, ZBRC , ZHPD, and ZPP is compared for the single-countmodel in Table 1. Also listed are the posterior expectation values of the directedKullback-Leibler divergences:

d1 =

∫ ∞

0

dσ πR1(σ |n) κ0 |σ, (6.4)

d2 =

∫ ∞

0

dσ πR1(σ |n) κσ | 0. (6.5)

In principle one could use d1 or d2 as a measure of significance. Note that BRC is notnecessarily equal to the minimum of d1 and d2, even though this seems to be the casefor the examples reported in the table, at least within the numerical accuracy of thecalculations. From a simple geometrical argument one expects that BRC≤ mind1, d2.

In contrast with the p value, the Bayesian measures of evidence depend on boththe signal model and the background model. However, BRC, d1, and d2 depend onthe signal model only through the relative uncertainty δε on the effective luminosity

21

Background Eff. Lum. Nobs d1 d2 BRC ZBRC ZHPD ZPP

1 1.0± 0.0 1.0± 0.0 10 15.68 7.20 7.20 3.76 4.92 5.18

2 1.0± 0.2 1.0± 0.2 10 14.93 6.12 6.12 3.46 4.50 4.943 1.0± 0.2 1.0± 0.4 10 18.56 4.47 4.47 3.03 4.25 4.944 1.0± 0.2 1.0± 0.6 10 28.54 3.30 3.30 2.86 4.18 4.94

5 1.0± 0.2 1.0± 0.2 10 14.93 6.12 6.12 3.46 4.50 4.946 1.0± 0.2 2.0± 0.8 10 18.56 4.47 4.47 2.96 4.16 4.947 1.0± 0.2 3.0± 1.8 10 28.54 3.30 3.30 2.63 3.95 4.94

8 1.0± 0.2 1.0± 0.2 10 14.93 6.12 6.12 3.46 4.50 4.949 1.0± 0.4 1.0± 0.2 10 11.62 6.08 6.08 3.45 4.04 4.4510 1.0± 0.6 1.0± 0.2 10 8.85 6.04 6.04 3.44 3.56 4.0311 1.0± 0.8 1.0± 0.2 10 6.87 6.01 6.01 3.42 3.17 3.70

12 1.0± 0.2 1.0± 0.2 10 14.93 6.12 6.12 3.46 4.50 4.9413 2.0± 0.4 1.0± 0.2 10 8.66 4.55 4.55 2.96 3.26 3.6614 3.0± 0.6 1.0± 0.2 10 5.52 3.39 3.39 2.53 2.45 2.8315 4.0± 0.8 1.0± 0.2 10 3.65 2.52 2.52 2.15 1.84 2.20

Table 1: Examples of significance calculations for the single-count model. Columns 2and 3 show the prior mean ± the prior standard deviation for background and effectiveluminosity. The other column headers are defined in the text. To help identify trends,line 2 is repeated at lines 5, 8, and 12.

(compare lines 2–4 with 5–7 in Table 1). This is a direct consequence of the factthat the posterior (2.33) and the marginal data model (2.20) only depend on σ and athrough the ratio σ/a, and that the σ integration in (6.1) goes from 0 to ∞. Thus,BRC depends on the signal model only through x = 1/δε2 − 1/2.

It is interesting that d1 and BRC behave oppositely under changes in the signalmodel: an increase in δε makes BRC smaller and d1 larger. To see which behavioris correct, consider that if our prior belief in our ability to detect signal has largeuncertainties (e.g. due to poor knowledge of the effective luminosity), then the evidenceprovided by the data should be downgraded, making us less willing to claim discovery.Hence only BRC seems to behave correctly.

Lines 8 to 15 in the table illustrate the variation of significance measures withchanges in the background model. Not surprisingly, changes in the background meanhave more effect than corresponding changes in the background uncertainty. A morepuzzling feature is that d1, ZHPD, and ZPP are much more sensitive to the backgrounduncertainty than d2, BRC, and ZBRC .

The search procedure described at the beginning of this section is partially mo-tivated by physicists’ fondness for the classical complementarity between hypothesistesting and interval construction. In terms of significance measures it corresponds tousing ZBRC , but this is not equivalent to using BRC (for example, ZBRC depends onboth x and a, whereas BRC only depends on x). Further study is needed to determinewhich measure of significance best serves the needs of high energy physics research.

22 ACKNOWLEDGMENTS

7 Open Questions

Reference prior methodology has now been around for about 30 years, during whichit has been continuously refined by attempts to apply it in an increasing number ofsituations. One result of these refinements is the so-called compact support argument,which we discussed in sections 2.2 and 2.5. As we showed, application of this argumentis not always trivial, and one would welcome a more deterministic approach to theconstruction of the required compact sets. This would be especially useful for codingpurposes.

Experimental high energy physicists spend a considerable amount of time tryingto obtain useable information about the nuisance parameters in their measurements,and they are generally successful in this, to varying degrees. It is therefore somewhatsurprising that most of the reference analysis literature deals with nuisance parametersabout which nothing is known a priori. One of the few exceptions is Ref. [17]. However,there remain some open issues, as we indicated in section 5, with the treatment ofpartially known nuisance parameters in the definition of intrinsic loss. We proposedtwo approaches, labeled conditional and marginal. Most of our work on the single-count model was done with the marginal approach, but we wonder whether a casecould be made for using the conditional approach, in spite of the objection mentionedin section 5.

In section 6 we discovered that some care is needed in the selection of a Bayesiantest procedure, because the many measures of significance that one can think of tendto disagree rather sharply. References [3, 4, 5] lay out strong theoretical argumentsfor using the BRC. What is still needed is an argument for selecting an appropriatediscovery threshold based on this criterion, and this hinges on a proper understandingof the various effects that physicists are concerned about. For example, does the look-elsewhere effect, which motivates the 5σ threshold on p values, affect the BRC, and ifso, how? Next, we need to build confidence in the behavior of BRC with respect touncertainties on the background and signal models. The numerical examples discussedin section 6 indicate that this behavior may be correct, but one ought to verify thismore generally.

Acknowledgments

I wish to thank Supriya Jain and Harrison Prosper for many meetings and discussionson the topic of reference analysis. Their Monte Carlo calculations helped check someof the analytical results presented here. Thanks also to Harrison for suggesting themarginal approach to intrinsic intervals in section 5.2.

FIGURES 23

Figures

Figure 1: Marginalized likelihood of equation (2.20) as a function of the signal crosssection σ, for several values of the observed number of events n.

24 FIGURES

Figure 2: Method 1 reference prior for the cross section σ, for three values of the shapeparameter y of the background prior (left), and for four values of the shape parameterx of the efficiency prior (right). The priors are normalized by the requirement thatπR1(1) = 1.

Figure 3: Marginal Method 2 reference prior for the cross section σ, for three valuesof the shape parameter y of the background prior (left), and for four values of theshape parameter x of the efficiency prior (right). The priors are normalized by therequirement that πR2(1) = 1.

FIGURES 25

Figure 4: Comparison of Method 1 and Method 2 reference priors for various values ofx and y. Both types of prior are normalized to 1 at σ = a/b.

26 FIGURES

Figure 5: Marginal Method 1 reference posterior for the cross section σ, for three valuesof the shape parameter y of the background prior (left), and for three values of theshape parameter x of the efficiency prior (right), as the number of observed events ngoes from 0 to 10 (top to bottom).

FIGURES 27

Figure 6: Method 2 reference posterior for the cross section σ, for three values of theshape parameter y of the background prior (left), and for three values of the shapeparameter x of the efficiency prior (right), as the number of observed events n goesfrom 0 to 10 (top to bottom).

28 FIGURES

Figure 7: Upper limits and central intervals at the 68 and 95% credibility levels,calculated from Method 1 posteriors. The prior for the background µ and the effectiveluminosity ε is given by eq. (2.14), with x = y = a = b = 1; this corresponds to a meanof 1.5 and a coefficient of variation of 82% for both µ and ε. Method 2 intervals areindistinguishable from Method 1 on this scale.

FIGURES 29

Figure 8: Upper limits and central intervals at the 68 and 95% credibility levels,calculated from Method 1 posteriors. The prior for the background µ and the effectiveluminosity ε is given by eq. (2.14), with x = y = 24.5 and a = b = 25.0; this correspondsto a mean of 1.0 and a coefficient of variation of 20% for both µ and ε. Method 2intervals are indistinguishable from Method 1 on this scale.

30 FIGURES

Figure 9: Difference between the Method 1 and Method 2 upper limits at the 68 and95% credibility levels.

Figure 10: Variation of the Method 1 reference posterior upper limit with mean back-ground for several values of the observed number of events n. The relative uncertaintyon the background and on the effective luminosity is 20% for the left plot and 50% forthe right one.

FIGURES 31

Figure 11: Method 1 reference posterior: coverage of 68% credibility central intervals(top left), 68% upper limits (top right), 95% central intervals (bottom left), and 95%upper limits (bottom right) for the cross section σ, as a function of σ. The coverage isaveraged over the prior for the effective luminosity ε and the background µ. This prioris given by eq. (2.14), with x = y = a = b = 1. Note the offset zero in the bottomplots. The solid horizontal lines indicate the credibility level of the constructions.

32 FIGURES

Figure 12: Method 2 reference posterior: coverage of 68% credibility central intervals(top left), 68% upper limits (top right), 95% central intervals (bottom left), and 95%upper limits (bottom right) for the cross section σ, as a function of σ. The coverage isaveraged over the prior for the effective luminosity ε and the background µ. This prioris given by eq. (2.14), with x = y = a = b = 1. Note the offset zero in the bottomplots. The solid horizontal lines indicate the credibility level of the constructions.

FIGURES 33

Figure 13: Method 1 reference posterior: coverage of 68% credibility central intervals(top left), 68% upper limits (top right), 95% central intervals (bottom left), and 95%upper limits (bottom right) for the cross section σ, as a function of σ. The coverageis averaged over the prior for the effective luminosity ε and the background µ. Thisprior is given by eq. (2.14), with x = y = 24.5 and a = b = 25.0. Note the offsetzero in the bottom plots. The solid horizontal lines indicate the credibility level of theconstructions.

34 FIGURES

Figure 14: Construction of 95%, 99%, and 99.9% credibility intrinsic intervals for anew particle cross section when 1 event (left) or 10 events (right) are observed overa background of 1.0 ± 0.2 events and with an effective luminosity of 1.0 ± 0.2 (thiscorresponds to x = y = 24.5 and a = b = 25). The dashed curves represent theMethod 1 reference posteriors (rescaled by a factor of 5 and 50, respectively) and thesolid curves show the posterior expected intrinsic loss.

Figure 15: Left: boundaries of minimum posterior expected intrinsic loss intervalswith 95% credibility as a function of the observed number of events. Right: frequentistcoverage of these intervals as a function of the true value of the cross section σ (notethe offset zero on the y axis).

REFERENCES 35

References

[1] Abramowitz, M., and Stegun, I. A., “Handbook of mathematical functions withformulas, graphs, and mathematical tables,” United States Department of Com-merce, National Bureau of Standards Applied Mathematics Series — 55, tenthprinting, December 1972, with corrections; see http://www.math.sfu.ca/~cbm/

aands/. 11

[2] Berger, J. O., private communication. 9

[3] Bernardo, J. M., “Nested hypothesis testing: the Bayesian reference criterion,”in Bayesian Statistics 6 (J. M. Bernardo, J. O. Berger, A. P. Dawid and A. F.M. Smith, eds), Oxford University Press, 1999, pg. 101–130 (with discussion);http://www.uv.es/~bernardo/Valencia6.pdf. 19, 22

[4] Bernardo, J. M., and Rueda, R., “Bayesian hypothesis testing: a reference ap-proach,” Int. Statist. Rev. 70, 351-372 (2002); http://www.uv.es/~bernardo/

IntStatRev.pdf. 19, 22

[5] Bernardo, J. M., “Reference analysis,” Handbook of Statistics 25 (D. K. Dey andC. R. Rao eds.), Amsterdam: Elsevier, 17 (2005); see also http://www.uv.es/

~bernardo/RefAna.pdf. 5, 6, 19, 22

[6] Bernardo, J. M., “Intrinsic credible regions: an objective Bayesian approachto interval estimation,” Test 14, 317 (2005); http://www.uv.es/~bernardo/

2005Test.pdf. 15, 16

[7] Cousins, R. D., private communication. 5

[8] Datta, G. S., and Sweeting, T. J., “Probability matching priors,” ResearchReport No. 252, Department of Statistical Science, University College London(March 2005); http://www.ucl.ac.uk/Stats/research/Resrprts/psfiles/

rr252.pdf. 15

[9] Demortier, L., “Dealing with data: signals, backgrounds, and statistics,” Lecturesat the 2008 Theoretical Advanced Study Institute, Boulder, CO, to be publishedby World Scientific Publishing Co.; see also http://physics.rockefeller.edu/

~luc/proceedings/TASI2008_statistics.pdf. 4, 5

[10] Demortier, L., Jain, S., and Prosper, H., “Reference priors for high energyphysics,” in preparation. 13

[11] Feldman, G. J., and Cousins, R. D., “Unified approach to the classical statisticalanalysis of small signals,” Phys. Rev. D 57, 3873 (1998); arXiv:physics/9711021[physics.data-an] 16 Dec 1999 (http://arxiv.org/abs/physics/9711021). 4

[12] Giunti, C., “New ordering principle for the classical statistical analysis of Pois-son processes with background,” Phys. Rev. D 59, 053001 (1999); arXiv:hep-ph/9808240v2, 6 May 1999 (http://arxiv.org/abs/hep-ph/9808240). 5, 14

36 REFERENCES

[13] Lindley, D. V., “Introduction to probability and statistics from a Bayesian view-point; Part 2: Inference,” Cambridge University Press, 1965 (292pp.). 20

[14] Mayo, D. G., and Cox, D. R., “Frequentist statistics as a theory of inductiveinference,” IMS Lecture Notes — Monograph Series: 2nd Lehmann Symposium— Optimality, Vol. 49, pg.77-97 (2006); arXiv:math/0610846v1 [math.ST] 27 Oct2006 (http://arxiv.org/abs/math.ST/0610846). 4

[15] Pratt, J. W., “Bayesian interpretation of standard inference statements ,” J. R.Statist. Soc. B27, 169 (1965). 14

[16] Rosenfeld, A. H., “Are there any far-out mesons or baryons?,” in Meson Spec-troscopy. A collection of articles, C. Baltay and A. H. Rosenfeld, eds., W.A. Ben-jamin, Inc., New York, Amsterdam, 1968, pg. 455. 4

[17] Sun, D., and Berger, J. O., “Reference priors with partial information,” Biometrika85, 55 (1998); see also http://www.stat.duke.edu/~berger/papers/sun.html.6, 7, 22