Ockham’s Razor and Bayesian Analysis

Maryam Zolghadr

Carlo Alberto College

Turin, April 21st, 2015

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Ockham’s razor principle

“Pluralitas non est poneda sine neccessitate.”

— William Ockham

It can be translated as Plurality must not be posited withoutnecessity.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Various interpretations of O.R. principle

“Entities should not be multiplied without necessity.”“It is vain to do more what can be done with less.”“An explanation of the facts should be no morecomplicated than necessary.”“Among competing hypothesis, favor the simplest one.”

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Connection between O.R. and Bayesian Analysis

O.R. enjoins us to favor the simplest hypothesis that isconsistent with the data, but determining which hypothesisis simplest is often no simple matter.Bayesian analysis can offer concrete help in judging thedegree to which a simpler model is to be preferred.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Law of accelerated motion

The law describing the motion of falling bodies proposed byGalileo and familiar to students of physics can be expressed asa quadratic equation:

s = a + ut +12

gt2

where a, u and g are adjustable parameters that can beassigned arbitrary values in order to fir the empirical data.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Ballistic trajectory is Parabola

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Question

Why is it, that the quadratic law is the choice of physicseverywhere?

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Prior Probability

Galileo and a modern student of physics would favor thequadratic law since it is simpler, whereas higher-degreepolynomials are unnecessarily complicated.Jeffreys suggested that the reason for favoring the simplerlaw is that it has a higher prior probability. In other words, itis consider the likelier explanation at the outset of theexperiment, before any measurements have been made.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Jeffreys and Dorothy Wrinch’ssuggestion for a measure of simplicity that depends onprior probabilities

For laws that can be expressed as differential equations, theysuggested a straightforward algorithm for counting parameters.Having sorted all possible laws according to this criterion, onecan try the simpler laws first, only moving on to morecomplicated laws as the simple ones prove inadequate torepresent the data.

Thus the ordering of hypothesis provides a kind of rationalizedO.R.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Problem with Jeffrey’s appeal to prior probability

Defining the simplest law as the one with the fewest adjustableparameters is a useful strategy, but it cannot be extended toyield a clear, universal rule for assigning prior probabilities.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Galileo’s Problem: Jeffreys suggestion for a measureof simplicity that does not depend on prior probabilities

If a law has any adjustable parameters, then it will besignificantly preferred to the simpler law only if its prediction areconsiderably more accurate.

If the prediction of the two models are roughly equivalent, thesimpler law can have greater posterior probability.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

To Catch a cheat: Objective quantification of O.R. byBerger and Jeffreys (1992)

Suppose a friend who has a reputation as a prankster offers toflip a coin to decide who will perform a little chore: Heads hewins, tails he loses.{

HHH : The hypothesis that the coin has two heads (unfair coin).HHT : The hypothesis that the coin is fair.

The hypothesis HHH that the coin has two heads is a simplerone than the hypothesis HHT that the coin is fair.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

To Catch a cheat: Objective quantification of O.R. byBerger and Jeffreys (1992)

Before the coin is flipped, you might believe that thehypothesis HHH and HHT are equally likely.After series of coin tosses, if heads appears invariably in along series of tosses, the hypothesis of two headed coinbecomes more attractive.The fair-coin hypothesis is consistent with every possibleobservation. However, the two-heads hypothesis would befalsified by a single appearance of tails.Since the two-heads hypothesis makes such sharppredictions, it is given greater (belief) credit when thosepredictions come to pass.Back to Galielo’s problem

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis

Conclusion

The key idea links Bayesian analysis to O.R. is a notion ofsimplicity in a hypothesis.We have discussed two ways in which Ockham’s razor canbe interpreted in Bayesian terms.

1 By choosing the prior probabilities of hypotheses, one canquantify the scientific judgment that simpler hypotheses aremore likely to be correct.

2 Bayesian analysis also shows that a hypothesis with feweradjustable parameters automatically has an enhancedposterior probability, because the predictions it makes aresharp about what data will be observed, and it is morereadily falsified by arbitrary data.

Both of these ideas are in agreement with the intuitivenotion of what makes a scientific theory powerful andbelievable.

Maryam Zolghadr Ockham’s Razor and Bayesian Analysis