Richard Carrier: Proving history or idiocy?

In preparation for the release of the sequel to Dr. Richard Carriers Proving History, I thought it would be good to address the foundations of the sequel as presented in Proving History. Namely, Dr. Carriers arguments about Bayes Theorem or BT (as presented in his book, anyway). There are several reasons for this. First, Dr. Carrier argues that BT should be THE method for all historical research. Second, the entire point of Proving History was to argue that the approach we will find in the sequel is the best. The third and most important reason, however, is that Dr. Carrier doesnt understand BT, how it differs from the method he actually advocates (Bayesian inference/statistics), or how fundamentally inaccurate basically every claim he makes about his methodology is.

We can begin our journey through Dr. Carriers failure to understand his own method by looking at his flawed proof. At one point, Dr. Carrier states we can conclude here and now that Bayes Theorem models and describes all valid historical methods. No other method is needed (p. 106). His proof, though, stands or falls with the first proposition in it: BT is a logically proven theorem (ibid.). This is true. However, Dr. Carrier doesnt seem to have read the sources he cites. For example, on p. 50 Dr. Carrier refers the reader via an endnote (no. 9) to several highly commendable texts on BT. The one he states gives a complete proof of formal validity of BT is Papoulis, A. (1986). Probability, Random Variables, and Stochastic Processes. (2nd Ed.). I dont have the 2nd edition, but I do have the 3rd and as this proof is trivial I really could use any intro probability textbook. Papoulis begins his complete proof of formal validity (as opposed to proof of informal validity? or incomplete proof?) by defining a set and probability function for which the axioms of probability hold. A key axiom is that any set of possible outcomes must sum or integrate to 1 (simplistically, for those who havent taken any calculus, integration is a kind of summation). For example, imagine an individual names Anna is drawing cards from the pack. Lets imagine that It wasnt the Jack of Diamonds Nor the Joker she drew at first It wasnt the King or the Queen of Hearts But the Ace of Spades reversed The probability of drawing the card she did is 1/52. This is true for the other 51 cards as

well. The probability that she would draw a card from the deck that was in the deck is 52/52 or 1. This is intuitive and obvious, but the important point is that it also follows from the fact that there are 52 cards and the probability for drawing any one of them is 1/52, hence the probability of drawing a card is given by the sum of the probabilities of drawing each individual card, or 1/52 summed 52 times. In Dr. Carriers appendix (p. 284) he notes that probability functions must sum to 1. What he apparently doesnt understand is what this entails. It means that in order to use BT to evaluate how probable some outcome, result, historical event, etc., is, one must consider every single one.

Let me make this simpler with a simple example. I can calculate the probability that, given a full deck of cards, a random draw will yield an ace because I know in advance every possible outcome. If, however, someone mixed together 10 cards drawn at random from 300 different decks and asked me to pick a card, I cant calculate the probability anymore. Even if I were told that the new deck contained 3,000 cards, I have no idea how many are aces. Incidentally, this is the perfect situation for Bayesian statistical inference, which works (simplistically) by assuming e.g., a certain distribution of aces and then changing my model of how likely it is that the next draw will yield an ace as I learn more about the distribution of cards in the entire 3,000 card deck.

Dr. Carrier wishes to use what he thinks BT is to evaluate the probability that particular events occurred ~2,000 years ago. For example, on pp. 40-42 he considers the possibility that Jesus was a legendary rabbi in terms of the class of legendary rabbis and information we have on such a class. We are in a far worse position than in the mixed card deck example above, because we dont even know the number of legendary rabbis still less who they might be (if we did wed have the answer: Jesus either would be one or wouldnt).

There is another basic property of BT Dr. Carrier seems to have missed. As Papoulis clearly states, BT is only valid for events/outcomes that are mutually exclusive. Often, both of these requirements (the sum to 1 and mutually exclusivity) are given together: the set of outcomes must be collectively exhaustive and mutually exclusive, or BT is only valid if 1) all possible outcomes are known 2) one and only one outcome can occur.

This makes BT useless for most purposes including historiography. However, Dr. Carrier isnt really using BT. As his references show (as well as his formulation of the theorem shows), he is actually using something called Bayesian inference/Bayesian analysis. However, this negates his entire proof because it doesnt matter if BT is a logically proven theorem and there is no complete proof of formal validity for some Bayesian inference/analysis theorem Dr. Carrier could use in place of his first proposition.

Ok, so we cant use BT, but that doesnt mean we cant use Bayesian methods. However, in order to use Bayesian methods Dr. Carrier would have to understand Bayesian statistics (and statistics in general). He doesnt. We can see this clearly when Dr. Carrier, whose expertise is ancient history, addresses the frequentist vs. Bayesian debate. To keep things simple, lets just say that this is an ongoing debate arguably going back to Bayes but definitely is over a century old. Dr. Carrier is apparently so confident in his mathematical acuity he resolves the dispute with almost no reference to math or the literature in a few pages: The whole debate between frequentists and Bayesians, therefore, has merely been about what a probability is a frequency of, the rules are the same for either (p. 266). Hm. Amazing that generations of the best statistical minds missed this. Oh wait. They didnt.

Lets look at how Carrier describes the dispute: The debate between the so-called frequentists and Bayesians can be summarized thus: frequentists describe probabilities as a measure of the frequency of occurrence of particular kinds of event within a given set of events, while Bayesians often describe probabilities as measuring degrees of belief or uncertainthy. (p. 265). This is laughably wrong:

Frequentist statistical procedures are mainly distinguished by two related features; (i) they regard the information provided by the data x as the sole quantifiable form of relevant probabilistic information and (ii) they use, as a basis for both the construction and the assessment of statistical procedures, long-run frequency behaviour under hypothetical repetition of similar circumstances. Bernardo, J. M. & Smith, A. F. (1994). Bayesian Theory. Wiley.

Undoubtedly, the most critical and most criticized point of Bayesian analysis deals with the choice of the prior distribution, since, once this prior distribution is known, inference can be led in an almost mechanic way by minimizing posterior losses,

computing higher posterior density regions, or integrating out parameters to find the predictive distribution. The prior distribution is the key to Bayesian inference and its determination is therefore the most important step in drawing this inference. To some extent, it is also the most difficult. Indeed, in practice, it seldom occurs that the available prior information is precise enough to lead to an exact determination of the prior distribution, in the sense that many probability distributions are compatible with this informationMost often, it is then necessary to make a (partly) arbitrary choice of the prior distribution, which can drastically alter the subsequent inference. Robert, C. P. (2001). The Bayesian Choice: From Decision-Theoretic Foundations to Computational Implementation (Springer Texts in Statistics). (2nd Ed.). Springer.

The frequency part of frequentist does have to do with kinds of events, but frequencies are the measure of probability, not the reverse. To illustrate, consider the bell curve (the graph of the normal distribution). Its a probability distribution. Now imagine a standardized test like the SATs which is designed such that scores will be normally distributed and have this bell curve graph. What does the graph tell us? It tells us that the most people who take the test get very close scores, but very infrequently some test-takers will get high scores and other will get low. In other words, the bell curve is the graph of a probability function (technically, a probability density function or pdf), and it is formed by the frequency of particular scores. We know that it is very improbable for a persons score to fall in either of the ends/tails of the bell curve because these are very infrequent outcomes.

What does this mean for frequentist methods? Well, Kaplan, The Princeton Review, and other test prep companies try to show their methods work by using this normal distribution. They claim that people who take their classes arent distributed the way the population is, because too frequently students taking their class obtain scores above average (i.e., those who take the classes have test scores that arent distributed the way the population is). They use the frequency of higher-than-average scores to argue that their class must improve scores.

Whats key is that the data are obtained and analyzed but the distribution is only used to determine whether the values the analysis yielded are statistically significant. Bayesian inference reverses this, creating fundamental differences. The process starts with a probability distribution. The prior distributions obtained represent uncertainty and make predictions about the data that will be obtained. Once the new data is obtained,

the model is adjusted to better fit it. This is usually done many, many times as more and more information is tested against an increasingly more accurate model. The key differences are 1) the iterative process 2) the use of models which make predictions 3) the use of distributions to represent unknowns and (in part) the way the model will learn or adapt given new input.

So why dont we find any of this in Dr. Carriers description of Bayesian methods? Why do we always find ad hoc descriptions of priors? Because Dr. Carrier wants to use Bayesian analysis but apparently doesnt understand what priors actually are or how complicated they can be in even simple models: In many situations, however, the selection of the prior distribution is quite delicate in the absence of reliable prior information, and generic solutions must be chosen instead. Since the choice of the prior distribution has a considerable influence on the resulting inference, this choice must be conducted with the utmost care. Marin, J. M., & Robert, C. (2007). Bayesian Core: A Practical Approach to Computational Bayesian Statistics. (Springer Texts in Statistics). Springer.

While the axiomatic development of Bayesian inference may appear to provide a solid foundation on which to build a theory of inference, it is not without its problems. Suppose, for example, a stubborn and ill-informed Bayesian puts a prior on a population proportion p that is clearly terrible (to all but the Bayesian himself). The Bayesian will be acting perfectly logically (under squared error loss) by proposing his posterior mean, based on a modest size sample, as the appropriate estimate of p. This is no doubt the greatest worry that the frequentist (as well as the world at large) would have about Bayesian inference that the use of a bad prior will lead to poor posterior inference. This concern is perfectly justifiable and is a fact of life with which Bayesians must contendWe have discussed other issues, such as the occasional inadmissibility of the traditional or favored frequentist method and the fact that frequentist methods dont have any real, compelling logical foundation. We have noted that the specification of a prior distribution, be it through introspection or elicitation, is a difficult and imprecise process, especially in multiparameter problems, and in any statistical problem, suffers from the potential of yielding poor inferences as a result of poor prior modeling. Samaniego, F. J. (2010). A Comparison of the Bayesian and Frequentist Approaches to Estimation. (Springer Texts in Statistics). Springer.

The stubborn and ill-informed Bayesian is in a much better position than Dr. Carrier. Dr. Carrier has confused BT with Bayesian analysis and the Bayesian approach with the frequentist all because he apparently hasnt understood any of these. Instead of prior distributions his priors are best guesses. Instead of real belief functions we find heres what I believe. No considerations are given to the nature of the data (categorical, nominal, and in general none numerical data require specific models and tests, Bayesian or not).

So instead of the universally valid historical method Dr. Carrier argues BT provides, all that hes actually done is butcher mathematics in order to plug values in to a formula that is as mathematical as numerology but apparently seems impressive if you have no clue what you are talking about. Perhaps thats why Dr. Carriers CV indicates hes been lecturing on Bayes Theorem since 2003, but his 2008 dissertation contains no reference to Bayes Theorem.