ESTIMATING FIELDING ABILITY IN BASEBALL PLAYERSOVER TIME

James Martin Piette III

A DISSERTATION

in

Statistics

For the Graduate Group in Managerial Science and AppliedEconomics

Presented to the Faculties of the University of Pennsylvania in PartialFulfillment of the Requirements for the Degree of Doctor of Philosophy

2011

Supervisor of Dissertation

Shane Jensen, Associate Professor, Statistics

Graduate Group Chairperson

Eric Bradlow, Professor of Marketing, Statistics, and Education

Dissertation CommitteeEdward George, Professor of StatisticsDylan Small, Professor of Statistics

Acknowledgements

Professor Shane Jensen served as the perfect advisor to me. It was a foregone

conclusion upon my arrival at Penn that you would be my advisor. I appreciate all

of your advice on research, video games and baseball, as well as introducing me to

fantasy baseball. I will miss our talks about how Alfonso Soriano and the Nationals’

closers make us cry.

I want to thank all of my fellow classmates in the Wharton Statistics department.

Being able to dump my grad school worries on people like Mike Baiocchi, Sivan

Aldor-Noiman, Jordan Rodu, Oliver Entine and Dan Yang helped me over the

bumps in the road towards my Ph.D.

I am fortunate to have had some excellent co-authors, like Blakely McShane and

Alex Braunstein, who pushed me to be a better researcher and writer, as well as

introduce me to the great things that Flipadelphia has to offer.

I am grateful to have shared an office with two of the finest people I know.

I will always be in Kai Zhang’s debt for both his wisdom and kindness; without

him contributing to my first paper, my Erdos number would be ∞. Sathyanarayan

ii

Anand likely knows me better than anyone else, and has provided me with endless

conversation as a roommate. Thank you for keeping me sane, grounded and affable,

while going through the four very trying years of my life.

I appreciate all that my home town friends, Austin Cowart and Alex Sprague,

did for me. Playing video games and sharing great beers stands as some of the most

fun moments in my life, and serve as reminders of who I am and where I am from.

Some of my best years during graduate school were working with and helping

start up Krossover with Vasu Kulkarni, Alex Kirtland and Sandip Chaudhari. I

could not imagine a better way to start my career, working for a company in the

sports industry with three smart, driven individuals who I can also call dear friends.

I would like to thank the people who have always been there with their love and

support: my family. To my sister, mom and dad, I love you all. Without you, I

would have never pursued, much less finished, my Ph.D, missing out on one of the

best experiences of my life. I have always wanted to make you proud, and I hope

that I will continue to do so.

My beautiful, soon-to-be wife Lisa Pham, who has given me nothing but love,

despite the fact that I am a lanky spaz that is easily excitable and loud. This

dissertation literally would not have been written without your encouragement. For

the past four years, anything I have achieved is because you gave me the confidence

to do it. I will always love you, jitterbug and all, with all my heart. And, as I

promised, I will get you a puppy as soon as possible.

iii

ABSTRACT

ESTIMATING FIELDING ABILITY IN BASEBALL PLAYERS OVER TIME

James Martin Piette III

Shane T. Jensen (Advisor)

It is commonplace around baseball to involve statistical analysis in the evaluation

of a player’s ability to field. While well-researched, the question behind how best

to model fielding is heavily debated. The official MLB method for tracking field-

ing is riddled with biases and censoring problems, while more recent approaches to

fielding evaluation, such as Ultimate Zone Rating [Lichtman 2003], lose accuracy

by not treating the field as a single continuous surface. SAFE, Spatial Aggregate

Fielding Evaluation, aims to solve these problems. Jensen et al [2009] took a rig-

orous statistical approach to this problem by implementing a hierarchical Bayesian

structure in a spatial model setting. The performance of individual fielders can be

more accurately gauged because of the additional information provided via sharing

across fielders. I have extended this model to three new specifications by building

in time series aspects: the constant-over-time model, the moving average age model

and the autoregressive age model. By using these new models, I have produced

a more accurate estimation of a player’s seasonal fielding performance and added

insight into the aging process of a baseball player’s underlying ability to field.

iv

Contents

Title Page i

Acknowledgements ii

Abstract iv

Table of Contents v

List of Figures viii

1 Introduction and Motivation 1

1.1 An Introduction to Baseball and Fielding . . . . . . . . . . . . . . . 2

1.2 The State of Fielding Analytics . . . . . . . . . . . . . . . . . . . . 5

1.3 Spatial Analysis in Sports . . . . . . . . . . . . . . . . . . . . . . . 11

1.4 Related Time Series Techniques . . . . . . . . . . . . . . . . . . . . 13

1.5 Dissertation Organization . . . . . . . . . . . . . . . . . . . . . . . 17

2 Original Bayesian Hierarchical Model 19

v

2.1 Description of the Data . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.2 Data Manipulation . . . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.2.1 Flyballs/Liners . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.2.2 Grounders . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2.2.3 Eligible BIP × Fielder Combinations . . . . . . . . . . . . . 28

2.3 Original Hierarchical Model . . . . . . . . . . . . . . . . . . . . . . 29

3 New Model Specifications 34

3.1 Constant-Over-Time Model . . . . . . . . . . . . . . . . . . . . . . 35

3.2 Moving Average Age Model . . . . . . . . . . . . . . . . . . . . . . 39

3.3 Autoregressive Age Model . . . . . . . . . . . . . . . . . . . . . . . 43

3.3.1 Model Description . . . . . . . . . . . . . . . . . . . . . . . 44

3.3.2 State Sampling: FFBS . . . . . . . . . . . . . . . . . . . . . 47

3.3.3 Remaining Sampling: Gibbs Sampler . . . . . . . . . . . . . 49

4 Results 53

4.1 Calculating SAFE Values . . . . . . . . . . . . . . . . . . . . . . . . 54

4.1.1 Choosing the Parametric Curves . . . . . . . . . . . . . . . . 54

4.1.2 Weighted Integration . . . . . . . . . . . . . . . . . . . . . . 56

4.2 Analyzing SAFE Estimates . . . . . . . . . . . . . . . . . . . . . . 59

4.2.1 General Analysis . . . . . . . . . . . . . . . . . . . . . . . . 59

4.2.2 Model Differences and Specific Players . . . . . . . . . . . . 63

vi

4.2.3 Age Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 69

4.3 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.3.1 Derek Jeter . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.3.2 Vernon Wells . . . . . . . . . . . . . . . . . . . . . . . . . . 78

5 Model Validation 81

5.1 Predicted Deviations . . . . . . . . . . . . . . . . . . . . . . . . . . 82

5.1.1 Methods of Calculation . . . . . . . . . . . . . . . . . . . . . 82

5.1.2 Graphical Survey . . . . . . . . . . . . . . . . . . . . . . . . 86

5.2 Comparing to Existing Metrics . . . . . . . . . . . . . . . . . . . . 89

6 Conclusions and Future Work 92

A Kalman Filter 95

B Backward Sampling 97

vii

List of Figures

1.1 Baseball positions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.2 Field of play broken up into the zones needed when calculating UZR 9

2.1 Heatmaps of all in play flyballs and liners . . . . . . . . . . . . . . . 22

2.2 Empirical angle densities for groundballs . . . . . . . . . . . . . . . 23

2.3 Defensive alignments by fielding teams . . . . . . . . . . . . . . . . 25

2.4 Examples of distance to a flyball/liner and angle on a grounder . . 27

3.1 Hierarchy of the Constant-Over-Time Model . . . . . . . . . . . . . 35

3.2 An Example of the Moving Average Age Model . . . . . . . . . . . 40

3.3 A Visualization of the Autoregressive Age Model . . . . . . . . . . 47

3.4 An Example of the Solution for Missing Seasons in the Autoregressive

Age Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

4.1 Histograms of the Posterior Means for Age-Specific SAFE Estimates 60

4.2 SAFE Results for a Group of Fielders under the Original Model . . 64

viii

4.3 SAFE Results for a Group of Fielders under the Constant-over-Time

Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66

4.4 SAFE Results for a Group of Fielders under the Moving Average Age

Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

4.5 SAFE Results for a Group of Fielders under the Autoregressive Age

Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.6 Position-Age SAFE Estimates . . . . . . . . . . . . . . . . . . . . . 72

4.7 Posterior Means and 95% Intervals on Autoregressive Terms over Time 74

4.8 Acrobatic Fielding Play by Derek Jeter . . . . . . . . . . . . . . . . 77

4.9 Catch at the Wall by Vernon Wells . . . . . . . . . . . . . . . . . . 79

5.1 Mosaic Plot of Winning Percentage via Predicted Deviations for Each

Ball In Play Type and Position Combination . . . . . . . . . . . . . 88

5.2 Histograms of the Average Ranking of Each Model . . . . . . . . . 90

6.1 Automatic Tracking Data by Field F/X . . . . . . . . . . . . . . . . 94

ix

Chapter 1

Introduction and Motivation

Evaluating a baseball player’s fielding ability using objective means has been an

ongoing problem amongst Major League Baseball (MLB) enthusiasts and profes-

sionals. During the sport’s infancy, the only data available for measuring a player’s

fielding contribution was fielding percentage, an antiquated metric based on data

acquired through subjective means. Thanks to technological advancements giving

way to new data in the past couple of decades, metrics meant to gauge the value

of a fielder have brought a fresh perspective to fielding’s contribution to winning

baseball games. However, most of these tools do not fully utilize the potential that

this new data has to offer, especially with regard to its continuous nature. SAFE,

Spatial Aggregate Fielding Evaluation, is an approach that attempts to fully uti-

lize these continuous measurements. The methods, both frequentist and Bayesian,

to calculate the rudimentary version of SAFE have been highlighted in past work

1

[18]. I expand on these approaches by adding time series elements to the previous

Bayesian model to produce more accurate estimates of fielding ability, delineate the

performance of a fielder on a season-to-season basis versus their entire career and

provide new insight into how baseball players age at the various fielding positions.

1.1 An Introduction to Baseball and Fielding

Baseball is a sport played between two teams of nine players each. The goal of a

team is to score more runs than the opposing team given the allotted resources1.

Teams score runs by having their players successfully touch a series of four bases.

To get on base, the team on offense attempts to bat a ball thrown to them by the

opposing team. Their opponent, also known as the fielding team, tries to prevent

players on offense, or hitters, from reaching these bases by getting them “out.” An

“out” can occur by either a pitcher throwing three strikes to a batter, or a player

on the fielding team (fielder) tags or forces out the batter and/or other offensive

players on the bases. In this dissertation, our focus is on the proficiency of fielders

at gathering and/or producing outs.

An out can be generated by the fielding team in a variety ways. I choose to

focus on the three methods with which the fielding team is the sole driver2: (i) a

force out, (ii) a tag out and (iii) a caught out. A force out is an out where the

1Baseball is a game that is not based on time, but on the number of resources (i.e. “outs”)each team has in their disposal.

2There are outs where the fielding team is not responsible. These usually involve the playerson offense breaking a rule, such a baserunning interference or running outside of the basepath.

2

Figure 1.1: The fielding positions taken by players whose team is on defense.

batter fails to reach first base on a ball hit into play before a fielder controling the

batted ball touches first base. Similarly, any batter who reached base, known as a

baserunner, is forced out by a fielder touching a non-first-base bag if the baserunner

is forced to advance to that base. This occurs when a baserunner or batter behind

them must advance to the base that they previously resided. A tag out occurs when

a fielder gathers a batted ball into their glove and touches one of the baserunners

or the hitter with their glove. Any ball hit into the air that is caught by a fielder

before that ball reaches the ground is known as a caught out.

A fielding team is comprised of two groups of players, the infielders and the

outfielders. I illustrate these positions in figure 1.1. The positions in the outfield

are left fielder, center fielder and right fielder. Outfielders are in charge of catching

a ball hit into the air for an out and relaying a successful hit into the outfield

back to an infielder. There are five infield positions, excluding the pitcher. Four of

3

Table 1.1: The conventional shorthand (Short) and numbering (Num.) used when ref-erencing a fielding position (Pos. Name).

Pos. Name Short Num.

First baseman 1B 3Second baseman 2B 4Third baseman 3B 5Shortstop SS 6Left fielder LF 7Center fielder CF 8Right fielder RF 9

those positions locate themselves around the non-home bases: the third baseman,

shortstop, second baseman and first baseman. These infielders must make plays on

both balls hit into the air (but stay in the infield) and balls hit on the ground. The

last infield position, the catcher, resides behind home plate and has the most on-field

responsibilities of any fielder. Catchers are not only responsible for fielding batted

balls adjacent to them, but they are also tasked to direct the other infielders on

the field and call specific plays to the pitcher. Shorthand for each fielding position,

along with each position’s corresponding position number that is used for scoring

purposes, can be seen in table 1.1.

The necessary mechanics required to properly field one’s position range from

fundamental techniques, such as throwing a ball, to more subtle and complex move-

ments, like transferring the ball from the glove to the throwing hand. Add to it

the raw athleticism needed to reach the ball quickly, often referred to as a player’s

range, and it can be sen there are many moving parts driving this one activity.

4

There is something else to consider when evaluating a player’s fielding ability, how-

ever. By improperly placing himself or herself on the field of play before a batter

hits, a fielder is at a disadvantage and might be unable to make a play on a batted

ball, even with the best mechanics. The fielder who is perfectly positioned may

need little or no effort to get the same out; this makes field positioning an entirely

separate skill from fielding mechanics. Given that the data does not relay where

each fielder is standing at the time of the at-bat, I am unable to sufficiently account

for this skill as I discuss in more detail in chapter 2.

1.2 The State of Fielding Analytics

At the most fundamental level of fielding analytics are baseball talent evaluators,

or scouts. Scouts are tasked with subjectively grading fielding talent based on

visual observations alone. These evaluators arrive at live games and practices to

watch potential (or current) talent. Once they are comfortable with making a value

statement about that particular player, the player is assigned a series of grades

ranging from 20 to 80; these grades represent the predicted ability of that player

by that scout over a range of fielding assets, such as arm strength and glove work.

A 20 on this scale corresponds to a poor grade at that skill, while receiving an 80

suggests that the fielder’s corresponding skill is elite. These grades serve the useful

purpose of projecting the future ability of a young fielder with little or no data

associated with them. For obvious reasons, this approach lacks the consistency and

5

preciseness desired for a proper analytical evaluation of a fielder’s ability.

For over a century, the only quantitative fielding metric available around base-

ball was fielding percentage, which is still the official method of tracking fielding

performance in MLB. Fielding percentage is the proportion of successfully fielded

balls hit into play. A ball in play (BIP) that is “unsuccessfully fielded” is called an

error. It is up to the scorekeeper, a person in charge of keeping an official tally of a

game, to make a call on every failed attempt by the fielding team, whether or not

the failure was due to a specific fielder’s negligence or just circumstance.

Studies based on this basic measure have suggested that defense may play an

integral part in a team’s success. Work by Hakes and Sauer [15] went into the

relative importance of fielding on game outcomes. While Hakes and Sauer’s findings

were unable to prove that fielding percentage had a significant effect on a team’s

winning percentage, there was evidence of a significant difference between per game

error rates for teams with winning records compared to teams with losing records

[15].

Many people in baseball have critiqued errors for being flawed [19]. Given the

ambiguous nature of placing blame after an error by a fielding team3, recording this

statistic is often a highly debatable and questionable undertaking. Complicating

the problem is that scorekeepers often have little or no training and receive no

3Often times, a failed attempt by the fielding team can not be attributed to just one player. Forexample, consider the scenario where one fielder makes a poor but catchable throw and anotherfielder fails to properly receive that throw. In cases like these, the scorekeeper is forced to makea subjective choice on whether the lapse was due to the throw, the catch or in rare cases, both.

6

formal evaluation of their performance. These individuals are also permitted to

drink while on the job and join rotisserie leagues, where members gamble money on

real baseball outcomes [26]. Kalist and Spurr [19] found that there were systematic

biases in calls made by scorekeepers, where the home team was given fewer errors,

on average, than the away team4.

The most glaring problem associated with errors and fielding percentage is that

fielders are only penalized for bad plays; there is no reward for good fielding plays

when using errors. Fielders who make errors of commission (e.g. make an errant

throw on a difficult play) are charged more than fielders that make errors of omission

(e.g. allowing a “playable” ball to pass in fear of making a mistake). For example,

Adrian Beltre, the former Seattle Mariner, is widely known in baseball circles to be

a great fielding third baseman. However, Beltre was tied for the sixth most errors

(18) in baseball during the 2007 season. By playing less cautiously and attempting

throws on difficult-to-field batted balls, he generated more outs for his team at the

expense of accruing more errors.

In the 1980’s, Bill James began to calculate a statistic he called Defensive Effi-

ciency Ratio5, or DER [17]. DER is the percentage of balls in play that a defense

successfully turns into outs. The major advantage of this statistic was the ease

with which it could be calculated given the basic data available at the time. While

certainly not useful at the player level, DER provided a basic way for evaluating a

4During more tenuous and crucial game situations, this bias nearly disappeared.5In the literature, it is sometimes referred to as the Defensive Efficiency Rating; the two names

reference the same statistic.

7

team’s defense over a season. Any analysis based on this metric suffers from one

glaring confounder: not all balls in play are created equal.

All balls in play can be categorized as one of the following ball in play types:

flyball, liner and groundball. A flyball is any ball hit into the air that takes a high

arc. This is juxtaposed with a liner, which is any batted ball that takes a lower

trajectory through the air6. Groundballs, or grounders, are balls hit on the ground;

that is, a ball that is batted into play which begins on the ground.

It has been shown that a ball in play hit on the ground is fielded by a team with

a much higher success rate than a ball hit in the air [10]. Some studies such as [30]

have shown that pitchers have an effect on the type of balls in play. Thus, teams

with pitchers who tend to induce more groundballs than average would be unfairly

biased upward, over-estimating their defensive efforts. This same phenomenon has

been observed for specific ballparks that have significant effects on the frequency of

certain types of balls in play [11].

More recently, a plethora of fielding statistics have been proposed to address

many of these existing problems, thanks in large part to newly available data

recorded by proprietary entities such as Baseball Info Solutions (BIS). BIS is able

to use advance technology to map the location of every ball hit into play, along with

a number of other characteristics associated with that particular event [9]. The first

such measure to employ this data was the Plus-Minus system [8].

6There is some abiguity surrounding what is regarded as a low versus high trajectory, but forthe purposes of this thesis, I do not address this ambiguity.

8

Figure 1.2: The field of play broken up into the zones 78 zones needed when calculatingUZR.

The Plus-Minus system is a straightforward tabulation of all plays made in a

season. For each out by a fielder in a predesignated bin, that fielder is credited with

a “plus” on that type of ball in play. Likewise, they are penalized a “minus” if no

out is recorded on a ball in play hit into that bin area of that particular type. By

summing these pluses and minuses across all bins in the field of play and all ball

in play types, I arrive at what is known as a fielder’s “plus/minus” [8]. A fielder’s

plus/minus is the number of outs that fielder made above (or below) the average

player at that position. In this way, the Plus-Minus system is a slight modification

on James’ DER by grouping additional bins by the type of batted ball allowed.

Ultimate Zone Rating (UZR) is a metric that quantifies the amount of runs

a player saved (or cost) their team through fielding [21]. To calculate UZR, the

9

field of play is split up into 78 zones. The 14 zones furthest away from home plate

are thrown out and are not considered in the UZR calculation. Next, the league

average out rate in each zone is compared to that same rate for an individual

fielder. These differences are summed over all the remaining 64 zones, weighted

by run expectancies and adjusted for factors such as ball speed. The final result

represents that fielder’s UZR in a given season. In cases where the fielder has not

fielded that position for a full season, their UZR is multiplied by a factor of the

average number of balls in play a fielder at that position would observe over 150

games; this projection is known as UZR/150. I illustrate how the field is zoned in

figure 1.2, along with further explanation about UZR’s calculation. Unlike existing

fielding statistics, UZR has an easily interpretable unit of measure (runs), affording

talent evaluators a better means of analyzing a fielder’s effect on their overall value

as a player.

The Probabilistic Model of Range (PMR) is a fielding statistic developed by

Pinto [25]. PMR incorporates parameters like batted ball direction, velocity and

type into estimating the likelihood of a player successfully fielding an out over 18

pie slices of the baseball field. For each ball in play by a fielder, a sum is taken over

the likelihood of that ball in play being turned into an out. Taking that sum across

all balls in play gives an estimate of the expected number of outs by that fielder

[25]. The ability to evaluate the fielding proficiency of both a specific fielder or an

entire team gives PMR an advantage over other modern techniques.

10

All of these methods serve as vital improvements on existing metrics. However,

they collectively suffer from a very similar problem. The major drawback to these

proposed metrics is in the discretization of the field of play. By enforcing a grid

where balls in play close to each other are binned together, the methods’ accuracies

are damaged by not utilizing the resolution of the data. Little is known about the

variance in these measures, especially within a season. The few studies done on the

subject by [34] and [22] have noted that sample size issues are critical in getting

an accurate assessment. Also, these metrics are only measures of how the player

actually performed, not an estimate of a player’s underlying ability.

1.3 Spatial Analysis in Sports

I take an alternative approach to addressing the issues surrounding current fielding

analytics. To properly utilize the continuous nature of the playing surface, I fit

smooth curves to the field of play. The smooth curves represent the probability

of a player successfully fielding a ball in play at any location on the field, given

covariates related to that batted ball. The approach taken in this dissertation is

novel, in that it is geared towards defense in baseball. Spatial analysis has been

applied to other sports, primarily basketball.

A problem of interest in the basketball community surrounds a common form of

basketball data organization called shot charts. Shot charts are 2D visualizations

of shots taken by a player or a team on the basketball court. Hickson and Waller

11

[16] search for patterns in the shooting behaviour of Michael Jordan, one of the

most popular basketball players of all-time. Using Jordan’s shot chart data from

his 2001-2002 season, spatial point process techniques, specifically non-parametric

kernel smoothing, are applied to estimate the relative risk of Jordan making a shot

(i.e. the ratio between making a shot and missing a shot) from any location on

the court in a specific game [16]. Their analysis, while specific to both situation

and player, provides teams facing Michael Jordan with a shooting “profile” and a

strategy for defending it.

Reich et al [28] uses Bayesian hierarchical models in estimating the spatial dis-

tribution to the success of making a shot at any location on the basketball court.

Reich et al choose to convert the original coordinate system of the shot charts into

polar coordinates, where the process of binning the court is done by angle and dis-

tance, similar to the approach taken when calculating UZR [21]. A conditionally

autoregressive (CAR) distribution can be defined on top of this newly defined space,

allowing for sharing among adjacent neighbours. Included in that specification are

sets of covariates that describe who is on the court, what team is winning and how

much time is left in the game. Using Monte Carlo Markov Chain (MCMC) proce-

dures to estimate the unknown model parameters, Reich et al provide estimates for

a player’s shooting ability on the court, while adjusting for spatial correlations and

teammate interactions [28].

Piette et al [24] take a non-parametric approach towards evaluating the shoot-

12

ing and scoring ability of professional basketball players. They develop two new

statistics, SCAB (SCoring Ability to Baseline) and SHTAB (SHooTing Ability to

Baseline), which serve to (respectively) measure the efficiency of a player at scoring

and shooting compared to a baseline player given the same opportunities. This

baseline player is determined via K-means clustering by players’ shooting patterns,

providing a more accurate standard with which to compare shooters. To calculate

these efficiencies, a kernel smoother is applied to one-dimensional shot data (i.e.

the distance away from the basket for each shot taken) [24]. Much like with UZR

and PMR, these techniques ignore the resolution of the data.

1.4 Related Time Series Techniques

There is great potential improvement on fielding techniques by fully utilizing the

sharing of data across time. For most any skilled activity, it is reasonable to assume

that some players are fundamentally better fielders than their peers of the same age

or over the span of several seasons.

Player trends over time have been studied previously by baseball researchers,

but the work is exclusive to batting and pitching ability. Kaplan [20] provides a

thorough treatment of available time series models on batting performance data in a

frequentist setting. The variety of models tested in [20] include both univariate (e.g.

moving average) and multivariate (e.g. vector autoregressive) versions of common

time series models. Their findings suggest that with regard to offensive metrics,

13

lag-1 models fit applied to the past century of baseball data adequately predict a

player’s current ability, with the exception of some measures [20].

In Null [23], a Bayesian approach is taken to the problem of predicting offensive

abilities of baseball players by implementing a Bayesian hierarchical model. At

the bottom of the hierarchy, a nested Dirichlet prior is placed on the underlying

probability of batting outcomes per plate appearance. One of the drawbacks to the

original implementation in [23] is that there is no incorporation of age adjustments.

Null explores these effects by fitting a fixed effects, asymmetric7 quadratic additive

model to age. Null estimates that such effects do exist and are significant, with

baseball players’ offensive peaks most often occurring between the ages of 28 and

32 [23].

There exists a plethora of additional, non-baseball related literature to draw

upon for time series applications in Bayesian modeling. One natural setting for this

type of modeling is in atmospheric research, where managing spatial, cyclical and

time dependencies is crucial to generating accurate estimates of underlying tem-

perature. Wikle et al [33] specify a five-stage Bayesian hierarchical model, where

the first two levels represent the data and the temperature effects of interest. In

the third level of the hierarchy lies the priors setup for capturing time trends. The

method of choice by Wikle et al is a space-time autoregressive moving average

(STARMA) specification, where a vector of covariates at time t depend on some

7The need for asymmetry is to allow for different rates of improvement and deterioration beforeand after a player’s peak, respectively.

14

combination of the observed covariates from time 1 to t−1. Concerned with identifi-

ability problems, common with these complicated Bayesian models, the matrices of

autoregressive coefficients are assumed to be diagonal matrices [33]. This Bayesian

model is a natural choice in this setting; estimates for all of these unknowns are

(relatively) easily calculated using a Gibbs sampler, yet there is enough flexibility

in the system due to the flatness of the priors placed on the parameters at each

level.

Other similar Bayesian hierarchical models have been developed to perform spa-

tial data analysis. Gelfand and Vounatsou [12] exchange the STARMA elements in

a similar Bayesian model with a conditional autoregressive (CAR) process to deal

with multivariate outcomes.

An alternative to the CAR model discussed above when controling for temporal

variation is a state space model. State space models take physical systems with a

set of inputs, outputs and state variables and permit a convenient derivation and

form to analyze these systems. To separate the data from the parameters of interest

(i.e. the states of the system), two fundamental equations are borne from the inputs

and outputs: the measurement equation, which captures what is actually observed,

and the transition equation, which describes the unobserved movement happening

during time [29]. The general form of these equations is:

(Measurement Eq.) yt = f(xt, εt, γt),

(Transition Eq.) xt = g(xt−1, ηt, φt),

15

where t represents discrete moments in time, yt is the observable outcome and xt

is the unobserved state. An important assumption is that the functions, f and g,

and the nuisance coefficients, γt and φt, are known apriori [29].

Seminal work by Carter and Kohn [5] discuss the implementation of a Gibbs

sampler in a Bayesian hierarchical setting using a special case of the state space

model. In their work, the measurement and transition functions are assumed to be

linear in the state variables and the errors are drawn from a mixture distribution,

where an unknown switch variable dictates the exact distribution. To properly sam-

ple the state variables, Carter and Kohn use a forward filter, backwards sampling

(FFBS) technique. A forward pass of the data is made, where the expectation of

each state given all of the data and states up to that time period (i.e. state) is

computed using the Kalman filter [2]. This is not sufficient for generating samples,

however, since the conditional posterior distribution of each state includes all states

and outcomes, both past and future. Carter and Kohn [5] propose a backwards step

to calculate the expectation of each state given future observations and states by

properly utilizing the calculated expectations from the forward pass. Samples are

then generated from these final values; these samples are representative of the con-

ditional posterior distributions for the corresponding state. Samples of these state

variables may be iteratively generated, along with the other parameters of inter-

est, including the nuisance parameters involved in the measurement and transition

equations, by incorporating this method into a Gibbs sampler of the entire model

16

[5]. Extensions on this work by Geweke and Tanizaki [14] give a solution for cases

where the state space functions are non-linear.

A direct application of this specific FFBS technique in conjunction with spatial

analysis is Ecology, specifically work done by Wikle [32]. Wikle employs Bayesian

hierarchical models for predicting outcomes from certain ecological processes. The

levels of the model hierarchy described in [32] are broken up into a data model and

a process model. From there, parameters that control for spatiotemporal trends

and latent processes are specified to fully capture both the time trend of the mean

House Finch clusters and the various physical barriers and landscape subtleties that

force the birds to spread in certain directions. Random draws from a Gibbs sampler

are taken to generate the desired posterior estimates and intervals. While admit-

ting that the numbers are far from perfect, Wikle does explain that this Bayesian

hierarchical approach lends itself to these applications where an understanding of

uncertainty over such a high-dimension of data of varying types is necessary [32].

1.5 Dissertation Organization

The remainder of the dissertation will be organized as follows. In chapter 2, I briefly

review the original “SAFE” model taken to evaluate fielding in [18]. I discuss the

specifics behind parametrizing the smooth curves fitted to the field of play, details

about the data used for estimation and the original Bayesian hierarchical model

[18]. I detail new model extensions which contain time series aspects in chapter 3.

17

The models I introduce are as follows:

1. a constant-over-time model,

2. a moving average age model and

3. an autoregressive age model.

A description of the results begin in chapter 4, along with an in-depth study of the

performance and evaluation of specific players with the different models. Then, I

detail the procedures and findings for internal and external model validation across

all the current models in chapter 5. Finally, in chapter 6, I conclude the dissertation

with our final thoughts and proposals for future work surrounding these new models.

18

Chapter 2

Original Bayesian Hierarchical

Model

The basic approach used in this paper is to model the outcome of every ball in play as

a binary variable (i.e. successfully fielded or not). The underlying probability of that

outcome is a function of the characteristics related to that ball in play, which include

landing location, velocity and type. Smooth curves representing that probability

are fit onto the baseball field of play using a similar parametrization to Reich et al

[28]. This type of Bayesian hierarchical model is used to share information across

space and over time for each fielder. After estimation of the model parameters, an

integration is performed over the entire playing surface, weighting each point by the

expected proportion of balls in play at that point and the run consequence of a ball

in play landing in that location. The final number produced from this process is an

19

estimate of the number of runs a fielder saved/cost their team through their fielding

in a given season: this statistic is called the Spatial Aggregated Fielding Evaluation

(SAFE) for each fielder [18]. My contribution will be to extend this approach to

share information within a player over time, using three model specifications.

2.1 Description of the Data

Fielding estimation in this thesis is based upon a high-resolution data source from

Baseball Info Solutions [9]. The data consists of all events that occurred in every

MLB game across seven seasons of play (2002 - 2008). All non-batted ball events

(e.g. strikeouts, walks) are removed, as well as batted balls that are caught for an

out in foul territory1 or batted balls that leave the field of play for a home run. The

remaining events, which I refer to as balls in play, are any batted ball that could

feasibly be fielded for an out, or multiple outs, by the fielding team.

Over the seven season span, nearly 930,000 such balls in play are observed. A

breakdown of that total by season and by batted ball type is located in table 2.1.

Bunts are partitioned into its own ball in play type. The defense by a fielding team

against a bunt is vastly different from the normal defensive positioning. While a

fielding team can be taken by surprise when a player bunts, it is usually apparent

apriori to the fielders that a batter will bunt based on the game situation and

1I do not include these batted balls because these observations are censored events; I onlyobserve batted balls that are caught in foul territory, leaving balls that drop (i.e. are not caught)in foul territory as unobserved.

20

Table 2.1: Breakdown of the data by season and by ball in play type.

Flyballs Grounders Liners Bunts

‘02 43,093 58,052 27,879 2,926(32.7%) (44.0%) (21.1%) (2.2%)

‘03 41,696 58,698 29,792 3,000(31.3%) (44.1%) (22.5%) (2.1%)

‘04 45,186 59,872 24,891 2,936(34.0%) (45.1%) (18.7%) (2.2%)

‘05 42,757 59,892 27,793 2,909(32.1%) (44.9%) (20.8%) (2.2%)

‘06 44,651 59,297 26,310 2,835(33.5%) (44.6%) (19.8%) (2.1%)

‘07 46,601 58,949 25,099 2,730(34.9%) (44.2%) (18.8%) (2.1%)

‘08 43,489 58,696 26,712 2,718(33.0%) (44.6%) (20.3%) (2.1%)

Total 307,473 413,456 188,476 20,054(33.1%) (44.5%) (20.3%) (2.1%)

batting stance. Add to that the infrequency with which bunts are attempted, I

exclude them from the analysis of SAFE.

For each of the remaining balls in play types (flyballs, liners, grounders), I focus

on three covariates for fitting our smooth curves: the ball in play’s (x, y) landing

location2, velocity3 and type (i.e. groundball, flyball or liner). For flyballs and

liners, the landing location literally represents the point at which the ball either

(a) landed on the field or (b) was caught by the fielder. Heat maps for the landing

locations of each flyball and liner in our dataset can be found in figure 2.1. This

2The coordinate system for the 2002 to 2005 seasons differs from the system used in the 2006to 2008 seasons, but a transformation to each can bring them to a universal coordinate systemmeasured in feet with reference to home plate.

3In our data, this is coded as an integer from 1 to 3, where 1 corresponds to balls hit at thelowest velocity and 3 corresponds to balls hit at the highest velocity.

21

Figure 2.1: Heatmaps of all in play (i.e. excluding homeruns and foul balls) flyballs(left) and liners (right).

definition is not applicable to groundballs, given that a groundball “starts” on

the ground after being hit. Instead, the coordinates for grounders represent the

locations where a fielding attempt was made. For reasons I reveal in Section 2.2,

it is more informative to report the angle at which the ball was hit. To visualize

the groundball data, I graph empirical histograms and densities on the angle from

third base where a groundball was fielded by one of the different infield positions in

figure 2.2.

Along with these covariates, other information about that ball in play is pro-

vided, such as who is on the field, how the play was scored (i.e. which fielder(s) did

what) and the result of the play (e.g. hit, out). One piece of information that is only

available in the three most recent seasons is whether or not the fielding team had

put on the “shift.” In fielding, when a left-handed batter who is known for pulling

22

Figure 2.2: The curves here represent the empirical histogram and densities of the anglesfor all groundballs hit in our dataset that were fielded by one of the infieldpositions. The angle here is the angle at which the groundball was hit withrespect to third base.

23

the ball (i.e. hitting the ball to the right side of the field)4 is at-bat, the infielders

often concentrate on the right side of the infield, leaving the space between third

base and second base almost entirely open (see figure 2.3). This extreme positioning

method is rare, however, as most batters adjust their style of play to incorporate

hitting balls to both sides of the field. Given the frequency with which it occurs

(under 1%) and the unavailability of such data during the first four seasons of data,

I choose to ignore this defensive shift variable entirely from our analysis.

There are other cases for more subtle shifts to the defensive alignment for both

infielders and outfielders. In particular, with a man on first base, the first baseman

is forced to play closer to first base, in case a pickoff throw is attempted5. The

middle infielders also position themselves near second base, given that there is a

chance of “turning a double play,” or getting two outs off of a batted ball (see figure

2.3). I explored this positioning movement by fitting models with a runner on first

base and models without a runner on first base. I found there to be no significant

difference between the estimation of the two models, so I disregard this effect and

simply fit one model.

4This phenomenon is exclusive to left-handed hitters due to the restraint on the first baseman.It is necessary that a fielder is near first base in order to force out the runner coming out of thebatters box; otherwise, first base would go uncovered and the batter would be able to safely reachbase, regardless if the ball was properly fielded in time.

5A pickoff throw is a throw by the pitcher to keep the baserunner from gaining too much of alead off of the base. By letting a runner go unchecked from this strategy, baserunners are morelikely to score on softly hit balls in play and could potentially take a base without a ball being hit(i.e. steal a base).

24

Figure 2.3: These are three examples of the different defensive alignments used by field-ing teams. Often these alignments depend on both the batter and the gamesituation. I show one instance related to left-handed batters who tend to pullthe ball (defensive shift) and the fielding arrangement when a runner is onfirst base (runner on 1st). Finally, for comparison purposes, the alignmentmost often used is included (normal).

25

2.2 Data Manipulation

In order to properly attribute the level of impact a fielder had on their team, I need

to model the probability of whether that fielder made an “out”6 on any given ball in

play. For each model, I start by preparing the data with two different procedures:

one for flyballs/liners and one for groundballs.

2.2.1 Flyballs/Liners

I treat flyballs and liners as synonymous in our analysis, as their landing location

data are measured in the same way. To properly model the probability of catching a

flyball or liner, the distance from where the fielder was standing to where the ball in

play landed, or was caught is required. Since I do not have data on where the fielder

was standing before the ball was hit, I estimate each position’s starting point, or

centroid, by finding the location where the highest number of outs are converted.

Another factor needed is whether the player is moving forward to reach the ball in

play. From observing the analysis, it is much easier to field a ball moving forward

than it is to run backwards to catch a batted ball. An example of this procedure

in practice can be seen on the left-side of figure 2.4.

6An “out” here could refer to a play in which no out is recorded. This scenario only occurswhen the fielder in question successfully fields the ball and makes an accurate throw, but thethrow’s recipient fails to convert. Note that this can only occur on a groundball, not on a liner orflyball.

26

Figure 2.4: A visual representation behind the procedures used to calculate the distanceto a flyball or liner in one example and the angle away to a grounder for an-other. Also shown is how moving forward on a flyball or liner is determinedand how ranging to the right is found for grounders.

2.2.2 Grounders

As I mention in Section 2.1, it is inappropriate to model the probability of fielding

a groundball using distance, because a grounder begins on the ground and the

coordinates recorded for it correspond to the location where a fielding attempt is

made on the groundball. Instead, I model the angle the grounder took away from

home plate. I find each fielder’s starting position, using the angle with which there

is the highest concentration of fielded batted balls (see the vertical red lines in figure

2.2). The direction that the fielder ranged to get to that ball in play is needed; that

is, whether that player ranged to their left or their right to field the grounder. I

show an example of gathering this information on the right-side of figure 2.4.

27

Table 2.2: Summary of the models.

Flyballs Grounders Liners

1B X X2B X X3B X XSS X XLF X XCF X XRF X X

2.2.3 Eligible BIP × Fielder Combinations

I carry out the model implementations for seven of the nine fielding positions,

eliminating catchers and pitchers from our analysis. I do not consider either pitchers

or catchers since they have very few fielding chances in a given season. I fit each

ball in play type separately for a given position. Some ball in play types, however,

are not modeled for certain fielding positions. I provide a listing of all eligible ball

in play and position combinations in table 2.2. In general, I do not model grounders

for outfielders, given that they would rarely be able to make an out on a groundball.

I also do not fit flyballs for infielders. Infield position players do have chances on

flyballs hit to the infield, often referred to as pop ups, but the probability of catching

them for an out is close to 1, regardless of their landing location in the infield. Thus,

the difference between an infielder who poorly fields pop ups and another infielder

who fields them perfectly is negligible. In total, this makes for 7 + 3 + 4 = 14

different fits per model specification.

28

2.3 Original Hierarchical Model

The Bayesian hierarchical model featuring no time dependent elements is the orig-

inal model for SAFE detailed in [18]. I start by considering some fielder i in season

t. I refer to the number of balls in play hit while player i is fielding as nit7. The

outcome of the play on the jth BIP is denoted by Sitj:

Sitj =

1 if the jth ball hit to the ith player in season t is fielded successfully,

0 if the jth ball hit to the ith player in season t is not successfully fielded.

These observed variables are modelled as Bernoulli realizations from an under-

lying, event-specific probability:

Sitj ∼ Bernoulli(pitj).

The BIP-specific probabilities, pitj, are modelled as a probit function of covariates:

pitj = Φ(Xitj · βββit),

where Φ(·) is the cumulative distribution function for the Normal distribution and

Xitj is the vector of covariates for BIP j. I have five different covariates related to

each BIP. For flyballs/liners, these are:

1. an intercept, 1,

2. the distance to the BIP from the fielder’s starting position, Ditj,

7In practice, I do not consider all balls in play hit while player i is fielding, but any ball inplay that player i could have a chance at fielding. I determine whether a player has a chance atfielding a ball in play by eliminating any balls in play that are hit past the furthest ball caughtby any fielder at that player’s position.

29

3. the interaction between distance and the ordinal velocity measurement, Ditj ·

Vitj,

4. the interaction between distance and a binary variable for whether the fielder

needs to range forward for the BIP, Ditj · Fitj,

5. and the interaction between all three of these covariates, Ditj · Vitj · Fitj.

A similar list of covariates are associated to each groundball:

1. an intercept, 1,

2. the angle to the BIP from the fielder’s starting position, Aitj,

3. the interaction between angle and the ordinal velocity measurement, Aitj ·Vitj,

4. the interaction between angle and a binary variable for whether the fielder

need to range to their right for the BIP, Aitj ·Ritj,

5. and the interaction between all three of these covariates, Aitj · Vitj ·Ritj.

The ability of individual player-seasons is represented by their coefficient vector βββit;

this is estimated separately for each season t. This vector, βββit, implies a potentially

different probability curve of making an out for each player-season combination.

Due to the number of possible balls in play, the separate estimation of each βββit

leads to some unstable and highly variable estimates. This problem is addressed

in [18] by modeling each player-season coefficient vector as a draw from a common

30

distribution shared by all players (and all seasons) at a position:

βββit ∼ Normal(µµµ,σ2σ2σ2).

I assume that the components of each βββit are a priori independent, which forces

σ2σ2σ2 to be a matrix with diagonal elements σ2k and off-diagonal elements of zero. I

use the non-informative prior distribution suggested by Gelman et al [13] for the

common parameters shared across all players at that position:

p(µk, σ2k) ∝ σ−1

k , for k = 0, . . . , 4.

[18] use a Gibbs sampling approach for the estimation of our full posterior dis-

tribution of all unknown parameters for a particular position and BIP type. In the

outline below, I suppress the t notation, since I am ignoring the time series effects in

this implementation. Thus, i in this section indexes all player-seasons in our dataset.

Recall that Xij is the vector of all location, velocity and interaction covariates for

BIP j for player-season i. Following the strategy outlined by Albert and Chib [1],

the data is augmented with random variables Z, where Zij ∼ Normal(Xij · βββi, 1)

and observe that:

P (Sij = 1) = Φ(Xij · βββi) = P (Zij ≥ 0).

Our posterior distribution of interest then becomes:

p(βββ,µµµ,σ2σ2σ2|S,X) ∝∫p(S|Z) · p(Z|βββ,X) · p(βββ|µµµ,σ2σ2σ2) · p(µµµ,σ2σ2σ2)dZ,

31

where:

p(S|Z) =m∏i=1

ni∏j=1

(I(Yij = 1, Zij ≥ 0) + I(Yij = 0, Zij ≤ 0)) ,

p(Z|βββ,X) ∝m∏i=1

ni∏j=1

exp

(−1

2(Zij −Xij · βββi)2

),

p(βββ|µµµ,σ2σ2σ2) ∝4∏

k=0

(σ2k)−m

2 ·m∏i=1

exp

(− 1

2σ2k

(βik − µk)2

),

p(µµµ,σ2σ2σ2) ∝4∏

k=0

(σ2k)− 1

2 .

I obtain samples of βββ,µµµ and σ2σ2σ2 from the joint posterior distribution using a Gibbs

sampling algorithm, where I iteratively sample for:

1. p(Zij|βββ,S,Xij), for each i = 1, . . . ,m and j = 1, . . . , ni,

2. p(βββi|Z,µµµ,σ2σ2σ2,X), for each i = 1, . . . ,m,

3. p(µµµ|βββ,σ2σ2σ2),

4. p(σ2σ2σ2|βββ,µµµ).

Note that m in this setting refers to the total number of player-seasons.

Step 1 of the algorithm is a sample from a truncated normal distribution for

each Zij:

p(Zij|βββ,S,X) ∝ exp

(−1

2(Zij −Xij · βββi)2

)· [I(Sij = 1, Zij ≥ 0) + I(Sij = 0, Zij ≤ 0)] ,

which means that I keep sampling Zij ∼ Normal(Xij ·βββi, 1) until either the condition

(Sij = 1, Zij ≥ 0) or the condition (Sij = 0, Zij ≤ 0) is met. Step 2 of the algorithm

32

samples the player-season parameters βββi from the distribution:

βββi ∼ Normal((Σ−1 + X′iXi)

−1 · (Σ−1µµµ+ X′iZi), (Σ−1 + X′iXi)

−1),

where Xi collects {Xij : j = 1, . . . , ni} and Σ is a 5 × 5 matrix with diagonal

elements σ2k and zeroes in the off-diagonals. Step 3 of the algorithm samples the

prior means µk, where k = 0, . . . , 4, from the distribution:

µk ∼ Normal

(βk,

σ2k

m

),

where βk =∑m

i=1 βik/m. The final step, step 4, samples the prior variances σ2k

(k = 0, . . . , 4) by sampling Ak from a Gamma distribution:

Ak ∼ Gamma

(m− 1

2,

∑mi=1(βik − µk)2

2

)

and then setting σ2k = A−1

k .

33

Chapter 3

New Model Specifications

I propose three new models for sharing information within a player over time. Our

basic extension of the SAFE model treats time as a constant, but considers each

season played by a player to be drawn from a parameter describing that player’s

“overall” underlying ability. I implement a similar moving average model seen in

[20]. In this model, I allow for shrinkage across age, where each player is drawn

from an ability average at that player’s age. Finally, the most sophisticated model I

choose to implement is autoregressive, similar to that used by Carter and Kohn [5],

where the states relate to each player’s underlying ability and the observed outcome

is whether a ball was successfully fielded.

34

Figure 3.1: A pictorial representation of the three-level hierarchy involved in theconstant-over-time ability model.

3.1 Constant-Over-Time Model

The original model treats each βββit as a separate coefficient vector, and ignores the

fact that some of these vectors represent the same player across multiple years. To

share this information across seasons by the same player, I propose an additional

model level where player ability is constant over time. This extension adds another

level to the former hierarchical model, giving us three tiers: an overall (or league

average) position curve, a player-specific curve for each player and a season-specific

curve for each season within each player. The added season-specific level does not

refer to the curves fit for all players in a specific season, but the curves fit for each

player in a specific season. I present a pictorial representation of this hierarchy in

figure 3.1.

Similar to the original model, the observed BIP outcomes are modeled as Bernoulli

35

realizations from an underlying, event-specific probability:

Sitj ∼ Bernoulli(pitj).

The BIP-specific probabilities, pitj, are modeled as a probit function of covariates:

pitj = Φ(Xitj · βββit),

where the curve coefficients βββit are season-specific for a particular fielder i. The

coefficients of the season-specific curve are drawn from a Normal distribution around

their player-specific abilities γγγi. Here, βββit is defined as the vector of coefficients for

this new season-specific curve for player i in season t. With the introduction of a

new prior, I define the variance of this prior for time-specific coefficients as τ 2τ 2τ 2; this

is the year-to-year variation from the player-specific means. I denote the player-

specific ability coefficients as γγγi for a player i, with corresponding variance σ2σ2σ2. Using

this nomenclature, our model can be written as the following:

p(µk, σ2k, τ

2k ) ∝ σ−1

k τ−1k , for any k = 0, . . . , 4,

γγγi ∼ Normal(µµµ,σ2σ2σ2),

βββit ∼ Normal(γγγi, τ2τ 2τ 2).

I use a similar Gibbs sampling approach for the estimation of our full posterior

distribution of all unknown parameters for a particular position and BIP type.

However, I must now track both players i and seasons t (t = 1, . . . , Ti) by each player

i. Assume I have T =∑

i Ti total seasons of data across all m players. Again, I begin

36

by augmenting our data with random variables Z, where Zitj ∼ Normal(Xitj ·βββit, 1),

which yields a similar equation:

P (Sitj = 1) = Φ(Xitj · βββit) = P (Zitj ≥ 0).

Our posterior distribution is now:

p(βββ,γγγ,µµµ,σ2σ2σ2, τ 2τ 2τ 2|S,X) ∝∫p(S|Z) · p(Z|βββ,X) · p(βββ|γγγ, τ 2τ 2τ 2)

· p(γγγ|µµµ,σ2σ2σ2) · p(µµµ,σ2σ2σ2, τ 2τ 2τ 2)dZ,

where:

p(S|Z) =m∏i=1

Ti∏t=1

nit∏j=1

(I(Sitj = 1, Zitj ≥ 0) + I(Sitj = 0, Zitj ≤ 0)) ,

p(Z|βββ,X) ∝m∏i=1

Ti∏t=1

nit∏j=1

exp

(−1

2(Zitj −Xitj · βββit)2

),

p(βββ|γγγ, τ 2τ 2τ 2) ∝4∏

k=0

(τ 2k )−

T2 ·

m∏i=1

Ti∏t=1

exp

(− 1

2τ 2k

(βitk − γik)2

),

p(γγγ|µµµ,σ2σ2σ2) ∝4∏

k=0

(σ2k)−m

2 ·m∏i=1

exp

(− 1

2σ2k

(γik − µk)2

),

p(µµµ,σ2σ2σ2, τ 2τ 2τ 2) ∝4∏

k=0

(τk · σk)−1.

I obtain samples βββ,γγγ,µµµ,σ2σ2σ2 and τ 2τ 2τ 2 from the posterior distribution using a Gibbs

sampling algorithm, where I iteratively sample from:

1. p(Zitj|βββ, Sitj,Xitj) for each i = 1, . . . ,m, t = 1, . . . , Ti and j = 1, . . . , nit,

2. p(βββit|Zit,Xit, γγγi, τ2τ 2τ 2) for each i = 1, . . . ,m and t = 1, . . . , Ti,

3. p(γγγi|βββ,µµµ,σ2σ2σ2) for each i = 1, . . . ,m,

37

4. p(µµµ|γγγ, τ 2τ 2τ 2,σ2σ2σ2),

5. p(τ 2τ 2τ 2|βββ,γγγ),

6. p(σ2σ2σ2|γγγ,µµµ).

Step 1 of the algorithm is still a sample from a truncated normal distribution for

each player i, season t and BIP j:

p(Zitj|βββit,S,X) ∝ exp

(−1

2(Zitj −Xitj · βββit)2

)· (I(Sitj = 1, Zitj ≥ 0) + I(Sitj = 0, Zitj ≤ 0)) ,

which implies I keep sampling Zitj ∼ Normal(Xitj ·βββit, 1) until either the condition

(Sitj = 1, Zitj ≥ 0) or (Sitj = 0, Zitj ≤ 0) is met. Step 2 of the sampling procedure

samples the player curves for each specific season, or player-season curve βββit, from

the distribution:

βββit ∼ Normal((T−1 + X′iXi)

−1 · (T−1µµµ+ X′iZi), (T−1 + X′iXi)

−1),

where T is a 5×5 matrix with diagonal elements τ 2k and zeroes in the off-diagonals.

In step 3 of the algorithm, γγγi is sampled, which are the player-overall curves located

at the second level of the model’s hierarchy. The sampling distribution is:

γik ∼ Normal

(βik · Tiτ2k + µk · 1

σ2k

Tiτ2k

+ 1σ2k

,1

Tiτ2k

+ 1σ2k

),

where βik =∑Ti

t=1 βitk/Ti. Step 4 samples each µk as:

µk ∼ Normal

(γk,

σ2k

m

),

38

where γk =∑m

i=1 γi/m. These sampled coefficients represent the league average

estimates. I sample τ 2k as A−1

k in step 5, where Ak is distributed as:

Ak ∼ Gamma

(T − 1

2,

∑mi=1

∑Tit=1(βitk − γik)2

2

).

In the final step in this sampling procedure, step 6, I generate samples of σ2k as B−1

k ,

where Bk is distributed as:

Bk ∼ Gamma

(m− 1

2,

∑mi=1(γik − µk)2

2

).

3.2 Moving Average Age Model

The moving average age (MAA) model suggests that a player’s fielding ability in a

given season is a function of their age during that season. Information about each

player’s curve coefficients is shared across all seasons played by a fielder who was of

that same age. Instead of promoting sharing within players as in the constant-over-

time model (i.e. between seasons by the same player), the MAA model focuses on

this time-dependent sharing across players. In other words, I remove any sharing of

information within players and add sharing of information between players of the

same age.

This new model extension may also be explained by making reference to the

original model in [18]. The same two-level hierarchical structure enforced in the

original SAFE model exists in the MAA model with one major difference: there are

multiple overall means, or µµµ’s. For each age in which at least one fielder is observed

39

Figure 3.2: A visual realization of the structure behind the MAA model.

in our sample, a unique position-age mean, and thus, a clone of the original SAFE

model hierarchy, is fit for all players of that age (see figure 3.2 for a diagram of the

MAA model’s structure).

Under the MAA model, I refer to the season-specific curve coefficients as age-

specific, reflecting the different “scale” in which time is treated. These age-specific

curve coefficients are indexed by βββia, where a is the age of fielder i during the

observed season. A Normal prior is placed on the βββia with mean µµµa and variance

σ2σ2σ2. This position-age mean, µµµa, can be interpreted as the average fielding curve for

fielders of age a at a particular position. Once again, the data levels of the model

are the same, where:

Siaj ∼ Bernoulli(piaj),

piaj = Φ(Xiaj · βββia),

and βββia are the age-specific curve coefficients. The hierarchy ends here by placing a

flat prior on the remaining unknown parameters. The MAA model is summed up

40

below:

βββia ∼ Normal(µµµa,σ2σ2σ2),

p(µak, σ2k) ∝ σ−1

k , for any k = 0, . . . , 4 and all a ∈ Age,

where Age is the set of all ages at which there is at least one fielder who played

that position at that particular age.

To estimate all of the unknown parameters, I sample estimates from the full

posterior distribution via Gibbs sampling for a given position and BIP type. As

mentioned above, I no longer index time by season t, but by the age of that fielder

during season t, call it a. Also, I let Agei be the subset of Age containing all ages at

which fielder i was observed and Ma be the set of player indices for all fielders that

have played a season at age a. Then, I augment our data with random variables

Z, where Ziaj ∼ Normal(Xiaj · βββia, 1). Thus, the probability corresponding to the

success on a ball in play j by fielder i at age a is:

P (Siaj = 1) = Φ(Xiaj · βββia) = P (Ziaj ≥ 0).

The posterior distribution for the MAA model is:

p(βββ,µµµ,σ2σ2σ2|S,X) ∝∫p(S|Z) · p(Z|βββ,X) · p(βββ|µµµ,σ2σ2σ2) · p(µµµ,σ2σ2σ2)dZ,

where µµµ is the collection of µµµa at all ages a in Age. The individual components of

41

the posterior distribution described above are as follows:

p(S|Z) =m∏i=1

∏a∈Agei

nia∏j=1

(I(Siaj = 1, Ziaj ≥ 0) + I(Siaj = 0, Ziaj ≤ 0)) ,

p(Z|βββ,X) ∝m∏i=1

∏a∈Agei

nia∏j=1

exp

(−1

2(Ziaj −Xiaj · βββia)2

),

p(βββ|µµµ,σ2σ2σ2) ∝4∏

k=0

(σ2k)−T

2 ·m∏i=1

∏a∈Agei

exp

(− 1

2σ2k

(βiak − µak)2

),

p(µµµ,σ2σ2σ2) ∝4∏

k=0

(σk)−1.

I retrieve samples βββ,µµµ and τ 2τ 2τ 2 from the posterior distribution by iteratively sampling

from:

1. p(Ziaj|βββ, Siaj,Xiaj) for each i = 1, . . . ,m, a ∈ Age and j = 1, . . . , nia,

2. p(βββia|Zia,Xia,µµµa,σ2σ2σ2) for each i = 1, . . . ,m and a ∈ Age,

3. p(µµµa|βββ·a,σ2σ2σ2) where βββ·a is the collection of all the age-specific curve coefficients

at age a,

4. p(σ2σ2σ2|βββ,µµµ).

Step 1 of the algorithm for the MAA model is the same as step 1 for the constant-

over-time model, with the only difference being in how time is referenced:

p(Ziaj|βββia,S,X) ∝ exp

(−1

2(Ziaj −Xiaj · βββia)2

)· (I(Siaj = 1, Ziaj ≥ 0) + I(Siaj = 0, Ziaj ≤ 0)) .

This means that I continue sampling Ziaj ∼ Normal(Xiaj · βββia, 1) until either the

condition (Siaj = 1, Ziaj ≥ 0) or (Siaj = 0, Ziaj ≤ 0) is met. In step 2 of the Gibbs

42

sampling procedure, I sample all five of the age-specific curve coefficients for each

player i. For any a ∈ Agei, the conditional posterior distribution of the age-specific

curve coefficients for fielder i, βββia, is:

βββia ∼ Normal((Σ−1 + X′iaXia)

−1 · (Σ−1µµµa + X′iaZia), (Σ−1 + X′iaXia)−1).

For all a ∈ Age, the position-age curve coefficients at age a, µµµa, are sampled during

step 3. The sampling distribution for coefficient k is:

µk ∼ Normal

(βak,

σ2k

ma

),

where βak =∑

i∈Maβiak/ma andma is the number of players who played the position

at age a in our dataset, or simply |Ma|. I sample the variance parameters σ2σ2σ2 in the

final step, step 4. For a particular coefficient k, σ2k is sampled via A−1

k , where Ak is

distributed as:

Ak ∼ Gamma

(T − 1

2,

∑a∈Age

∑i∈Ma

(βiak − µak)2

2

).

3.3 Autoregressive Age Model

The MAA model is a first attempt at engaging a time-sensitive aspect in the dataset

by enforcing sharing across players of similar ages. The disadvantage with this

approach is that there is a lack of continuity between seasons for a given player. In

the constant-over-time model, a player’s season-specific curve coefficients are drawn

from a distribution centred around that player’s player-specific curve coefficients,

43

which relies heavily upon the idea of incorporating between-season sharing for each

player. However, there are no time series features in the constant-over-time model

that utilize our understanding that a player’s fielding assets flourish and degenerate

with age. The final proposed model, an autoregressive age model, combines elements

from the strengths of both the constant-over-time and moving average age models.

3.3.1 Model Description

Much like in the MAA model, the curve coefficients of a fielder during a particular

season are indexed via age. Thus, the age-specific curve coefficients for player i

during season t are referred to using the vector βββiait . Drawing on work by [5] on

state space models in Bayesian settings, I treat these age-specific curve coefficients

as states in a system. After each time period (i.e. as a player ages), a fielder’s

current age-specific curve coefficients are a linear function of their past age-specific

curve coefficients. The actual observations of these states are tied to the latent

variables Z. While technically unknown, I sample these latent variables using the

actual on-the-field outcomes, S, and its associated covariates, X.

The autoregressive age model’s framework can be summarized by defining the

output and state evolution equations for this new model as follows:

(Output Eq.) Ziait = Xiaitβββiait + eiait ,

(State Evolution Eq.) βββiait = φφφaitβββiai(t−1)+ uiait ,

where Ziait are the latent variables, Xiait are the covariates, βββiait are the age-specific

44

curve coefficients, eiait are standard normals that represent the error associated with

the latent variables, φφφait are the age-dependent discounts/premiums shared across

all players, uiait are the error terms associated with the lag curve coefficients and

are distributed by N(0,σ2σ2σ2). The subscript ait is a mapping that represents the

age at which player i plays in their tth season. Note that when referencing these

equations, t > 1. Also, φφφait is a matrix of values, where all non-diagonal elements

are zero. This is done for model simplicity, as I do not consider the case where

each curve’s coefficients contain interaction. The latent variables from the “output

equation” Z are generated from the observed outcome variables. Those Bernoulli

variables and their underlying, event-specific probabilities are defined as:

Sia+itj ∼ Bernoulli(piaitj) and

piaitj = Φ(Xiaitj · βββiait).

To generate samples of βββiait when t ≥ 1, I need a few additional assumptions

on top of this state space structure. In particular, I must define initial state mean

and variance conditions, call them αααi and τ 2τ 2τ 2, respectively. Keeping with the same

flavor of the state evolution equation, I describe the initial state distribution to be:

(Initial State Dist.) βββiai1 ∼ N(φφφai1αααi, τ2τ 2τ 2),

where αααi and τ 2τ 2τ 2 are both 5 by 1 vectors. These parameters, αααi, for all i, and τ 2τ 2τ 2,

are treated as partly unknown with no associated priors. As a result, I do not

sample these initial state parameters, but instead, semi-estimate them as I move

45

through our Gibbs sampling chain. The remaining state space assumption is that

conditioned on the unknown parameters inside the state space domain, all of the

distributions surrounding the errors eiait and uiait are understood, for all i and t.

Finally, I place uniform priors on the remaining unknown parameters, specifi-

cally:

p(φa, σ2k) ∝ σ−1

k , for any k = 0, . . . , 4 and all a ∈ Age,

where Age is the set of all ages at which there is at least one fielder who played that

position at that particular age. A visualization of the autoregressive age model is

provided in figure 3.3, detailing the movement of each player from their initial state

to their final state, given that there are no missing observations between the two.

Due to nature of the data, where players can suffer injuries putting them out of

commission for an entire season, there are unavoidable cases of missing observations.

In the event that there are these gaps in a player’s career over the observed dataset,

the seasons separated by a gap are treated as adjacent. For example, if a fielder

played seasons 2002 and 2004, but missed 2003 due to injury, then the fielder’s state

during 2002 skips their state during 2003 and transitions directly into their 2004

state. In these cases, however, an alteration is made to those fielders’ transition

parameters, φφφ, as they are carried through twice, one for their age in the missing

season and one for their current season. Referring back to the example above,

the player’s age-specific curve coefficients in 2002 are multiplied by two φφφ when

transitioning into their 2004 state: the φφφ at their age in 2003 and the φφφ at their age

46

Figure 3.3: These are visual examples of how fielders’ curve coefficients change as theyage. An arrow in this model represents movement in time and the associatedφa’s are the modifiers, or autoregressive terms, that are permuted with thefielder’s previous age-specific curve coefficients.

in 2004. For more details on this phenomenon, I present an example similar to the

one discussed in figure 3.4.

Carrying out Bayesian inference on this model requires more than the Gibbs

sampler detailed for previous models. Due to the complexity behind linear state

space models, I must incorporate a method known as Forward Filtering Backward

Sampling, or FFBS, when sampling the season-specific curve coefficients βββiait , or

states.

3.3.2 State Sampling: FFBS

Before I begin to highlight this process, I need to add more notation. Let ZTii =

{Z′iai1 , . . . ,Z′iaiTi}′ be the vector of latent variables, XTi

i = {X′iai1 , . . . ,X′iaiTi}′ be

47

Figure 3.4: In this example, fielder i is missing observations during their second season,or the season at age ai2. To solve this problem, seasons 1 and 3 are treatedas adjacent and instead of the discount/premium transition coefficient beingjust φai2 , it is the product of φai2 and φai3 .

the matrix of covariates and let βββi

= {βββ′iai1 , . . . ,βββ′iaiTi}′ be the total state vector.

The following equation shows how to generate the whole of βββi

given ZTii , XTi

i and

the remaining unknown parameters, θθθ:

p(βββi|ZTi

i ,XTii , θθθ) = p(βββiaiTi |Z

Tii ,X

Tii , θθθ)

Ti−1∏t=1

p(βββiait|Zti,X

ti, θθθ,βββiai(t+1)

).

These probabilities are Gaussian densities, so in order to generate all of the

βββiait , I make two passes through the state space portion of our model: a forward

and backward pass. I calculate the following expectation in the forward pass:

E(βββiait|Zti,X

ti, θθθ), V ar(βββiait|Zt

i,Xti, θθθ),

which starts at t = 1 and ending in the last season for fielder i, t = Ti. In the

backward pass, I sample current states conditional on all ”future” states using:

E(βββiait|Zti,X

ti, θθθ,βββiai(t+1)

), V ar(βββiait|Zti,X

ti, θθθ,βββiai(t+1)

).

This sampling begins at the state t = Ti − 1 and finishes with the first sea-

son, t = 1. The output from this last pass, as well as E(βββiaiTi |ZTii ,X

Tii , θθθ) and

48

V ar(βββiaiTi |ZTii ,X

Tii , θθθ) from the first pass, serve as the means and variances that are

used when I sample all of our states for each player i (i.e. βββi).

Forward Pass

During the forward pass, I use the Kalman filter to find the conditional means

βββi(ait|ait) and the conditional variances Si(ait|ait), for t = 1, . . . , Ti, where:

βββi(ait|ait) = E(βββiait|Zti,X

ti, θθθ), Si(ait|ait) = V ar(βββiait|Zt

i,Xti, θθθ).

The details of that process can be found in appendix 6.

Backward Pass

The backward pass is performed by using predict and update steps similar to a

method known as the Kalman smoother. By conditioning on future states, I am

including 5 additional observations of the state vector βββiait , one for each curve

coefficient. After each update, I sample the current state from the conditional

mean and variances generated through that update, which are also passed on to the

future update steps. For a thorough description of what this pass entails, refer to

appendix A.

3.3.3 Remaining Sampling: Gibbs Sampler

FFBS covers the sampling of the states or individual age-specific curve coefficients.

A traditional Gibbs Sampler is used to obtain samples of the remaining unknown

49

parameters. Here is the ordering from which samples are iteratively taken from the

posterior, slotting the FFBS procedure into the appropriate spot:

1. p(Ziaitj|βββ·a·t ,Siaitj,Xiaitj) for each i = 1, . . . ,m, t = 1, . . . , Ti and j = 1, . . . , nit,

2. p(βββiait|αi, τ 2τ 2τ 2,σ2σ2σ2, φ,Z,X) for each i = 1, . . . ,m and t = 1, . . . , Ti,

3. p(αi|βββi, τ 2τ 2τ 2,σ2σ2σ2, φ,Z,X) for each i = 1, . . . ,m,

4. p(τ 2τ 2τ 2|βββ,ααα,σ2σ2σ2, φ,Z,X),

5. p(φφφa|βββ·a,αααa,σ2σ2σ2) for each a ∈ Age, where βββ·a refers to age-specific curve coef-

ficients played at age a and αααa is the set of initial state curve coefficients for

fielders whose first season was at age a,

6. p(σ2σ2σ2|βββ,ααα,φφφ).

I start by sampling the latent variables for each player and season, Ziait , in the

same manner in the previous models:

p(Ziaj|βββia,S,X) ∝ exp

(−1

2(Ziaj −Xiaj · βββia)2

)· (I(Siaj = 1, Ziaj ≥ 0) + I(Siaj = 0, Ziaj ≤ 0)) ,

where I sample Ziaj ∼ Normal(Xiaj · βββia, 1) until either the condition (Siaj =

1, Ziaj ≥ 0) or (Siaj = 0, Ziaj ≤ 0) is met.

Next, I employ the FFBS previously detailed in section 3.3.1 to sample our

age-specific curve coefficients for each observed season, βββiait . Also, as previously

50

mentioned, ααα and τ 2τ 2τ 2 are semi-estimated; this is done during steps 3 and 4 and

coincides with the end of the FFBS for step 2. For simplicity sake, I use the MLE

values produced during the FFBS process to estimate each ααα and τ 2τ 2τ 21.

Since each coefficient in the diagonal matrix φφφa is set to be the same (i.e.

φak = φa ∀ k), sampling φφφa in step 5 is the equivalent to sampling φa. This

setup encourages sharing across each curve coefficient when sampling the autore-

gressive term for each age, which allows for a more comprehensive interpretation

of the φa parameter as a cost/discount due to age on a player’s previous season’s

performance. The conditional posterior distribution for φa is proportional to:

φa|βββ·a,αααa,σ2σ2σ2 ∼ Normal

(∑4k=0Eak∑4k=0 Fak

,1∑4

k=0 Fak

),

where:

Eak =

∑i∈Aa

βi(a−1)kβiak +∑

i∈Baαikβiak

σ2k

,

Fak =

∑i∈Aa

β2i(a−1)k +

∑i∈Ba

α2ik

σ2k

,

Aa is the set of all players who played a non-debut season at age a and Ba is the

set of all players whose age during their debut season was a. This distinction for a

player’s first season is due to the initial state distribution.

In the final step, I sample the variance related to uiait , σ2σ2σ2, which can be inter-

preted as the season-to-season variation in a player’s age-specific curve coefficients.

The corresponding conditional posterior distribution relies on the seasonal evolu-

1This is labeled as semi-estimation because the maximum likelihood calculation is based onthe samples produced during the backward sampling in step 2.

51

tion of all the states of the age-specific curve coefficients, which for a particular

coefficient k, is sampled via C−1k , where Ck is distributed as:

Ck ∼ Gamma

(T − 1

2,

∑a∈Age

[∑i∈Aa

(βiak − φaβi(a−1)k)2 +

∑i∈Ba

(βiak − φaαik)2]

2

).

52

Chapter 4

Results

Given the breadth and scope involved in the three new models, I choose to split

up the task of presenting and analyzing the results into three sections. In the

first section, I explain how I get to the SAFE estimates through the posterior

distributions of the corresponding season- or age-specific curve coefficients. Next,

I go through the bulk of our analysis focused on the SAFE estimates from all

the models, both old and new. I make some observations about the estimates in

general, then explore deeper into more specifics regarding the different models. The

last section contains case studies of two players of particular interest. I dissect how

the popular baseball audience views these players and juxtapose that with respect

to the SAFE findings.

53

4.1 Calculating SAFE Values

The main goal of this dissertation is to evaluate fielding ability, and the numerical

value that would help us best do that is the number of runs a particular fielder saved

or cost their team while on the field. Up to this point, the unknown parameters of

interest estimated via their conditional posterior distributions are on a scale inter-

pretable only for the probit function. The conversion required to get these estimates

into runs is a weighted integration between two parametric curves, one using the

season- or age-specific curve coefficients and the other being the “average” curve

coefficients, across either the entire field of play for flyballs/liners or all possible an-

gles from third base for grounders. Borrowing from the notation (and methods) in

[18], the resulting numerical estimate is called SAFE, or Spatial Aggregate Fielding

Evaluation.

4.1.1 Choosing the Parametric Curves

In each model, there is either a set of season-specific curve coefficients that rep-

resent the fielding ability of a player in a given season, like in the original model

or the constant-over-time model, or a group of age-specific curve coefficients that

correspond to the fielding ability of a player at a given age, like with the moving

average age model or the autoregressive age model. Fundamentally, the two sets

are no different as both refer to a single year and player combination; the major

difference lies in how the curves are indexed within their respective model. For

54

simplicity, I refer to the age-specific curve coefficients of fielder i at age a from

MAA and autoregressive age models simply as βββit, keeping to the same notation

for season-specific curve coefficients.

Our implementation of the methods described in chapters 2 and 3 produce sam-

ples from the conditional posterior distribution of βββ. After properly vetting these

sampled chains via thinning and removing burn-in1, I can calculate the full poste-

rior distribution of βββit for each player i in season t from these remaining samples.

By using the probit function, these samples of coefficients from the full posterior

distribution produce probability of success curves, one for each of the following

combinations (for each model):

1. by position,

2. by fielder,

3. by season,

4. by velocity and

5. by the remaining covariates.

The remaining covariates depend on the ball in play type. For grounders, this is over

all possible angles from 0◦ to 90◦ and whether the balls are hit to a player’s left hand

side. For flyballs and liners, the remaining covariates include all (x, y)-locations in

the field of play and if the ball hit into play was in front of the fielder.

1The amount of thinning and burn-in removed differs from model to model.

55

An average probability curve is needed as a reference to produce the run values

of interest. I experimented with several curves to use as the reference for “aver-

age player.” A natural choice when implementing the original or constant-over-time

model is the posterior means of the curves produced by the position curve coeffi-

cients, µµµ. For the MAA model, a given fielder’s age-specific curves also could be

compared to the position-age curve at that age a, µµµa. The latter suggestion is too

specific as a form of reference and the former is no different than the maximum

likelihood estimates aggregated over all players at that position, as proposed by

[18]. I choose to use the maximum likelihood estimates βββ+ to keep a consistent

means of comparison across all of the proposed models. With it, I calculate the

necessary probability curves, which are over the same combinations as mentioned

in the above list, less (2) fielder and (3) season.

4.1.2 Weighted Integration

When comparing a player’s fielding performance in a given season to the average

player at the position, I am interested in the differences between the corresponding

probability curves across each covariate combination and each velocity v. The

calculation for a fielder i in a season t, once again, depends on the ball in play type.

When performed on grounders, this calculation is done using [pit(θ, v) − p+(θ, v)]

at each angle θ, while for flyballs and liners, it is [pit(x, y, v) − p+(x, y, v)] at each

(x, y)-location on the field, where pit is the probability function calculated using

56

curve coefficients βββit.

These differences provide the basis for evaluating the fielding success of an in-

dividual player in a given season versus the average player’s performance, but it

does not provide the overall numerical evaluation that I want (i.e. SAFE). I must

incorporate these differences into a weighted integration for each ball in play type

to produce the desired run estimates, which are detailed in [18]. The integration

is done over all (x, y)-locations or angles θ and velocities v2, and is broken up into

two integrals: SAFEait for flyballs and liners and SAFEb

it for grounders.

SAFEait =

∫f(x, y, v) · r(x, y, v) · s(x, y, v)

·[pit(x, y, v)− p+(x, y, v)]dxdydv,

SAFEbit =

∫f(θ, v) · r(θ, v) · s(θ, v) · [pit(x, y, v)− p+(x, y, v)]dθdv,

where f is a kernel density estimate of the frequency of balls in play at that location

(flyballs/liners) or angle (grounder), r is the run consequence of that ball in play

and s is the shared responsibility by the fielders at that position on that ball in

play.

These added functions are the weights that convert the differences into runs

above/below the average fielder at that position, and each has a very specific pur-

pose. I include f to weigh success (or failure) at locations or angles with a higher

concentration of balls in play (as per the average fielder) in contrast to locations

or angles with a lower concentration of balls in play. Since all nine players on the

2There are only three available in the dataset provided by BIS.

57

field at any given moment have a chance to affect the outcome of a ball that lands

in play, fielders at each position share responsibility of fielding any given ball in

play. At locations and angles where this is especially true, the fielder’s ability is

downplayed, as it could be expected that a fellow teammate could also be respon-

sible for allowing hits. When I produce the overall numerical evaluation, it is most

interpretable in terms of runs saved or cost. I accomplish this by placing a run

value on balls in play at each location/angle and each velocity, which is determined

by the average run consequence over all observations. This run consequence is the

result of multiplying the ball in play outcome distribution by the corresponding

linear weights3.

The integration for a particular player in a given season and ball in play type is

interpreted as the number of runs saved (or allowed) above (or below) the average

player at that position for one ball in play4. To properly project this to an evaluation

that reflects an entire season of play, I scale it by the average number of balls in

play of that type seen in a given season by the 15 most “used”5 fielders. Finally, I

sum the projected SAFE values according to player i’s position in that season:

SAFEoutit = SAFEfly

it · nfly,posit + SAFElnrit · nlnr,posit ,

SAFEinfit = SAFEgnd

it · ngnd,posit + SAFElnrit · nlnr,posit ,

3Linear weights are the collection of run consequences for each ball in play outcome. I chooseto use the linear weights estimated by Tom Tango in [31].

4By one ball in play, I literally mean one ball in play distributed over all the different locationsat a rate given by the empirical frequency observed in the entire dataset.

5“Used” refers to the number of attempts by a fielder.

58

where fly is flyballs, gnd is grounders, lnr is liners, posi is the position of fielder

i, out refers to outfield positions and inf refers to infield positions. Each of these

numerical summaries are the final SAFE estimates I used to evaluate the fielding

ability of baseball players.

4.2 Analyzing SAFE Estimates

4.2.1 General Analysis

Tables displaying all of the SAFE estimates are not included in this dissertation;

instead, the spreadsheets for each position and each model can be found at the SAFE

website6. There are general aspects about SAFE estimates that are important to

note. I start by presenting seven histograms, one for each position, of all posterior

means for the age-specific SAFE estimates from the MAA model in figure 4.1. I

choose the moving average age model given its effectiveness, which will be explained

in more detail in chapter 4.3.2.

In terms of runs created, it is apparent that batting skill is a much more crucial

element to a position player ’s repertoire (i.e. non-pitcher’s skill set) than their

fielding ability. Some elite batters are estimated to consistently deliver seasons of

60+ runs added above average [31], while SAFE places the best fielders around

10 to 12 runs saved during their peak years. This is further substantiated by the

6The direct URL for the SAFE website is http://www-stat.wharton.upenn.edu/ stjensen/research/safe.html.

59

1b

Runs

Den

sity

0.00.10.20.30.40.50.60.7

−15 −10 −5 0 5 10

2b

Runs

Den

sity

0.000.020.040.060.080.100.120.14

−15 −10 −5 0 5 10

3b

Runs

Den

sity

0.000.050.100.150.200.25

−15 −10 −5 0 5 10

ss

RunsD

ensi

ty0.000.020.040.060.080.10

−15 −10 −5 0 5 10

lf

Runs

Den

sity

0.00

0.05

0.10

0.15

−15 −10 −5 0 5 10

cf

Runs

Den

sity

0.000.020.040.060.080.100.120.14

−15 −10 −5 0 5 10

rf

Runs

Den

sity

0.00

0.05

0.10

0.15

−15 −10 −5 0 5 10

Figure 4.1: Seven histograms, one for each position, of the posterior means taken fromthe MAA model’s SAFE estimates on the age-specific curve coefficients.

60

left skew in nearly every histogram; very poor fielders survive to play enough at

each position because their other skills, specifically batting, greatly outweigh their

shortcomings in fielding.

Also, the differences in magnitude between the SAFE estimates for each po-

sition are stark. The most influential position is the shortstop, with players at

that position costing their team as much as approximately 17 runs and others sav-

ing their squad around 14 runs. The remaining infielder positions vary in degree

of run contribution, where second basemen are at the top of those with various

players spanning ±10 runs and first basemen having the least effect with no one

player costing/saving their team more than 4 runs. Outfielders’ SAFE estimates

suggest that they have a lesser effect on defense when compared to infielders as the

best/worst at each position contribution around ±8 runs.

To summarize, the magnitude of effectiveness by each fielding position is as

follows, from largest to smallest effects: shortstop, second baseman, center fielder,

left fielder, right fielder, third baseman and first baseman. This ordering conflicts

with common knowledge, specifically a theory laid out by Bill James called the

Defensive Spectrum. The Defensive Spectrum is a ranking of fielding positions,

ordered from easiest position to field to the hardest [17]. Table 4.1 is the current

Defensive Spectrum as it stands.

I expect that the difficulty of fielding a position is directly related to that posi-

tion’s impact. Consider the case where a position requires little skill but has a large

61

Table 4.1: Bill James’ Defensive Spectrum.

Rank Position

Easiest 1. DH2. 1B3. LF4. RF5. 3B6. CF7. 2B8. SS9. C

Hardest 10. P

impact. Teams would adjust to this inefficiency by placing more skilled fielders at

that position to gain a strategic advantage and, thus, making the position require

more skill to play. Disregarding the ends of the defensive spectrum (i.e. DH, C and

P), the ordering from the SAFE estimates mimics the hardest half of the spectrum,

but disagrees with the easiest half. The biggest difference is in the placement of

the left fielder. As seen in table 4.1, a left fielders’ impact is nearly identical to

that of a right fielder. This is not surprising, and it does not predicate a change

in the ranking, as one element not captured by SAFE is the ability to throw out

runners after a catch in the outfield; throwing out runners attempting to advance

on a flyball out is more difficult to do from right field than left field, given their

location with respect to the bases. The more shocking results is that left fielders

have a larger influence on the game than third basemen. This might be due to the

noise factor surrounding the position. Third basemen have much less time to react

to liners hit in their vicinity, as they play closer to the point of contact than any

62

other position (less the pitcher), making their starting position a crucial element.

Our lack of positioning data (i.e. the positioning of each fielder before a ball is

hit into play) may be attributing to the lower-than-expected contribution by third

basemen.

4.2.2 Model Differences and Specific Players

I graphically display the SAFE estimates calculated using the samples produced

by the various models to gain a better sense of model differences. The vertical

lines in this section’s figures span the lower and upper bounds of the 95% posterior

intervals for the SAFE estimates corresponding to that fielder/season and specific

model. Located on those lines in the same color are solid circles; these are the

posterior means of the afore mentioned SAFE estimates.

In these figures, I choose to present the SAFE estimates from each model for only

seven fielders, one for each position. This decision is not due to lack of candidates, as

the total number of unique fielders for which SAFE estimates exist at each position

varies between 50 to 80 players. Instead, the motivation behind the graphics is to

get a better general sense of how the outlined models differ, and how each model’s

interpretation relays different information on a fielder’s ability and career trajectory.

General information describing the “vitals” of each of the seven players chosen can

be found in table 4.2.

1This measurement is taken as of the last observed season, 2008.

63

−15

−10

−5

05

10

Infielders: Combined (Grounders + Liners)

SA

FE

run

s ● ● ●● ●

● ●

● ● ●

●

●

●

●

● ● ●

●●

●●

●

Albert Pujols Dan Uggla Derek Jeter Adrian Beltre

Seasonal

05

10

Outfielders: Combined (Flyballs + Liners)

SA

FE

run

s

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

● ●

●

Adam Dunn Andruw Jones Vladimir Guerrero

Seasonal

Figure 4.2: SAFE results for a group of fielders under the original model.

64

Table 4.2: The vitals of the seven fielders chosen for demonstration purposes.

Name Position Throws Weight1 Height1 DOB

Albert Pujols 1B R 230 6’ 3” 1980-01-16Dan Uggla 2B R 205 5’ 11” 1980-03-11Adrian Beltre 3B R 220 5’ 11” 1979-04-07Derek Jeter SS R 195 6’ 3” 1974-06-26Adam Dunn LF R 285 6’ 6” 1979-11-09Andruw Jones CF R 230 6’ 1” 1977-04-23Vladimir Guerrero RF R 235 6’ 3” 1976-02-09

The first figure, figure 4.2, are the SAFE results gathered from the original SAFE

model for the seven chosen fielders. The dark green lines shown here are simply

the posterior means and 95% intervals for SAFE estimates taken from the season-

specific curve coefficients. Nearly all of the 95% posterior intervals for the SAFE

estimates from the original model cover the zero-line (i.e. the intervals contain

zero), which suggests that there is still copious noise drowning out player’s true

fielding ability. Once again, I notice the huge differences in magnitude between the

SAFE estimates for each position, with Derek Jeter’s (negative) ability having a far

greater impact than Albert Pujol’s (positive) effect.

In figure 4.3, I present the SAFE estimates from the constant-over-time model.

The solid black lines in figure 4.3 represent a fielder’s SAFE calculations based

on their player-specific curve coefficients, while the gray vertical lines correspond

to their season-specific curve coefficients. The constant-over-time model has sig-

nificantly less shrinkage to the average player compared to the original model, as

there are fewer 95% posterior intervals containing zero. The player-specific intervals,

65

−20

−10

010

Infielders: Combined (Grounders + Liners)

SA

FE

run

s

● ●

●

●● ● ●

● ●●

●

●

● ●

●

●

●

●

●

●

●

●

●●

●

●

Albert Pujols Dan Uggla Derek Jeter Adrian Beltre

OverallSeasonal

−5

05

1015

Outfielders: Combined (Flyballs + Liners)

SA

FE

run

s

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●●

●

Adam Dunn Andruw Jones Vladimir Guerrero

OverallSeasonal

Figure 4.3: SAFE results for a group of fielders under the constant-over-time model.

66

which can be thought of as a fielder’s ability over all seasons observed in the dataset,

are much wider than their season-specific counterparts. I expect that how a player

has performed at their position over a span of multiple seasons would be much nois-

ier than how they fared during a specific season. There are a few exceptions to this

where a fielder’s player-specific intervals are the same width as their season-specific

intervals, including Albert Pujols and Adrian Beltre. This phenomenon is likely a

combination of these two players consistent output at those positions, as well as the

consistent performance of other players at those positions.

Figure 4.4 displays the SAFE estimates taken from the moving average age

model’s samples. The blue vertical lines are the 95% posterior intervals for the av-

erage player of that age at that position, which are found via the position-age curve

coefficients for that age. The corresponding 95% posterior intervals for the age-

specific curve coefficients for each fielder are shown with the red vertical line. The

numbered time line below each player’s name refers to the age at which that player

was during each of the seven observed seasons. The reported posterior means and

95% intervals of the MAA model do not differ much from the constant-over-time

model for these seven fielders. As expected, there is a degree of shrinkage pulling

the individual age-specific SAFE estimates towards their respective position-age

estimates; however, instances do exist where the age-specific estimates are substan-

tially different. I see this happening for both poor (e.g. Derek Jeter at age 30 and

31) and exceptional (e.g. Albert Pujols, Adrian Beltre and Andruw Jones during

67

−20

−10

010

20

Infielders: Combined (Grounders + Liners)

SA

FE

run

s

● ● ● ● ●● ●

| | | | | | |23 24 25 26 27 28 29

● ● ●● ●

●

●●

●●

●

●

| | | | | | |23 24 25 26 27 28 29

●

●

●

●

●

●●

●

●

●

| | | | | | |28 29 30 31 32 33 34

● ●

●

●

●

●

●

● ● ● ●

● ●●

| | | | | | |23 24 25 26 27 28 29

●●

●

●●

●

●

Albert Pujols Dan Uggla Derek Jeter Adrian Beltre

AgeAveragePlayer

−5

05

1015

Outfielders: Combined (Flyballs + Liners)

SA

FE

run

s

●

●

● ●

●

●●

| | | | | | |22 23 24 25 26 27 28

●

●

●

●

● ●

●

●

●● ●

●

●●

| | | | | | |25 26 27 28 29 30 31

●

●

●

●

●●

● ●

●

●

●

●

●

| | | | | | |27 28 29 30 31 32 33

●●

●

●

●

●

●

Adam Dunn Andruw Jones Vladimir Guerrero

AgeAveragePlayer

Figure 4.4: SAFE results for a group of fielders under the moving average age model.

68

their mid-20’s) seasonal performances.

Finally, the seven fielders’ posterior means and 95% intervals for the SAFE es-

timates from the autoregressive age model are located in figure 4.5. The purple

squares are the semi-estimated means from each player’s initial state curve coef-

ficients. The yellow dots and lines are the SAFE estimates corresponding to the

age-specific curve coefficients for that player. The autoregressive age model agrees

less with the other proposed extensions, and its estimates trend more from year-to-

year, a natural result of the autoregressive feature. The fielders most effected are

the outfielders that display gradual and incremental improvement/degradation as

they age.

4.2.3 Age Analysis

One of the goals set out in this dissertation is to gain a better understanding of how

aging affects players’ fielding ability at the different positions. There are several

issues surrounding this question. As is true with all physical activity, humans reach

their peak fitness relatively early in life. Since fielding is a athletic task, it is not

unreasonable to expect that after a certain age, fielders no longer have the same

agility and quickness that make them as able to field their position. However,

experience is necessarily learned over time; during or after their athletic peak, a

player could still overcompensate for their dwindling dexterity with greater skill.

To further explore the age effect, I take a look at the two proposed model extensions

69

−20

−10

010

Infielders: Combined (Grounders + Liners)

SA

FE

run

s

● ●●

● ●

| | | | | | |24 25 26 27 28 29 30

●

●

●

| | | | | | |26 27 28 29 30 31 32

●●

●

●

●

●

●

| | | | | | |28 29 30 31 32 33 34

●●

●

●●

●

●

| | | | | | |23 24 25 26 27 28 29

Albert Pujols Dan Uggla Derek Jeter Adrian Beltre

AgesInitialSeasonal

−10

−5

05

1015

20

Outfielders: Combined (Flyballs + Liners)

SA

FE

run

s

●

●●

●

●●

●

| | | | | | |22 23 24 25 26 27 28

●

●

●

●

● ●

| | | | | | |25 26 27 28 29 30 31

●

●

●

●

●●

●

| | | | | | |28 29 30 31 32 33 34

Adam Dunn Andruw Jones Vladimir Guerrero

AgesInitialSeasonal

Figure 4.5: SAFE results for a group of fielders under the autoregressive age model.

70

that link ability to age: the moving average age model and the autoregressive model.

An advantage of the MAA model’s structure is the ability to easily to capture

average trends in ability across age via the position-age elements. I visualize the

posterior means and 95% intervals of the SAFE estimates for the position-age curve

coefficients in figure 4.6. The position-age SAFE estimates translate to the runs

saved/sacrificed by a team who utilize an average player of that specific age for that

position. If there were indeed strong effects related to age, then I would expect there

to be patterns in the posterior means and 95% intervals of the SAFE estimates; that

is not the case. Some movement is seen in the posterior means that could possibly

attributed to age, but their corresponding posterior 95% intervals are too large to

make any definitive statements. The closest that these estimates get to a significant

peak is with right fielders at ages 24 and 25. Given that their are approximately 140

posterior 95% intervals tested, it is natural to expect at least two intervals would

not cover zero simply by chance, if not more.

The autoregressive age model features an autoregressive term, φa, that repre-

sents the discount/cost for a fielder at age a on the previous year’s curve coefficients.

Intuitively, a φa greater than 1 corresponds to an age at which fielders improve on

their previous year, whereas fielding ability is declining during an age with a φa

less than 1. Our findings from analyzing the posterior means and 95% intervals for

φ across all position and BIP type combinations are inconclusive. I illustrate why

in figure 4.7, where the posterior values of φ are shown for the two position and

71

20 25 30 35

−50

5

Position−Age SAFE Intervals: lf

Age

Run

s

25 30 35 40

−15

−50

510

Position−Age SAFE Intervals: cf

Age

Run

s

20 25 30 35

−50

5

Position−Age SAFE Intervals: rf

Age

Run

s

25 30 35−8

−6−4

−20

2

Position−Age SAFE Intervals: 1b

Age

Run

s

25 30 35 40

−20

−10

05

10

Position−Age SAFE Intervals: 2b

Age

Run

s

25 30 35

−10

−50

5

Position−Age SAFE Intervals: 3b

Age

Run

s

25 30 35

−20

010

2030

Position−Age SAFE Intervals: ss

Age

Run

s

Figure 4.6: The posterior means and 95% intervals of the SAFE estimates for eachposition-age curve across each position.

72

BIP type pairs with the largest observations (i.e. balls in play), flyballs by center

fielders and grounders by shortstops. In neither plot do I see any sort of pattern

between age and the autoregressive terms. Any signal coming from a player’s aging

process is heavily outweighed by the amount of noise involved in estimating these

unknowns.

There are a few confounding factors that likely influenced the unclear results

into how age effects fielding ability. Survival bias is certainly present, as team

management often force their players to move from their position when they begin

to show signs of decline. Also, some players who post poor fielding numbers at

any age might still be playing that position because of their exceptional batting

and/or base running ability. The biggest confounder effecting the study of ageing is

the lack of data. Without more observations from contiguous seasons, a few great

fielders who are still playing at an advance age, like Omar Vizquel, may serve as

highly influential outliers skewing any results over such a small span of time.

4.3 Case Studies

I deliver case studies of two well-known fielders, Derek Jeter and Vernon Wells, to

complete the analysis of results. They are chosen not for their popularity or “star”

status, but because of the analysis by many baseball writers and authorities that

both are among the top defenders at their position. In fact, these two players have

combined for a total of eight Rawlings Gold Glove awards. These awards are given

73

Figure 4.7: The posterior means and 95% intervals of each φa, for all ages a observed ofthat position and BIP type combination, for (1) flyballs by center fieldersand (2) grounders by shortstops.

74

annually to the MLB players judged by the managers and coaches from around

the league to have exhibited the most superb individual fielding performances at

each fielding position in both the National League (NL) and the American League

(AL). There is a clear disagreement between these popular opinions and the SAFE

estimates produced by our models.

4.3.1 Derek Jeter

Derek Jeter debuted as the New York Yankees shortstop in 1995 and has remained

at that position for the Yankees to present day. He is well known for his offensive

prowess, especially as a player whose career was spent at a challenging position like

shortstop, but his fielding ability has always been debatable. He has won a total

of five Gold Glove awards as a shortstop, so people around baseball certainly think

highly of his defensive gifts. On the other hand, there have been countless articles

by experts in sabermetrics, research coming from the SABR (Society of American

Baseball Research) community, that question his fielding ability [8] and the fielding

awards he has won [6]. The SAFE estimates, as seen in table 4.3, seem to agree

with the sabermetric experts. In fact, the SAFE estimates surrounding the three

years Derek Jeter won Gold Gloves available in our dataset are abysmmal across all

models, representing some of the worst posterior means observed at shortstop. A

few of his earlier seasons suggest that he was only slightly below average, but the

remaining years’ SAFE estimates put him among some of the worst fielders in the

75

game.

Part of the problem in evaluating Derek Jeter objectively by eye is that he

is extremely athletic. When watching him field, he is regularly seen pulling off

amazingly acrobatic plays, similar to the one shown in figure 4.8. In that play,

Derek Jeter is attempting to throw out a base runner going from first base to second

base after a grounder is hit to his right side. To have a chance on the play, he is

forced to throw against his momentum, which is carrying him towards third base,

away from his throw’s intended target. To the viewer, the play looks fantastic as

he converts an out on what is seemingly an impossible attempt. The contention by

many baseball experts who claim Derek Jeter’s fielding metrics are justified is that

an average shortstop would be able to get that out more easily, either by reacting

sooner or being more quick, which would allow them to get the out without the

need of a spectacular-looking play. The end result being that Derek Jeter has more

trouble making plays on balls that the average shortstop finds easier to field, while

simply not having any chance of getting the out on tougher ball in play. Thus, it

appears as if Derek Jeter is fielding his position well, when he might not actually

be doing so.

2“GG?” refers to whether or not that player won the Rawling Gold Glove award for theirfielding skill at their position in a given season.

3This is the max number of balls in play observed of that player during that season over allmodels, as there are changes in sample size for the same players across different models. Thevariation is due to the necessary removal of some outliers and aberrations in the data for modelstability purposes.

76

Figure 4.8: An acrobatic throw by Derek Jeter attempting to get an out on a play thatmight have been made more easily by someone who had better range orreaction time.

Table 4.3: Posterior means and 95% intervals of the SAFE estimates for Derek Jeter atshortstop by all models.

Original ConstantAge / Season GG?2 Sample3 Mean Interval Mean Interval

28 / 2002 899 0.1 (-4.9, 6.0) -0.5 (-7.5, 6.3)29 / 2003 329 0.4 (-1.1, 2.2) -0.3 (-1.3, 1.0)30 / 2004 X 879 -3.6 (-9.7, 2.0) -12.3 (-12.3, 17.8)31 / 2005 X 1035 -9.9 (-14.7, -3.5) -17.0 (-22.6, -11.6)32 / 2006 X 949 0.0 (-4.7, 4.5) -4.6 (-9.5, 1.0)33 / 2007 1032 -5.0 (-9.2, -0.3) -7.1 (-12.3, -1.3)34 / 2008 898 0.5 (-4.6, 5.1) -2.3 (-7.8, 3.6)

MAA Autoregress

28 / 2002 899 -0.2 (-6.6, 5.7) -1.1 (-8.3, 6.5)29 / 2003 329 -0.7 (-1.7, 0.4) -0.2 (-1.5, 1.2)30 / 2004 X 879 -12.6 (-18.3, -6.6) -10.3 (-15.7, -4.7)31 / 2005 X 1035 -17.0 (-22.6, -11.4) -16.6 (-22.3, -11.7)32 / 2006 X 949 -5.2 (-11.9, 0.5) -4.6 (-10.2, 1.0)33 / 2007 1032 -7.3 (-12.9, -1.6) -7.7 (-13.4, -2.0)34 / 2008 898 -2.5 (-8.4, 3.0) -3.0 (-8.4, 3.2)

77

4.3.2 Vernon Wells

As a center fielder, Vernon Wells is decidedly not below-average at fielding, but

the results from the SAFE estimates suggest that he is likely not above-average.

From 2002 to 2006, Vernon Wells was the starting center fielder for the Toronto

Blue Jays. During the latter half of that span, he won three consecutive Gold

Glove awards for being one of the three best outfielders in the American League

in each season. Toronto signed him at the end of the 2006 season to a seven-year

contract worth $126 million. After the 2008 season, many baseball experts and

writers soured on the deal, calling it a bad signing [7]. Most of the complaints

were based on a retrospective analysis of Vernon Wells’ offensive production, not

his fielding ability, which was quoted by some as “very good” [27]. However, while

his fielding performance in 2005 was notable with all models agreeing that he saved

his team about 2 or more runs, it is fairly unanimous across all SAFE estimates

that his fielding ability was certainly not better, and in many cases poorer, than

the average center fielder, as seen in table 4.4.

Vernon Wells is well-known for successfully fielding attempts on balls caught at

or over the outfield walls, as seen in figure 4.9. On first glance, an observer might

believe that his exceptional ability to catch these balls in play outweighs his poor

play on other balls in play, especially considering the high run consequence of failing

at these attempts. This is highly unlikely considering the number of these chances

in any given season represent less than 1% of the total balls in play. A possible

78

Figure 4.9: Vernon Wells catches a line drive right at the wall for an out. If he had notsuccessfully caught the ball, the play would have had a high run consequence,either becoming a double, a triple or even a home run. However, theseattempts are few and far between.

79

Table 4.4: Posterior means and 95% intervals of the SAFE estimates for Vernon Wellsat center field by all models.

Original ConstantAge / Season GG?2 Sample3 Mean Interval Mean Interval

23 / 2002 641 -2.3 (-6.2, 1.2) -4.9 (-9.9, 0.5)24 / 2003 683 0.2 (-2.9, 3.6) 0.4 (-4.4, 5.3)25 / 2004 X 567 0.6 (-2.5, 4.7) 0.9 (-4.4, 6.6)26 / 2005 X 614 2.3 (-0.8, 5.4) 2.1 (0.4, 3.9)27 / 2006 X 608 -1.6 (-4.9, 1.7) -2.5 (-7.0, 2.9)28 / 2007 573 -2.3 (-5.8, 1.3) -4.6 (-10.1, 0.5)

MAA Autoregress

23 / 2002 641 -8.8 (-12.7, -4.6) -3.2 (-9.1, 1.9)24 / 2003 683 -3.3 (-7.2, 1.3) 2.0 (-2.2, 6.2)25 / 2004 X 567 0.8 (-5.0, 6.1) -1.0 (-5.1, 5.9)26 / 2005 X 614 3.2 (-2.5, 8.1) 4.3 (-1.4, 8.7)27 / 2006 X 608 -2.5 (-8.2, 3.7) -1.7 (-6.8, 3.6)28 / 2007 573 -5.7 (-11.8, -0.1) -3.4 (-8.2, 1.6)

explanation in the disagreement between the SAFE estimates and popular opinion

is that Vernon Wells’ inability to successfully field balls in high traffic areas (i.e.

high frequency of balls in play over all observations) overwhelms his spectacular

catches on balls in play at low traffic areas.

80

Chapter 5

Model Validation

In this chapter, I aim to perform proper validation to all the presented models,

both internal and external. For internal validation, I propose a new measure of a

model’s ability to predict future outcomes known as predicted deviation. Graphical

summaries based on values calculated via predicted deviation provide a method

to gauge which method best predicts hold out data: our model extensions, the

original model in [18] or maximum likelihood estimates. The external validation is

performed doing comparisons to existing fielding metrics. Given the differences in

units amongst some of these measures, this exercise is simply an attempt to contrast

the predictive nature of each and determine whether the SAFE approach is stacking

up to the current compendium.

81

5.1 Predicted Deviations

In order to validate each model internally, I develop an intuitive metric for gauging

the prediction accuracy of each model in our setting; I call it predicted deviation.

Simply put, the predicted deviation for a fielder is the average deviation in the

predicted chance of fielding an out and the actual outcome per BIP (i.e. out or no

out). There are several such values for each fielder and ball in play type combination,

and their calculations differ between models.

5.1.1 Methods of Calculation

I begin by splitting up the dataset into training and holdout parts. Since the BIS

data spans from the 2002 to the 2008 season, it is natural to split up that dataset

by making the training portion range from 2002 to 2007 and having the holdout set

be the 2008 season. By forcing the holdout season to be the bookend of our entire

data set, it lends itself to our time series approach1. An outline of the steps involved

in calculating the predicted deviations for any of the proposed SAFE models, given

a ball in play type and position combination, is as follows:

1. Sample the curve coefficients for that player during the holdout season, βββit,

from the appropriate distribution given the training data.

2. For each BIP j by fielder i, calculate the probability of a success on that BIP,

1I only calculate the predicted deviation of a fielder if that fielder (1) met a minimum require-ment for BIPs seen across this training data and (2) had an opportunity to make a play on atleast one BIP in the holdout season.

82

pij, using the sampled βββti.

3. Take a weighted sum over all BIPs by fielder i of the differences between the

predicted probabilities of a success, pij’s, and the actual outcomes, yij, to get

the predicted deviation for fielder i at iteration t, Dti .

4. Repeat steps 1 to 3 for each qualifying fielder at that position and each iter-

ation.

5. Calculate the average predicted deviation for each fielder Di by averaging over

each iteration.

In step 1, I mention that the sampling of βββit

is done using the “appropriate” dis-

tribution. I say “appropriate” because the explicit sampling distribution depends

on which model is being evaluated; this is where differences exist in the predicted

deviation calculation between models. However, the rest of the process (i.e. steps

2 to 5) remains the same throughout all models.

Original Model

Consider the original model iteration. Suppose I want to calculate the predicted

deviation of player i, a center fielder, for liners2. The first step in the calculation is to

sample the vector of curve coefficients, βββit, from the following sampling distribution

2The specifics about the fielder and ball in play type are completely arbitrary here; there isno difference in the steps for calculating predicted deviation when considering different types offielders or batted ball types.

83

given the training data:

βββit|y’02:’07,Θ

t ∼ Normal(βββti,’07, (σ2σ2σ2)t),

where Θt is the set of sample iterations t of all unknown parameters taken from

running the model on the training dataset.

For each ball in play j attempted by player i in the holdout dataset, I calculate

the probability of that BIP being fielded using the above sampled curve:

pij = Φ(yij|Xij, βββit),

where pij is the probability of ball-in-play j (in the holdout set) being caught by

player i, yij is the actual ball-in-play outcome for player i and ball-in-play j and

Xij are the ball-in-play covariates for player i and ball-in-play j.

With this vector of probabilities, the predicted deviation for player i and itera-

tion t, or Dti , follows as:

Dti =

∑i

(∑j |yij − pij|

ni

),

where ni is the total number of balls in play fielder i had a chance to field in the

holdout season.

I repeat the previous steps 100 times. This gives a vector of predicted deviations

Di = (D1i , D

2i , . . . , D

100i ). For purposes of accurately gauging the prediction error

for a given player and BIP type, I take the average of that vector, resulting in the

predicted deviation for player i, or simply Di.

84

Constant-over-Time Model

Instead of relying on the previous season’s season-specific curve coefficients (i.e.

2007 season-specific curve coefficients) to properly sample βββi, I sample using a

mean corresponding to that fielder’s player-specific curve coefficients, γγγti. These

coefficients, along with the sampled variation at the player-specific level in iteration

t, (τ 2τ 2τ 2)t, are used in determining the sampling distribution for βββit:

βββit|y’02:’07,Θ

t ∼ Normal(γγγti, (τ2τ 2τ 2)t).

The remaining procedure continues at step 2.

Moving Average Age Model

In the MAA model, none of the past information about the player is carried forward

when predicting their future abilities. Instead, sampling depends on the age of that

player during the holdout season. Thus, the sampling distribution used to generate

βββit

for a given iteration t uses the position-age curve coefficients:

βββit|y’02:’07,Θ

t ∼ Normal(µµµai,’08 , (σ2σ2σ2)t).

There is a caveat to this sampling approach; there are instances where for a

particular age a, µµµa is not sampled in the training set, yet a player may be of

age a during the holdout season. To correct for this, I use the position-age curve

coefficients from the first previous age, µµµa′ , where a′ < a and a′ was estimated on

the training data.

85

Autoregressive Age Model

The sampling distribution for βββit

is identical to the sampling distribution in the

original model with the addition of the autoregressive term, φφφta. For a given sampling

iteration t, I incorporate φφφa into the model by multiplying it to the lag age-specific

curve coefficients, specifically βββiai,’07 :

βββit|y’02:’07,Θ

t ∼ Normal(φφφai,’08 · βββiai,’07 , (σ2σ2σ2)t).

Exceptions present themselves in employing this sampling distribution. In cases

where a fielder’s last season was prior to the ′07 season or for whatever reason, a

fielder did not play enough in ′07 to meet the minimum ball in play threshold (e.g.

injury), I apply a similar solution to the season gaps described in section 3.3. A

problem similar to the exception in the sampling distribution for the MAA model

occurs if there are no observations of a player at age a during the training seasons,

but a fielder is at that age in the holdout season. The above sampling distribution

is not available in that case as no estimates exist for φφφa. The fix I employ is to

use the autoregressive age coefficients for the first previous age, φφφa′ , and substitute

them in for φφφa.

5.1.2 Graphical Survey

I can best communicate the results from calculating the predicted deviations of each

model through a collection of summary statistics. I highlight one such statistic in

this dissertation, which is called winning percentage. I define winning percentage

86

for model L, call it Win%L, as the proportion of players in which model L’s average

predicted deviation is the minimum of all five models, or:

Win%L =1∑m

i=1 ni,’08

m∑i=1

ni,’08 · I(

DiL

= minl∈Λ

(Dil)

),

where Λ is the collection of models considered. A sampling size adjustment is

applied to weight each “win” for a model L by the total number of balls in play by

the player related to the “win” in the holdout season.

A mosaic plot of the adjusted winning percentages for each model, including

the MLE, is shown in figure 5.1. It is clear that there is a difference in prediction

performance between the models. Note that if all of the models randomly predict

the outcome of every ball in play, then I would expect the Win%L for any model

L to be 20%. The MAA model has the highest Win% in almost all flyball and

grounder combinations. The BIP type and position pair with the most samples,

grounders by shortstops, is dominated by the moving average age model, having a

Win% of over 75%. The winning model for the remaining grounder fits is the MAA

model, except for grounders by first basemen, in which the MAA model’s Win%

is a distant second to the autoregressive age model. Even for the flyball BIP type

I see the moving average age model consistently performing as the best predictive

model when compared to the rest. This is not as true for liners, in which the leader

in Win% is varies. Part of this can be attributed to the noisy nature of liners

(see section 2). Overall, however, the moving average model is the most dominant

with the constant-over-time and autoregressive age models rounding out the top 3,

87

Figure 5.1: A mosaic plot displaying the winning percentage via predicted deviations foreach ball in play type and position combination with sample size adjustment.

respectively.

Another summary statistic of predicted deviations used to highlight predictive

ability is called average ranking. Similar to Win%, I calculate average ranking

for a particular ball in play type and position by comparing all of the models

over each sample iteration and player. The models are ordered by their predicted

deviation from smallest to largest for each sample and assigned a numerical rank

based on where they are in that ordering (e.g. 1 for smallest, 2 for second smallest,

etc.). These ranks are averaged across all iterations and adjusted for sample size.

Unlike the sample size adjustment described for Win%, I correct for sample size

88

considerations in the holdout season by simply adding observations corresponding

to the number of balls in play attempted by that fielder on that ball in play type

during the holdout season.

I present histograms of the average rankings for each model in figure 5.2. There

is very little to take from this graphic. The average rankings do support that the

MAA model is great for prediction, but it also seems to suggest that the MLE is not

atrocious, either. Given the ambiguity in the diagnostics of this summary statistic,

I disregard the average ranking findings, instead focusing on the Win% statistic.

5.2 Comparing to Existing Metrics

I choose two metrics of comparison for the external validation of the SAFE model:

UZR (Ultimate Zone Rating) and DRS (Defensive Runs Saved). The data on these

metrics is provided by FanGraphs [3]. UZR, as described in chapter 1, quantifies

the amount of runs a player saved (or cost) their team through fielding [21]. The

measurement I choose to use is prorated over 150 games, a typical amount of games

by a starting fielder. DRS is a spawn of the previously mentioned Plus-Minus

system [8]. DRS, or total defensive runs saved, indicates how many runs a player

saved/hurt his team on the field compared to the average fielder at this position.

To get the runs estimates, weighted pluses and minuses are summed. For example,

a given fielder with a success on a play made that is made on average 20% of the

time by other fielders at that position receives +0.80. On the other hand, a failure

89

MLE

Rank%

Den

sity

0.0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5

Original

Rank%

Den

sity

0.0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5

Constant−Over−Time

Rank%

Den

sity

0.0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5

Moving Average Age

Rank%

Den

sity

0.0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5

Autoregressive Age

Rank%

Den

sity

0.0

0.2

0.4

0.6

0.8

1.0

1 2 3 4 5

Figure 5.2: These are the histograms of the average rankings of each model with a redline representing the median value.

90

Table 5.1: Table containing correlations (Corr) and standard deviations (SD) be-tween/for each metric. Note that the units on the standard deviations are inruns.

Corr SAFE DRS UZRSAFE - 0.55 0.64DRS 0.55 - 0.78UZR 0.64 0.78 -SD 3.26 10.03 10.63

on that same play would result in a -0.20.

Correlations between each of the three metrics (SAFE, DRS and UZR), as well

as the standard deviations on the estimates of each model, are presented in table

5.1. It is apparent from this table that SAFE is positively related to the other

fielding measures, but not to the same extent that DRS and UZR are correlated

to themselves, suggesting that SAFE is explaining something about fielding ability

that the other measures are not capturing. The other significant point is the dra-

matic difference in standard deviations between SAFE and the other metrics. DSR

and UZR have inflated standard deviations of around 10 runs, while the standard

deviation on SAFE estimates is a little higher than 3 runs. I would expect that

SAFE estimates vary less than other metrics, given that SAFE evaluates talent

under “normal” circumstances and UZR/DRS capture what actually happened.

However, a 7 run discrepancy in standard deviation implies there could be some

systematic overestimating by the existing fielding metrics.

91

Chapter 6

Conclusions and Future Work

In this dissertation, I present three new model extensions on top of the original

SAFE model. I explore and validate all models in an exhaustive way, and those

diagnostics point to the newly created moving average age model as the model

with the best predictive power. Using the newer versions of ball in play data [9], I

found that the current data related to defensive positioning is either too variable or

infrequent to have a substantial effect on the results as a whole. Finally, I attempt

to understand how aging may affect fielders at the different positions, but a lack of

data and existing confounders made the task too difficult with the data available.

There is plenty of future work and improvements regarding SAFE and fielding

evaluation. New models incorporating aspects of the most successful proposed

model extensions may boost predictions. One such example is a variable selection

model. In this new model, during a given season, a fielder’s curve coefficients are

92

sampled from either a player-specific mean or from a mean shared by players of

that same age, depending on an unknown mixture parameter; features of both the

constant-over-time model (i.e. player-specific mean) and moving average age model

(i.e. position-age mean) are melded together using a common variable selection

parameter. The most important improvement on the SAFE model is getting more

accurate and descriptive data, and the upcoming addition of Field F/X to every

major league baseball park may be the answer. According to [4], the new Field

F/X system that has been employed at only one park thus far (AT&T Park in San

Francisco) is able to track the movement of every player on field, along with the

trajectory, speed and direction of each ball in play. An example of the kind of data

that Field F/X is said to provide is in figure 6.1. The majority of the noise regarding

the samples of the full posterior distributions for each SAFE model is related to

three issues:

1. improperly estimating the initial positioning of each player,

2. categorical nature of the velocity data and

3. no information about the travel time (i.e. hang time) of each ball in play.

With Field F/X, information would be available to surpass all three obstacles, which

should lead to much more accurate fielding evaluation when using SAFE estimates.

93

Figure 6.1: A graphical representation created for Popular Science magazine [4] of thetype of data that is automatically tracked by the Field F/X system.

94

Appendix A

Kalman Filter

The Kalman filter is a recursive method for producing estimates of states by looping

over two steps from the initial state to the most recent one. These two steps are

broken down into a prediction step and an update step. In the prediction step, the

value of the next state is predicted based on all previous observations and (estimates

of) states. That prediction is then updated in the update step using the observation

from that state.

To apply this method, I need to initialize at time t = 1. Given the assumptions

made, this is straightforward:

βββi(ait|ait) = αααi, Si(ait|ait) = T,

where T is a 5× 5 diagonal matrix of τ 2τ 2τ 2. Our prediction step simply becomes

βββi(ait|ai(t−1)) = φφφaitβββi(ai(t−1)|ai(t−1)),

Si(ait|ai(t−1)) = φφφaitSi(ai(t−1)|ai(t−1))φφφ′ait,

95

for t = 2, . . . , T . After each prediction step, following update is made to these

values:

βββi(ait|ait) = βββi(ait|ai(t−1)) +Kiait(Ziait −Xiaitβββi(ait|ai(t−1))),

Si(ait|ait) = (I −KiaitXiait)Si(ait|ai(t−1)),

where

Kiait = Si(ait|ai(t−1))X′iait

(XiaitSi(ait|ai((t−1))X′iait

+ I).

96

Appendix B

Backward Sampling

The technique employed for this calculation and sampling is a minor variation on

the Kalman smoother, where the update steps differ by being run on each element

in the state vector, instead of time. This same method was described in [5]. The el-

ements of our state vector in this model setting are the individual curve coefficients.

Since there is no dependence between these coefficients, this greatly simplifies all

computations. For each coefficient k = 1, . . . , 5, let

βββi(ait|ait, k) = E(βββiait|Zti, βiai(t+1)1, . . . , βiai(t+1)5),

Si(ait|ait, k) = V ar(βββiait|Zti, βiai(t+1)1, . . . , βiai(t+1)5).

Also, assume that, for k = 0, βββi(ait|ait, 0) = βββi(ait|ait), the conditional mean esti-

mated by the original filter step, and Si(ait|ait, 0) = Si(ait|ait). Finally, let

εi(ait, k) = βiai(t+1)k − φφφ′ai(t+1)k

βββi(ait|ait, k − 1),

Ri(ait, k) = φφφ′aitkSi(ait|ait, k − 1)φφφaitk + Σiai(t+1),

97

where φφφaitk is the kth row of φφφait and Σiait is a diagonal matrix of uiait elements.

Then, the altered observation update step of the Kalman smoother is as follows:

βββi(ait|ait, k) = βββi(ait|ait, k − 1) + Si(ait|ait, k − 1)φφφaitkεi(ait, k)/Ri(ait, k),

Si(ait|ait, k) = (I − φφφaitkφφφ′aitkSi(ait|ait, k − 1)/Ri(ait, k))Si(ait|ait, k − 1).

By executing these 5 update steps backwards through time (that is, t = Ti −

1, . . . , 1), I obtain estimates for each t of

βββi(ait|ait, 5) = E(βββiait|Zti,βββiai(t+1)

), Si(ait|ait, 5) = V ar(βββiait|Zti,βββiai(t+1)

),

which serve as the conditional means and variances from which I will sample each

state, or age-specific curve coefficients.

98

Bibliography

[1] J.H. Albert and Siddhartha Chib. Bayesian analysis of binary and poly-

chotomous response data. Journal of the American Statistical Association,

88(422):669–679, June 1993.

[2] Brian Anderson and John Moore. Optimal Filtering. Dover Publications, 2005.

[3] David Appelman and Mitchel Lichtman. FanGraphs, 2011.

[4] Bjorn Carey. New Camera System Takes the Guesswork out of Baseball Stats,

February 2010.

[5] C. K. Carter and R. Kohn. On Gibbs sampling for state space models.

Biometrika, 81(3):541–553, 1994.

[6] James Click. Did Derek Jeter Deserve the Gold Glove? 2007.

[7] Jerry Crasnick. Wells running dry in an awful big way, 2009.

[8] John Dewan. The Fielding Bible. ACTA Sports, 2006.

[9] John Dewan. Baseball Info Solutions, 2009.

99

[10] Chris Dutton and Peter Bendix. Batters and BABIP, 2008.

[11] David Gassko. Batted balls and park effects, 2008.

[12] Alan E Gelfand and Penelope Vounatsou. Proper multivariate conditional

autoregressive models for spatial data analysis. Biostatistics (Oxford, England),

4(1):11–25, January 2003.

[13] Andrew Gelman, John Carlin, Hal Stern, and Donald Rubin. Bayesian Data

Analysis. Chapman and Hall/CRC, 2nd editio edition, 2003.

[14] J Geweke. Bayesian estimation of state-space models using the Metropol-

isHastings algorithm within Gibbs sampling. Computational Statistics & Data

Analysis, 37(2):151–170, August 2001.

[15] J.K. Hakes and R.D. Sauer. Effects and Valuation of Fielding in Major League

Baseball: A Play-by-Play Analysis. Technical report, Clemson University

mimeo, 2005.

[16] D.M.A. Hickson. Spatial analyses of basketball shot charts: An application to

Michael Jordans 2001-2002 NBA season. PhD thesis, 2002.

[17] Bill James. The New Bill James Historical Baseball Abstract. Free Press,

revised edition, 2001.

100

[18] Shane T. Jensen, Kenneth E. Shirley, and Abraham J. Wyner. Bayesball: A

Bayesian hierarchical model for evaluating fielding in major league baseball.

The Annals of Applied Statistics, 3(2):491–520, June 2009.

[19] David E Kalist and Stephen J Spurr. Baseball Errors. Journal of Quantitative

Analysis in Sports, 2(4), October 2006.

[20] David Kaplan. Univariate and Multivariate Autoregressive Time Series Models

of Offensive Baseball Performance: 1901-2005. Journal of Quantitative Analy-

sis in Sports, 4(3), July 2008.

[21] Mitchel Lichtman. Ultimate Zone Rating (UZR), 2003.

[22] Mitchel Lichtman. The FanGraphs UZR Primer, 2010.

[23] Brad Null. Modeling Baseball Player Ability with a Nested Dirichlet Distribu-

tion. Journal of Quantitative Analysis in Sports, 5(2), May 2009.

[24] James Piette, Sathyanarayan Anand, and Kai Zhang. Scoring and Shooting

Abilities of NBA Players. Journal of Quantitative Analysis in Sports, 6(1),

January 2010.

[25] David Pinto. Probabilistic Model of Range, 2003.

[26] Bill Plaschke. Hit or Error? A Question of Judgement.

[27] Joe Posnanski. Understanding Ricciardi?, 2009.

101

[28] Brian J Reich, James S Hodges, Bradley P Carlin, and Adam M Reich. A

Spatial Analysis of Basketball Shot Chart Data. The American Statistician,

60(1):3–12, February 2006.

[29] R. Roesser. A discrete state-space model for linear image processing. IEEE

Transactions on Automatic Control, 20(1):1–10, February 1975.

[30] David Studeman. Groundballs, flyballs, and line drives., 2004.

[31] Tom Tango, Mitchel Lichtman, Andrew Dolphin, and Pete Palmer. The Book:

Playing the Percentages in Baseball. Potomac Books Inc., 2007.

[32] C.K. Wikle. Hierarchical Bayesian Models for Predicting the Spread of Eco-

logical Processes. Ecology, 84(6):1382–1394, 2003.

[33] C.K. Wikle, L.M. Berliner, and N. Cressie. Hierarchical Bayesian space-time

models. Environmental and Ecological Statistics, 5(2):117–154, 1998.

[34] Jeff Zimmerman. UZR’s (and Most Other Defensive Metrics) Limitation in

Year to Year Analysis, 2009.

102