Journal of Quantitative Analysis in Sportsbobconnolly.web.unc.edu/files/2017/09/Journal-of...w o udl have only an 80% c h a n c e of qualifying. A t the same time, players of m u c

Volume 7, Issue 4 2011 Article 7

Journal of Quantitative Analysis inSports

Going for the Green: A Simulation Study ofQualifying Success Probabilities in

Professional Golf

Robert A. Connolly, University of North Carolina at ChapelHill

Richard J. Rendleman Jr., Tuck School of Business atDartmouth and University of North Carolina at Chapel Hill

Recommended Citation:Connolly, Robert A. and Rendleman, Richard J. Jr. (2011) "Going for the Green: A SimulationStudy of Qualifying Success Probabilities in Professional Golf," Journal of QuantitativeAnalysis in Sports: Vol. 7: Iss. 4, Article 7.

©2011 American Statistical Association. All rights reserved.Brought to you by | University of North Carolina at Chapel Hill

AuthenticatedDownload Date | 8/7/17 5:45 PM

Going for the Green: A Simulation Study ofQualifying Success Probabilities in

Professional GolfRobert A. Connolly and Richard J. Rendleman Jr.

AbstractEach year, over 1,300 golfers attempt to qualify for the PGA TOUR through Q-School.

Using simulation, we estimate the probabilities that Q-School correctly identifies high-skill golfers.We show that players with skill equivalent to the very best on the PGA TOUR would havehigh probabilities of qualifying, but others, equal in skill to many active PGA TOUR members,would have low odds of qualifying. We explore the impact of variations in Q-school structureon qualifying probabilities for players with different skill levels, but most of the variations thatimprove tournament efficiency are largely impractical.

KEYWORDS: golf, PGA TOUR, simulation, tournaments, Q-School, type I and II errors, skill,luck, smoothing spline

Author Notes: Robert A. Connolly is Associate Professor, Kenan-Flagler Business School,University of North Carolina, Chapel Hill. Richard J. Rendleman, Jr. is Visiting Professor, TuckSchool of Business at Dartmouth and Professor Emeritus, Kenan-Flagler Business School,University of North Carolina, Chapel Hill. The authors wish to acknowledge with gratitude thePGA TOUR for providing the ShotLink data used in connection with this study, Max Grender-Jones, who provided invaluable assistance in the collection of an earlier version of the data,Steve Carman of the PGA TOUR for assistance with the collection and cleaning of Q-Schooldata, Kelli Anderson of the PGA TOUR, who helped to clarify many of the rules regarding Q-School competition and PGA TOUR eligibility requirements, Steve Powell, Ben Rosen, and HarveyWagner, who provided helpful comments, and Kin Lo, Steven Nahmias and other participants inthe “Hole in One: Using Operations Research to Analyze Golf II” session of the 2009 INFORMSconference. Please address comments to Robert Connolly (email: [email protected];phone: (919) 962-0053) or to Richard J. Rendleman, Jr. (e-mail: [email protected];phone: (919) 962-3188).

Brought to you by | University of North Carolina at Chapel HillAuthenticated

Download Date | 8/7/17 5:45 PM

1. Introduction

Each fall the PGA TOUR conducts the PGA TOUR Qualifying Tournament– better known as Q-School – to identify a group of players to participate onthe TOUR the following season who would otherwise be ineligible. Q-Schoolconsists of four stages, Pre-Qualifying, which started in 2006, and Stages 1-3.Except for the final stage, each stage of Q-School is broken into a number ofcompetitions conducted at different tournament sites, with a portion of thefield at each site continuing on to the next stage. Those who do not advancefrom early-stage competition are eliminated and have no opportunity to re-enter the competition until the following year. Although many professionally-untested golfers must begin Q-School in Pre-Qualifying, a number of golfersenter Q-School in its later stages based on their success in prior Q-Schoolcompetitions and/or positions on the PGA TOUR and affiliated NationwideTour money lists.

The problem that the PGA TOUR is attempting to solve through itsannual Q-School competition is not unlike the problems of talent selectionand employee performance evaluation routinely faced by business organiza-tions and academic institutions, among others. But unlike the promotionaland talent selection practices of most organizations, the rules of Q-School areunambiguous. Both those who are being evaluated and those doing the eval-uation know the “rules of the game.” Once a player is evaluated, the TOURcannot change the outcome based on subjective or political criteria. We areunaware of any set of rules for identifying the relative rankings of high skillindividuals for entry into or promotion within an organization that is cleanerand more transparent than Q-School. The question we study here is whetherQ-School itself is effective in identifying the most highly skilled among itsentrants.

Although it is unlikely that “the next Tiger Woods” will enter Q-Schoolon an annual basis, one would expect a regular flow of new TOUR-capableplayers to enter the professional golfing talent pool; otherwise the TOURcould not sustain itself over the long run.1 As part of the “regular flow,”we estimate that players with scoring characteristics comparable to a num-ber of well-known professional golfers would have substantially less than a50% chance of qualifying for the PGA TOUR through Q-School if they had

1We compiled lists of players who participated in at least 15 PGA TOUR events in years2000-2010 to proxy for PGA TOUR membership in each year. For years t = 2000 to 2009,we computed the percentage of players who were TOUR members in year t but not in yeart + 1 as a measure of annual membership turnover. The turnover measure ranged from18.8% to 29.9%, with a mean of 25.5% (or 38 to 64 players, with a mean of 51).

1

Connolly and Rendleman: Going for the Green

Published by De Gruyter, 2011



to start at the beginning in Pre-Qualifying. Moreover, our results show thata player starting in Pre-Qualifying with the scoring characteristics of SteveStricker, ranked eighth in the Official World Golf Rankings as of this writing,would have only an 80% chance of qualifying. At the same time, players ofmuch lower skill can make it through Q-School and qualify for the TOUR bysimply having a run of favorable random variation in their scoring.

The effectiveness of Q-School in selecting the most highly skilled playersto participate on the PGA TOUR depends in large part on its ability to resolvethe inherent random variation in scoring to reveal, or efficiently estimate, trueplayer skill. Ideally, Q-School would select the most highly skilled among itsparticipants, and luck would play a minimum role in determining those whoqualify for the TOUR.

In Connolly and Rendleman (2008, 2009 and 2011), we show that golferswho win PGA TOUR events generally experience substantial favorable ran-dom variation in scoring (‘good luck’) relative to their levels of skill. Weestimate that it takes approximately 10 strokes of cumulative abnormally fa-vorable performance over four rounds of play to win a typical PGA TOURevent. Moreover, almost all who finish among the lowest scoring 25 to 30experience some degree of favorable abnormal performance. If ‘luck’ plays arole in regular PGA TOUR competition, then clearly it should also play arole in determining those among Q-School participants who are successful andunsuccessful qualifying for the TOUR.

Although no practical qualifying mechanism that the PGA TOURcould devise could eliminate the luck factor entirely, it is instructive to knowjust how well the current structure works. In this study we employ simulationto help answer this question. Our simulations suggest that among the mosthighly skilled players in a given year’s Q-School competition, many will notsucceed in qualifying for the TOUR due to unfavorable random variation inscoring (Type I errors), and many players of substantially lower skill will qual-ify (Type II errors) due to favorable random variation. Of course, the tradeoffbetween Type I and II errors is what drives the design of all organizationalpromotion activities. Unlike Q-School, however, quantifying these potentialerrors in other organizations is particularly difficult, especially when the rulesof promotion are less clear and subject to manipulation in many forms. More-over, it may be easy to identify Type II error rates in a business organization,that is, identify those who were promoted who turned out to be low skill, butidentifying Type I error rates associated with high skill individuals who wereincorrectly identified as low skill and forced to leave an organization can bemuch more problematic. In our study, however, we are able to estimate both

2

Journal of Quantitative Analysis in Sports, Vol. 7 [2011], Iss. 4, Art. 7

DOI: 10.2202/1559-0410.1308



types of error rates associated with the process used by the PGA TOUR toselect high-skill golfers to join the PGA TOUR.

From a statistical standpoint, determining the relative skill rankings ofprofessional golfers could be thought of as a problem in rank statistics. How-ever, the complexity of the statistical problem suggests that an analytical eval-uation of Q-School efficiency using rank order statistics would be intractable.Q-School is conducted over four stages with new participants entering the com-petition in later stages based on prior performance on the PGA TOUR, on theaffiliated Nationwide Tour and in previous Q-School competitions. Also, allbut the final stage is conducted over multiple tournament sites, with those ad-vancing to the next stage being determined based on players’ relative rankingsat the sites where they participate. Taken together, this structure suggests asimulation-based approach may be best suited to evaluate the selection effi-ciency of Q-School.

Although we conduct simulations of some obvious alternative tourna-ment structures, such as having all players in a given Q-School stage competehead-to-head on the same course and extending the number of rounds of com-petition, from a practical standpoint, some of these structures are not feasible.For example, it would be physically impossible to fit 900 or more Stage 1 par-ticipants on the same golf course at the same time. However, these alternativestructures do provide insight into the extent to which limited rounds of com-petition over numerous tournament sites contribute to Type I and II errors.In no sense do we attempt to find optimal alternatives to Q-School.

The remainder of our paper is organized as follows. In the next sectionwe review the literature of tournaments relevant to our study of Q-School selec-tion efficiency. In Section 3, we describe the data used in the study. In Section4, we describe the Q-School qualification process. We provide evidence on thesuccess rates of golfers who qualify for the PGA TOUR through Q-School andthe Nationwide Tour in Section 5 and find that those who have qualified forthe PGA TOUR via the Nationwide Tour have been more successful.

In Section 6 we summarize observed qualifying characteristics of Q-School competition and variation in scoring, which, in turn, serve as the basisfor the calibration of our Q-School simulation model. We discuss how wemodel individual golfer skill in Section 7. In Sections 8 and 9, we explain thegeneral structure of our Q-school simulation model our method of simulatingQ-School scoring. In Section 10, we discuss the simulation results, and inSection 11, compare the potential performance of hypothetical alternative Q-School structures. A final section presents a summary of the paper and ourconclusions.

3





2. Related Work

Two approaches to the study of tournament structure embody very differentinterpretations of the stochastic element in measured tournament performance.According to tournament theory, both the level of effort competitors chooseto exert and the degree of risk they choose to bear reflect the performancecharacteristics of others in the competition, the tournament’s competitive de-sign and the structure of tournament prizes.2 Tournament theory predictsthat deviations from one’s “best efforts” level of performance reflect both arandom component and an endogenous component, which, in turn, reflects thedesign characteristics of the tournament. Much of the focus in the theoreticalliterature of tournaments is on designing ‘tournaments,’ such as corporate andacademic promotion rules, that will maximize the overall level of output of anorganization over the long run.

The second line of tournament study, which draws on the statisticaldecision theory literature (for example, Gibbons, Olkin and Sobel (1977),Narayana (1979) and David (1988)), concerns the ability of a given tournamentstructure to identify the most highly skilled among its competitors. In this lineof work, characterized by the recent papers of Ryvkin and Ortmann (2008) andRyvkin (2010), players’ efforts and risk-taking levels are not choice variables;tournament participants are assumed to play their best. As such, deviations ofperformance from a competitor’s “best efforts” output are due to exogenousrandom factors alone. As noted by Ryvkin (2010, p. 668), “this assump-tion is realistic for an important class of selection situations involving humansubjects, such as the final stages of recruitment tournaments and other envi-ronments where stakes are high, and significant prior investment has alreadybeen made by competitors.” Examples cited by Ryvkin include formation ofOlympic teams, innovation races, elections, and high level sports tournaments.And, most certainly, Q-School would also fit this characterization. In manycases, those participating in Q-School have spent years investing both timeand money in skill development. For those involved, the stakes couldn’t behigher – being chosen as one of 25 participants from a pool of over 1,300 com-petitors to participate on the following year’s PGA TOUR.3 Moreover, unlike

2The seminal paper on tournaments is Lazear and Rosen (1981). Useful surveys includeMcLaughlin (1988) and Prendergast (1999). Frick (2003) provides a summary of how tour-nament theory, as developed by Lazear and Rosen (1981) and Rosen (1986), applies in sportssettings. Hvide and Kristiansen (2003) study contests where participants choose how muchrisk to take rather than the amount of effort to exert.

3For example, at the end of the 2010 PGA TOUR season, average and median prizemoney for players who competed in 15 or more tournaments, a rough estimate of PGA

4


DOI: 10.2202/1559-0410.1308



promotion tournaments in commonly-studied theoretical models applied to thepromotion process in firms, many in Q-School have no baseline income fromplaying professional golf to fall back on in the event they do not qualify forthe TOUR. For a golfer, earning the right to play on the tour is like findingthe proverbial pot of gold at the end of the rainbow.

In an important set of papers, Ryvkin and Ortmann (2009) and Ryvkin(2010) study the selection efficiency of multiple tournament formats, contests,knockout tournaments, and round-robin tournaments. They explore the in-teraction of tournament structures, the distribution of player skills, and thelevel of noise in designing efficient tournaments. In their work, the tourna-ment selection efficiency criterion is generally focused on identifying the ‘best’player, not a group of the most highly skilled, as in Q-School. Based on theiranalytical and simulation results involving contests, binary tournaments, andround-robin tournaments, several predictions emerge. Of particular relevanceto our work is the following finding: when players have similar skill levelsand the random component in scoring is substantial, the optimal length ofthe tournament is much longer than in a setting with low noise and diverseabilities. Put in statistical terms, the sample size has to be very large beforethe distribution centers on the ‘right’ outcome.4,5

3. Data

Our study draws primarily on two distinctly different datasets. The first,which we refer to as the Q-School sample, is derived from 18-hole Q-School

TOUR membership, was $1,227,376 and $837,416, respectively, with 86 players within thisgroup earning over $1 million. It should be noted that a large portion of a PGA TOURplayer’s income can come from endorsements which, in turn, reflect success in tournamentcompetition. Sirak (2008) provides an analysis of 2007 on-course and estimated off-courseincome for 50 professional golfers, 42 of which were active PGA TOUR members. Among the42, the estimated off-course income for 27 golfers exceed their tournament prize winnings,and within this group, we compute a median ratio of off-course to on-course income of 1.72.

4We show that the error rates observed for the Q-School qualifying scheme could bereduced substantially by increasing the number of 18-hole rounds in the competition. How-ever, the number of rounds required to bring a reasonable level of efficiency to the qualifyingscheme is too high to be implemented in practice.

5Other studies on selection efficiency focus on algorithms for organizing and conductingparticular tournament formats (knockout vs. contest vs. round-robin, to name a few) thatwill yield desired outcomes. For example, McGarry and Schutz (1997) use Monte Carlomethods to study the ability of knockout and round-robin tournament formats to put theith player in the ith rank at the end of the tournament. Other examples of research onoptimal tournament structure in this vein include Appleton (1995) and Clarke, Norman,and Stride (2009).

5





scoring data obtained from the PGA TOUR for each stage of Q-School over the1999-2010 period. The raw data required extensive processing to ensure thateach player was identified consistently throughout the sample and assigned theproper PGA TOUR identification number, when applicable. Except for ouranalysis of Official World Golf rankings in section 5, our analysis of Q-Schoolfocuses exclusively on the portion of the data that starts in 2006, the year Q-School introduced Pre-Qualifying.6 Hence, we generally refer to the Q-Schooldata in terms of the 2006-2010 period.

Our second dataset, which we refer to as the PGA TOUR sample, isderived from the PGA TOUR’s ShotLink database and covers the 2003-2010PGA TOUR seasons. It includes 18-hole scores for every player in every strokeplay event sanctioned by the PGA TOUR for years 2003-2010 for a total of151,954 scores distributed among 1,878 players. We limit the sample to playerswho recorded 10 or more 18-hole scores. The resulting sample consists of148,145 observations of 18-hole golf scores for 699 PGA TOUR players over366 stroke-play events. Most of the omitted players are not representative oftypical PGA TOUR players. For example, 711 of the omitted players recordedjust one or two 18-hole scores.7

We also obtained data on Official World Golf Rankings from the PGATOUR. We now turn to a description of Q-School structure. Understandingthis structure is critical to understanding our simulation design. Moreover,the structure should make it clear that an evaluation of Q-School selectionefficiency could not be formulated analytically.

6We eliminated eight non-representative scores of 100 or higher in the 2006-2010 portionof the data.

7Generally, these are one-time qualifiers for the U.S. Open, British Open and PGAChampionship who, otherwise, would have little opportunity to participate in PGA TOURsanctioned events.

6


DOI: 10.2202/1559-0410.1308



Figure 1: Relationship among Q-School, Nationwide Tour and PGA TOUR at the end of the 2008 season. Darkframed boxes indicate Q-School qualifying stages. Within these boxes N/M indicates that N players out of atotal of M participants qualified for the next stage of Q-School. Q-School Pre-Qualifying and stages 1 and 2consist of four 18-hole rounds. Stage 3 consists of six 18-hole rounds. Positions for the PGA and NationwideTours are based on 2008 Official Money winnings. Positions for Q-School are based on total score.

7



Brought to you by | University of N

orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

4. Q-School Structure

Figure 1 illustrates the various stages of Q-School competition as structuredin 2008, typical of the 2006-2010 Q-School sample period, how Q-School feedsthe PGA and Nationwide Tours and how these two tours, in turn, feed sub-sequent Q-School competitions. In 2008, 434 players participated in four 18-hole rounds of Pre-Qualifying competition at six different tournament sites,and 238 of the 434 advanced to Stage 1. In Stage 1, the 238 were joined forfour 18-hole rounds of play across 12 tournament sites by 713 golfers whoseeligibility to begin in Stage 1 was based primarily on their success in previousQ-School competitions or their placing 71st or worse on the 2008 NationwideTour Money List. Of the 951 Stage 1 participants, 305 advanced to Stage 2.They, in turn, were joined for four rounds of play across six tournament sitesby 157 players whose eligibility to begin Q-School competition in Stage 2 wasbased on performance on the 2008 PGA and Nationwide Tours and variousmiscellaneous exemptions. Of the 462 participants in Stage 2, 126 advancedto the final stage, a competition at a single tournament site consisting of six18-hole rounds, three each on two different courses. These players were joinedby those in positions 126-150 on the 2008 PGA TOUR Money List and inpositions 26-40 on the Nationwide Tour Money List along with a few otherplayers who were given miscellaneous exemptions, bringing the total numberof Stage 3 competitors to 163. The lowest scoring 25 players and ties (28 to-tal) earned 2009 PGA TOUR cards. Those in positions 29 to 79 in the finalstage (the next closest to 50) earned fully-exempt playing privileges on the2009 Nationwide Tour.8 All remaining players earned partially-exempt statuson the Nationwide Tour, with initial preference given to those with the bestscores in the final stage of Q-School competition.9 As it turns out, only six ofthe 434 participants in Pre-Qualifying made it to Stage 3, and among the six,the two best finishers, Joseph Sykora and Martin Piller, earned fully-exemptstatus on the Nationwide Tour.

Figure 1 shows that in addition to the (approximately) 50 annual PGATOUR qualifiers from Q-School and the Nationwide Tour, the top 125 playerson the PGA TOUR Money List get to return to the PGA TOUR the followingyear. The players in positions 126-150 on the Official PGA TOUR Money List

8The term, “fully exempt player,” as it is applied to the PGA TOUR or NationwideTour, is a player who has earned the right to play on the tour without restriction, otherthan being subject to tournament priority lists based on past performance.

9A “partially exempt player” on the PGA TOUR or Nationwide Tour is one who isallowed to fill out a tournament field after all fully-exempt players have been given theopportunity to participate.

8


DOI: 10.2202/1559-0410.1308



advance immediately to Stage 3 of 2008 Q-School. Even if they had not quali-fied for the 2009 PGA TOUR through 2008 Q-School competition, they wouldhave received partially-exempt status on the 2009 PGA TOUR. PGA TOURplayers in positions 151 and higher on the 2008 Official PGA TOUR MoneyList advance immediately to Stage 2 of Q-School, while the first 50 players inthis group also receive partially-exempt status on the following year’s Nation-wide Tour. It should be noted that no official PGA TOUR member can do anyworse than returning to Stage 2 of Q-School. This is a key characterization ofQ-School competition in structuring our simulations.

Figure 1 also shows that players in positions 26-60 on the 2008 OfficialMoney List of the Nationwide Tour were able to return to the NationwideTour in 2009 as fully-exempt members. The first 15 players in this groupadvanced immediately to Stage 3 of 2008 Q-School. Nationwide players inpositions 61-100 earned the right to return to the Nationwide Tour in 2009as partially exempt members. We leave the remaining observations about therelationships among Q-School, PGA TOUR and Nationwide Tour qualificationto the reader. We note, however, that there are other ways of making thePGA and Nationwide Tours, which are not included in Figure 1, such aswinning a PGA TOUR event during the past two years or being a non-PGATOUR member who earns as much official prize money as the 125th player onthe Official PGA TOUR Money List. Figure 1 is intended to illustrate theinterconnections among Q-School, PGA TOUR qualification and NationwideTour qualification only.

5. Do Q-School and the Nationwide Tour Iden-

tify the Best Players?

Table 1 summarizes the distribution of Official World Golf Rankings (OWGR)as of December 26, 2010 among those who qualified for the PGA TOURthrough Q-School and the Nationwide Tour between 1999 and 2008. We notethat some players qualified for the TOUR more than once over this period, andsome qualified for the TOUR in different years from both Q-School and theNationwide Tour. To avoid multiple counting, we take every player who qual-ified for the TOUR over the 1999-2008 period and classify him as qualifyingfrom Q-School or the Nationwide Tour based on his last TOUR qualificationexperience. We also note that the Official World Golf Rankings are based ona rolling two-year performance window. Therefore, the two-year gap between2008 and 2010 eliminates any World Golf Ranking points accumulated priorto our final qualifying period.

9





Table 1: Official World Golf Rankings as of 12-26-2010 for PGA TOUR Qual-ifiers from Q-School and Nationwide Tour, 1999-2008

World Golf Q-School NW TourRanking N % N %

1-100 20 7.49 16 12.60101-200 23 8.61 10 7.87201-300 20 7.49 13 10.24301-400 22 8.24 11 8.66401-500 16 5.99 14 11.02501-600 17 6.37 9 7.09601-700 9 3.37 7 5.51701-800 8 3.00 6 4.72801-900 9 3.37 1 0.79901-1000 7 2.62 6 4.72

Over 1000 116 43.45 34 26.77

Total 267 100.00 127 100.00

Of particular note is the high proportion, 43%, of Q-School graduates(those who qualify for the PGA TOUR through Q-School competition) whofell outside the top 1,000 of the World Golf Rankings. By comparison, only27% of the Nationwide Tour graduates fell outside the top 1,000. Moreover, thehigher proportion of players graduating from the Nationwide Tour who wereamong the OWGR’s top 500 (50% vs. 38%), along with the lower proportionranked worse than 1,000, suggests that since 1999, the Nationwide Tour hasdone a better job identifying the most highly-skilled players, despite the factthat Nationwide Tour participants did not perform as well in Q-School to beginwith.

These findings suggest a non-negligible ‘luck’ factor associated withmaking it onto the PGA TOUR via Q-School, implying a relatively high TypeII error rate among players identified through Q-School as highly skilled. Bycontrast, Nationwide Tour competition extends presently over 26 tournaments,and, therefore, it is less likely that relatively weak players will qualify forthe PGA TOUR through Nationwide competition. Moreover, since favorablerandom variation in scoring tends to play a substantial role in qualifying for

10


DOI: 10.2202/1559-0410.1308



the PGA TOUR through Q-School, many highly-skilled players in Stage 3who do not finish among those who qualify for the PGA TOUR but go onto the Nationwide Tour could be as skilled, if not more skilled, than many ofthe Stage 3 players who actually qualify for the TOUR. With 26 additionaltournaments to be played the following year on the Nationwide Tour, manywho qualify for the PGA TOUR through Nationwide competition are likely toend up being more successful in the long run than some who qualified for thePGA TOUR from Q-School ahead of them.

6. Properties of Q-School Competition and Scor-

ing

In this section we describe general properties of Q-School qualifying successrates and scoring which, in turn, serve as the basis for calibrating our simu-lations of Q-School competition. Although we draw some general inferencesfrom the data, none are supported by tests of statistical significance, since theinferences drawn are not the primary focus of this study.

6.1. Qualifying Success Rates

Table 2 shows how the players who participated in Q-School from 2006 to2010 advanced through its various stages. All values within the table, as wellas Tables 3 - 6, reflect the properties of Q-School over the 2006-2010 period.Over this period, 2,114 players entered Q-School in the Pre-Qualifying stage.10

Only 26 of the 2,114 reached Stage 3, but none within this group qualified forthe PGA TOUR, and very few who entered Q-School in Stage 1 earned theirTOUR cards (43 of 3,448).

Table 3, derived from the entries in Table 2, shows qualifying successrates by stage of play, conditional upon Q-School entry stage. By design,52.6% of the participants in Pre-Qualifying advanced successfully to Stage 1.Stage 1 participants, in turn, consist of two groups, those who started in Pre-Qualifying and those who started in Stage 1. Among the Stage 1 participantswho started in Pre-Qualifying, 20.7% (230 of 1,112) advanced to Stage 2, while38.2% of the Stage 1 participants who started Q-School in Stage 1 advancedto the next stage. We note that one of our main objectives in structuring

10If a player entered Q-School in Pre-Qualifying in each year, 2006-2010, he is counted asfive players.

11





simulations of Q-School competition is to produce qualifying success rates insimulated data that closely approximate those shown in Table 3.

Table 2: Aggregate Performance of Q-School Entrants, 2006-2010

Final Stage ReachedInitial Pre- Stage 3Stage Qual Stage 1 Stage 2 NW (PE) NW (FE) PGA TOUR Total

Pre-Qual 1,002 882 204 15 11 0 2,114Stage 1 0 2,131 992 186 96 43 3,448Stage 2 0 0 488 132 85 61 766Stage 3 0 0 0 83 68 44 195Total 1,002 3,013 1,684 416 260 148 6,523

NW (PE) = Qualified for partially-exempt status on Nationwide Tour, NW (FE) =Qualified for fully-exempt status on Nationwide Tour, PGA TOUR = Qualified for fully-exempt status on PGA TOUR. The 1,002 players who participated in Pre-Qualifying butdid not advance to Stage 1 is net of 7 players who actually qualified for Stage 1 but didnot continue. The 3,013 players who did not advance from Stage 1 is net of 11 playerswho actually qualified for Stage 2 but did not continue.

Table 3: Q-School Qualifying Success Rates by Playing Stage/Entry Stage,2006-2010

Playing StageEntry Pre-Stage Qual Stage 1 Stage 2 Stage 3

Pre-Qual 0.526 0.207 0.113 0.000Stage 1 0.382 0.247 0.132Stage 2 0.363 0.219Stage 3 0.226

Values in the table represent proportions of par-ticipants playing in a given stage who qualifiedfor the next stage. The proportions shown ad-vancing from Stage 3 are those who earned fully-exempt status on the PGA TOUR.

12


DOI: 10.2202/1559-0410.1308



The entries in Table 3 suggest that the level of skill among those whoparticipated in a given stage of Q-School is an increasing function of entrystage. For example, only 11.3% of the Stage 2 participants who began Q-School in Pre-Qualifying advanced to Stage 3. By contrast, 24.7% of theStage 2 participants who started in Stage 1 advanced to Stage 3, and 36.3%of those who entered Q-School in Stage 2 advanced to the final stage.

6.2. Mean Scoring of Q-School Participants

Table 4 summarizes the difference between mean 18-hole scores, conditionalupon Q-School entry stage, and mean scores in Stage 3 for players who enteredQ-School in Stage 3. Without exception, in each playing stage mean scores arethe lowest among players who entered Q-School in the later stages. Consistentwith the qualifying success rates summarized in Table 3, this suggests that, onaverage, players who are allowed to bypass early-stage competition and enterQ-School in the later stages are more highly skilled than the players againstwhom they are competing who entered Q-School in earlier stages.

Table 4: Normalized Mean Scores by Playing Stage/Entry Stage, 2006-2010



Mean scores in each entry stage/playing stagecategory represent the difference between thecategory mean score and the mean score in Stage3 for players who entered Q-School in Stage 3.

Also, players who advance in Q-School competition tend to record lowerscores on average in advanced-stage competition than all players in their sameentry group scored in earlier competition. This suggests that players whoadvance in Q-School are more highly skilled as a group than those in theirsame entry group who do not advance. However, differences in average scores

13





from one stage to the next for a given entry group could also reflect the relativedifficulty of courses rather than differences in skill.

6.3. Mean Scores Adjusted for Relative Round and CourseDifficulty

We have insufficient player connections within the Q-School data to estimatesimultaneously both player skill and the relative difficultly of each round-course combination (interaction) as in Connolly and Rendleman (2008). Forexample, in 2008, we cannot estimate the inherent difficulty of round 1 ofPre-Qualifying at Kinderlou Forest Country Club (Course Rating of 76.6)relative to round 3 of Stage 1 played at Martin Downs (Course Rating of 73),while simultaneously estimating the skill of the players participating on thosecourses.11 If Pre-Qualifying entrants who reached Stage 1 recorded lower scoreson average than all players who competed in Pre-Qualifying, is it because thosewho advanced to Stage 1 are better players, or is it because a portion of Stage1 competition was conducted on a relatively easy course? We simply cannottell.

To control for relative course difficulty and playing conditions, we com-pute the difference between each 18-hole score and the mean score for thesame round on the course on which the round is played. We refer to suchscores as “round-course-adjusted” scores. For each playing stage, we then(“playing stage-”) normalize the mean of the round-course-adjusted scores bysubtracting the mean associated with players entering Q-School in the sameplaying stage. Table 5 summarizes the playing stage-normalized, round-course-adjusted mean scores for years 2006-2010 combined. Again, we observe that ineach playing stage, mean (round-course-adjusted) scores are the lowest amongplayers who entered Q-School in the later stages.

6.4. Variation in Round-Course-Adjusted Scores

Table 6 summarizes standard deviations of round-course-adjusted scores foreach playing stage/ entry stage category. The standard deviations of round-course-adjusted scores within each playing stage category (column) exhibit

11According to the USGA (www.usga.org), a USGA Course Rating is the evaluation of theplaying difficulty of a course for scratch golfers under normal course and weather conditions.It is expressed as the number of strokes taken to one decimal place (72.5), and is based onyardage and other obstacles to the extent that they affect the scoring difficulty of the scratchgolfer. A male scratch golfer is a player who can play to a Course Handicap of zero on anyand all rated golf courses.

14


DOI: 10.2202/1559-0410.1308



little variation conditional upon entry stage. Generally, standard deviationsfor each entry group tend to be lower as players within the group advance fromone stage of Q-School to the next.12

Table 5: Playing-Stage-Normalized Mean Round-Course-Adjusted Scores byPlaying Stage/Entry Stage, 2006-2010



Mean scores by playing stage represent the differ-ence between the entry stage/playing stage meanscore and the mean score of players who enteredQ-School in the same stage.

Table 6: Standard Deviations of Round-Course-Adjusted Scores by PlayingStage/Entry Stage, 2006-2010



12As noted in Connolly and Rendleman (2008, p. 82), and as we also show in Section9.2, standard deviations of round-course-adjusted scores tend to be lower for players withlower mean scores (i.e., higher skill players play with more consistency). Therefore, if themore highly-skilled players within an entry group are advancing through Q-School, standarddeviations of round-course-adjusted scores for a given entry group should be lower in thelater stages of Q-School competition.

15





The standard deviation of round-course-adjusted scores is much higherin Pre-Qualifying than in the other stages of play. We believe that thishigh standard deviation reflects more than just greater variation in round-to-round scoring among Pre-Qualifying participants. Instead, it appears thatPre-Qualifying consists of at least two distinctly different groups of players –those who are legitimate candidates to participate in professional golf compe-tition and those who have no realistic chance of qualifying for the TOUR.

In our simulations, we are unable to replicate the pattern of standarddeviations shown in Table 6 in the Pre-Qualifying stage without modeling Pre-Qualifiers as a bimodal group, with 72% of the Pre-Qualifiers being modeledas ‘legitimate’ players and 28% consisting of a second group whose scores, onaverage, are 4.77 strokes higher. We note, however, that the qualifying successrates that we estimate in our simulations are virtually the same, whether weassume that 28% of the players in Pre-Qualifying are among those in thesecond group or whether there is no second group at all.

7. Estimating Skill of Individual Q-School Par-

ticipants

The primary focus of this study is to estimate the probabilities associatedwith individual players qualifying for the PGA TOUR through Q-School asa function of their skill levels. In 2010, 1,370 players recorded 8,309 scoresin Q-School competition, an average of 6.06 scores per player. Unfortunately,this represents an insufficient number of observations to estimate individualskill with any degree of precision. Moreover, due to the nature of Q-Schoolcompetition, there is inherent bias in individual player sample mean scoresrelative to true mean scores. Although many who advance in the competitionare likely to advance for being more skilled compared with those who do notadvance, others may advance for having experienced more favorable randomvariation in their scoring. Similarly, those who do not advance are likely tobe less skilled, but also are likely to have experienced unfavorable randomscoring variation. Therefore, in a given stage, as a group, the sample meanscores of players who advance to the next stage of competition should be biaseddownward relative to true mean scores, and the sample mean scores of thosewho do not advance should be biased upward.

In addition, we would like to estimate the probabilities that players withskill comparable to successful PGA TOUR players would make it through Q-School if they had to begin Q-School competition in Pre-Qualifying. There issimply no way to draw such inferences by studying the 2006-2010 Q-School

16


DOI: 10.2202/1559-0410.1308



data in isolation without also connecting the Q-School data to scoring datacharacteristic of regular players on the TOUR.

The key to making this connection is through the regular PGA TOURplayers who participate in Q-School after unsuccessful PGA TOUR seasons.Defining a regular PGA TOUR player as one who participated in at least 15stroke play events during the PGA TOUR season immediately preceding entryto Q-School, we find that n2 ∈ {40, 33, 47, 39, 39} such players entered Stage2 of Q-School in years 2006-2010, respectively, and n3 ∈ {21, 20, 16, 17, 20},respectively, entered Stage 3.13 Using the statistical methodology of Connollyand Rendleman (2008, 2011) to estimate the scoring characteristics of regularTOUR players, including those who had to go back to Q-School, we can linkthe scoring characteristics of Q-School participants coming off the PGA TOURto those of more successful TOUR players such as Tiger Woods and PhilMickelson who do not participate in Q-School.

Using the Connolly and Rendleman (2008) model as modified in Con-nolly and Rendleman (2011), we estimate time-varying skill functions for agroup of professional golfers representative of PGA TOUR participants dur-ing the eight-year period 2003-2010. We employ the cubic spline methodologyof Wang (1998), as described in Connolly and Rendleman (2008), to estimateskill functions and autocorrelation in residual errors for players with 91 or morescores. We employ a simpler linear representation without autocorrelation, asin Connolly and Rendleman (2011), for players with 10 to 90 scores over thefull sample period.14

13We are not concerned with whether a player was an official member of the PGA TOURduring the PGA TOUR season immediately preceding his entry to Q-School but, instead,whether the statistical properties of his play were characteristic of those of regular TOURplayers who entered Q-School. Therefore, some players so identified might not have beenofficial members of the TOUR, while others who participated in less than 15 stroke playevents could have actually been official PGA TOUR members. We note, however, that thePGA TOUR does require its official members to participate in at least 15 events.

14We established the 91-score minimum in Connolly-Rendleman (2008) as a compromisebetween having a sample size sufficiently large to employ Wang’s (1998) cubic spline model(which requires 50 to 100 observations) to estimate player-specific skill functions, whilemaintaining as many established PGA TOUR players in the sample as possible. The cen-soring of a sample in this fashion will have a tendency to exclude older players who areending their careers in the early part of the sample and younger players who are beginningtheir careers near the end. If player skill tends to vary with age, such a censoring mechanismcan create a spurious relationship, where mean skill across all players in the sample appearsto be a function of time. (Berry, Reese and Larkey (1999) show that skill among PGATOUR golfers tends to improve with age up to about age 29 and decline with age startingaround age 36. Thus, ages 30-35 tend to represent peak years for professional golfers.) Toeliminate any type of age-related sample bias arising from a censored sample, we employ

17





Simultaneously, the model accounts for random variation in scoringdue to differences in relative round difficulty (round-course effects) and thepropensities of individual players to perform better or worse on certain courses(player-course effects). (To conserve space, we refer the reader directly to Con-nolly and Rendleman (2008, 2011) for a description of the statistical methodol-ogy, its relationship to previous literature on performance measurement in golfand other athletic contests, and its efficiency relative to alternative modelingstructures.) We note that the model does not take account of specific infor-mation about playing conditions (e.g., adverse weather as in Brown (2011),pin placements, morning or afternoon starting times, etc.) or, in general,the particular conditions that could make scoring for all players more or lessdifficult, when estimating random round-course effects. Nevertheless, if suchconditions combine to produce abnormally high or low scores in a given 18-hole round, the effects of these conditions should be reflected in the estimatedround-course-related random effects.15

Estimated random round-course effects range from -4.45 to 7.89 strokesper round, implying more than a 12-stroke difference between the relativedifficulty of the most difficult and easiest rounds played on the TOUR duringthe 2003-2010 sample period. Estimated random player-course effects rangefrom -0.186 to 0.146, too small to have a meaningful impact on the overallscores in a typical 72-hole PGA TOUR event.

When estimating skill functions for PGA TOUR players, we also obtainsets of player-specific residual scoring errors, denoted as θ and η. The θ er-rors represent potentially autocorrelated differences between a player’s actual18-hole scores, reduced by estimated random round-course and player-courseeffects, and his predicted scores. The η errors represent θ errors adjusted forestimated first-order autocorrelation, and are assumed to be white noise.

We have identified the players among the 699 who participated in Stages2 and 3 of Q-School during the 2006-2010 period. Using the estimated skillfunctions for these players during the PGA TOUR seasons immediately pre-ceding their entry to Q-School and the sets of their estimated θ errors, we areable to build scoring distributions characteristic of regular PGA TOUR playerswho entered Q-School in Stages 2 and 3. Inasmuch as skill functions for these

a 10-score minimum, rather than a 91-score minimum, and use simpler linear functions toestimate skill for those who recorded between 10 and 90 scores.

15Round-course effects, with similar justification, are also estimated in Berry, Reese andLarkey (1999) and Berry (2001). A similar approach is taken by Caulkins et. al (1993) inestimating on-time performance of airlines, where empirically observed on-time performancerates reflect “not just the difficulty of flying into [a given airport], but also the skill of theairlines that serve it.” (pg. 713)

18


DOI: 10.2202/1559-0410.1308



players are estimated simultaneously with those of the remaining players inthe 699-player PGA TOUR sample, we are able to connect the scoring charac-teristics of those among the 699 who entered Q-School between 2006 and 2010to those of Tiger Woods and other highly-skilled players on the TOUR.

8. Q-School Simulation Structure

8.1. Creating a Prototypical Q-School Structure

In our simulations, we employ a Q-School structure that reflects the numberof total and successful participants by stage of play observed in actual Q-School competition over the 2006-2010 period. Each simulation consists of1,000 trials, with each trial representing the simulation of a single season ofQ-School competition.

Although not shown in Table 2, Pre-Qualifying was conducted at fourto six tournament sites in years 2006-2010. Stage 1 was conducted at 11 to 13sites. Stage 2 was conducted at six sites in each year, and Stage 3 was alwaysconducted at a single site, although competition in the final stage was alwaysconducted on two courses, with each competitor playing half of his rounds oneach course. For the purposes of our simulations, we assume 6, 12, 6 and 1sites, respectively, for Pre-Qualifying through Stage 3, respectively, and referto this as a 6-12-6-1 venue structure.

In our simulations, we employ a prototypical Q-School structure thatreflects the average number of total and successful participants by stage ofplay in the 2006-2010 Q-School data, spread over a 6-12-6-1 venue struc-ture. We then calibrate the parameters of assumed scoring distributions em-ployed in the simulations so that the simulations produce qualifying successrates, playing-stage-normalized mean round-course-adjusted scores, and stan-dard deviations of round-course-adjusted scores by playing stage/entry stagethat closely match those observed in the 2006-2010 empirical Q-School sampleas summarized in Tables 3, 5 and 6. We describe the calibration process inSection 9.2.

Table 7 summarizes the Q-School structure used in calibrating our sim-ulations. Pre-Qualifying is spread over six tournament sites, with an averageof 70.47 players participating at each site and 37.07 qualifying for Stage 1 com-petition.16 In Stage 1, an average of 25.78 of 76 participants per venue advanceto Stage 2. Stage 2 is comprised of an average of 77.10 players per venue, with

16In the simulations we allocate extra players on a random basis so that these and similarvalues for Stages 1-3 are obtained on average.

19





an average of 20.97 players at each site advancing to Stage 3. The single tour-nament site in Stage 3 hosts an average of 164.8 players, where, on average,29.60 advance to the PGA TOUR. We do not spread the Stage 3 players overtwo different courses, as is the case in actual Q-School competition.

Table 7: Prototype Q-School Structure Employed in Simulation Calibrations

QualifiersPre-Qual Stage 1 Stage 2 Stage 3

Total next-stagequalifiers 2006-2010 1,112 1,547 629 148

Qualifiers/year 222.40 309.40 125.80 29.60Tournament sites 6 12 6 1Qualifiers/tournament site 37.07 25.78 20.97 29.60

ParticipantsPre-Qual Stage 1 Stage 2 Stage 3

Total participants 2006-2010 2,114 4,560 2,313 824

New entrants/year 422.80 689.60 153.20 39.00Qualifiers/year from previous stage 0.00 222.40 309.40 125.80Total participants/year 422.80 912.00 462.60 164.80Tournament sites 6 12 6 1Participants/tournament site 70.47 76.00 77.10 164.80With the exception of the number of tournament sites per stage, which variedslightly by year, all entries reflect actual Q-School experience for the 2006-2010period. The 1,112 next-stage qualifiers in Pre-Qualifying and 1,547 in Stage 1are net of 7 and 11 players, respectively, who qualified for the next stage but didnot continue.

8.2. Player Groups

In each simulation trial, we pick a year y ∈ {2006, 2007, ..., 2010} at random.We then assume that the participants in simulated Q-School competition aredrawn from the following six distinctly different player groups, with an indi-cation of the average number of players per group in each simulation trial:

1. Non-PGA TOUR players entering Q-School competition in Pre-Qualifying(422.8 players on average, see Table 7)

20


DOI: 10.2202/1559-0410.1308



2. Non-PGA TOUR players entering Q-School competition in Stage 1 (689.6players on average)

3. Players entering Q-School competition in Stage 2 coming directly fromthe PGA TOUR (We denote the number of such players as n2 (y), in-dicating that the number of players depends on the randomly selectedyear, y.)

4. Non-PGA TOUR players entering Q-School competition in Stage 2 (153.2−n2 (y) players on average)

5. Players entering Q-School competition in Stage 3 coming directly fromthe PGA TOUR, denoted as n3 (y)

6. Non-PGA TOUR players entering Q-School competition in Stage 3 (39−n3 (y) players on average)

As described in Section 9, the statistical properties of scoring for thetwo groups of players coming directly from the PGA TOUR (groups 3 and5, above) reflect the scoring characteristics, as estimated by the Connolly-Rendleman (2008, 2011) model (discusseed above in Section 7), of TOURplayers who actually entered Q-School in the randomly selected year. Thestatistical properties of scoring for non-PGA TOUR players entering Stage 3are based on a modification (described in Section A of the Appendix) of thescoring properties of Stage 3 players coming directly from the TOUR. Thestatistical properties of all other non-PGA TOUR player groups are based onmodifications of scoring distributions associated with Stage 2 players comingfrom the TOUR (also described in Section A of the Appendix).

8.3. Simulation of Q-School Competition

If a player enters Q-School in Pre-Qualifying and makes it to Stage 3, he willhave participated in 18 rounds of golf, four rounds each in Pre-Qualifyingthrough Stage 2 and six rounds in Stage 3. Using the procedure described inSection 9, we generate 18 random 18-hole scores for each player in simulatedQ-School competition but use only the portion of the scores so generated thatare relevant to a player’s stage of play. For example, if a player enters Q-Schoolin Stage 3, we would use only the last six of his 18 randomly generated scores.

The following steps, which reflect the prototype Q-School structuresummarized in Table 7, describe the structure of simulated Q-School compe-tition:

21





1. Select an average of 422.8 players at random for Pre-Qualifying andgenerate 18 random 18-hole scores for each player using the proceduresdescribed below in Section 9.

2. Randomly assign each of the players so selected to each of six Pre-Qualifying venues, with an average of 70.47 players assigned to eachvenue.

3. Compute the total of the first four randomly generated scores for eachplayer and rank each player from lowest total score to highest total withineach venue.

4. From each ranked list, select an average of 37.07 players with the lowesttotal scores from each venue as venue winners who move on to Stage 1(on average, 222.4 total).17

5. Combine the Pre-Qualifying winners (222.4 players, on average) with anaverage of 689.6 new Stage 1 entrants to produce a pool of 912 playersfor competition in Stage 1.

6. Distribute the 912 players randomly across 12 venues of 76 players each.

7. Generate 18 random 18-hole scores for each new player entering Q-Schoolin Stage 1.

8. Compute the total of scores 5-8 for each of the 912 Stage 1 players anddetermine venue winners who move on to Stage 2 (an average of 25.78players per site).18

9. Proceed in likewise fashion through Stage 3, using the number of newstage entrants, venues and venue winners as indicated in Table 7.

10. The lowest scoring 29.6 players (on average) in Stage 3 qualify for thePGA TOUR.

We explore variations on this general simulation structure in Section 10.

17We also remove an average of 7/5 players per year who qualify for Stage 1 to account forthe seven players who qualified for Stage 1 over the 2006-2010 period but did not continue.

18We also remove an average of 11/5 players per year who qualify for Stage 2 to accountfor the 11 players who qualified for Stage 2 over the 2006-2010 period but did not continue.

22


DOI: 10.2202/1559-0410.1308



9. Simulated Scoring Distributions for Q-School

Participants

9.1. Regular PGA TOUR Players Entering Q-School

In each simulation trial, we select i = 1 to n3 (y) players at random withreplacement from the set of n3 (y) regular PGA TOUR players who actuallyentered Stage 3 of Q-School in randomly selected year y. These become theregular PGA TOUR players in simulated Stage 3 Q-School competition. Therandomly selected PGA TOUR player associated with Q-School participant iis denoted as Stage 3 TOUR player k(i, 3).

We recognize that a player’s official money winnings in a given PGATOUR season, which, in turn, reflect his scoring, determine whether he mustparticipate in Q-School at the season’s end to qualify for PGA TOUR play thefollowing year. Therefore, in building the simulated scoring distribution forsimulation player i, we include only the portion of randomly selected playerk(i, 3)’s estimated skill function associated with actual PGA TOUR season y.We then select a point at random on the portion of randomly selected playerk(i, 3)’s estimated skill function covering PGA TOUR season y as an estimateof simulation player i ’s mean skill level.

To simulate random variation in scoring for Q-School participant i,we select a starting θ error at random from the entire distribution of Stage 3TOUR player k(i, 3)’s θ errors estimated over the entire 2003-2010 PGA TOURsample period. We then select 28 η errors at random (with replacement) fromplayer k(i, 3)’s entire distribution of η errors. Using the initial θ error, thevector of 28 randomly-selected η errors, and player k(i, 3)’s first-order auto-correlation coefficient as estimated in the Connolly-Rendleman (2008, 2011)model, we compute a sequence of 28 estimated θ errors. We apply the last 18estimated θ errors to Q-School player i ’s randomly selected mean score, yield-ing a sequence of 18 simulated 18-hole scores for player i. We do not employthe first ten simulated θ errors in our simulated scores in order to give the au-tocorrelated component of residual scoring errors time to ‘burn in.’ Simulatedscores for the n2 (y) regular PGA TOUR players who enter Q-School in Stage2 are produced in similar fashion. By construction, all of the simulated scoresare ‘neutral’ with respect to variation in scoring due to relative round-coursedifficulty and a player’s propensity to score better on some courses than others.

23





The two right-most histograms in the bottom section of Figure 2 showdistributions of mean skill levels, based on 1,000 simulation trials, for PGATOUR players entering Q-School in Stages 2 and 3, respectively. Histogramsin corresponding positions in Figure 3 show distributions of simulated 18-holescores for both groups. Simulated 18-hole scores range from 60.82 to 90.67among those shown in the Stage 2 histogram (712,206 scores total) and from59.85 to 85.40 among those shown in the histogram for Stage 3 (338,166 scorestotal).

Note from Figure 3 that there is a 72.15−71.57 = 0.58 stroke-per-roundaverage scoring difference between the scoring distribution of PGA TOURplayers entering Q-School in Stages 2 and 3. Therefore, in our simulations,PGA TOUR players who enter in Stage 3 have a built-in mean 0.58 stroke-per-round advantage over those who enter in Stage 2.

9.2. Non-PGA TOUR Players Competing in Q-School

As mentioned in Section 8.2, we build simulated distributions of scoring forall Stage 3 entrants not coming from the PGA TOUR by modifying simulatedscores that would otherwise be associated with Stage 3 entrants coming directlyfrom the TOUR. Similarly, we build simulated distributions of scoring for allPre-Qualifying and Stage 1 entrants and for those entering Q-School in Stage2 who are not coming from the PGA TOUR by modifying simulated scoresthat would otherwise be associated with Stage 2 entrants coming directly fromthe TOUR.

The modified distributions are calibrated to produce simulated quali-fying success rates (see Table 3), playing-stage-normalized mean round-course-adjusted scores (see Table 5), and standard deviations of round-course-adjustedscores by playing stage/entry stage (see Table 6) that closely approximatethose observed empirically. We summarize the player scoring distributionmodification process and the sensitivity of our results to parameter selectionin Sections A and B of the Appendix.

24


DOI: 10.2202/1559-0410.1308



Figure 2: Distributions of simulated mean skill levels.

25




orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

Figure 3: Distributions of simulated 18-hole scores.

26


DOI: 10.2202/1559-0410.1308


orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

We calibrate the simulations by selecting a set of 10 parameters, x =x1, ..., x10, associated with modified player scoring distributions that minimizesthe following function:

Z = c3Q3(x)W′ + c5Q5(x)W′ + c6Q6(x)W′

c3 = 50; c5 = 0.02; c6 = 1

W =

0.215 0.113 0.023 0.003

0 0.351 0.134 0.0330 0 0.078 0.0280 0 0 0.020

Qt(x) is a 4×4 matrix of squared differences between the empirically observedvalues in Table t and corresponding simulated values. Each element of W cor-responds to the proportion of all empirically-observed Q-School participantsassociated with each entry stage/playing stage combination. We employ theconjugate gradients method of Fletcher and Reeves (1964) to determine opti-mal parameter values, x = x1, ..., x10. We also employ simulating annealingwith various starting parameter values but cannot improve upon the solution.The coefficients c3, c5, and c6 are selected to account for differences in scaleassociated with qualifying proportions in Table 3 and the means and standarddeviations of golf scores in Tables 5 and 6 as well as our perception of the rela-tive importance of matching the values in the three tables. For example, sincemuch of our “story” has to do with the probabilities that players of certaintypes would qualify for the TOUR through Q-School if they had to start inPre-Qualifying, we feel that matching the values in Table 3, which gives qual-ifying proportions, is the most important consideration. We note, however,that solution values are relatively insensitive to the value of c3. For example,we obtain very similar parameter solution values whether c3 = 50 or c3 = 5.

Panels A, B and C of Table 8 show how well qualifying success rates,playing-stage-normalized mean round-course-adjusted scores, and standard de-viations of round-course-adjusted scores by playing stage/entry stage in thesimulated data match those observed in the 2006-2010 Q-School sample. Al-though there are some relatively large differences between empirical and simu-lated values, especially in playing Stage 3, the largest differences tend to occurwhere the relative weights are the smallest. For example, entry Stage 0/play-ing Stage 3, where differences for Tables 3 and 5 are large, gets only 0.3% ofthe weight in our objective function. Overall, we believe that the simulations,

27





calibrated with the parameters summarized in Appendix Table 1, reproducethe properties of the 2006-2010 Q-School sample in its most important dimen-sions.

The center section of Figure 2 shows simulated distributions of meanskill for all player groups derived from the Stage 2 PGA TOUR sample, whilethe right-most section shows scoring distributions based on the Stage 3 sample.Corresponding sections of Figure 3 show distributions of simulated 18-holescores.

10. Simulation Results

10.1. Qualifying Success Rates

In each simulation trial we split Q-School participants into groups of 25 playersbased on mean skill and estimate the frequency of success in Q-School for eachplayer group across 1,000 simulation trials. We place the 25 most highly-skilledplayers per trial in group 1, the next 25 in group 2, etc. through group 20. Wethen lump all the remaining players, those ranked 501 and above, into group21 (approximately 805 players).

We break participants into groups of 25 since, as of 2010, the lowest25 finishers and ties at the conclusion of play in Stage 3 qualify for the PGATOUR. The remainder of players in Stage 3 are eligible for Nationwide Tourmembership. The next number of finishers closest to 50 qualify as “fully-exempt” members of the Nationwide Tour, eligible to play in any NationwideTour event the following year. The remaining Stage 3 participants becomepartially-exempt Nationwide players. Ideally, the 25 most highly skilled playersin Q-School competition would have high probability of qualifying for the PGATOUR.

As shown in Table 9, which provides qualifying success rates for eachof the 21 player groups, this is not the case. Only 30.7% of the 25 mosthighly-skilled Q-School participants finish Stage 3 among the 25 lowest scoringplayers. For those finishing among the lowest-scoring 25, an average of only7.7 players would have come from skill group 1.

By finishing among the lowest-scoring 75, 62% of the players in group1 would have done no worse than earning fully-exempt Nationwide status, andby participating in Stage 3, 80% would have done no worse than earning apartial exemption on the Nationwide Tour. The remaining 20% would haveearned nothing more than the right to enter Q-School the following year inStage 1 (see Figure 1).

28


DOI: 10.2202/1559-0410.1308



Table 8: Calibration: Simulation Results vs. Characteristics of Q-School Sample

Panel A: Qualifying Success Rates per Playing Stage/Entry StageSimulation 2006-2010 Q-School Sample Simulation Minus Q-School Sample

Entry Pre-Stage Qual Stage 1 Stage 2 Stage 3 Pre-Qual Stage 1 Stage 2 Stage 3 Pre-Qual Stage 1 Stage 2 Stage 3

Pre-Qual 0.529 0.200 0.139 0.282 0.526 0.207 0.113 0.000 0.003 -0.007 0.026 0.282Stage 1 0.389 0.247 0.134 0.382 0.247 0.132 0.007 0.000 0.001Stage 2 0.372 0.219 0.363 0.219 0.009 -0.001Stage 3 0.260 0.226 0.034

Panel B: Playing-Stage-Normalized Round-Course-Adjusted Mean Scores per Playing stage/Entry stageSimulation 2006-2010 Q-School Sample Simulation Minus Q-School Sample


Pre-Qual 0.000 0.991 1.463 1.531 0.000 0.991 1.438 1.025 0.000 0.000 0.024 0.507Stage 1 0.000 0.549 0.759 0.000 0.540 0.453 0.000 0.009 0.306Stage 2 0.000 0.172 0.000 0.136 0.000 0.036Stage 3 0.000 0.000 0.000

Panel C: Standard Deviation of Round-Course-Adjusted Scores per Playing Stage/Entry StageSimulation 2006-2010 Q-School Sample Simulation Minus Q-School Sample


Pre-Qual 4.037 3.191 3.016 2.939 4.075 3.190 3.039 2.993 -0.038 0.000 -0.023 -0.054Stage 1 3.189 2.986 2.889 3.119 3.079 2.809 0.071 -0.092 0.080Stage 2 2.999 2.836 2.978 2.692 0.021 0.145Stage 3 2.910 2.757 0.153

Results based on 1,000 simulation trials using the simulation parameters summarized in Appendix Table 1. The center portions ofPanels A-C are Tables 3, 5 and 6, respectively.

29




orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

Table 9: Qualifying Success Rates by Skill Group

Percentages per Category by Skill Group Number of Players per Category by Skill GroupSkill Lowest Lowest Stage 3 Lowest-Scoring 25 Lowest-Scoring 75 Stage 3 Participant

Group Scoring 25 Scoring 75 Participant Players Cumul. Players Cumul. Players Cumul.1 30.7 62.0 80.0 7.7 7.7 15.5 15.5 20.0 20.02 19.0 47.2 70.0 4.7 12.4 11.8 27.3 17.5 37.53 12.8 37.0 60.0 3.2 15.6 9.2 36.6 15.0 52.54 10.0 31.2 57.5 2.5 18.1 7.8 44.3 14.4 66.95 6.9 23.6 47.8 1.7 19.8 5.9 50.2 12.0 78.86 4.8 18.2 40.2 1.2 21.1 4.5 54.8 10.0 88.97 3.7 15.8 36.6 0.9 22.0 3.9 58.7 9.2 98.08 3.1 13.2 34.6 0.8 22.7 3.3 62.0 8.6 106.79 2.4 10.4 28.9 0.6 23.3 2.6 64.6 7.2 113.910 1.5 7.9 24.2 0.4 23.7 2.0 66.6 6.1 119.911 1.2 6.2 21.0 0.3 24.0 1.6 68.2 5.3 125.212 1.0 5.5 19.9 0.2 24.3 1.4 69.5 5.0 130.213 0.7 4.5 17.7 0.2 24.4 1.1 70.6 4.4 134.614 0.5 3.8 16.0 0.1 24.6 0.9 71.6 4.0 138.615 0.3 2.7 13.9 0.1 24.6 0.7 72.3 3.5 142.116 0.3 2.1 11.1 0.1 24.7 0.5 72.8 2.8 144.817 0.2 1.8 10.3 0.1 24.8 0.4 73.2 2.6 147.418 0.2 1.3 8.8 0.0 24.8 0.3 73.6 2.2 149.619 0.2 1.2 7.8 0.0 24.9 0.3 73.9 1.9 151.620 0.1 1.0 7.1 0.0 24.9 0.2 74.1 1.8 153.321 0.0 0.1 1.4 0.1 25.0 0.9 75.0 11.5 164.8

Results based on 1,000 simulation trials using the simulation parameters summarized in Appendix Table 1. Skill group 1consists of the first 25 players per simulation trial based on the ranking of individual player skill. Group 2 consists of thenext 25 players, etc., with group 20 consisting of players ranked 476-500. Group 21 consists of the last 1, 305− 500 = 805players (approximately).

30


DOI: 10.2202/1559-0410.1308


orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

As one would expect, the qualifying success rates fall monotonically byskill group. Taking groups 1-3 together, only 36.6/75 = 48.8% of the 75 mosthighly-skilled players in Q-School competition would have finished among the75 lowest-scoring players at the end of Stage 3.

Although the qualifying success rates in individual skill categories 4-21 are not high, collectively we see that many players of lower skill wouldhave finished among the lowest scoring 25 who would have qualified for thePGA TOUR, many would have earned no worse than fully-exempt Nationwidestatus, and at least 24.2% in each of skill groups 1-10 would have done no worsethan earn partially-exempt Nationwide status. 1.4% of the players ranked 501or higher (those in skill group 21) would have earned the right to play on theNationwide Tour in some capacity. Moreover, an average of 11.5 players inskill positions 501 or worse among the (approximately) 165 who participate inStage 3 would have earned no worse than partially-exempt Nationwide status.

10.2. Success Rates for ‘The Next Tiger Woods’ and FiveOther Highly-Skilled Players

It is interesting to consider how well players of skill comparable to that ofTiger Woods and other highly-skilled PGA TOUR professionals would fare ifthey had to begin Q-School in Pre-Qualifying. To address this question, wereplace six randomly-selected players in Pre-Qualifying with a set of playerswhose scoring characteristics are comparable to those of “Tiger Woods” andfive other “good” players.19 In each simulation trial, the first “good” player is aplayer whose simulated score is drawn from the distribution of “Tiger Wood’s”scores from the 2006-2010 PGA TOUR sample in randomly selected year y.The second good player is a player with scoring characteristics identical to thatof a player selected randomly from skill positions 2-21 in year y. In determiningthese skill positions, a player’s skill is defined as the average value of his 18-hole score as estimated over year y only. The third good player is selectedrandomly from skill positions 22-41, etc., with the sixth selected randomlyfrom positions 82-101. While it is unrealistic to assume that a ‘Tiger Woods’will come along every year, it is not unrealistic to assume that players like thosein skill quintiles 1-5 will enter Pre-Qualifying on a regular basis. Otherwise,there would be no permanent talent pool on the TOUR. Simulated mean skill

19Until 2010, Tiger Woods was the most highly skilled player in each Q-School year, 2006-2009, by a wide margin. However, in 2010 Woods was the fourth-most skilled player, andLee Westwood was first. In the remainder of the paper, when we refer to “Tiger Woods,”we are really referring to a player with Woods’ scoring characteristics in 2006-2009 andWestwoods’ in 2010.

31





levels and variation in scoring among the six good players are determined usingthe same sampling procedure described in Section 9.1 for PGA TOUR playersentering Q-School in Stages 2 and 3 and, therefore, are not modified using themethods described in Appendix Section A, which apply to non-TOUR playersonly.

Histograms showing the distributions of skill and 18-hole scores for the“good” player group are shown on the far left of Figures 2 and 3. Note that theskill distribution clearly is bimodal, reflecting that players with Tiger Woods’scoring characteristics are substantially more skilled than the other “good”players. Also note that as a group, the mean score in the “good” player groupis 71.57 − 70.06 = 1.51 strokes less than that of players in the next-mosthighly-skilled group, PGA TOUR players entering Q-School in Stage 3.

If Q-School is working properly, we believe that a player of Tiger Wood’sability who enters Q-School in the Pre-Qualifying stage should almost alwaysqualify to play on the PGA TOUR. Moreover, one would hope that playersof skill comparable to those in skill positions 2-101 should have little troublequalifying. Panel A of Table 10 summarizes the probabilities of making itthrough the four stages of Q-School for players of Tiger’s Wood’s ability andfor randomly selected players from skill quintiles 1-5. In Table 10, “making itto the next stage” from Stage 3 refers to finishing among those who qualifyfor the TOUR, 29.6 players per year, on average.

The top panel of Table 10 shows that a player with skill comparable tothat of Tiger Woods would earn his TOUR card 94.4% of the time. However,the qualifying success rate for those in skill quintile 1 is only 74.2%, withsuccess rates falling monotonically by skill group to 39.8% for group 5.

It should be noted that all of the success rates shown in Table 10 areconditional upon a Tiger-type player and players in all five skill quintile groupsplaying simultaneously in the Q-School competition. Therefore, the successrates for each of the six player groups should be lower than if no other groupswere included. However, the estimated success rates are not much lower.For example, with all six groups in the competition, 74.2% of the players inquintile 1 earn TOUR cards. Although not shown in the table, if players inquintile 1 are the only “good” players in Q-School competition, their successrate increases to 76.7%.

Note that among players comparable to Tiger Woods and the players inthe various skill quintiles, almost none are eliminated from Q-School competi-tion in Pre-Qualifying. However, starting in Stage 2, many players other thanthe Tiger-type player start to get knocked out. In fact, by the end of Stage 2,24% and 26.5% of the players in quintiles 4 and 5, respectively, are eliminated

32


DOI: 10.2202/1559-0410.1308



Table 10: Estimated Probabilities of Success at Each Stage of Q-School when‘Tiger’ and Quintile Players Enter Q-School in Pre-Qualifying

Panel A: Standard Q-School StructureStage

Player Pre-Q 1 2 3Tiger 1.000 0.998 0.976 0.944

Quintile 1 0.999 0.985 0.903 0.742Quintile 2 1.000 0.976 0.848 0.618Quintile 3 1.000 0.963 0.797 0.518Quintile 4 1.000 0.951 0.760 0.438Quintile 5 1.000 0.946 0.735 0.398

Panel B: Number of Roundsin Each Stage of Q-School is Doubled

StagePlayer Pre-Q 1 2 3Tiger 1.000 1.000 1.000 0.996

Quintile 1 1.000 0.999 0.984 0.915Quintile 2 1.000 1.000 0.948 0.815Quintile 3 1.000 0.993 0.916 0.701Quintile 4 1.000 0.987 0.884 0.617Quintile 5 1.000 0.988 0.857 0.552Results based on 1,000 simulation trials us-ing the simulation parameters summarized inAppendix Table 1.

players who have experienced favorable random variation in their performance.The tendency for favorable variation to be a determining factor in Q-

School qualifying success could be reduced if the number of rounds of playper stage of Q-School were increased. Panel B of Table 10 summarizes successrates for a player comparable to Tiger Woods and players in skill quintiles 1-5,assuming that the number of rounds per stage is doubled. Here we observehigher qualifying success rates (e.g., 91.5% vs. 74.2% for quintile 1 and 55.2%vs. 39.8% for quintile 5), but rates that are lower than what we believe mostwould consider to be acceptable.

from Q-School. Generally, these players are being eliminated by lesser-skilled

33





10.3. Success Rates for the Top 25 Players in the PGATOUR Sample, 2006-2010

We now conduct a number of separate 1000-trial simulations where, in eachsimulation, we replace a single randomly selected player in Pre-Qualifying witha player whose scoring profile is identical to that of one of the 25 most highly-skilled players in the PGA TOUR sample. For the purposes of identifying the25 players, we first limit the possible players to those who recorded at leastone score in each PGA TOUR season, 2006-2010. Among these players, wecompute the mean skill estimate for each player by year, 2006-2010, and thencompute the average of the five annual mean scoring estimates. This overallaverage reflects the simulation procedure in which each Q-School year, 2006-2010, is equally likely to be selected in a given simulation trial, and withinthe selected year, each player’s skill estimate is equally likely to be selectedas his simulation skill estimate. A player’s skill ranking is the ranking of hisaverage skill value so computed among the players who recorded at least onescore in each of years 2006-2010. We also show simulation results for Q-Schoolparticipants with scoring profiles equivalent to five well-known players who arenot as highly ranked.

Table 11 summarizes estimated success rates for each of the 30 playersin each stage of Q-School, listed in the order of mean estimated skill. Thestandard deviation of η errors for each player is shown along with the first-order autocorrelation coefficient associated with his θ errors, denoted as ρ.

Among those entering Q-School in Pre-Qualifying with skill comparableto a top-25 player, the median qualifying rate is 70.3%. Individual qualifyingsuccess rates are not strictly monotonic with average skill values. For example,a player with Adam Scott’s scoring characteristics, the 17th ranked player, hasthe lowest qualifying success rate, 63.7% (tied with Camilo Vilegas), amongthe top 25 and also the highest standard deviation of η errors. (It appearsthat higher standard deviations of η errors are associated with lower quali-fying success rates.) Generally, the rates of qualifying success among the 30players reflect a combination of mean skill, round-to-round variation in scor-ing, autocorrelation in θ errors, and variation in mean skill within each player’sestimated skill function.

34


DOI: 10.2202/1559-0410.1308



Table 11: Estimated Probabilities of Success at Each Stage of Q-School for Selected PGA TOUR Players

Skill Average Standard Estimated Probabilities of SuccessRank Player Skill Deviation η ρ Pre-Q Stage 1 Stage 2 Stage 3

1 Woods, Tiger 68.099 2.574 0.034 1.000 0.998 0.981 0.9672 Furyk, Jim 69.356 2.429 -0.016 1.000 1.000 0.954 0.8813 Mickelson, Phil 69.406 2.829 -0.012 1.000 0.994 0.962 0.8914 Els, Ernie 69.624 2.664 0.000 1.000 0.991 0.937 0.8315 Stricker, Steve 69.632 2.638 0.011 1.000 0.989 0.918 0.8006 Donald, Luke 69.653 2.597 -0.011 1.000 0.992 0.939 0.8427 Singh, Vijay 69.890 2.551 0.062 0.999 0.985 0.901 0.7648 Garcia, Sergio 69.892 2.585 -0.052 0.999 0.976 0.904 0.7709 Harrington, Padraig 69.986 2.752 0.102 1.000 0.988 0.906 0.75010 Westwood, Lee 70.011 2.910 -0.019 1.000 0.984 0.877 0.69611 Goosen, Retief 70.013 2.636 -0.021 1.000 0.969 0.864 0.67512 Allenby, Robert 70.076 2.625 -0.028 1.000 0.983 0.886 0.70313 Casey, Paul 70.079 3.010 0.084 1.000 0.977 0.871 0.66514 Rose, Justin 70.087 2.847 -0.013 1.000 0.986 0.879 0.72015 Ogilvy, Geoff 70.088 2.826 0.051 1.000 0.990 0.917 0.76316 Cink, Stewart 70.098 2.439 0.003 0.999 0.983 0.865 0.66917 Scott, Adam 70.125 3.025 0.009 0.999 0.951 0.849 0.63718 Clark, Tim 70.138 2.746 0.030 1.000 0.984 0.872 0.67819 Johnson, Zach 70.139 2.600 0.016 0.999 0.973 0.871 0.68420 Toms, David 70.145 2.588 0.068 1.000 0.987 0.895 0.74821 Kim, Anthony 70.162 2.928 0.011 1.000 0.981 0.884 0.68422 Choi, K.J. 70.171 2.627 0.039 1.000 0.970 0.851 0.65223 Villegas, Camilo 70.232 2.762 0.066 1.000 0.965 0.847 0.63724 Perry, Kenny 70.252 2.748 0.028 1.000 0.982 0.869 0.64225 Poulter, Ian 70.255 2.698 -0.046 1.000 0.989 0.879 0.701

31 O’Hair, Sean 70.381 2.523 0.134 0.997 0.936 0.744 0.46439 Kuchar, Matt 70.522 2.722 0.046 1.000 0.978 0.833 0.59548 Appleby, Stuart 70.658 2.752 -0.010 1.000 0.953 0.776 0.51469 Hoffman, Charley 70.824 2.842 0.082 0.999 0.959 0.753 0.41585 Curtis, Ben 70.943 2.937 0.033 1.000 0.945 0.707 0.379

Results based on 1,000 simulation trials using the simulation parameters summarized in Appendix Table 1. ρ denotes theestimated first-order autocorrelation coefficient associated each player’s residual θ errors as estimated over the 2003-2010period, and the standard deviation of η is the standard deviation of each player’s η errors as estimated over 2003-2010.All players in the table recorded at least one score in each PGA TOUR season, 2006-2010. A player’s average skill iscomputed by first averaging his estimated score in each year of simulated Q-School competition, 2006-2010, and thenaveraging the five average scoring estimates. This average reflects the simulation procedure in which each Q-School year,2006-2010, is equally likely to be selected in a given simulation trial, and within the selected year, each player’s skillestimate for the selected year is equally likely to be selected as his simulation skill estimate. A player’s skill ranking isthe ranking of his average skill value so computed among the players who recorded at least one score in each of years2006-2010.

35




orth Carolina at C

hapel Hill

AuthenticatedD

ownload D

ate | 8/7/17 5:45 PM

We believe that most who follow golf would find it disturbing that aplayer with Steve Stricker’s scoring characteristics would have only an 80%chance of earning his TOUR card and that the next K.J. Choi would haveonly a 65% of qualifying for the TOUR. Players entering Pre-Qualifying withscoring characteristics comparable to lesser-ranked players Sean O’Hair, MattKuchar, Stuart Appleby, Charley Hoffman and Ben Curtis would have betweena 37.9% and 59.5% chance of qualifying for the TOUR. But we believe theseestimated success rates are quite realistic. For example, it took Sean O’Hair,ranked 43rd in the Official World Golf Rankings as of December 26, 2010,six tries in Q-School between 1999 and 2004 before qualifying for the PGATOUR. Since earning his TOUR card, O’Hair has won three PGA TOURevents. Similarly, Zach Johnson, ranked 22rd, spent five years in Q-Schoolbetween 1998 and 2002, with his best finish coming in 2002, a tie for 94th inStage 3. This earned Johnson a partial exemption on the 2003 NationwideTour, where he finished the season in first place, thereby earning the right toplay on the PGA TOUR starting in 2004. Since then, he has won seven timeson the PGA TOUR, including the 2007 Masters Tournament.

11. Alternative Q-School Structures

Except for Stage 3, each stage of Q-School is conducted in separate compe-titions, with winners at each competition site advancing to the next stage.For example, in our prototypical Q-School structure, summarized in Table 7,Pre-Qualifying is conducted at six separate tournament sites, with an averageof 422.8 players competing for 222.4 spots in Stage 1 competition. (In theliterature of contest design, such a structure is known as a “subcontest”(Fuand Lu (2009), p. 147)). However, since the winning players are determinedin six separate Pre-Qualifying competitions, the 222 who played the best inPre-Qualifying do not necessarily qualify for Stage 1. In our first test of al-ternative Q-School structure, we estimate the probabilities that highly-skilledplayers are eliminated in Q-School competition because they happened to havebeen assigned to tournament sites for which they had lower chances of advanc-ing.

To do so, we restructure our simulations into a series of “grand con-tests,” so that each stage of Q-School is conducted at a single tournament site.Such a structure guarantees that the lowest-scoring players in each stage earnnext-stage qualifying positions (although, due to random variation in scoring,it does not guarantee that the most highly-skilled players in each stage getto advance). We run the simulations two ways, first without seeding Pre-

36


DOI: 10.2202/1559-0410.1308



Qualifying with any special players with scoring characteristics comparableto specific PGA TOUR players and second, by including ‘Tiger Woods’ andplayers from skill quintiles 1-5, as in Table 10. As it turns out, the qualify-ing success rates are essentially the same as those obtained when Q-School isconducted using the prototype structure. Therefore, we conclude that usinga number of separate tournament sites for stage qualifying is not a significantfactor in strong players being eliminated in Q-School competition.

We also consider a Q-School structure in which all players compete ina single stage in multiple rounds on the same course. Although impractival,such a structure would solve the problem of strong players being eliminatedfrom competition prior to Stage 3 and also the minor problem, addressedabove, associated with conducting Pre-Qualifying through Stage 2 at multipletournament sites.

Panel A of Table 12 shows qualifying success rates for skill groups 1-6when Q-School is conducted with the prototypical structure summarized inTable 7. Throughout the table, success rates are shown with and withoutthe inclusion of ‘the next Tiger Woods’ and players from skill quintiles 1-5.Panel B shows success rates when Q-School is conducted in a single stage on asingle course over 18 rounds, the same number of total rounds in the prototypestructure. In the single-stage structure, the equivalent to reaching Stage 3 isfinishing among the lowest scoring 164.8 players, on average, the same numberof Stage 3 players in the prototype structure.

Comparing Panels A and B, we see that conducting Q-School over 18rounds of play at a single tournament site, rather than over 18 rounds in fourstages of single elimination competition, increases the probability that the 25most highly-skilled players in Q-School will end up among the 25 players withthe lowest scores. For example, when ‘the next Tiger Woods’ and playersfrom skill quintiles 1-5 are not included, the probability of finishing amongthe lowest scoring 25 players increases from 30.7% to 39.8% for players in skillgroup 1. When ‘Tiger’ and his five counterparts are included, the probabilityincreases from 36.0% to 47.7%.

We also consider the effects of doubling the number of tournamentrounds from 18 to 36 (Panel C) and, again, from 36 to 72 (Panel D). With 72rounds of play, 61.6% of those in skill group 1 finish among the lowest scoring25 when ‘Tiger’ and counterparts are not competing, and 67.3% finish amongthe lowest scoring 25 when the Tiger group is competing. With or withoutthe Tiger group, almost all players in group 1 finish among the lowest scoring164.8 (on average), meaning that they would at least earn partially-exemptstatus on the Nationwide Tour.

37





Table 12: Qualifying Success Rates in Single-Stage Q-School Competition

Panel A: Standard Staging Structure

Without Tiger and Quintile Players With Tiger and Quintile PlayersSkill Lowest Lowest Stage 3 Lowest Lowest Stage 3

Group Scoring 25 Scoring 75 Participant Scoring 25 Scoring 75 Participant1 30.7 62.0 80.0 36.0 65.7 81.12 19.0 47.2 70.0 18.6 48.6 72.13 12.8 37.0 60.0 11.7 36.3 60.04 10.0 31.2 57.5 9.1 30.7 57.45 6.9 23.6 47.8 6.3 23.8 49.26 4.8 18.2 40.2 4.4 18.3 40.5

Panel B: Single Stage, 18 Rounds

Without Tiger and Quintile Players With Tiger and Quintile PlayersSkill Lowest Lowest Lowest Lowest Lowest Lowest

Group Scoring 25 Scoring 75 Scoring 164.8 Scoring 25 Scoring 75 Scoring 164.81 39.8 72.2 91.7 47.7 78.2 94.32 20.8 52.6 81.5 19.5 52.8 82.53 12.8 40.7 72.8 11.4 40.4 73.84 8.3 30.4 64.1 6.9 30.4 64.05 5.3 23.1 54.4 4.4 22.3 54.66 3.5 18.1 47.3 2.9 17.4 47.5

Panel C: Single Stage, 36 Rounds



Panel D: Single Stage, 72 Rounds



Results based on 1,000 simulation trials using the simulation parameters summarized inAppendix Table 1. In Sections B-D, all Q-School participants compete together at a singletournament site for the number of rounds shown. Skill group 1 consists of the first 25 playersper simulation trial based on the ranking of individual player skill. Group 2 consists of thenext 25 players, etc. Only the first six groups are shown.

38


DOI: 10.2202/1559-0410.1308



Although qualifying success rates are improved when the number ofrounds of tournament play rises, less than 2/3 of the players in skill group 1,the 25 most highly-skilled players in Q-School competition, would finish amongthe lowest scoring 25 after 72 rounds of play when ‘Tiger’ and counterparts arenot included in the competition. Thus, even after 72 rounds of competition,it would not be possible to identify the 25 most highly skilled players with ahigh level of precision.

There is no practical way to conduct Q-School for 72 rounds on a singlecourse. However, competition for a full year on the Nationwide Tour roughlyapproximates such a scenario, albeit with fewer competitors. In 2010, the 75leading money winners on the Nationwide Tour averaged 76 18-hole roundsof play (median of 79 rounds) over the 2010 season, approximately the sameas the 72 rounds in our alternative Q-School structure. PGA TOUR cardsare awarded to the 25 leading money winners on the Nationwide Tour, notexactly the same as the 25 players with the lowest total scores over 72 rounds.Moreover, not all Nationwide players participate in the same events, they of-ten miss cuts and are eliminated from individual tournaments, and moneywinnings are highly non-linear in scoring. Nevertheless, we believe that Na-tionwide competition is much closer to our 72-round single-stage structurethan regular Q-School structure conducted in four stages at multiple tourna-ment sites. In this respect, it is not surprising that players who have earnedtheir TOUR cards through Nationwide competition have tended to be moresuccessful than those who earned their cards through Q-School.

12. Summary and Conclusions

This study uses simulation to estimate Type I and Type II error rates associ-ated with the Q-School qualifying mechanism employed by the PGA TOURto identify golfers of high skill to join the TOUR. We calibrate the simulationsto reproduce important statistical properties of scoring actually observed inthe various stages of the Q-School qualifying process.

From simulation, we estimate that the current Q-School qualifyingstructure leads to high Type I and II errors. For example, we find that themedian estimated qualifying success rate for golfers of skill comparable to thatof the 25 most highly skilled players on the PGA TOUR is only 70.3%, butthat many players of substantially less skill could qualify for the TOUR in agiven year. Although skill plays a substantial role in determining qualifyingsuccess rates, for example, a player of Tiger Woods’ ability, as estimated overthe 2006-2010 period, would get through almost 97% of the time, random

39





variation in scoring also plays a substantial role. This suggests that in anygiven year, one would expect a portion of those who play in the final stage ofQ-School and end up qualifying for the TOUR would be less skilled relativeto final stage participants who only qualify to play on the Nationwide Tour.This helps to explain why the long-term success of golfers who qualify for thePGA TOUR through Q-School is not as high as that of golfers who qualify forthe TOUR through Nationwide competition.

We explore the impact of alternative Q-School structures on the effi-ciency of the Q-School qualifying process. We find that a single-stage tour-nament conducted with substantially more rounds than the 18 in the currentstructure would be a more efficient mechanism for identifying high-skill golfers.Although impractical, this finding is consistent with the prediction in Ryvkinand Ortmann (2008) that a longer tournament should be the most efficient.This alternative structure resembles reasonably well the current NationwideTour, where the top 25 money winners over a season of 26 tournaments auto-matically qualify to play the next season on the PGA TOUR.

The existing PGA TOUR qualification process has some features thatwe have not explored in detail here. As we noted in the introduction, the PGATOUR seems to have recognized that Q-School and the Nationwide Tour bothplay an important role in identifying high-skill golfers for participation on theTOUR. Over the past few years, the PGA TOUR has changed the allocationof TOUR cards among Q-School and Nationwide qualifiers. Although theoptimal allocation of TOUR cards to each qualifying mechanism isn’t known,an extension of the simulation framework developed in this study could proveuseful for making this assessment.

Finally, it seems that through Q-School, and partially through the Na-tionwide Tour, the PGA TOUR is attempting to solve the general problem ofhuman performance evaluation, given limited track records, that faces virtu-ally all organizations. These evaluation activities are typically constrained bytime, money, and other resources. But unlike Q-School and the NationwideTour, the systems of performance evaluation found in organizations may befar more complex and ambiguous. Compared with human performance eval-uation activities found in academic departments, professional service firms,internship programs and other settings, the fixed set of rules associated withthe Q-School selection metric provides a level of simplicity not likely to befound in the aforementioned settings. To the extent that our findings general-ize into such settings, our results raise serious questions as to whether typicaltournament-style selection mechanisms generate the right outcomes in noisyand ambiguous environments. We believe that further study of these issues isboth intellectually challenging and of immense practical value.

40


DOI: 10.2202/1559-0410.1308



Appendix

A. Scoring Distribution Modification Process

for Non-PGA Tour Players

In equations (1) through (6) below, we summarize the process by which weproduce scoring distributions for non-PGA Tour players entering Q-School.

hi,j = µj + κj

(hk(i,j) − µj

)+ φj (1)

µj =

{µ∗2 (y) , for j = 0, 1, 2µ∗3 (y) , for j = 3

(2)

σk(i,j) = −1.939 + 0.066hk(i,j) (3)

σi,j = −1.939 + 0.066hi,j (4)

ξi,j = σi,j/σk(i,j) (5)

θi,j = θk(i,j)ξi,j (6)

In (1) through (6), the subscript i denotes a non-PGA TOUR playerin a simulation trial who enters Q-School in Stage j = 0, ..., 3, with Stage 0denoting Pre-qualifying. If the player’s entry stage is 0, 1, or 2, his level ofskill and variation in scoring is based on that of a randomly selected player(with replacement) from the sample of n2 (y) Stage 2 players coming directlyfrom the TOUR in randomly selected year y. The TOUR player so selectedis denoted as player k(i, j). If the player enters Q-School in Stage 3, his levelof skill and variation in scoring is based on that of a randomly selected playerfrom the sample of n3 (y) Stage 3 players coming directly from the TOUR,denoted as player k(i, 3). We refer to the appropriate set of Stage 2 or 3 playerscoming directly from the PGA TOUR upon whom the scoring characteristicof simulation player i are based as player i ’s “base sample.”

41





˜

˜˜

In (1), hk(i,j) denotes a mean skill level drawn randomly from the year

y portion of the skill function of base sample player k(i, j). This becomes thebase skill level from which Q-School player i ’s mean skill level and variation inscoring is derived. The population mean, µj, of individual skill function valuesthat could be drawn from this group in randomly selected year y is µ∗2 (y) ifplayer i ’s skill is based on that of a Stage 2 player and µ∗3 (y) if his skill is

based on that of a Stage 3 player. When κj > 1, the κj

(hk(i,j) − µj

)portion

of (1) has the effect of widening the distribution of potential mean skill levelsdrawn initially from the base sample while preserving the population mean ofthe base sample from which the random skill level is drawn. Similarly, whenκj < 1, the effect is to narrow the distribution of potential mean skill levels.The idea in using this spreading device is that for some entry stages, thedistribution of mean skill levels might be more disperse than that associatedwith PGA TOUR players entering Q-School in the stage from which the skilllevel of the i th simulated Q-School competitor is derived. We also include amean displacement parameter, φj, which allows the overall population meanscores for simulated Q-School participants in a given entry stage to be higheror lower than those of their associated base samples.

The coefficients −1.939 and 0.066 in (3) and (4) are the estimatedintercept and slope of a least squares regression of the standard deviation ofη errors on the means of individual player skill function values across the 699players in the 2003-2010 PGA TOUR sample, with adjusted R2 = 0.047 andp < 0.0001. Consistent with this relationship, when a Q-School participant’ssimulated mean skill level is higher than the mean skill level drawn randomlyfrom his base sample (i.e., when hi,j > hk(i,j)), the θ and η errors which drivethe random component associated with player i ’s simulated score should bemore disperse than those of his base sample player k(i, j). In (3) through (6)we adjust simulated θ errors to capture this general relationship.

We model Pre-qualifying entrants as a bimodal group. The proportionγ of Pre-qualifying entrants is assumed to come from a group of lesser-skilledplayers who have only a minimal chance of qualifying for the TOUR, whilethe proportion 1− γ of Pre-qualifying entrants are assumed to be ‘legitimate’players. In our simulations, both groups can have their own unique values ofsimulation parameter φ0.

To simulate random variation in scoring for non-PGA TOUR Q-Schoolparticipant i, we select a starting θ error at random from the distribution ofbase player k(i, j)’s θ errors estimated over the entire 2003-2010 PGA TOURsample period. We then select 28 of base player k(i, j)’s η errors at random(with replacement). Using the initial θ error, the 28 randomly-selected η

42


DOI: 10.2202/1559-0410.1308



errors, and player k(i, j)’s first-order autocorrelation coefficient as estimatedin connection with the Connolly-Rendleman (2008, 2011) model, we computea sequence of 28 estimated θ errors. We then multiply the 28 θ errors by ξi,j,as in (6), to obtain a set of modified θ errors. We apply the last 18 modifiedθ errors to simulation player i ’s mean skill level as estimated in equation (1),yielding a sequence of 18 simulated 18-hole scores for simulation player i.

As described in Section 9.2, we calibrate our simulations by choosing avalue of the parameter γ, together with parameter values κj and φj, for entrystages 0,...,3, including separate parameter values of φ0 for the two groups ofPre-qualifying entrants, to produce simulated qualifying success rates, playing-stage-normalized mean round-course-adjusted scores, and standard deviationsof round-course-adjusted scores by playing stage/entry stage that closely ap-proximate those observed empirically. Appendix Table 1 provides a summaryof parameter values employed in the simulations.

Appendix Table 1: Parameters of Calibrated Simulation Model

Entry StagePre-qualifying

Parameter Legitimate Lesser-Skilled Stage 1 Stage 2 Stage 3φ 2.93 7.74 1.59 0.53 0.60κ 2.14 2.14 1.87 1.58 1.07γ n/a 0.29 n/a n/a n/a

B. Sensitivity to Parameter Selection

In this section we explore the sensitivity of qualifying success rates to theselection of simulation parameters. Appendix Table 2 summarizes qualifyingsuccess rates for skill groups 1-6 for eight different parametrization schemes.In the simulations underlying Appendix Table 2, we employ the prototype Q-School structure summarized in Table 7 and do not seed Pre-qualifying withany highly-skilled players from the PGA TOUR such as Tiger Woods andplayers from quintiles 1-5.

Panel A of Appendix Table 2 shows the same success rates for skillgroups 1-6 as reported in Table 10. In Panel B, all κ values are set to 1.0,which has the effect of not spreading the distributions of mean scores fornon-PGA TOUR players relative to those of the base PGA TOUR samples

43





from which they are drawn. Although not shown in the table, this alterna-tive parametrization substantially reduces the standard deviation of round-course-adjusted scores. The qualifying success rates are generally higher inskill groups 1-3 and lower in groups 4-6 compared with those associated withthe original calibrated parameters.

In Panel C, the qualifying success rates reflect the calibrated parametriza-tion, except that Pre-qualifying does not include a group of players who haveno realistic chance of qualifying for the TOUR. Here, the qualifying successrates are essentially the same as those associated with the calibrated parame-ters.

In Panel D, the qualifying success rates reflect the calibrated parametriza-tion, except that the intercept in equation 4 is increased by 1.0. This has theeffect of increasing the standard deviation of round-course-adjusted scores fornon-PGA TOUR players but again, the qualifying success rates are essentiallythe same as those associated with the calibrated parameters.

In Panel E, the qualifying success rates reflect the calibrated parametriza-tion, except that all mean displacement values, φ, are doubled. This creates agreater difference in skill among player groups, thereby increasing the successrates of the most highly-skilled players. Nevertheless, these success rates aresufficiently similar that we would change the basic conclusions of this study.

In Panel F, the qualifying success rates reflect the calibrated parametriza-tion, except that all mean displacement values, φ, are halved. This creates asmaller difference in skill among player groups, thereby decreasing the quali-fying success rates of the most highly-skilled players.

In Panel G, the qualifying success rates reflect the calibrated parametriza-tion, except that all κ values are doubled. This has the effect of creatinga significant amount of spread in the distributions of mean scores for non-PGA TOUR players relative to those of the base PGA TOUR samples fromwhich they are drawn, thereby creating substantially greater differences inskill between highly-skilled and lesser-skilled players. As a result, qualifyingsuccess rates are substantially higher than those associated with the standardparametrization. But again, we do not believe that these success rates are sohigh that we would change the basic conclusions of this study.

Finally, in Panel H, the qualifying success rates reflect the standardparametrization, except that all κ values are halved. This has the effect ofcreating substantially less spread between the distributions of mean scores fornon-PGA TOUR players relative to those of the base PGA TOUR samplesfrom which they are drawn. Here, the qualifying success rates are generallyhigher in skill groups 1-3 and lower in groups 4-6 compared with those associ-ated with the original calibrated parameters.

44


DOI: 10.2202/1559-0410.1308



Taken as a whole, our general conclusion that many highly-skilled play-ers would not qualify for the PGA TOUR through Q-School appears to be rel-atively insensitive to the calibration parameters employed in our simulations.

Appendix Table 2: Qualifying Success Rates with Changes in SimulationParameters

Panel A: Original Calibration Parameters Panel B: All κ = 1Skill Lowest Lowest Stage-3 Skill Lowest Lowest Stage-3

Group Scoring 25 Scoring 75 Participant Group Scoring 25 Scoring 75 Participant1 30.7 62.0 80.0 1 35.2 70.2 87.72 19.0 47.2 70.0 2 21.1 52.3 75.13 12.8 37.0 60.0 3 14.6 42.0 67.74 10.0 31.2 57.5 4 9.7 32.8 60.25 6.9 23.6 47.8 5 5.8 23.0 47.86 4.8 18.2 40.2 6 3.4 15.2 36.6

Panel C: Leave Out Bimodal Group (γ = 0) Panel D: equation 4 Intercept Increased by 1.0Skill Lowest Lowest Stage-3 Skill Lowest Lowest Stage-3


Panel E: All φ Doubled Panel F: All φ HalvedSkill Lowest Lowest Stage-3 Skill Lowest Lowest Stage-3


Panel G: All κ Doubled Panel H: All κ HalvedSkill Lowest Lowest Stage-3 Skill Lowest Lowest Stage-3


Results based on 1,000 simulation trials using the prototype simulation structure summarized inTable 7 without seeding Pre-qualifying with any highly-skilled players from the PGA TOUR. PanelA results based on calibrated simulation parameters summarized in Appendix Table 1. Results in allother panels based on calibrated parameters except for changes indicated. Skill group 1 consists of thefirst 25 players per simulation trial based on the ranking of individual player skill. Group 2 consistsof the next 25 players, etc. Only the first six groups are shown.

45





References

Appleton, David R., 1995, May the Best Man Win? The Statistician 44, no.4:529 - 538.

Berry, Scott M. 2001. How Ferocious is Tiger. Chance 14, no. 3:51-56.

Berry, Scott M., C. Shane Reese and Patrick D. Larkey. 1999. Bridging Dif-ferent Eras in Sports. Journal of the American Statistical Association94, no. 447:661-676.

Birkinshaw, Julian. 2001. Strategies for Managing Internal Competition. Cal-ifornia Management Review 44, no. 1:21 - 38.

Brown, Jennifer. 2011. Quitters Never Win: The (Adverse) Incentive Ef-fects of Competing with Superstars. Unpublished manuscript, KelloggGraduate School of Management, Northwestern University, Evanston,IL.

Caulkins, Jonathan P., Arnold Barnett, Patrick D. Larkey, Yuehong Yuan,Jesse Goranson. 1993. The On-Time Machines: Some Analyses ofAirline Punctuality. Operations Research 41, no. 4:710-720.

Clarke, S. R., J. M. Norman, and C. B. Stride. 2009. Criteria for a Tourna-ment: the World Professional Snooker Championship. Journal of theOperational Research Society 60, no. 12:1670 - 1673.

Connolly, Robert A. and Richard J. Rendleman, Jr.. 2008. Skill, Luck andStreaky Play on the PGA Tour. The Journal of The American Statis-tical Association 103, no. 1:74-88.

Connolly, Robert A. and Richard J. Rendleman. 2009. Dominance, Intimida-tion and ‘Choking’ on the PGA Tour. Journal of Quantitative Analysisin Sports 5, no. 3:1-32.

Connolly, Robert A. and Richard J. Rendleman. 2011. What it Takes to Winon the PGA TOUR (If Your Name is “Tiger” or if it isn’t). Interfacesforthcoming.

Davenport, Thomas H. and John C. Beck. 2000. Getting the Attention YouNeed. Harvard Business Review 78, no. 5:118 - 126.

David, H. A. 1998. The Method of Paired Comparisons. Oxford Press, NewYork.

46


DOI: 10.2202/1559-0410.1308



Frick, Bernd. 2003. Contest Theory and Sport. Oxford Review of EconomicPolicy 19, no. 4:512 - 529.

Gibbons, Jean Dickinson, Ingram Olkin, and Milton Sobel. 1977. Selectingand Ordering Populations: A New Statistical Methodology. Wiley, NewYork.

Fletcher, R. and C.M. Reeves. 1964. Function Minimization by ConjugateGradients. Computer Journal 7, 2:148-154.

Fu, Qiang and Jingfeng Lu. 2009. The Beauty of Bigness: On OptimalDesign of Multi-Winner Contests. Games and Economic Behavior 66,no. 1:146-161.

Hvide Hans K. and Eirik G. Kristiansen. 2003. Risk Taking in SelectionContests. Games and Economic Behavior 42, no. 1:172 - 179.

Lazear, Edward P. and Sherwin Rosen. 1981. Rank-Order Tournamentsas Optimum Labor Contracts. Journal of Political Economy 89, no.5:841-864.

Ljung, G. M. and Box, G. E. P. 1978. On a Measure of Lack of Fit in TimeSeries Models. Biometrika 65, no. 2:297-303.

McGarry, T. and R. W. Schutz. 1997. Efficacy of Traditional Sport Tourna-ment Structures. Journal of the Operational Research Society 48, no.1:65 - 74.

McLaughlin, Kenneth J. 1988. Aspects of Tournament Models: A Survey.Research in Labor Economics 9, no. 1:225 - 256.

Narayana, T. V., 1979, Lattice Path Combinatorics with Statistical Applica-tions. University of Toronto Press, Toronto.

Prendergrast, Canice. 1999. The Provision of Incentives in Firms. Journal ofEconomic Literature 37, no. 1:7 - 63.

PGA TOUR. 2008. Application to Enter 2008 PGA TOUR Qualifying Tour-nament (for the PGA TOUR and Nationwide Tour) Conducted by thePGA TOUR.

Rosen, Sherwin. 1986. Prizes and Incentives in Elimination Tournaments.American Economic Review 76, no. 4:701 - 715.

Ryvkin, Dmitry. 2010. The Selection Efficiency of Tournaments. EuropeanJournal of Operational Research 206, no. 3:667 - 675.

47





Ryvkin, Dmitry. and Andreas Ortmann. 2008. The Predictive Power ofThree Prominent Tournament Formats. Management Science 54, no.3:492-504.

Sirak, Ron. 2008. The Rich get Richer: The Money on Tour gets anotherBoost. No Wonder Tiger’s Approaching $1 billion. Golf Digest 59, no.2:96-100.

Wang, Y., 1998, Smoothing Spline Models with Correlated Random Errors.The Journal of The American Statistical Association 93, no. 1:341-348.

48


DOI: 10.2202/1559-0410.1308



Documents

Journal of Quantitative Analysis in Sportsbobconnolly.web.unc.edu/files/2017/09/Journal-of...w o udl have only an 80% c h a n c e of qualifying. A t the same time, players of m u c