Report of the NACAC Ad Hoc Committee on U.S. News & World Report Rankings
September 23, 2011
2
ContentsI. Introduction 3
II. RankingsHistoryandBackground 4
III. TheNACACDiscussion 7
IV. MethodologyMatters 10
V. DistortingEffectsandProfessionalPractice 15
VI. TakingtheNextSteps 20
VII. AdHocCommittee’sRecommendations 23
VIII. Appendix 24
IX. Bibliography 25
3
IntroductionIn2010,theNationalAssociationforCollegeAdmissionCounseling(NACAC)convenedagroupofmemberstoform
anAdHocCommitteeonU.S.News&WorldReportRankings.TheCommitteewasconvenedtoconductdiscussions
withU.S.Newsstaffforthepurposeofofferinganorganizationalconduitthroughwhichtoexchangeideas,convey
concerns,andrespondtoquestionsabouteachorganization’srespectiveconstituency.
TohelpinformtheAdHocCommittee,NACACconductedasurveyofassociationmembersinMay2010togauge
attitudesofcollegeadmissioncounselingprofessionalstowardtheU.S.News&WorldReportrankingspublication,
“America’s Best Colleges.”1 This final report incorporates survey findings from the member survey, as well as
informationgleanedfromcommitteediscussionsandmeetingswithmembersofNACAC’sstateandregionalaffiliates
fromaroundthecountry.
Forpurposesof this report, the term“Committee”refersonly to theNACACmemberswhoserveon theAdHoc
Committee.Representatives ofU.S.News&WorldReportmeetwith theAdHocCommittee to ensure an open
dialoguewiththeassociation,buttheirviewsarenotrepresentedinthisreport.Fortheofficialpublicsummariesof
meetingsbetweenbothNACACandU.S.News&WorldReportrepresentatives,visittheAdHocCommittee’sWeb
page at http://www.nacacnet.org/AboutNACAC/Governance/Comm/Pages/NACACUSNewsAdHocCommittee.aspx.
TheCommitteewouldliketoexpressitsappreciationtoRobertMorse,DirectorofDataResearchatU.S.News&
WorldReport,forhisparticipationindiscussionswiththeAdHocCommittee.
Committee Members
PeterCaruso,Chair
BostonCollege,MA
BruceChamberlin
GeorgetownUniversity,DC
RafaelS.Figueroa
AlbuquerqueAcademy,NM
PamHorne
PurdueUniversity,IN
IreneLogan
VirginiaStateUniversity
LeeMelvin
UniversityofConnecticut
JosephPrieto
HinsdaleCentralHighSchool,IL(Retired)
KrisGettingRoach
UniversityofSt.Thomas,MN
MichaelSexton
SantaClaraUniversity,CA
4
Rankings History and BackgroundThe Evolution of Modern Rankings
Throughoutthelatenineteenthcentury,theU.S.BureauofEducationpublishedannualreportscontainingstatistical
dataoncollegesanduniversitiesacrossthenation.Followingthediscontinuationofthesereportsin1890,various
(andvaried)attemptshavebeenmadetorankinstitutionsofhighereducationbasedonacademicquality.2TheU.S.
News&WorldReport’srankingspublication,thoughunprecedentedinitspopularity,isinmanywaysaproductof
therankingsthatcamebeforeit.
Historically,itwascommoninEuropetocountthenumberofeminentmenassociatedwithauniversityinorderto
measure itsworth.Thismethodwasuse in thefirstAmericancollege rankings in1910andremained influential
throughthemid-1960s.4 At leastninesuchrankingswerepublishedbetween1910and1964byJamesCattell,
thefirsttorankU.S.collegesanduniversities,andbyothers.5ReputationalrankingswereintroducedbyRaymond
Hughesin1925.Theirprominencedevelopedslowlyovertime,eventuallyeclipsingthatoftheprevious“eminent
men”rankings.Regardlessofthemethodology,however,mostearlyrankingswereusedprimarilywithininstitutionsof
highereducationandassociationsofcollegesanduniversities.Theygarneredonlyminimalattentionfromthepublic
upuntiltheintroductionoftheU.S.News&WorldReportrankingspublication.6
2Walleri,R.DanandMarshaK.Moss,EvaluatingandRespondingtoCollegeRankings,1995.3CompiledfromWalleri,1995;Meyers,LukeandJonathanRobe,“CollegeRankings:History,CriticismandReform,”CenterforCollegeAffordabilityandProductivity,2009;Rauhvargers,Anrejs,“GlobalUniversityRankingsandTheirImpact,”EuropeanUniversityAssociation,2011;andWebster,David,“ReputationalRankingsofColleges,Universities,andIndividualDisciplinesandFieldsofStudy,FromTheirBeginningstothePresent,”AgathonPress,1992.4Meyers,2009.5Ibid.6Walleri,1995;Meyers,2009.
Table 1. Timeline of Major Developments in Academic Quality College Rankings3
1910 Psychologist James Cattell publishes the first American college rankings in American Men of Science based on the number of identified "eminent men" who had earned degrees from, or were employed at, the ranked institutions.
1925 Chemistry professor Raymond Hughes publishes the first reputational rankings in A Study of the Graduate Schools of America.
1934 Hughes, now chair of the American Council of Education, publishes a second, more extensive, set of reputational rankings of graduate schools.
1957 Journalist Chesly Manly publishes six college rankings in the Chicago Tribune: ten best universities, co-educational colleges, men’s colleges, women’s colleges, law schools and engineering schools.
1959 Humanities professor Hayward Keniston reintroduces reputational rankings in the appendix of a report prepared for the University of Pennsylvania.
1966 Allan Cartter, with the American Council for Education, publishes An Assessment of Quality in Graduate Education, also known as the Cartter Report. It is based on reputational surveys, features improved methodology and sells approximately 26,000 copies.
1967 Jack Gourman begins publishing a college ranking titled The Gourman Report, which applies an undisclosed methodology and is widely criticized for its statistical anomalies.
1970 Kenneth Roose and Charles Anderson replicate Cartter’s methodology in A Ranking of Graduate Programs and attempt to downplay the “pecking order” of institutions.
1973 Peter Blau and Rebecca Margulies publish a reputational ranking of professional schools. 1977 Allan Cartter and Lewis Solomon publish another reputational ranking of professional schools in
response to dissatisfaction with the Blau and Margulies rankings. 1981 Lewis Solomon and Alexander Astin publish an undergraduate reputational ranking. 1982 The National Academy of Sciences and National Research Council publish the Assessment of
Research-Doctorate Programs in the United States, which includes ratings, rather than rankings, of institutions, and is based on both reputational survey responses and quantifiable data.
1983 U.S. News & World Report first publishes its college and university rankings. 2003 Shanghai Jiao Tong University publishes the first global university rankings.
5
Encouragingly,thehistoryofreputationalcollegerankingsismarkedbyvariousimprovementsinmethodologyand
expansionsinscope.Earlyreputationalsurveyswerecarriedoutonaverysmallscaleandthereforeproducedless
reliableresultsthatwerelessinclusive.Forexample,Hughes’firstranking(1925)wasbasedsolelyontheresponses
ofMiamiUniversityfacultymemberswhowereaskedtolist40-60instructorswhotaughtintheirdisciplineandto
ratethequalityofthedepartmentsofonly36institutions.7AlanCartter’s1966rankings,conversely,werebasedon
surveyssenttoseniorandjuniorscholarsanddepartmentchairpersonsfromacrossthenation.Respondentswere
askedtorateinstitutionsseparatelyonthequalityoftheirgraduatefacultiesandthequalityoftheirdoctoraltraining
programs,addingalayerofspecificitytotherankings.Cartter’sfinalproductincluded106institutions,morethanany
previousranking.8In1982,theNationalResearchCouncil’sratingscombinedquantitativedatawiththeresultsof
reputationalsurveysregarding228institutions,makingitmorecomprehensivethananypreviousefforttomeasure
academicqualityatcollegesanduniversitiesintheU.S.9
TheU.S.News&WorldReportrankingspublicationhas,sinceitsfirstpublicationin1983,gonethroughsignificant
changes and developments. The first three editions were based entirely on reputational surveys sent to college
presidentsandwerereleasedeveryotheryear(1983,1985and1987).In1988,anumberofimportantprecedents
wereset: therankingspublicationbegantobereleasedonanannualbasis, itsreputationalsurveysweresent to
academicdeansandadmissionofficersaswellascollegepresidents,and itbased75%of therankingsonnon-
reputationalinputandoutputvariables.Aroundthesametime,U.S.Newsalsobeganreleasingtheirrankingsina
standaloneguidebook,whichprovidedmoreinformationthantherankingsissueofthemagazine.10
Sincetheybeganrankingcolleges,U.S.Newshasimprovedtheintegrityoftherankingsbyconfirmingself-reported
datawithothersourcesandstandardizingandclarifyingvariabledefinitions.Theyalsobeganincludinginformation
onunrankedinstitutionsandroundingaggregatescoresuptowholenumbers(ratherthanfirstdecimalplaces)to
determineties.Duringthemid-1990s,outputmeasuresreceivedaddedemphasis,astheweightonthegraduation
ratevariablewasincreasedandthe“valueadded”(nowreferredtoas“graduationrateperformance”)variablewas
added.Inthe2004editionof“America’sBestColleges,”U.S.Newsrespondedtogrowingcriticismfromthosein
highereducationandabandoned theuseof theyield variable indeterminingselectivity.11 Thepopularityof the
publicationgrewasthemethodologywasrefined.AccordingtoCollege Rankings: Democratized College Knowledge
for Whom?,“USNWRsold485,000reprintsofthatfirstissueandcurrentlysellsanestimated2.3millionofitsregular
collegerankingsissueandanestimated700,000copiesofitsstand-alonecollegerankingsguide.”12
The Effects of Rankings and Our Desire for Order
Rankings seem to fulfill twodemands inmodern society, oneperhapsmoreunderstandable thananother. First,
consumersviewrankingsasamethodfordeterminingvalue.InformationaboundsintheInternetage,thoughfinding
atrustedsourcetomakesenseoftheinformationcanbeadifficulttask.Indeed,theInstituteforHigherEducation
Policy(IHEP)notesthat“stakeholdersgravitatetowardsystemsthatprovidesomeinterpretationoftheinformation”
available about postsecondary education.13 Second, many people, as evidenced by the commercial success of
U.S.News’“America’sBestColleges,”craveanauthoritativesourcetotellthemwhatis‘best,’whoisnumberone.
Rankorderingtoastersorcarsmaymakesomesenseinthiscontext,astherearemoreorlesstangibleoutcomes
whichdepend lessonhuman tendenciesandmoreonmechanicalprecision.However,collegesanduniversities
arecomplex institutionswithmultiplepurposes that, toa largeextent, relyonhuman interactionsand individual
determinationtofunction.
7Meyers,2009.8Walleri,1995;Meyers,2009.9Walleri,1995;10Sanoff,AlvinP.,“TheU.S.NewsCollegeRankings:AViewfromtheInside,”InstituteforHigherEducationPolicy,2009;Meyers,2009.11Ibid.12McDonough,PatriciaM.,etal,“CollegeRankings:DemocratizedCollegeKnowledgeforWhom?”ResearchinHigherEducation,1998,Vol.39.5.13Sponsler,BrianA.,“IssueBrief:TheRoleandRelevanceofRankingsinHigherEducationPolicymaking,”InstituteforHigherEducationPolicy,2009.
6
Thetroublewithrankingsuchcomplexinstitutionsusingsimplemeasuresiswell-documented.Rankingscanbeself-
fulfillingprophecies,theprocessbywhichreactionstosocialmeasuresconfirmtheexpectationsorpredictionsthat
areembeddedinmeasuresorwhichincreasethevalidityofmeasuresbyencouragingbehaviorthatconformstoit.14
Evidenceoftheself-fulfillingprophecyisfoundintheinfluenceofpriorrankingsonresponsestopeerreviewsurveys,
acorecomponentoftheUSNWRrankings.
Rankingscanalsoimproperlyreduceandsimplifycomplexconcepts,decontextualizinginformationthatcouldbe
essentialtoconsumers.AnexcellentexampleofthedecontextualizingeffectsoftheUSNWRrankingswasexpressed
bynumerous respondents to theNACACmembersurvey. In response to thequestion,“Does the title ‘America’s
BestColleges’accuratelydescribethecontentsofthepublication?,”NACACmembersresponded,“Bestforwhom?”
Rankings, according to Sauder and Espeland, are “the culmination of many commensurative practices,” where
complex concepts are reduced to single data points that are deemed indicative of quality or success.15 Such
commensurationleadstoscrutinyoverminisculedifferences,aphenomenoncommonlyascribedtotherankordering
ofinstitutionsusingtheUSNWRmethodology.
Thecumulativeeffectsoftherankings,combinedwiththedistillationofjudgmentsaboutqualityintodatapointsof
varyingdegreesofdefensibility,createquestionableincentivesforinstitutionsandleadtosignificantconfusionamong
consumersaboutthedifferencesbetweeninstitutions.Suchconfusionamongconsumersriskscreatingmismatches
betweenstudentsand institutions,aswellas reputationalbiases thatmaydiscouragestudents frompursuingan
educationataninstitutionthatmaybethe“best”collegefortheirneedsortastes.
14Sauder,MichaelandWendyNelsonEspeland,“RankingsandReactivity:HowPublicMeasuresRecreateSocialWorlds,”AmericanJournalofSociol-ogy,2007,Vol.113.1.15Ibid.
7
The NACAC DiscussionThe Ongoing Discussion
Thediscussionaboutacademicrankingsandtheireffectsisextensiveandongoing.Throughoutmuchofthelast
decade,membersofNACAC,aswellasotherobservers,haveexpressedamyriadofconcernsaboutrankingsona
consistentbasis.TheNACACBoardofDirectorsviewedtheappointmentofanAdHocCommitteeasanopportune
momenttoconsolidatethemembers’concernsintoasingle,organizedeffort.
Thiscommittee’sobservationsandrecommendationsarebestseenassymptomaticoflong-standingconcernsthat
lingerinourprofessionalcommunity,aswellasothers.Aninitialandexpectedobservationofthecommitteewasa
generaldislikefortherankingsoverall.AspartoftheCommittee’ssurvey,NACACmemberswereaskedtoindicate,
onascalefromoneto100,theirgeneralattitudetowardtheU.S.News&WorldReportrankings.Ascoreofone
representsastrenuousobjectiontotheU.S.Newsrankings,while50representsacompletelyneutralattitudeand
100indicatesstrongsupport.Themeanscores,whicharepresentedinFigure1,revealgenerallynegativeopinions
oftherankingsamongrespondents.
Highschoolcounselors(meanscoreof28.67)expressedlowerregardfortherankingsthancollegeadmissionofficers
(meanscoreof38.54),butbothgroupsheldnegativeviews(meanscoreunderfifty)towardtherankings.16Public
highschoolcounselorsviewedtherankingsslightlymorecharitably(meanscoreof35.11)thanprivatehighschool
counselors(meanscoreof23.92).
How Influential Are the Rankings?
Ifcollegeadmissioncounselingprofessionalsare,onthewhole,negativelyinclinedtowardthem,istheinfluenceof
rankingssuchthatweshouldbemoreconcerned,orlessconcerned,abouttheireffectsoncollegeadmissionand
counseling?NACACmemberswereaskedabouttheiropinionsonthechangeinprominenceoftheU.S.News&
WorldReportrankingsoverthelastfiveyears.AsFigure2suggests,amajorityofNACACmembersbelievethatthe
prominenceoftheU.S.News&WorldReportrankingshasincreasedoverthepastfiveyears.
0 20 40 60 80 100
Figure 1. Average Perceptions of U.S. News Rankings on "Feeling Thermometer"
College
High School
Total
16Toensurethatthemeanwasnotmaskinghighlypolarizedopinions,weexaminedthemedianscoresaswell.Highschoolcounselors’medianfeelingthermometerscorewastwentyfive,whilecollegeadmissionofficers’medianscorewasthirtyfive.
8
Best for Whom?
NACACmembersexpressedsomethingapproachingaconsensusonthequestionofwhetherthetitleofU.S.News
&WorldReport’sannualpublication, “America’sBestColleges,”accurately represents the informationpresented
therein.
Only2.9percentofall respondents(2.4percentofhighschoolcounselorsand3.3percentofcollegeadmission
professionals)believedthatthetitleofthepublicationaccuratelyrepresentsthecontentdeliveredbythepublication.
Themajorityofcollegeadmissionofficers(51.3percent)andhighschoolcounselors(61.9percent)reportedthatthe
titleisnotatallaccurate(Figure3).
AsFigure4indicates,publichighschoolcounselorswereslightlymorelikelytobelievethatthetitleatleast“somewhat
accurately”describesthecontentinthepublication.
0.0%5.0%
10.0%15.0%20.0%25.0%30.0%35.0%40.0%
Muchmore
Somewhatmore
Same Somewhatless
Much less
Figure 2. Change in Prominence of USNWR Rankings Over Past Five Years
College
High School
Total
0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%
Accurately Somewhataccurately
Not at allaccurately
Figure 3. How accurately does the title "America's Best Colleges" describe content in
publication?
College
High School
Total
9
Morethan600NACACmembersofferedcommentsonthisquestioninadditiontotheirmultiplechoiceresponses.
The most common themes in the open ended responses add substance to the general notion that the title of
“America’sBestColleges”doesnotaccuratelyconveytheinformationcontainedinthepublicationtoconsumers.
Commonthemesincluded:
• “The Best for Whom?”—Many members stated that the best college for an individual student will be
determinedbythequalityoffitbetweeninstitutionandstudent.
• “What’s inaRank?”—Membersargued that thedifference innumeric rankbetweencolleges isat least
somewhatarbitrarybyvirtueoftheweightingsystemusedinthemethodology,thatmakingsubsequentrank
orderdistinctionsbetweencollegesdoesnotprovethatonecollegeis“better”thananother,andthatthe
weightsofthefactors,whenchanged,haveproducedandwillproducedifferentrankings.
• Inputsvs.Outputs—Membersfinditdifficulttoexplainwhattheterm“best”describes—manybelievethat
therankings’useof“input”variables(includingselectivityandtestscores)andothervariablesnotrelated
todirectlymeasurableoutputs(suchasthepeerassessments)leadconsumerstomakedecisionsbasedon
informationunrelatedtothequalityofeducationprovidedattheinstitution.
Overarching Themes
During the committee’s discussion, review of the member survey, and review of the academic literature, three
overarchingthemesemerged.
1) Methodology matters. Ordinalrankingscreateflawedconclusionsandresultingmisperceptionsabout
differencesincollegequality.
2) Distorting Effects and Professional Practice. Whilethereislittleconclusiveevidenceofwidespread
gamingofrankings,thetendencytoconformtorankingsmethodologycreatesincentivestofocus
disproportionateresourcesondataelementsthatcanchangerankingswithoutnecessarilychanging
thequalityoftheinstitution.Moreover,surveyresultsandCommitteediscussionssuggestedtheneedfor
moreprofessionaleducationandconsumerresourceswithregardtorankingspublications.
3) Taking the Next Step. Asthehistoryofrankingssuggests,changeisagradualandincremental
process—butonethattakesplacenonetheless.Therearemanyrecommendationsforchange,including
recentrecommendationsfromtheAmericanAssociationofCollegiateRegistrarsandAdmissionsOfficers
(AACRAO),thattheCommitteesupports.ThisCommitteefoundparticularinterestintheideaofcorrecting
formethodologicalflawsbyde-emphasizingthesingle,ordinallistthatU.S.News&WorldReportpublishes,
andinsteadencouragingmore‘openaccess’useofthedatatoallowstudentsandfamiliestocreatetheir
owncollegelistsusingcriteriatheyfindimportant.
4) Committee Recommendations. ThefinalsectionofthereportofferstheCommittee’srecommendationsas
aresultofitsdiscussions.
0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%
Accurately Somewhataccurately
Not at all accurately
Figure 4. How accurately does the title "America's Best Colleges" describe content in publication?
HS Results Only
Public
Private
10
Methodology MattersAccordingtoaNationalOpinionResearchCenter(NORC)evaluationoftheU.S.News&WorldReportrankingsof
undergraduate institutions,“theprincipalweaknessof thecurrentapproach is that theweightsused tocombine
thevariousmeasuresintoanoverallratinglackanydefensibleempiricalortheoreticalbasis.”17Indeed,thereisno
objectivewaytoselectofcriteriaforinclusioninarankingmethod.Oncethecriteriaareselected,thereisfurtherno
objectivemannerforassigningweightstoeachcriterion.Asaresult,thereareaninfinitearrayofpossiblerankings
for colleges anduniversities.Moreover, because thecriteriaused to rankcolleges oftendonotdifferwidely, the
placementofweightstodifferentiateinstitutionswithsimilarcharacteristicsmakesspecificplacementinaranking
“essentiallyarbitrary.”18
The Institute for Higher Education Policy (IHEP) notes that “ordinal rankings are inherently misleading...[T]rue
differencesamongcloselyrankedinstitutionsmaybeminimalorquitelarge,butallrankingarepresentedashaving
thesamemagnitudeofdifference.”19
IHEPfurthernotesthat“[t]heinputsincommonrankingsystems—andtheindicatorsselectedtomeasurethem—
reflectimplicitlyvalue-ladendecisionsabouthowtoappropriatelydefineeducationalquality.Mostoftheindicators
used in theconstructionofcollegerankingshave little todowithpolicygoals relating toaccessandequity; they
createuniformnotionsofeducationalqualityandoverlookimportantdistinctionsineducationalpreparation,personal
experiences,andhistoricaltreatmentofvariousstudentpopulationsinhighereducation.Policymakersandthepublic
areill-servedbyrankingsthatrelyondataindicatorsthatbytheirnatureareexclusionary.”20
Student Selectivity
Withitsrankingmethodology,USNWRturnsahighlysubjectivecollectionofintangiblesintoonequantifiablevalue
thatcanbearrangedagainstotherlikevalues.Theresultingorderedlistattractsaneagerpopulationofinformation
seekerswithitssimpleapproachtoquantifyingaveryimportant,andexpensive,decision.Themethodologyisthe
backboneofthe“BestColleges”publication,andconsumersoftheinformationshouldunderstanditscomponents
beforeusingitasacollegesearchtool.Tounderstandthemethodologyistounderstandwhicheducationalvariables
USNWRdeems themost importantmeasuresof institutionalquality. Inacompanionpiece to the2010rankings
issue,USNWReditorialstaffcalledtherankingsa“startingpointforthecollegesearch.”Therankingsactuallyreflect
moreofanopinionofonenewssourceabouttheappropriatemeasuresofeducationalquality.
ThecurrentUSNWR“BestColleges”rankingformulaisbrokendownintoseveralprimarycomponents:undergraduate
academicreputation,studentselectivity,facultyresources,graduationandretentionrates,financialresources,alumni
giving,andgraduationrateperformance(thedifferenceoftheactualsix-yeargraduationrateandtheratepredicted
byUSNWR).Eachof thesecategories isbrokendown into “subfactors” that occasionallydiffer according to the
institutiontype.Table2focusesontheselectivityandreputationalcomponents.
17NationalOpinionResearchCenter,“TheNORCReportonU.S.NewsandWorldReport,”WashingtonMonthly,1997.18Huggins,PeterandLiorPachter,“SelectingUniversities:PersonalPreferenceandRankings,”CornellUniversity,2008.19Sponsler,2009.20Ibid.
11
This assemblage of factors and subfactors represents the data chosen by USNWR staff to measure institutional
quality,butthepresenceofthesevariablesalsohighlightstheabsenceofotherintangibles.Dataonteachingquality,
studentinvolvement,andabilitytoobtainajobdoesnotexistinapublication-readyformat,soUSNWRfindsother
measurablefactorstorepresentacademicrigor.USNWRcombinesinputvariables(standardizedtestsandclassrank),
processvariables(facultyresources)andoutputmeasures(graduationrates)togenerateafinalrating.However,the
newspublicationdoesnot includedetailedstatistical analysis thatwould lend thenecessary scientificcredibility.
JeffreyEvansStakemakesthispointinananalysisoftheimpactlawschoolrankingshaveoninstitutionalpolicy:
Likemanyrankings,thosepublishedbyU.S.Newsarebasedonanumberoffactors,mostofwhich
makeuseofreadilyavailabledata.Manyofthesecriteriaareofnoinherentinteresttothereadersofthe
rankings.Aprospectivelawstudentwantingtoworkasalawyerhasnoparticularinterestintheamountof
moneyaschoolspends,thenumberofvolumesinthelibrary,thegradesoftheotherstudentsinthe
class,oreventhereputationoftheschoolamongacademicsbecausesuchreputationsarebuiltprimarily
onfacultypublicationsandnotteachingquality.22
Furthermore,theheavyrelianceoninputvariablessuggestsamisinterpretationofthepurposeofhighereducation.
Wheninterpretingtherankingsasatruemeasureofinstitutionalquality,oneequatestheeducationalinputsthesame
astheeducationaloutputs.23Thus,themissionoftheuniversitybecomesirrelevant.
Table 2. Weights of Selectivity and Reputational Components of USNWR Methodology21 Ranking Category
Category Weight Subfactor Subfactor Weight National Universities and National Liberal Arts Colleges
Regional Universities and Regional Colleges
National Universities and National Liberal Arts Colleges
Regional Universities and Regional Colleges
Undergraduate Academic Reputation
22.5% 25% Peer assessment survey
66.7% 100%
High School Counselors’ rating
33.3% 0%
Student selectivity for fall 2010 entering class
15% 15% Acceptance rate
10% 10%
High school class standing in top 10%
40% 0%
High school class standing in top 25%
0% 40%
Critical Reading and Math portions of the SAT and the composite ACT scores
50% 50%
21U.S.NewsStaff,“BestColleges2012:AbouttheRankings/Methodology,”U.S.News&WorldReport,2011.22Stake,JeffreyEvans,“TheInterplayBetweenLawSchoolRankings,Reputations,andResourceAllocation:WaysRankingsMislead,”IndianaLawJournal,2006,Vol.81.299.23Henderson,WilliamD.andAndrewP.Morriss,“StudentQualityasMeasuredbyLSATScores:MigrationPatternsintheU.S.NewsRankingsEra,”IndianaLawJournal,2006,Vol.81.163.
12
Studentselectivityaccountsfor15%oftheoverallranking.Asidefromtheacceptancerate,twounreliablemetrics
wieldmostoftheinputvariable’semphasisontherestoftheranking:classrankandstandardizedtestscores.Ina
2009surveyofadmissionprofessionals,only16percentofrespondentsattributedconsiderableimportancetoclass
rank,24afactorthatcomprises40percentoftheselectivityrating.Theinclusionofclassrankinthemethodology
promotesacriterionthatpenalizesstudentsforselectingacademicallyrigoroussecondaryschools.25
Whilesignificantlymorerespondentsratedstandardizedtestsasconsiderablyimportant(58percent)intheadmission
process,industryexpertshaveagreedontheirexclusionfromanyevaluativemeasuresofeducationalsystems.The
two principle test developers, the College Board and ACT, discourage the use of their test data as measures of
educationalquality.TheNACACTestingCommission,madeupof19admissionandcounselingprofessionalsfrom
bothpublicandprivateinstitutions,concluded“thatthecontinueduseofadmissiontestscoresasaranking-related
measurecreatespressureonadmissionofficestopursueincreasinglyhightestscores.”
BecauseUSNWRmustrelyonquantifiablemeasureslikestandardizedtestdataandclassranktoofferanumerical
schoolrating,intrinsicethicaldilemmasarisefromthecollaterallyproducedincentives.Astheprevalenceofrankings
increases,sodoestheirpressuresonthepolicydecisionsofpostsecondaryadministrators.Admissiondeansand
directorsarefacedwiththechoiceof leadingtheircollegeoruniversityaccordingto institutionalphilosophyor to
rankingmethodologies.
TheimpactoftheLawSchoolAdmissionTest(LSAT)onlawschooladmissionpolicyprovidesatemplateforpolluting
theholisticadmissionprocess.LawschoolshavebeenclingingprecipitouslyclosetotheirrankingintheUSNWR
guide,andoneoftheeasiestmethodstoimproveaninstitution’sstandingintheannuallististhroughthemedian
LSATscore.“Notonlyisittangibleandinternalinawaythattheothervariablesarenot,itiscertain,andthatcertainty
makesitimportantinwaysdifferentfromtheothervariables.”26Wheninstitutionsneedabumpofseveralpointsin
theirmedianscore,theycanquicklyturntheirattentiontonextyear’sapplicantpool.Lendingunduesignificanceto
testscoresinanevaluativeformulaencouragesthesubjectsoftheevaluationtodothesame.
While test scores provide a solution to the largely immeasurable metrics of the college experience, rankings
publicationswouldbeill-advisedinusingthemfortheirratings.Sincethereisastandarderrorofmeasurementfor
standardizedtests,usingthemcomparativelyinarankingformulaisreckless.Thestandarderrorofmeasurementfor
theSATMathematicsandCriticalReadingsectionsis30pointsandabout1pointfortheACTcompositescore.The
CollegeBoard’sGuidelinesontheUsesofCollegeBoardTestScoresandRelatedDataclearlystatethatthepractice
of“makingdecisionsaboutotherwisequalifiedstudentsbasedonlyonsmalldifferencesintestscores”shouldbe
avoided.Becausetherankingspublicationsmeasurethousandsofpostsecondaryinstitutions,theyfacetheveryreal
possibilityofcharacterizinginstitutionsbasedonstatisticallyinsignificantscoredifferences.
Reputational Survey
ThereputationalsurveycomprisesasizeableproportionoftheoverallUSNWRranking.ForNationalUniversitiesand
NationalLiberalArtsColleges,thereputationalsurveycarriesa22.5weighting,makingitthemostinfluentialcategory.
ForRegionalUniversitiesandRegionalColleges,itrepresentsone-quarteroftheoverallrating,whichisonlyequaled
bygraduationandretentionrates.InthecontextofUSNWR’srankinghistory,theweightingseemsappropriate.The
veryfirstattemptfromUSNWRtorankcollegesin1983reliedentirelyonareputationalsurvey.The“BestColleges”
guide,however,hasgrowninrelevancesincethen,increasingitssalesofitsregularcollegerankingissuebyover
2millioncopies in its first15years.27Theannual issuesoonoutgrewa standalone reputational survey, and the
publicationbeganupdatingitsmethodology.
24NACACAdmissionTrendsSurvey,2009.25Ehrenberg,RonaldG.,“ReachingfortheBrassRight:TheU.S.News&WorldReportRankingsandCompetition,”TheReviewofHigherEducation,2002,Vol.26.2.26Johnnson,AlexM.,Jr.,“TheDestructionoftheHolisticApproachtoAdmissions:ThePerniciousEffectsofRankings,”IndianaLawJournal,2006,Vol.81.309.27McDonough,1998.
13
Secondaryandpostsecondaryprofessionalsareoverwhelminglyskepticalofthereputationalsurveycomponentof
theUSNWRrankingsmethodology.Inthecommittee’ssurveyofNACACmembers,onlyfivepercentofrespondents
called the peer assessments a “good indicator” of college quality. By comparison, nearly 40 percent of NACAC
membersagreedthatgraduationandretentionrates,similarlyweightedfactorsintheUSNWRrankings,weregood
indicatorsofcollegequality.
Thepeerassessmentsareinherentlysubjective,whichmightexplainthelowopinionratinginthemembersurvey
comparedtoaquantitativefactorlikegraduationrates.TheNORCreviewoftheUSNWRundergraduaterankings
methodologyhighlightsthelimitationsofusingacademicreputationasaproxyforinstitutionalquality.Inadditionto
addressingthefactthatacademicexcellencealonedoesnotdefinetheentirespectrumofstudentneeds,thereport
alsopointedoutthat“itisgenerallyassumedthatreputationschangemoreslowlythanrealchangeininstitutions,
thusovervaluinginstitutionsthat,infact,maybedecliningandundervaluinginstitutionsthatareimproving.”Inother
words,thereputationalsurveydoesnotserveasagoodindicatorofinstitutionalqualityinagivenyearbecausethe
respondent’sassumptionsmaybebasedonpastevaluations.
Whilesomeinstitutionshaveexperiencedpositionalmobilityintherankingsfromyeartoyear,whetherfromachange
ininstitutionalpolicyorachangeintheUSNWRmethodology,therelativelystaticnatureofthetoptierisrevealing
of attitudinal trendsofuniversity administrators. In theUSNWR lawschool rankings,whichalsoweight thepeer
assessmentsat25percent28,nineoutof10institutionsfrom1987to2004remainedinthetop10.29Assigningthe
largestoroneofthelargestweightstoameasurelikethereputationalsurveyisdangerousbecausetheresponses
submitted toUSNWRmay in factbepartofaself-perpetuatingsystem.Theschools rankedhighly in thewidely
distributed“BestColleges”publicationbenefitfromthereflexivenatureofthereputationalsurvey,whileschoolsin
thelowertierarepenalizedbasedonprevious,andpotentiallyoutdated,critiques.This“echoeffect”tendstopullthe
rankinginthedirectionofthepreviousyear’sassessment.30
Onemajorflawinincludingapeerassessment inarankingsmethodologyinvolvestheinabilityofrespondentsto
identifymeaningfulchangeatotherinstitutions.AsstatedintheNORCevaluation,“Thegreatvariancebothacross
andwithininstitutionsmakesitverydifficulttogetconsensusonqualitycriteriaoronmeasuresforundergraduate
programsingeneral,orevenforgroupsofcollegesoruniversitiesthatmightappearsimilar.”Withnoclearmethod
of identifyingeducationalquality, reputational survey respondents are encouraged to focus solely onmoreeasily
identifiablemetrics,likestandardizedtests.AsWilliamHendersonandAndrewMorrissaddressintheirresearchon
changesinmedianLSATscores,thefluctuationsinvisiblefactorslikestandardizedtestshavemoreofanimpact
0.0%
5.0%
10.0%
15.0%
20.0%
25.0%
30.0%
35.0%
40.0%
45.0%
Goodindicator
Fair indicator Poorindicator
Not anindicator
Figure 5. Peer Assessments
College
High School
Total
28Inthelawschoolrankings,onlyselectivityequalsthepeerassessmentsintermsofweight.Thelawschoolrankingsalsoincludeanassessmentscorebylawyersandjudgesweightedat15percent,whichbringsthetotal“qualityassessment”to40percentofthetotalranking.29HendersonandMorriss,2006.30Stake,2006.
14
on academic reputation than other intangibles like curriculum improvements. Therefore, the reliance on peer
assessmentsplacesadditionalweightontheselectivityrating.Onemightaskwhetherstandardizedtestsdatastarted
toaffectreputationsorreputationsstartedtoaffectselectivity,andultimatelystandardizedtests.Sincetherankings
publicationshavepervadedtheconsciousnessofthecollegeadmissionculture,itnolongermatterswhichcamefirst.
Thereputationshavebeenfirmlyestablishedthroughannualpublication.
Common Misconceptions
Thecommitteefoundthattherewassomemisunderstandingofrankingmethodology,aswellassomeperceptionsthat
werenotentirelyaccurate,amongsurveyresponsesandinconversationswithmembers.Respondentsoccasionally
citedtheneedformeasuresthatarecurrentlyincludedintheUSNWRmethodologyandcriticizedtheinclusionof
criteriathathavebeendroppedorchanged.Forexample,onerespondentstated,“Ithinkthe4yeargraduationrate
isabsurd,”when,infact,USNWRusesasix-yeargraduationrate.Similarly,somerespondentscitedtheneedfor
ameasurethattakesincomingstudentcharacteristicsintoaccountwhencomparinginstitutions’graduationrates.
Suchameasure iscurrently included inUSNWR’smethodologyunder thename“graduationrateperformance.”
Yieldwasrepeatedlycitedasasourceofnegativepressureoninstitutionsdespitethefactthatitisnolongerincluded
intherankingsmethodology.Respondentsalsooccasionallylistedretentionrates,student-to-facultyratios,andthe
percentagesoffacultywithterminaldegreesascriteriathatshouldbeincludedintherankingsmethodology.Though
thesefactorsarealreadyincludedintheUSNWRrankingsmethodology,themisunderstandingamongmembersmay
beinpartattributabletotheirrelativelyminimalweight.
15
Distorting Effects and Professional PracticeInstitutional Responses to the Rankings
Inareportontheeffectofrankingsinhighereducationpolicymaking,theInstituteofHigherEducationPolicy(IHEP)
noted:
Rankingshavethepotentialtoshiftinstitutionalbehaviorsinwaysthatmaynegativelyaffectpolicygoals.
Rankingscreateincentivesforinstitutionstotakeactionsdesignedtoimprovetheirpositions.Thisreactivity
createsconditionsinwhichinstitutionsrespondtotheconceptofeducationalqualityembeddedinrankings,
whichisnotalwaysalignedwithpublicpolicygoals,suchasequityanddiversity.31
Figure6demonstratesthatanoverwhelmingmajority(95.1percent)ofNACACmembersbelievethattheU.S.News
&WorldReportrankings“putpressureoninstitutionstoinvestinstrategiesandpracticesprimarilyforthepurposeof
maintainingorstrengtheningpositionintherankings,”eitherconsistentlyoroccasionally.
Highschoolmembersaremoresuspiciousofinstitutionalresponsestotherankings.Nearlytwo-thirds(63.6percent)
ofhighschoolrespondentsbelievethattherankings“consistently”putpressureoninstitutions,comparedtoonly
46.5percentofcollegerespondents.
Morethan300NACACmembersofferedcommentsonthisquestioninadditiontotheirmultiplechoiceresponses.
Themostcommonthemesintheopenendedresponsesaddsubstancetothegeneralbeliefinstitutionsandschools
arepressuredtomakeprogrammaticchangesineffortstoimprovetheirrankings.Commonthemesincluded:
• Manipulatingnumbers—Manymembersbelievethatschoolsmanipulatethedatathatisusedtocalculate
the U.S. News & World Report rankings, especially admit and yield32 rates, with wait lists, fast-track
applications,andearlydecisionprograms.
• Outsidepressure—Memberscommonlyreportedbeingpressuredbytheirinstitution’spresidents,trustees,
andfacultytoadoptstrategiesthatwouldincreasetheirrank.
• Benefits—Somemembersarguedthatthepressuretoimproverankingscanbenefitschools,colleges,and
studentsbyencouragingpoliciesthatimprovecertainstudent-centeredfeatures,includingretentionrate
andclasssize.
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
Yes, consistently Yes, occasionally No
Figure 6. Puts Pressure on Institutions to Invest in Strategies to Maintain/Improve Ranking
College
High School
Total
31Sponsler,2009.32YieldratesarenolongerusedintheU.S.Newsrankingsformula.
16
IncontrasttothedatashowninFigure6,54.1percentofNACACmembersrepresentingcollegesreportedthattheir
particularinstitutionsdonotmakeanyprogrammaticchangesbasedontherankings,asseeninFigure7.Because
theU.S.News&WorldReporthighschoolrankingsarelessprominentandinfluentialthanthecollegerankings,only
responsesfromNACACmembersrepresentingcollegesarediscussedforthisquestion.33
Very few NACAC college members (7.6 percent) report that their institutions consistently “make programmatic
changesat least inpartbecauseof their influenceontherankings.”Overone-thirdofcollegerespondents(38.4
percent)reportthattheirparticularinstitutionsdosooccasionally.ComparingFigures6and7yieldsaninteresting
contrast.Collegerespondents’beliefsthatinstitutionsare“gaming”therankingsgenerallyseemstoapplytoother
colleges,whereastheyarelesslikelytoperceivetheirowninstitutionasmanipulatingtheprocess.TheCommitteewill
explorethisfindingfurtherastheymeetandshareinformationwithmembersoverthecomingmonths.
Useful to College and University Recruiting Efforts?
AstheIHEPreportnotes,“[t]heuseofrankingsbypostsecondaryinstitutionshascontributedtotheirpopularity.”34
Indicativeofthecomplexrelationshipbetweenrankingsandinstitutions,thediversificationofdistinctionsconferred
byUSNWRhashadthedouble-edgedeffectsofaddressing(albeitonlypartly)concernsaboutaone-sized-fitsall
rankingandaffordingmoreinstitutionstheopportunitytopromotetheirrankingtothepublic.
ThemajorityofNACACmembersagreedwiththestatement,“U.S.Newsrankingsareusefultocollegeanduniversity
recruitingefforts.”Collegeswererelativelyevenlydividedonthisquestion,as55.6percenteithersomewhatagreed
oragreedand44.4percenteithersomewhatdisagreedordisagreed.Nearly73percentofhighschoolcounselors,
ontheotherhand,eithersomewhatagreedoragreedthattherankingsareusefultocollegeanduniversityrecruiting
efforts(Figure8).
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Yes, consistently Yes, occasionally No
Figure 7. School or Institution Makes Programmatic Changes Because of Rankings,
College Results Only
33Thehighschoolmemberresponsesforthequestionregardingthepromotionofrankingswereasfollows:6.7percentconsistentlymakechangesbasedontherankings,20.4percentoccasionallymakechangesbasedontherankings,and72.9percentdonotmakeanychangesbasedontherankings.34Sponsler,2009.
17
Rankings Create Confusion for Students and Families?
AlargemajorityofrespondentsfromallgroupsagreedthattheU.S.Newsrankingscreategreatconfusionforstudents
and families interested incollege information.Overall,83.4percentof respondentsagreedor somewhatagreed,
versusonly16.7percentofrespondentwhoexpressedsomelevelofdisagreement.Highschoolcounselors(86.6
percent)weremostlikelytosuggestthattherankingscreateconfusionforstudentsandfamilies(Figure9).
Privatehighschoolcounselorswereslightlymorelikelytobelievethattherankingscreateconfusionforstudentsand
families(Figure10).
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 9. Rankings Create Confusion for Students and Families
College
High School
Total
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 10. Rankings Create Confusion for Students and Families, HS Results Only
Public
Private
0.0%5.0%
10.0%15.0%20.0%25.0%30.0%35.0%40.0%45.0%50.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 8. Rankings Helpful for College and University Recruiting Efforts
College
High School
Total
18
Do Rankings Encourage Counter-Productive Behavior Within Colleges and Universities?
An overwhelming majority of the survey respondents (87 percent) either “somewhat agree” or “agree” that the
U.S.News&WorldReportrankingsencouragecounter-productivebehaviorwithincollegesanduniversities.High
school respondents were most likely to either “agree” or “somewhat agree” (89.4 percent) that rankings cause
counterproductivebehavioratcollegesanduniversities, thoughcollege respondentsweresimilarly inclined (84.7
percenteitheragreedorsomewhatagreed)(Figure11).
Amongcolleges,admissionofficersfrompublicinstitutionswereslightlylesslikelytobelievethattherankingscaused
counter-productivebehaviorthanadmissionofficersatprivateinstitutions(Figure12).
Rankings and Recruiting
AmajorityofNACACmembersbelievetherankingsare,generallyspeaking,usefultocollegeanduniversityrecruiting
efforts.Aspecificexampleoftherankings’utilityforcolleges—thepromotionofaninstitution’srankinitsmarketing
materials—isshowninFigure13.BecauseamajorityofNACACmembersrepresentinghighschoolsarenotpresented
atallintheU.S.News&WorldReportrankings,onlyresponsesfrommembersrepresentingcollegesarediscussed
forthisquestion.
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 11. Do USNWR rankings cause counter-productive behavior at colleges and universities?
College
High School
Total
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 12. Do USNWR rankings cause counter-productive behavior at colleges and universities?
College Results Only
Public
Private
19
Ofthe88.6percentofcollegememberswhoareincludedintherankings,only2.8percentreportbeingpresented
unfavorably.Amongtheremaining79.8percentofcollegememberswhoarepresentedfavorablyintheU.S.News
&WorldReportrankings,overtwo-thirds(71.3percent)promotetheirrank,thoughmostdosoinalimitedfashion.
Amongcolleges,publicinstitutionswereslightlymorelikelytopromotetheirrankingthanprivateinstitutions(Figure
14).
Rankings in Counseling and Admission Offices
NACACmemberswereaskedwhethertheyspend“agreatdealoftime,”“sometime,”or“notime”discussingor
answeringquestionsabout theU.S.News&WorldReport rankings.AsFigure15shows,amajorityofmembers
spendatleastsometimediscussingU.S.Newsrankingswithstudentsandfamilies.
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Yes,promotewidely
Yes,promote in
a limitedfashion
No, do notpromote
Notpresentedfavorably
Notpresented
at all
Figure 14. Promote Rankings in Marketing Strategy, College Results Only
Public
Private
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
Yes, promotewidely
Yes, promotein a limited
fashion
No, do notpromote
Not presentedfavorably
Not presentedat all
Figure 13. Promote Rankings in Marketing Strategy, College Results Only
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
Great deal of time Some time No time
Figure 15. Time Spent Discussing Rankings with Students and Families
College
High School
Total
20
Taking the Next StepMisleading Conclusions About Institutional Quality?
Anoverwhelming89.1percentofallrespondentsagreedorsomewhatagreedthattheU.S.News&WorldReport
rankingsoffermisleadingconclusionsaboutinstitutionalquality.AsFigure16shows,opinionsdidnotvarysubstantially
betweencollegeandhighschoolmembers.
Disaggregatedresultsamonghighschoolsagainrevealaslightdifferencebetweenpublicandprivatehighschool
opinionsabouttheconclusionsdrawnbytherankingsaboutinstitutionalquality,thoughthedifferencesinthiscase
areslight(Figure17).
Theseresults,combinedwiththeresponsestothequestionabouttheaccuracyofthetitle,“America’sBestColleges,”
offertheclearestindicationofwhereNACACmemberconcernsaregrounded.35
The Case for Personalized Lists
“Thepublicationofasinglerankingprovidesreaderswithasummarythatmaybemisleadingandfailstoproperly
accountforpersonalpreferenceanduncertaintyinweights.”36Asreferencedearlier,the“realproblem,”according
toanumberofacademicanalysesoftherankingsandamongmanysurveyrespondents, isUSNWR’s“arbitrary”
assignmentsofweightstoeachcategoryandsubcategory.37
0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 16. USNWR Rankings Offer Misleading Conclusions about Institutional Quality
College
High School
Total
0.0%10.0%20.0%30.0%40.0%50.0%60.0%70.0%80.0%90.0%
Agree Somewhatagree
Somewhatdisagree
Disagree
Figure 17. USNWR Rankings Offer Misleading Conclusions about Institutional Quality, HS
Results Only
Public
Private
35Therewasastrongandstatisticallysignificantcorrelation(-.521,p<.01)betweenresponsestothefeelingthermometerquestionandthequestionabouttheaccuracyofthetitle“America’sBestColleges.”36HugginsandPachter,2008.37Ehrenberg,RonaldG.,“MethodorMadness?InsidetheU.S.News&WorldReportCollegeRankings,”JournalofCollegeAdmission,2005,Vol.189.
21
In 2004, Marguerite Clarke, at the time an Assistant Professor of Research at Boston College, offered four
recommendationsbasedonherresearchworthrepeatinginthiscommittee’sreport:
1) TheeditorsatUSNWRneedtoreexaminethevalidityoftheindicatorsandweightsusedforeachranking.
2) Theeditorsshouldrefrainfromusinganoverallscoretorankschools.Thesescoresarenotreliableenough
toallowfinedistinctionsamongschools.Amoredefensibleapproachwouldbetouseschools’overallscores
toplacethemin“quality”bands,listingthemalphabeticallywithineachband.
3) USNWRcouldallowconsumerstocreatetheirownrankingformulas.
4) Consumersneedtobecomemorecriticaloftheassumptionsunderpinningrankingsaswellasthevalue
systembehindthechoiceofindicatorsandweights.38
TheCommitteebelieves that the researchanddiscussionabout the imprecisionofordinal rankings in the“Best
Colleges”listhasreachedapointwherenotproactivelyacknowledgingthelimitationsoftherankingformulathrough
vigorousconsumereducationandflexible‘ranking’optionsrisksmisleadingconsumersandhascompromisedthe
journalisticintegrityofthepublication.
TheCommittee’ssurveyresearchclearlyindicatesthatcollegeadmissioncounselingprofessionalsfindthedataand
informationavailable through theUSNWRrankingspublicationuseful.NACACmemberswereasked to rank the
featuresofthe“America’sBestCollege”publicationfromonetoseven,onebeingthemosthelpfulandsevenbeing
theleasthelpful.Perhapsreflectingtheirdisapprovaloftheprocessofplacingthingsinordinalrank,aboutathird
ofmembersrated,ratherthanranked,thefeatures(i.e.usedthesamenumberfortwoormorefeatures).Forthe
purposesofthisreport,theirresponseswereexcluded(Figure18).
Articlesonpreparingforandnarrowingthecollegesearch,aswellasthoseonhowtopayforcollege,receivedthe
highestmeanscores,withaveragemeansscoresof2.51,3.1,and3.19, respectively.Unsurprisingly, theannual
rankingsofcollegesarebelievedtobetheleasthelpfulfeatureofthepublicationwithanoverallmeanscoreof5.56.
NACACmemberswerelukewarmtorecommendingtherankingsifstudentscouldcreatetheirownweight,possibly
duetolingeringsuspicionordislikeforrankingsingeneral.Ifgiventheoptiontoselectindividualweightsforvarious
elementsoftheU.S.News&WorldReportrankings,mostmembers(58.6percent)wouldbeneithermoreorless
inclinedtorecommendtherankingstotheirstudents.Overaquarter(26.6percent)ofmemberswouldbemorelikely
torecommendtherankings,giventheoptiontochoosetheirownweights(Figure19).
1
2
3
4
5
6
7
Articles onPreparingfor College
Search
Articles onNarrowing
CollegeSearch
Articles onHow toPay forCollege
Directoryof Colleges
Articles onAdjustingto College
Articles on"How toGet In"
AnnualRankings
of Colleges
Mos
t Hel
pful
Le
ast H
elpf
ul
Figure 18. Member Rank of Features in Rankings Publicaton
College
High School
Total
38Clarke,Marguerite.“WeighingThingsUp:ACloserLookatU.S.News&WorldReport’sRankingFormulas,”CollegeandUniversity,2004,Vol.79.3.
22
However, the Committee believes that offering consumers the opportunity to create their own lists by weighting
dataelementsaccordingtotheirpreferenceswouldsimultaneouslyinformthepublicthatsuchrankingsarehighly
objective,andallowUSNWR theopportunity to feature listscreatedbystudents,counselors,colleges,andother
consumersaccordingtocriteriatheyhavedevised.
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
More likely torecommend
Neutral Less likely torecommend
Figure 19. If Students Could Create Own Weights...
College
High School
Total
23
Ad Hoc Committee’s RecommendationsGiventhelongandpunctuatedevolutionofrankingsintheU.S.,theCommitteebelievesitisimportanttopromote
recommendationsforpositivechangethatfollowthedirectionthatadaptationalpathsseemtolead.Moreover,the
Committeebelievesthatprofessionalpractice isanobligationandaprerequisiteforcollegeadmissioncounseling
professionals,whoareresponsiblefortheinterpretationofrankingstostudentsandfamilies,themaintenanceand
constructionof rankings throughparticipation in thesurveys,and theperpetuationof rankingsbyusing them in
professional settings.As such, theCommittee’s recommendations aredirected at bothNACACand theUSNWR
rankingspublication.Ourintentisalsoforseveraloftheserecommendationstoapplytoundergraduateinstitutional
rankingsofanytype.
Tothatend,theCommitteewishestodrawattentiontoasetofrecommendationsputforthbytheAmericanAssociation
ofCollegiateRegistrarsandAdmissionsOfficers(AACRAO),whichconvenedaspecialcommitteeatalmostthesame
timeasNACAC’sAdHocCommittee.(SeeAppendix)
For NACAC
Develop professional education resources for members about rankings.
• The committee found that therewas somemisunderstanding of rankingmethodology, aswell as some
perceptionsthatwerenotentirelyaccurate,amongsurveyresponsesandinconversationswithmembers.
• Thecommitteerecognizedtheneedfortrusted,accurateinformationtoconveytostudentsandfamilies
aboutrankings.
Work with education publishers and data outlets to encourage development of do-it-yourself lists for consumers.
• Asubstantialamountofacademicresearchsuggeststhatrankingsarehighlysensitivetotheunderlying
methodologies.Accordingly,consumersshouldbeabletoestablishtheirown‘weightsandmeasures’that
canresultinliststhatproducebetter“fit”forstudents.
For Rankings Publications
Remove the “class rank” and “standardized testing” metrics from rankings methodologies in favor of factors that
measure student satisfaction and engagement.
• The selectivity rating counts for 15 percent of the overall ranking, with high school class standing and
standardizedtestscoresmakingup90percentofthatrating.
• Only16percentofAdmissionTrendsSurveyrespondentsattributedconsiderableimportancetoclassrank.
• TheNACACTestingCommissionrecommendedthatU.S.News&WorldReporteliminatetestscoresasa
measureofinstitutionalquality,andhighlightedstatementsfromtheCollegeBoardandACTdiscouraging
theincorporationofstandardizedtestsinrankingmethodologies.
Reduce the weight of the reputational survey.
• Peer assessments make up 22.5 percent of the U.S. News National and Liberal Arts rankings and 25
percentoftheregionalrankings.
• Thepeerassessmentsarehighlysubjectiveandmaybedisproportionallyinfluencedbysocialfactorsthat
donotmeasureinstitutionalquality.
• In a NACAC survey, only five percent of respondents agreed that the peer assessments serve as good
indicatorsofinstitutionalquality.
Encourage emphasis on fit through customized rankings.
• NACACconsistentlyencouragesthefocusonfit incollegecounseling.Auser-basedmethodologywould
allowstudentstoexamineinstitutionsbasedontheirownnotionsofinstitutionalquality.
Continue to evolve rankings methodologies through the association’s communication channels.
• NACACwillprovideeducationalinformationthatsupportstheproperinterpretationsofdatafoundinranking
reports.
24
AppendixOnNovember7,2010,theAmericanAssociationofCollegiateRegistrarsandAdmissionsOfficers(AACRAO)hosted
a national executive forum during its annual Strategic Enrollment Management Conference. Titled “The College
RankingsDebateandFutureImplications:AssessingtheValueofanInstitution’sUndergraduateExperience,”the
executive forumwasdesigned to stimulate thoughtful discussionof andconsensusbuildingaroundhowbest to
providethepublicwithmeaningfuldataaboutinstitutions’dedicationto,andsuccessat,studentlearning.Thus,the
desiredoutcomewasaclearsetofprinciplesforcollegeassessmentsthatwouldleadtomeaningfulreformsofcurrent
rankingprograms.ThediscussionleadersandwhitepaperauthorswereGeorgeKuh,Chancellor’sProfessorofHigher
Education at IndianaUniversity andDirector of theCenter forPostsecondaryResearch and theCollegeStudent
ExperienceQuestionnaireResearchProgram;JohnPryor,DirectoroftheCooperativeInstitutionalResearchProgram
(CIRP)andManagingDirectoroftheHigherEducationResearchInstitution(HERI)atUCLA;WatsonScottSwail,CEO
oftheEducationalPolicyInstitute(EPI);JayGoff,ViceProvostofEnrollmentManagementatMissouriUniversityof
ScienceandTechnology;BobBontrager,SeniorDirectorofAACRAOConsultingandStrategicEnrollmentManagement
Initiatives;andand[sic]JasonLane,AssistantProfessorandSeniorResearch[sic],InstituteforGlobalEducation
PolicyStudies(IGEPS)atSUNY-Albany.Approximately75practitionersparticipatedintheforum.Participantswere
askedtodiscussthechallengesofcollegeanduniversityrankingprograms.Thesummarythatfollowsprovidesan
overviewoftheexecutiveforum’sprimaryareasofdiscussion.Theconcludingprinciplesareintendedtohelpguide
developmentofanassessmentsystemthatwouldbetteraidstudentsinthecollegeselectionprocess.
AACRAO-principlesforthecreationofaratingsystem
1) RatingwithoutRanking
2) RecognizeInstitutionalDifference
3) CreateCommonPost-AdmissionMilestones
4) TransparencythroughAgreementonDefinitions,DataInstruments,andCollectionProcesses
5) AccountfortheValue-AddedFeaturesofanEducationalExperience
6) GovernancebyaNon-ProfitEntity
AACRAO-goalsofthenewapproach
• Createpublicunderstanding thathundredsofqualitycollegesanduniversitiesexistand that theymeet
students’learninganddevelopmentalneedsindifferentways.
• Focusonvalue-addedoutcomesbytypesandnumbersofsuccessfulstudentsandbygraduates’satisfaction
with their overall undergraduate experience. This should mitigate the focus on traditional measures of
prestigeandselectivity.
• Provide students, parents, and employers with comparative learning data and information about the
particular learningobjectives/skillsetsthatareemphasized,therebymotivatingschoolstoembracetheir
missionsratherthanaligntheireffortswithcurrentrankingsystems.
• Provideameaningfulalternativetoexistingrankingsystems.
(Availableonlineathttp://www.aacrao.org/Files/Publications/CUJ8604_WEB_L.pdf)
25
BibliographyCarey,Kevin.“CollegeRankingsReformed:TheCaseforaNewOrderinHigherEducation.”Education Sector.Education
Sector,Sept.2006.Web.<http://www.educationsector.org/sites/default/files/publications/CollegeRankingsReformed.
pdf>.
Caron, Paul L. and Rafael Gely. “Dead Poets and Academic Progenitors: The Next Generation of Law School
Rankings.”Indiana Law Journal81.1(2006):1-12.Web.
Clarke,Marguerite.“WeighingThingsUp:ACloserLookatU.S. News & World Report’sRankingFormula.”College
and University79.3(2004):3-9.Web.
Ehrenberg,RonaldG.“MethodorMadness? InsidetheU.S. News & World ReportCollegeRankings.”Journal of
College Admission189(2005):29-35.Print.
---“ReachingfortheBrassRight:TheU.S. News & World ReportRankingsandCompetition.”The Review of Higher
Education26.2(2002):145-162.Project MUSE.Web.
Gladwell,Malcolm.“TheOrderofThings.”The New Yorker14Feb.2011:68-75.Web.<http://www.newyorker.com/
reporting/2011/02/14/110214fa_fact_gladwell>.
Griffith,AmandaandKevinRask. “The Influenceof theUSNewsandWorldReportCollegiateRankingson the
MatriculationDecisionofHigh-AbilityStudents:1995-2004.”Economics of Education Review26(2007):244-255.
ScienceDirect.Web.
Henderson,WilliamD.andAndrewP.Morriss.“StudentQualityasMeasuredbyLSATScores:MigrationPatternsin
theU.S. NewsRankingsEra.”Indiana Law Journal81.163(2006):163-204.Web.
Huggins, Peter and Lior Pachter. “Selecting Universities: Personal Preference and Rankings.” Cornell University
Library.CornellUniversity,7May2008.Web.<http://arxiv.org/PS_cache/arxiv/pdf/0805/0805.1026v1.pdf>.
Johnson,AlexM.,Jr.,“TheDestructionoftheHolisticApproachtoAdmissions:ThePerniciousEffectsofRankings.”
Indiana Law Journal81.309(2006):309-358.Web.
Lane,JasonE.“Rating(NotRanking)theUndergraduateExperience:PrinciplesfromaNationalDiscussion.”College
and University86.4(2011):2-7.Web.
McDonough,PatriciaM.,etal.“CollegeRankings:DemocratizedCollegeKnowledgeforWhom?”Research in Higher
Education39.5(1998):513-537.JSTOR.Web.13April2010.
Meredith,Marc.“WhyDoUniversitiesCompeteintheRatingsGame?AnEmpiricalAnalysisoftheEffectsoftheU.S.
News and World ReportCollegeRankings.”Research in Higher Education45.5(2004):443-461.Web.
Myers, Luke and Jonathan Robe. “College Rankings: History, Criticism and Reform.” Center for College
Affordability and Productivity. Center for College Affordability and Productivity. Mar. 2009. Web. <http://www.
centerforcollegeaffordability.org/uploads/College_Rankings_History.pdf>.
National Opinion Research Center. “The NORC Report on U.S. News and World Report.” Washington Monthly.
WashingtonMonthly,n.d.Web.<http://www.washingtonmonthly.com/features/2000/norc.html>.
26
Posner,RichardA.“LawSchoolRankings.”Indiana Law Journal81.13(2006):13-24.Web.
Rauhvargers,Andrejs.“GlobalUniversityRankingsandTheirImpact.”EUA.EuropeanUniversityAssociation,2011.
Web.<http://www.eua.be/pubs/Global_University_Rankings_and_Their_Impact.pdf>.
Sanoff,AlvinP.“TheU.S. NewsCollegeRankings:AViewfromtheInside.”College and University Ranking Systems:
Global Perspectives and American Challenges.Ed.InstituteforHigherEducationPolicy.InstituteforHigherEducation
Policy,2007.9-22.Web.<http://www.ihep.org/assets/files/publications/a-f/CollegeRankingSystems.pdf>.
Sauder, Michael and Wendy Nelson Espeland. “Rankings and Reactivity: How Public Measures Recreate Social
Worlds.”American Journal of Sociology113.1(2007):1-40.Web.
---“StrengthinNumbers?TheAdvantagesofMultipleRankings.”Indiana Law Journal81.205(2006):205-227.
Web.
- - -“The Discipline of Rankings: Tight Coupling and Organizational Change.” American Sociological Review 74
(2009):63-82.Web.
Sponsler,BrianA. “IssueBrief: TheRoleandRelevanceofRankings inHigherEducationPolicymaking.” IHEP.
Institute forHigher EducationPolicy, Sept. 2009.Web. <http://www.ihep.org/assets/files/publications/m-r/(Issue_
Brief)_The_Role_and_Relevance_of_Rankings.pdf>.
Stake,JeffreyEvans.“The InterplayBetweenLawSchoolRankings,Reputations,andResourceAllocation:Ways
RankingsMislead.”Indiana Law Journal81.299(2006):229-270.Web.
U.S.NewsStaff.Best Colleges 2012: About the Rankings/Methodology.U.S.News&WorldReport,12Sept.2011.
Web.<http://www.usnews.com/education/best-colleges/articles/2011/09/12/best-colleges-2012-about-the-rankings-
methodology?s_cid=related-links:TOP>.
Walleri,R.DanandMarshaKMoss,eds.Evaluating and Responding to College Guidebooks and Rankings.San
Francisco:Jossey-BassPublishers,1995.Print.
Webster,DavidS.“ReputationalRankingsofColleges,Universities,andIndividualDisciplinesandFieldsofStudy,
FromTheirBeginningstothePresent.”Higher Education: Handbook of Theory and Research Volume VIII.Ed.John
C.Smart.Bronx,NewYork:AgathonPress,1992.234-304.Print.