Editorial: The 13th Rothkopf Rankings: All-Time Interfaces

This article was downloaded by: [71.56.194.18] On: 12 November 2020, At: 21:15Publisher: Institute for Operations Research and the Management Sciences (INFORMS)INFORMS is located in Maryland, USA

INFORMS Journal on Applied Analytics

Publication details, including instructions for authors and subscription information:http://pubsonline.informs.org

Editorial: The 13th Rothkopf Rankings: All-TimeInterfaces/INFORMS Journal on Applied Analytics 50-YearAnniversary Special EditionMichael F. Gorman

To cite this article:Michael F. Gorman (2020) Editorial: The 13th Rothkopf Rankings: All-Time Interfaces/INFORMS Journal on Applied Analytics50-Year Anniversary Special Edition. INFORMS Journal on Applied Analytics 50(6):387-395. https://doi.org/10.1287/inte.2020.1049

Full terms and conditions of use: https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-Conditions

This article may be used only for the purposes of research, teaching, and/or private study. Commercial useor systematic downloading (by robots or other automatic processes) is prohibited without explicit Publisherapproval, unless otherwise noted. For more information, contact [email protected].

The Publisher does not warrant or guarantee the article’s accuracy, completeness, merchantability, fitnessfor a particular purpose, or non-infringement. Descriptions of, or references to, products or publications, orinclusion of an advertisement in this article, neither constitutes nor implies a guarantee, endorsement, orsupport of claims made of that product, publication, or service.

Copyright © 2020, INFORMS

Please scroll down for article—it is on subsequent pages

With 12,500 members from nearly 90 countries, INFORMS is the largest international association of operations research (O.R.)and analytics professionals and students. INFORMS provides unique networking and learning opportunities for individualprofessionals, and organizations of all types and sizes, to better understand and use O.R. and analytics tools and methods totransform strategic visions and achieve better outcomes.For more information on INFORMS, its publications, membership, or meetings visit http://www.informs.org

http://pubsonline.informs.org

https://doi.org/10.1287/inte.2020.1049


https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-Conditions

https://pubsonline.informs.org/Publications/Librarians-Portal/PubsOnLine-Terms-and-Conditions

http://www.informs.org

INFORMS JOURNAL ON APPLIED ANALYTICSVol. 50, No. 6, November–December 2020, pp. 387–395

http://pubsonline.informs.org/journal/inte ISSN 0092-2102 (print), ISSN 1526-551X (online)

Editorial: The 13th Rothkopf Rankings: All-Time Interfaces/INFORMS Journal on Applied Analytics 50-Year AnniversarySpecial EditionMichael F. Gormana

aDepartment of MIS, Operations, and Analytics, University of Dayton School of Business, Dayton, Ohio 45469Contact: [email protected], https://orcid.org/0000-0003-1295-8147 (MFG)

Received: June 15, 2020Accepted: June 17, 2020


Copyright: © 2020 INFORMS

Abstract. In 1996, Michael Rothkopf created an index of institutional contributions to thepractice of operations research (OR) andmanagement science (MS). Since then, Ron Frickerand I have continued to calculate, evolve, and extend this ranking. In this Interfaces (INT)and INFORMS Journal on Applied Analytics (IJAA) anniversary ranking, I analyze orga-nizational contributions to the practice of OR/MS in INT since the inception of the journalin 1971. I calculate all three prior ranking measures—visibility, yield, and academicyield—and a blend of the rankings. This analysis considers only articles in INT/IJAA, whichis a departure from prior rankings, which included applied work in some other journals.However, for the first time, we have a single database of every article ever published inINT/IJAA, allowing analysis of trends over time. I am also able to compute the all-timecontributions of nonacademic institutions as done by Fricker in 2012.

Keywords: professional: comments on • Rothkopf ranking

Introduction: 50th Anniversary EditionIn 1996, Michael Rothkopf developed an index ofwhich universities contribute most to the practiceliterature. This ranking is now known as the RothkopfRankings in his honor, and this article is the 13th in theseries in 25 years. The purpose of these rankings is toidentify the universities that contribute to the prac-tice literature. Michael Rothkopf conducted sevenapplied research rankings between 1997 and 2007(Rothkopf 1996, 1997, 1999, 2002, 2004, 2005, 2007),Ron Fricker provided three rankings from 2009 to2012 (Fricker 2009, 2011, 2013), and I provided themost recent Rothkopf Rankings in Gorman (2016,2019). Table 1 summarizes the date range evaluatedfor each ranking effort. Generally, these rankingsrepresent a rolling measure of such contributions, witha seven-year history for each ranking conducted ap-proximately every two years, with five overlappingyears. More recently, they have been conducted everythree years, with four overlapping years.

This ranking is the first to consider “all-time”contributions to Interfaces (INT) and INFORMS Jour-nal on Applied Analytics (IJAA) since its inception in1970, a time span of 50 years (1970–2019). As such, itmeasures not what an institution has contributed re-cently, but the institutional contribution to the journalover amuch longerperiod, the life of INT/IJAA. This all-in ranking offers a number of advantages.

• First, the original ranking (Rothkopf 1996) hada starting year of 1988; the major contributing

organizations of the first 18 years of the journal hadnever been measured. This analysis considers everysubmission for the full 50-year history of the journal,from 1970 to 2019.• Second, because the yield measure (Fricker 2009)

was created with the 8th ranking and the academicyield measure (Gorman 2016) was created with the11th ranking, they did not exist in earlier rankings. Inthis analysis, all three measures of contribution de-fined below (visibility, yield, and academic yield, aswell as a blended ranking across the measures) areincluded for the life of the journal.• Third, the prior time series analysis of the

visibility, yield, and academic yield presented inGorman (2019) was severely hampered by the lack ofavailability of data from some prior analyses as wellas overlapping time periods. For example, only thetop n universities had a reported rank in any singlereport; Gorman (2019) could only report an aver-age rank for those institutions whose rank was re-ported in prior analyses. Thus, this report provides amore complete time series, eliminating those miss-ing values.• It has been noted in prior Rothkopf Rankings

that, with a seven-year horizon, considerable shifts inthe rankings can occur, which may be due largely tonoise and the fortunate or unfortunate start and endpoints for the study. The full 50-year data set gives ameasure of a long-lasting commitment to appliedresearch over time.

387

http://pubsonline.informs.org/journal/inte

mailto:[email protected]

https://orcid.org/0000-0003-1295-8147

https://orcid.org/0000-0003-1295-8147


• Finally, because data on the other measures aremore complete, the newly proposed blendedmeasure(Gorman 2019) can now be calculated for the entirehistory of the journal given the presence of a completeset of data and calculations of the other measures,which allows for consistent time series analysis.

SampleThe starting point for this analysis is every digitalobject identifier (DOI) ever produced in Interfaces orthe INFORMS Journal on Applied Analytics (Freemanet al. 2020) in its full 50-year history since its inceptionin 1970 through the last issue of 2019. Over that timeframe, INT/IJAA has produced 3,416 unique DOIspenned by 7,437 (not necessarily unique) authors.Table 2 identifies the breakdown of the frequencyof article types in INT/IJAA. Consistent with priorrankings, this analysis will consider full-length,refereed articles and case studies, shorter contribu-tions (such as Art and Science of Practice, PracticeAbstracts, and Practice Summaries), and columns fromregular contributing editors (such as 20/30 Hindsight,The Ombudsman, The Fifth Column, People Skills,All Around the Model, Creativity in MS/OR, ModelWorld, The Easy Chair, and The Occasional Observer,among others). This subset represents 76% (2,587/3,416)of all the DOIs in the journal. Because full-length articlestend to have more authors than the various types ofexcluded submissions, the included DOIs constitute85% (6,300/7,437) of the authors. Consistent with priorRothkopf Rankings, excluded contributions from INT/IJAA include book and application reviews, edito-rials, announcements, contributor and editorial boardlists, calendars, calls for papers, issue introductions,and the like.

In some cases, the academic or professional affili-ation or nation could not be discerned for some au-thors. These authors are included in the counts abovebut are excluded from Rothkopf Rankings below. Forexample, among the included works in the rank-ings, 14 authors’ nationalities could not be discerned,and 4 author types (academic/practitioner) could notbe determined.

Exclusions from This StudyPrior Rothkopf Rankings included articles from theOR Practice section ofOperations Research and articlesin Manufacturing and Services Operations Management(MSOM) and Decision Analysis (DA), but they areomitted from this analysis. Historically, the frequencyof such papers is relatively low relative to the volumeof papers in INT/IJAA that they are generally notmaterial to the final rankings. Typically, Interfacesarticles make up 95% of the articles in each ranking(Gorman 2019); MSOM and DA generally have fewapplied articles. This is more true in early years forINT/IJAA, whenMSOM andOR Practice did not exist.Thus, there is little loss in comparability with priorrankings. Consistentwith the “anniversary” nature ofthis article and given that the vast majority of appliedresearch is in INT/IJAA, the focus of this report is oncontributions to practice solely based solely on con-tributions to INT/IJAA.

Measures of ContributionFor all measures described below, consistent withthe scoring proposed by Mike Rothkopf (Rothkopf1996), full-length articles and case studies count as afull point, and articles and columns count as onehalf-point.

Table 1. The Rothkopf Rankings

Ranking number Author Year Years of coverage

1 Rothkopf 1996 1988–19942 Rothkopf 1997 1990–19963 Rothkopf 1999 1992–19984 Rothkopf 2002 1994–20005 Rothkopf 2004 1996–20026 Rothkopf 2005 1998–20047 Rothkopf 2007 2000–20068 Fricker 2009 2002–20089 Fricker 2011 2004–201010 Fricker 2013 2006–201211 Gorman 2016 2009–201512 Gorman 2019 2012–201813 Gorman 2020 1971–2020

Notes. The Rothkopf Rankings tradition (so named in 2009 by RonFricker in honor of the rankings’ originator, Michael Rothkopf) beganin 1996 and has since been conducted roughly every two to threeyears. Prior rankings have evaluated themost recent seven years, on arolling, overlapping basis. This 13th ranking is the first to consider a50-year horizon of contributions.

Table 2. This Study Includes 2,587 (76%) Articles, CaseStudies, Practice Abstracts and Summaries, and ColumnsPublished in INT/IJAA Since Its Inception

Included Not included

Article 2,080 Book review 177Column 440 Contributors 89Practice abstracts 32 Editorial 85Case study 25 Applications reviews 56Practice summaries 10 Award 51Total 2,587 Letter 32

Introduction 30Calendar 43Management science roundup 25Misapplications reviews 24Call for papers 23Instructions to authors 20Note 12Teachers’ forum 12Other 150

Total articles 3,416 Total 829

Gorman: Editorial: The 13th Rothkopf Rankings 50th Anniversary Edition388 INFORMS Journal on Applied Analytics, 2020, vol. 50, no. 6, pp. 387–395, © 2020 INFORMS

As described in Gorman (2019), the three scor-ing methods are simple but varied. Four differentmethods of apportioning these points have beendeveloped over the years: visibility, yield, and aca-demic yield, and a blended ranking based on the otherthree. The relative merits of these measures are dis-cussed in detail in Gorman (2016) and are describedin brief below.

1. Visibility (Rothkopf 1996):Visibility (so dubbedby Fricker 2009) is the original measure created byRothkopf (1996). Visibility is ametric based on authorparticipation; each author on an article gets a fullpoint (half-point for columns and practice summa-ries). The sum of the yield measure is based on thetotal academic authors contributing to applied arti-cles in the date range. The “visibility” is a function ofthe number of academics participating in the project.

2. Yield (Fricker 2009): Yield splits a single pointfor an article (half-point for a column) proportionallyamong all of the authors contributing to an article.Yield is a metric based on proportional author con-tribution to a work. The sum is the proportion of allthe articles contributed by all authors, approximatelyequal to the number of articles plus one half of thesmaller contributions of summaries and columns.

3. Academic Yield (Gorman 2016):Academic yieldis a contribution metric based solely on the aca-demically produced portion of the article. Academicyield follows a similar formula as in the yield mea-sure; however, nonacademic authors do not countagainst the measured academic contribution to thearticle. The total academic points for each article areequal to one (0.5 for each column). The sum of theacademic yield is thus equal to the total applied ar-ticles plus one half the number of columns in the daterange. Because the Rothkopf Rankings were based onacademic support of applied research, the measurecaptures the portion of the contribution to an articleby academics without penalizing the participationof a nonacademic author.

4. Blended Measure (Gorman 2019): The blendedranking takes the average of the ranks achieved byeach measure. It captures the scoring contribution ofall three measures through an equal weighting of the

ranks from each measure: the awareness (visibility),productivity (yield), and academic contribution (ac-ademic yield) to each article. In this ranking, completedata for INT/IJAA allow for more complete rankingdata and time series analysis.Table 3 illustrates the differences of each measure

for a few illustrative publications with contributionsfrom “University A.”University A receives one pointper author with the visibility score and a proportionof a point in the yield score based on the fraction of itsauthors’ contribution to the article. The sum of thevisibility score is the total number of academic au-thors, the sum of the yield score is the total proportionof the authors on a paper that are from academia, andthe sum of the academic yield score is one for anyarticle with academic authors.

The 13th Rothkopf Ranking—50thAnniversary ResultsAs is the norm, the Rothkopf Ratings are presentedby U.S. and non-U.S. categories. Because we are in-cluding nonacademic contributions in this ranking,these, too, are split by U.S. and non-U.S. categories.Table 4 summarizes the total authors included, and

Table 3. Example Scoring for “University A” Shows the Difference in the Three Measures ofContribution

Article authors Visibility Yield Academic yield

1 from University A 1 1 12 from University A 2 1 13 from University A, 1 from University B 3 0.75 0.752 fromUniversity A, 2 fromUniversity B, 2 fromCompany C 2 0.33 0.51 from Company C 0 0 0

Table 4. U.S. Academic and Nonacademic InstitutionsContribute the Most to INT/IJAA

Count Visibility Yield Academic yield

United StatesAcademic 2,567 2,374 1,129 1,356Nonacademic 2,178 2,088 684

Non-U.S.Academic 959 935 360 435Nonacademic 592 578 170Total 6,296 5,975 2,343 1,791

United StatesAcademic 41% 40% 48% 76%Nonacademic 35% 35% 29%

Non-U.S.Academic 15% 16% 15% 24%Nonacademic 9% 10% 7%

Note. Non-U.S. academic institutions have participated, but non-U.S., nonacademic organizations are least prevalent.

Gorman: Editorial: The 13th Rothkopf Rankings 50th Anniversary EditionINFORMS Journal on Applied Analytics, 2020, vol. 50, no. 6, pp. 387–395, © 2020 INFORMS 389

the total of each score, by academic and nonacademicaffiliation andU.S. and non-U.S. location. Count is thetotal authors contributing to this set of articles. Vis-ibility is the author-based total point score, adjustedfor the lower point values of practice abstracts andcolumn contributions, and thus is slightly lower than

the total count of authors. Yield is the article-basedmeasure, after adjusting downward for practice andcolumns, and thus is slightly lower than the totalnumber of articles in the sample.In the journal’s 50-year history, 335 unique, identi-

fiableU.S.-based academic institutions have had authors

Table 5. The Top 50 U.S. Academic Institutions for Contributions to INT/IJAA over Its History

Rank U.S. academic institution Visibility YieldAcademic

yieldVisibilityrank

Yieldrank

Academic yieldrank

Averagerank

1 Colorado School of Mines 72.0 51.5 54.5 2 1 1 1.332 Massachusetts Institute of Technology 72.5 32.0 45.2 1 3 3 2.333 University of Pennsylvania 70.0 42.8 46.8 3 2 2 2.334 Naval Postgraduate School 65.0 21.4 30.1 4 7 5 5.335 University of Texas at Austin 56.5 21.4 32.8 6 6 4 5.336 Pennsylvania State University 54.0 23.0 27.3 7 4 6 5.677 Stanford University 45.5 22.7 25.9 8 5 8 7.008 Georgia Institute of Technology 61.0 18.1 26.5 5 11 7 7.679 University of Southern California 40.5 19.3 22.9 10 8 10 9.3310 Northwestern University 34.5 18.9 23.0 14 9 9 10.6711 Carnegie-Mellon University 44.0 16.3 19.2 9 14 13 12.0012 University of Maryland 32.5 18.3 22.8 16 10 11 12.3313 University of Pittsburgh 36.0 16.4 19.2 13 13 14 13.3314 University of Cincinnati 33.0 16.6 18.8 15 12 15 14.0015 University of Virginia 38.0 16.2 16.7 11 15 18 14.6716 University of Michigan 38.0 12.6 16.7 11 20 17 16.0017 Cornell University 29.5 12.5 17.6 18 21 16 18.3318 Temple University 21.5 15.5 19.3 26 17 12 18.3319 Columbia University 21.5 13.1 16.0 26 18 20 21.3320 Indiana University 22.5 12.8 13.5 22 19 24 21.6721 University of California, Los Angeles 22.5 12.3 13.8 22 22 23 22.3322 University of Rochester 19.5 16.1 16.3 32 16 19 22.3323 University of Alabama 27.0 10.1 13.3 19 28 25 24.0024 Harvard University 25.0 11.4 13.0 21 25 27 24.3325 University of Colorado 22.0 11.5 13.2 24 23 26 24.3326 Princeton University 22.0 10.0 15.9 24 29 21 24.6727 University of California, Berkeley 21.0 11.5 15.1 29 24 22 25.0028 University of South Carolina 27.0 10.4 11.7 19 27 31 25.6729 University of Tennessee, Knoxville 21.5 8.6 12.0 26 33 29 29.3330 Arizona State University 21.0 8.5 11.9 29 35 30 31.3331 University of Illinois at Urbana-Champaign 18.5 10.0 11.0 33 30 33 32.0032 University of Florida 16.5 9.2 12.5 38 32 28 32.6733 University of Dayton 17.0 9.5 11.4 37 31 32 33.3334 Purdue University 20.5 8.3 10.7 31 37 34 34.0035 University of Nebraska-Lincoln 14.5 8.2 9.3 42 38 37 39.0036 Case Western Reserve University 14.5 8.6 8.6 42 33 45 40.0037 Rensselaer Polytechnic Institute 30.5 5.3 8.4 17 60 46 41.0038 Oakland University 10.5 10.5 10.5 62 26 35 41.0039 George Mason University 14.5 7.8 9.0 42 43 39 41.3340 Southern Methodist University 14.5 7.8 9.0 42 42 41 41.6741 University of North Carolina at Chapel Hill 14.5 7.1 9.9 42 47 36 41.6742 University of Minnesota 18.5 6.3 8.3 33 50 47 43.3343 Duke University 16.5 7.4 8.0 38 44 48 43.3344 Texas Tech University 13.0 7.9 9.0 51 41 39 43.6745 Villanova University 18.0 7.0 7.5 35 48 51 44.6746 University of Massachusetts Boston 10.5 8.5 9.3 62 36 37 45.0047 West Virginia University 18.0 8.0 6.0 35 40 62 45.6748 Boston University 15.0 5.7 8.8 41 54 42 45.6749 University of Houston 13.0 7.3 8.8 51 45 43 46.3350 Yale University 13.0 7.3 8.8 51 46 44 47.00


who contributed to the journal. Over that same history,314 unique, identifiable non-U.S. academic institutionshave contributed to the journal. Table 5 shows the top50 U.S. academic institutions, sorted by their aver-age rank across the three measures. Table 6 shows the

top 50 non-U.S. academic institutions, sorted by theiraverage rank across the three measures.In the United States, the Colorado School of Mines

is the leading contributor to the journal over its his-tory, followed closely by Massachusetts Institute of

Table 6. The Top 50 Non-U.S. Academic Institutions for Contributions to INT/IJAA over Its History

Rank Non-U.S. academic institution Visibility YieldAcademic

yieldVisibilityrank

Yieldrank

Academic yieldrank

Averagerank

1 University of Chile 52.0 11.6 13.8 1 1 1 1.002 Erasmus University 30.0 9.1 11.8 2 2 2 2.003 University of Waterloo 17.0 8.6 9.5 4 3 3 3.334 University of British Columbia 20.0 6.8 9.3 3 6 4 4.335 Indian Institute of Management 15.0 7.4 8.7 5 4 5 4.676 Lancaster University 14.0 7.0 7.5 7 5 6 6.007 University of Toronto 15.0 5.3 7.0 5 8 7 6.678 University of Ottawa 11.0 5.0 5.8 8 9 10 9.009 University of Auckland 9.0 4.5 6.5 12 11 8 10.3310 Technion - Israel Institute of Technology 8.5 5.7 6.5 16 7 8 10.3311 University of Western Ontario 8.0 5.0 5.5 17 9 11 12.3312 Eindhoven University of Technology 9.0 3.8 4.8 12 13 13 12.6713 University College 8.0 4.4 5.0 17 12 12 13.6714 Katholieke Universiteit Leuven 9.0 3.6 4.0 12 14 16 14.0015 Laval University 10.0 3.3 4.3 9 21 15 15.0016 University of Alberta 8.0 3.3 4.7 17 18 14 16.3317 London School of Economics and Political Science 9.0 3.3 3.6 12 18 20 16.6718 INSEAD 7.5 3.2 3.8 23 23 19 21.6719 Ecole des Hautes Etudes Commerciales 7.0 3.5 3.1 24 16 26 22.0020 Manchester University 6.0 3.4 4.0 34 17 16 22.3321 KoreaAdvanced Institute of Science and Technology 7.0 2.5 3.0 24 27 27 26.0022 University of Waikato 6.0 3.3 3.3 34 20 24 26.0023 Pontifical Catholic University of Chile 6.5 3.3 3.3 32 22 25 26.3324 University of Groningen 7.0 2.3 3.0 24 36 27 29.0025 University of Montreal 5.5 2.8 3.5 40 26 21 29.0026 Royal Military College of Canada 4.5 3.5 3.5 51 15 21 29.0027 Dalhousie University 5.0 2.5 3.5 43 27 21 30.3328 Polytechnic University of Madrid 8.0 2.3 2.5 17 40 38 31.6729 Tsinghua University 10.0 1.7 2.6 9 50 37 32.0030 University of Buenos Aires 8.0 2.4 2.5 17 34 45 32.0031 Indian Institute of Technology 8.0 2.4 2.4 17 35 46 32.6732 University of Sao Paulo 6.0 2.3 3.0 34 37 27 32.6733 University of Melbourne 6.0 2.5 2.5 34 27 38 33.0034 University of Seville 7.0 1.8 3.0 24 49 27 33.3335 University of Bath 4.5 2.5 3.0 51 27 27 35.0036 Tilburg University 7.0 1.3 4.0 24 66 16 35.3337 American University 4.0 2.5 3.0 53 27 27 35.6738 University of Strathclyde 7.0 2.0 2.3 24 44 47 38.3339 Henley College 3.0 3.0 3.0 69 24 27 40.0040 University of Canterbury 3.0 3.0 3.0 69 24 27 40.0041 University of Calgary 3.0 2.5 3.0 69 27 27 41.0042 Public University of Navarre 5.5 1.6 2.6 40 53 36 43.0043 Bilkent University 5.5 1.5 2.5 40 54 38 44.0044 Cass Business School, City University of London 5.0 2.2 2.2 43 41 49 44.3345 University of Florence 6.0 1.8 1.8 34 45 56 45.0046 Norwegian University of Science and Technology 7.0 1.3 2.0 24 69 51 48.0047 National University of Singapore 3.0 2.3 2.5 69 37 38 48.0048 Imperial College 3.0 2.5 2.0 69 27 51 49.0049 Ecole Superieure des Sciences Economiques et

Commerciales3.0 2.0 2.5 69 42 38 49.67

50 Tel Aviv University 3.0 2.3 2.3 69 37 47 51.00


Technology and the University of Pennsylvania. Thosethree institutions are ranked in the top three in all threemeasures of contribution. The remainder of the top 10generally attain a ranking in the top 10 in all threerankings: Naval Postgraduate School, University ofTexas at Austin, Pennsylvania State University, Stan-ford University, Georgia Institute of Technology,University of Southern California, and Northwest-ern University.

Internationally, the University of Chile dominatesin all three measures over the life of the journal, andErasmus University holds the second rank in all threemeasures. The University of Waterloo, the Universityof British Columbia, the Indian Institute of Man-agement, and Lancaster University generally holdranks 3 through 6 in the measures. The University ofToronto, the University of Ottawa, the University ofAuckland, and Technion - Israel Institute of Tech-nology round out the top 10.

It is noteworthy that a top 10 finish in the UnitedStates requires significantly higher absolute scores; atop 10 finish in the United States requires somethingon the order of 20 articles, whereas the same rankinternationally requires something more like 6. Putanother way, the University of Chile would haveranked 16th worldwide if U.S. schools were included;Erasmus University would have ranked 18th.

Time Series AnalysisAlthough a primary benefit of this study is to take thelong view over the life of the journal, it is interesting tolook at behaviors over time. This study allows a first-time look at consistently developed measures overthe life of the journal. Figure 1 shows the top 20 U.S.schools in each of the five decades of INT/IJAA’sexistence. In total, 54 different schools have been inthe top 20 in any decade. Figure 2 shows the rankingsof the 26 schools who have made the top 20 in two ormoredecades. Theyare sortedby frequency in the top 20.Three U.S. institutions have been in the top 20 inall five decades of the existence of INT/IJAA: the

Colorado School of Mines, Massachusetts Institute ofTechnology, and the University of Texas at Arlington.The Colorado School of Mines was never rankedlower than sixth in any decade.The University of Pennsylvania and the University

of Southern California were in the top 20 in threedecades, but the contributions have been lighter inrecent decades. Seven institutions have appeared inthe top 20 in three decades, with Carnegie-MellonUniversity and the Naval Postgraduate School makingstrong contributions in the last 30 years on a consis-tent basis.Of the schools that have appeared in the top 20

twice, Cornell University, the Georgia Institute ofTechnology, the University of Dayton, and the Uni-versity of Tennessee, Knoxville have shown a recenthigh level of contribution.Figure 3 shows the ranking for the top 10 inter-

national schools by decade. (Ten was chosen becausethere are fewer international schools and generallymake lower contributions; the top 20 in any decademight be 2/3 of the total contributors.) Internation-ally, 35 different schools made the top 10 over the fivedecades. Only four schools appear in the top 10 inthree of the five decades: Waterloo in the 1970s throughthe 1990s; University of Chile, Erasmus University, andUniversity of Auckland in the 1990s through the 2010s;and University of British Columbia in the 1970s, 1990s,and 2000s.

Measures Applied toNonacademic InstitutionsFricker (2012) is the only attempt to measure the con-tribution of nonacademic institutions to the practiceliterature. That work focused on the contribution ofnonacademic institutions to the applied literature for theyears2005–2011. As he did, I follow themethodologiesdescribed above for each institution; however, in thiscase, as with the academic analysis, I do so solely forINT/IJAA and for the entire 50-year life of the journal.Unlike Fricker, I restrict the articles solely to INT/IJAA

Figure 1. Fifty-Four Different Schools Have Been in the Top 20 in the Last Five Decades


articles over its 50-year history; however, as notedabove, INT/IJAA makes up the vast majority of theapplied analytics literature (over 95% of the scoringin Fricker (2012) came from INT/IJAA articles). Amongpractitioners, INT/IJAA is undoubtedly a more likelyoutlet for applied research.

Ron considered “parent” organizations (e.g., all ofIBM, regardless of division; all of the U.S. govern-ment, regardless of which organization (Centers forDisease Control and Prevention, Environmental Pro-tection Agency, etc.)); I do differentiate the U.S. mili-tary divisions, but as Ron did, do not differentiate theother governmental organizations. As with Fricker,I do not differentiate by U.S. and non-U.S. compa-nies but rather provide a worldwide ranking, asmany of the leading companies are in fact multina-tional. Consistency in naming was attempted to somedegree. For example, over this large time frame,

“International BusinessMachines” came to be knownas IBM; these names were aggregated into a singlecompany. However, merged companies, such asSanta Fe and BNSF or Chessie System and CSX,were counted separately, as were companies thatwere divested or spun off, such as Sabre fromAmerican Airlines. Finally, I drop the “academicyield”measure in this ranking, as it does not apply tothese organizations.In total, 1,050 nonacademic organizations have

contributed to INT/IJAA, compared with 649 aca-demic institutions. This large number is not sur-prising, as often there is author participation from theorganization inwhich a project took place; many suchorganizations are a one-time participant in a researchproject in the journal. Also, many of the leadingcontributors are consulting and technology firms, ofwhich there are many.

Figure 2. U.S. Institution Ranking by Decade

Figure 3. (Color online) The Top 10 Non-U.S. Academic Institutions by Decade

Note. Schools in the top 10 at least three times are bolded, with arrows showing change in rank.


As with academic rankings, Table 7 sorts contrib-uting organizations based on the average rank of thetwo remaining measures that were applied. By anymeasure and by a considerable margin, IBM is thetop nonacademic contributor to INT/IJAA. IBMwouldrank ahead of any international academic institutionand in the top five of U.S. academic institutions fortheir contributions.

The close second biggest contributor is the U.S.government, which benefits in this method of scor-ing from the aggregation of its many departmentsand agencies. Of the 16 different U.S. governmentalagencies included in that category, major contribut-ing organizations include the Centers for DiseaseControl and Prevention, U.S. Environmental Protec-tion Agency, Energy Information Administration,

Table 7. The Top Contributing Nonacademic Institutions in the World Since the Journal’s Inception

Rank Organization name Visibility YieldVisibilityrank

Yieldrank

Averagerank

1 IBM 139.0 25.8 1 1 12 U.S. Government 56.0 18.3 3 2 2.53 General Electric 75.0 16.9 2 3 2.54 AT&T 42.0 13.7 6 4 55 U.S. Army 44.5 9.9 5 5 56 Hewlett Packard 53.0 7.5 4 8 67 American Airlines 17.0 9.3 14 6 108 Merrill Lynch 23.0 6.0 9 13 119 Intel 21.0 6.3 11 12 11.510 General Motors 27.0 5.6 7 17 1211 Xerox 16.5 6.5 17 10 13.512 City of New York 24.0 5.3 8 19 13.513 RAND Corporation 14.5 6.3 21 11 1614 U.S. Navy 15.0 5.7 19 15 1715 Sabre Corporation 17.0 4.3 14 21 17.516 Ford 22.0 3.9 10 25 17.517 Arthur D. Little 11.0 7.0 29 9 1918 Weyerhaeuser Company 14.0 5.7 22 16 1919 Systems Research Incorporated 9.0 8.8 33 7 2020 Procter & Gamble 18.0 3.0 12 32 2221 U.S. Air Force 13.0 4.9 25 20 22.522 Bethlehem Steel Corporation 11.0 4.3 29 21 2523 Schneider National 15.0 3.3 19 31 2524 United Airlines 8.0 4.0 40 23 31.525 ExxonMobil 8.5 3.5 38 27 32.526 Applied Decision Systems 6.0 6.0 54 13 33.527 Decision Focus Incorporated 10.0 2.8 31 38 34.528 Syngenta 16.0 2.2 18 51 34.529 Dell 18.0 2.0 12 58 3530 CSIRO 9.0 2.7 33 40 36.531 McKinsey & Company 7.0 3.4 45 30 37.532 Standard Oil Company 6.0 3.7 54 26 4033 Mayo Clinic 9.0 2.3 33 49 4134 U.S. Coast Guard 12.0 2.1 26 56 4135 Sandia National Laboratories 17.0 2.0 14 72 4336 Mau Trucking 6.0 3.0 54 33 43.537 Booz Allen Hamilton 6.0 2.9 54 37 45.538 Bristol-Myers Company 5.5 5.5 74 18 4639 INSIGHT 7.0 2.5 45 47 4640 Federal Express 8.0 2.1 40 55 47.541 DISTINCT Management Consultant 8.0 2.1 40 57 48.542 Bechtel Power Corporation 5.0 3.5 75 28 51.543 Phillips 6.0 2.2 54 53 53.544 Getty Oil 6.0 2.2 54 53 53.545 Delta Airlines, Inc. 9.0 1.9 33 74 53.546 U.S. WEST Advanced Technologies 6.0 2.0 54 59 56.547 National Science Foundation 5.0 2.7 75 39 5748 Norfolk Southern Corporation 9.0 1.8 33 81 5749 UPS 8.5 1.8 38 77 57.550 Chesapeake Decision Sciences 5.0 2.5 75 41 58


U.S. Postal Service, and Department of Energy, amongothers. Many of these would have been in the top 50based on their own individual contributions.

General Electric (GE) rounds out the top three. GE andthe U.S. government share second and third rankings,depending on the measure. AT&T, the U.S. Army,Hewlett Packard, and American Airlines fall ap-proximately into the fourth through seventh rank-ings. Merrill Lynch, Intel, General Motors, and Xeroxround out the top 10.

DiscussionThis anniversary edition of the Rothkopf Rankings isthe first to produce all three measures of researchproduction developed over the last 30 years (visi-bility, yield, and academic yield), with a blendedranking between them, for the entire history of thejournal. A number of observations can be made. Al-though we measure institutional contributions to thejournal as a way to measure that support, it seemsclear that such contributions are ephemeral and highlyvariable. Of course, what we are truly measuring isthe interest and the ability of faculty at those schools(or employees in those organizations, in the case ofnonacademic institutions) to pursue applied research,and only secondarily are we suggesting what the in-stitutional support for that research must be in orderfor it to be sustained. Of course, the level of appliedresearch can come and go over time as faculty arehired, retire, or change positions.

Nonetheless, there are some longtime institutionalsupporters of applied research identified. The Colo-rado School of Mines must be commended above allothers for the consistency and quality of its sup-port for applied research. Massachusetts Institute ofTechnology and the University of Texas at Austinsimilarly so. The University of Pennsylvania has beennearly as supportive. Recently, the Georgia Instituteof Technology, Cornell, and the Naval PostgraduateSchool have been strong supporters.

Internationally, there are generally lower total con-tributions to INT/IJAA and, therefore, some additionalvariability to the flow of applied research from aroundthe world. The University of Chile, Erasmus Univer-sity, and Auckland University should be commendedfor outstanding applied research in recent decades.Longstanding support for applied research can be seenfrom schools such as the University of Waterloo, theUniversity of British Columbia, the Indian Institute ofManagement, Lancaster University, the University ofToronto, and the University of Ottawa.

It is a pleasure to see the high level of contributionsfrom nonacademic institutions in the journal. Althoughthe Rothkopf Rankings were originally conceived tomeasure academic focus on applied research, themeasures can also be used to understand the extentof analytics in practitioner organizations that arewilling to share them. IBM, the U.S. government andarmed forces, GE, and many other organizationshave shown the willingness and ability to conductadvanced analytical projects and then share themwith the world.For the INFORMS Journal on Applied Analytics and

for analytics research as a whole, the need to dem-onstrate our tools in practice is essential to prove thevalue of what we do and continue to evolve thediscipline to more advanced and useful levels.

ReferencesFreeman NK, Keskin BB, McCullough C (2020) IJAA: Past, present,

and future. INFORMS J. Appl. Analytics 50(6):355–372.Fricker RD Jr (2009) Editorial: The eighth Rothkopf Rankings of

universities’ contributions to the INFORMS practice literature.Interfaces 39(6):533–539.

Fricker RD Jr (2011) Editorial: The ninth Rothkopf Rankings ofuniversities’ contributions to the INFORMS practice literature.Interfaces 41(6):590–598.

Fricker RD Jr (2012) Editorial: The first Rothkopf Rankings of non-academic organizations. Interfaces 42(6):585–590.

Fricker RD Jr (2013) Editorial: The 10th Rothkopf Rankings of uni-versities’ contributions to the INFORMS practice literature In-terfaces 43(6):572–577.

Gorman MF (2016) Editorial: The 11th Rothkopf Rankings: 28 yearsof tracking applied research in academia. Interfaces 46(3):264–276.

Gorman MF (2019) Editorial: The 12th Rothkopf Rankings. INFORMSJ. Appl. Analytics 49(4):295–303.

Rothkopf MH (1996) Editorial: Which universities contribute to thepractice literature? The first Interfaces ranking. Interfaces 26(2):16–21.

Rothkopf MH (1997) Editorial: Which universities contribute to thepractice literature? The second Interfaces ranking. Interfaces27(4):19–22.

Rothkopf MH (1999) Editorial: The third Interfaces ranking of uni-versities’ contributions to the practice literature. Interfaces 29(6):107–111.

Rothkopf MH (2002) Editorial: Leveling the field? The fourth Inter-faces ranking of universities’ contribution to the practice litera-ture. Interfaces 32(2):23–27.

Rothkopf MH (2004) Editorial: The fifth Interfaces ranking of uni-versities’ contributions to the practice literature. Interfaces 34(2):135–138.

Rothkopf MH (2005) Editorial: The sixth Interfaces ranking of uni-versities’ contributions to the practice literature. Interfaces 35(5):425–428.

Rothkopf MH (2007) Editorial: The seventh Interfaces ranking ofuniversities’ contributions to the practice literature. Interfaces37(6):566–569.


Documents

Editorial: The 13th Rothkopf Rankings: All-Time Interfaces