Measuring discursive influence across scholarshipAaron Gerowa,1, Yuening Hub, Jordan Boyd-Graberc,d,e,f, David M. Bleig,h,i, and James A. Evansj,k,1

aDepartment of Computing, Goldsmiths, University of London, New Cross, London SE14 6NW, United Kingdom; bCloud AI, Google, Sunnyvale, CA 94809;cDepartment of Computer Science, University of Maryland, College Park, MD 20742; dUniversity of Maryland Institute for Advanced Computer Studies,University of Maryland, College Park, MD 20742; eiSchool, University of Maryland, College Park, MD 20742; fLanguage Science Center, University ofMaryland, College Park, MD 20742; gDepartment of Statistics, Columbia University, New York, NY 10027; hDepartment of Computer Science, ColumbiaUniversity, New York, NY 10027; iData Science Institute, Columbia University, New York, NY 10027; jDepartment of Sociology, University of Chicago,Chicago, IL 60637; and kComputation Institute, University of Chicago, Chicago, IL 60637

Edited by Peter S. Bearman, Columbia University, New York, NY, and approved February 12, 2018 (received for review November 18, 2017)

Assessing scholarly influence is critical for understanding the col-lective system of scholarship and the history of academic inquiry.Influence is multifaceted, and citations reveal only part of it. Cita-tion counts exhibit preferential attachment and follow a rigid“news cycle” that can miss sustained and indirect forms of influ-ence. Building on dynamic topic models that track distributionalshifts in discourse over time, we introduce a variant that incor-porates features, such as authorship, affiliation, and publicationvenue, to assess how these contexts interact with content to shapefuture scholarship. We perform in-depth analyses on collections ofphysics research (500,000 abstracts; 102 years) and scholarship gen-erally (JSTOR repository: 2 million full-text articles; 130 years). Ourmeasure of document influence helps predict citations and showshow outcomes, such as winning a Nobel Prize or affiliation witha highly ranked institution, boost influence. Analysis of citationsalongside discursive influence reveals that citations tend to creditauthors who persist in their fields over time and discount credit forworks that are influential over many topics or are “ahead of theirtime.” In this way, our measures provide a way to acknowledgediverse contributions that take longer and travel farther to achievescholarly appreciation, enabling us to correct citation biases andenhance sensitivity to the full spectrum of scholarly impact.

scholarly influence | science of science | probabilistic modeling

Scholarship is a complex and chaotic process by which authorscreate, share, and promote new concepts, theories, meth-

ods, data, and findings. Subsequent research adopts scholarlyinnovations through direct contact with the original work orthrough follow-on research, review articles, seminars, and per-sonal conversation. Researchers may acknowledge such influ-ence by citing the work on which they draw or passively adopt-ing the vocabulary, ideas, assumptions, approaches, or insightswithout explicit references. In this way, citations constitute animportant but incomplete signal of influence, the full spec-trum of which includes direct and indirect—acknowledged andignored—influences from across the scholarly landscape. Wehold that a key dimension of influence involves the ability tochange scholarly discourse, which itself interacts with contextualfeatures, such as the status of publication venues, authors’ name-sakes, and institutions that host the research. Here, we seek todetect discursive influence and the factors that shape it.

Research impact is commonly assessed by the number ofexplicit references to a document, author, or journal. A refer-ence list represents a complex combination of considerations,including perceived influence of important arguments, methods,data, or findings but also, what authors, reviewers, and editorsbelieve should be cited to appease readers or enhance their ownstatus. Scientometrics provides measures of author, journal, andinstitutional impact (1), which tabulate functions of the citationdistribution over articles and time. New “alt metrics” go beyondcitations to assess online views, downloads, likes, and tweets orestimate readership, cocitation, and diversity (2, 3). Like cita-tions, these metrics provide an informative but distorted portraitof influence: they exhibit rich-get-richer feedback (4–6) and man-ifest manipulation by authors and editors (7–9). Authors may citetheir own work to drive traffic to prior articles (10, 11). Lessnefariously, citations may certify membership in a community,reveal intellectual alliances, or reflect the status aspirations of

a paper (12–14). Moreover, authors are often unable to cite allof their influences, selecting citations based on cognitive avail-ability and space constraints (15, 16). Over time, citations of apaper tend to decay due to a preference for recentness (17),even when a paper continues to be influential. Further compli-cating the matter, citation cultures differ across disciplines andtime periods (18, 19).

The primary artifact of scholarly work is not citations but text.Here, we offer a measure of influence based on text and contextas they shape future discourse. We use scholarly discourse to ref-erence the amalgam of language in academic publications, thattraces a sequence of communication and scholarly argument. Weadopt a broad notion of discourse as the overall state of schol-arship expressed in published work. This approach avoids somechallenges of discourse analysis while raising new ones. Analyz-ing specific aspects of scholarly conversation requires a selectionprocess, the result of which may not generalize. Alternatively,analyzing a vast number of publications presents technical chal-lenges. To model more general notions of discourse, we use aclass of probabilistic techniques called topic models that extractpatterns of term–document co-occurrence and yield semanticallyrelated word distributions called “topics.”

Here, we develop a dynamic topic model designed to explainnot only which documents are influential but how their influ-ence is derived from content and its interaction with features,like authorship, affiliation, and publication venue. We build onmodels of lexical change (20, 21) and in particular, the document

Significance

Scientific and scholarly influence is multifaceted, shifts overtime, and varies across disciplines. We present a dynamic topicmodel to credit documents with influence that shapes futurediscourse based on their content and contextual features. Wetrace discursive innovation in scholarship and identify theinfluence of particular articles along with their authors, affil-iations, and journals. In collections of science, social science,and humanities research spanning over a century, our mea-sure helps predict citations and reveals signals that recognizeauthors who make diverse contributions and whose contribu-tions take longer to be appreciated, allowing us to compen-sate for bias in citation behavior.

Author contributions: A.G., Y.H., J.B.-G., D.M.B., and J.A.E. designed research; A.G., Y.H.,J.B.-G., and J.A.E. performed research; A.G., Y.H., J.B.-G., and J.A.E. contributed newreagents/analytic tools; A.G. and J.A.E. analyzed data; and A.G., Y.H., J.B.-G., D.M.B., andJ.A.E. wrote the paper.

The authors declare no conflict of interest.

This article is a PNAS Direct Submission.

This open access article is distributed under Creative Commons Attribution-NonCommercial-NoDerivatives License 4.0 (CC BY-NC-ND).

Data deposition: The source code of the model’s implementation has been deposited onGitHub and is available at https://github.com/gerowam/influence.1 To whom correspondence may be addressed. Email: a.gerow@gold.ac.uk or jevans@uchicago.edu.

This article contains supporting information online at www.pnas.org/lookup/suppl/doi:10.1073/pnas.1719792115/-/DCSupplemental.

Published online March 12, 2018.

3308–3313 | PNAS | March 27, 2018 | vol. 115 | no. 13 www.pnas.org/cgi/doi/10.1073/pnas.1719792115

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0

SOCI

AL

SCIE

NCE

SA

PPLI

EDPH

YSIC

AL

SCIE

NCE

S

influence model (DIM) (22). In the DIM, documents receive ascore based on how they change future topics, but the modeloffers no mechanism to explain the composition of influence.Our variant offers such an explanation by explicitly modelingdocument-level covariates alongside content. This allows forcomparing different kinds of authors, institutions, and publica-tion venues for each topic individually. We affirm the value ofsuch features, as they help predict citations in a widely studiedcorpus of computational linguistics research (23). We also intro-duce a mechanism to assess a document’s contribution over timeby simulating future discourse without it. This dynamic measureof contribution helps address the limited scope of citations andprovide a composite understanding of scholarly influence.

Robust Model of InfluenceWe consider a document to be influential to the extent that itaffects future discourse. Probabilistic topic models enable theanalysis of such discourse in large text collections (24–26). Top-ics {k0, · · · , kK} are discovered from lexical co-occurrence ina set of documents and refer to probability distributions overobserved words, the most likely of which typify a topic (27).In time-ordered texts, dynamic topic models derive topics fromdocuments binned into periods {t0, · · · , tT}, allowing topics tochange over time (20). Each document is fit with a document–topic mixture, θd , and each word is fit with a topic assignmentzd,k ,n∈N over the vocabulary N . The first (and to our knowl-edge, only) model to explicitly measure how individual docu-ments change future topics is the DIM (22). DIM learns docu-ment influence in a topic, `d,k , based on how it changes futuretopics but does not explain its composition.

Topic models have proven effective for analyzing scientific lit-erature (21, 28, 29), although they make assumptions about thedata. In particular, such models assume that a specified numberof topics are present throughout. Statistically, words are assignedto topics as those topics semantically shift from “background”topics (composed of collection-wide words) to specific, coherenttopics. Choosing the number of topics, K , is important and hasempirical implications, but it can be interpreted as the number ofretained discursive dimensions (27, 30). Choosing the number oftopics was done in a theoretically motivated, data-driven fashionusing static models of the same data fit with many topics to iden-tify a reasonable threshold for how many topics the data used (SIAppendix). This approach approximates a Bayesian nonparamet-ric search over possible numbers of topics (31, 32) and allows usto discover the topic complexity for each corpus.

Our variant adds an important explanatory mechanism toestablish how influence arises. By incorporating a latent regres-sion on document covariates, the model estimates the marginaleffect of authorship, institution, and journal on influence. Al-though a modest technical innovation, this enables robust expla-nations of discursive influence and its origins. A document’sinfluence in a particular topic, `d,k , is a product of its content andits interaction with associated contexts traced in article metadata.The coefficients that solve the latent regression, denotedµk , offerestimates of how covariates add or detract from influence in eachtopic. As we will see, this opens a range of insights into how schol-arly influence unfolds.

ResultsThe model produces three important results: (i) topics (wordmixtures over time, βt,k , and topic mixtures within documents,θk ), (ii) variational estimates of influence (ˆd,k ), and (iii) esti-mated marginal effects of covariates on influence (µk ). A corpusof computational linguistics research (Association for Compu-tational Linguistics Anthology Reference Corpus; ACL-ARC)(23) was used to compare our variant with DIM. Despite theadditional complexity, our model performed comparably withDIM in terms of convergence in the likelihood bound and per-plexity. Estimates of influence in DIM correlated with citationcounts: ρ=0.22 (K =10), ρ=0.21 (K =20), ρ=0.21 (K =50),

and ρ=0.18 (K =75). Our variant yielded significantly strongerand substantially larger correlations: ρ=0.28 (K =10), ρ=0.27(K =20), ρ=0.23 (K =50), and ρ=0.22 (K =75; Fisher ztransform; all p< 0.01. This confirms that the exogenous fea-tures captured by our model have an observable effect on pre-dicting citations. Furthermore, modeling these additional fea-tures adds explanation to our notion of discursive influence.

Influence in Physics. A collection of research published by theAmerican Physical Society (APS) was used to estimate a static500-topic model, discover an optimal 37-topic solution, and then,fit our 37-topic dynamic influence model. The APS collectionprovides a rich citation environment, where 90% of documentsin the sample were cited by other APS publications. Most ofthe resulting topics typify subfields and allow us to trace theemergence of modern concepts in physics (SI Appendix). Wefound that document influence (Id ) (Eq. 8) correlated with cita-tion counts, Cd , at ρ=0.32 (p< 0.01). Some anecdotal resultsaffirm that discursive innovations are meaningful and substan-tive. For example, specialization is traced by authors’ outsizedmarginal influence in a few, related topics (Fig. 1A). Ed Wit-ten has driven advances in cosmology and string theory, ArnoPenzias’ radio telescope contributed greatly to understanding thebig bang and the development of new cosmic detector arrays, andPhilip Anderson cofounded spin-glasses as a model of magneticphase transitions. Witten’s byline lends a positive effect for influ-ence in cosmology-related topics as does Penzias’ byline in detec-tors and atomic physics and Anderson’s byline in lattice quan-tum chromodynamics and superconductors among others. NobelLaureates, overall, have a significantly more positive effect ontheir document’s influence than those without a Nobel (Fig. 1D).

Discursive influence provides a way to measure impact. Fig.2 lays out four kinds of papers based on their relative discur-sive influence and citation count. Each quadrant is characterizedby high–low bins. Empirically, we define these quadrants rela-tive to median influence and number of citations. Papers in thehigh/high and low/low quadrants participate in the standard sci-entific cycle of credit, where discursive contributions are citedby future work. Here, the signals from citations and discursiveinfluence substitute for one another, with neither adding muchinformation to the assessment of impact.

Papers in the off-diagonal quadrants, however, adhere lesscleanly to this pattern. Citations and discourse here constitutecomplementary signals in at least three ways. First, citationsexhibit preferential attachment, favoring authors who persist innarrow subfields of science, whereas discourse identifies greaterinfluence for itinerant scientists whose careers span diverse topics.Second, citations tend to follow a skewed, log-normal distributionover time with rapid uptake followed by a diminishing tail of atten-tion. As such, citation patterns favor articles that receive most oftheir references within this scientific news cycle. Document influ-ence, however, credits papers that are enduringly influential aswell as so-called “sleeping beauty” (SB) papers, which experiencea delay before discovery and tend to have high document influ-ence (Fig. 1C). Third, citations uniquely capture nondiscursivecontributions, like the provision of new data or critiques, whereasinfluence captures both direct and indirect innovations.

We propose that scientists and scholars experience profes-sional pressure to cite touchstone works of authors with anestablished presence in their field. Authors who contribute tomore diverse topics post high discursive influence but are lesslikely to receive high citations. We measure author establish-ment through persistence (Eq. 11), calculated as the inverseentropy of authors’ sum of document–topic mixtures, scaled bytheir prolificness. Author persistence correlates with their mostcited papers (ρ=0.57, p< 0.01), much more than with the sameauthor’s most influential paper (ρ=0.34, p< 0.01). More per-sistent authors receive more citations, but on average have lessinfluential documents (Fig. 1B). Authors who remain productivein a narrow range of topics are more likely to have a highly citedpaper than if they make more scattered contributions. Papers in

Gerow et al. PNAS | March 27, 2018 | vol. 115 | no. 13 | 3309

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0

BA

C

D

Fig. 1. (A) Box plots of author coefficients for each APS topic. Medians are shown as a red line within each box, the first quartiles are within the box, andthe second and third quartiles are within bands. CMP, condensed matter physics; HEP, high energy physics. Note the wider distributions for more generaltopics 1 and 36. Overlaid are coefficients for three physicists. Positive values mean that an author’s byline adds influence, whereas a negative value means itdetracts. A positive coefficient does not necessarily mean that a document is highly influential itself, only that it was more influential than if it had it beenwritten by the average author. (B) Locally weighted scatter plot smoothing (LOWESS) curves comparing document influence (dashed red line; left y axis) andcitations (solid blue line; right y axis) with author persistence (Eq. 11) (x axis). A consistent and statistically significant trend is established: more persistentauthors tend to produce more highly cited but less influential documents, whereas less persistent authors have more influential but less cited documents.(C) LOWESS curve fit to the plot of documents’ influence vs. their SB score. Error bars are±2 SE of the mean in both dimensions. (D) Distribution of authors’marginal effect on influence for Nobel Laureates compared with all other authors.

the high citation/low influence bin include those from authorswhose papers one “must cite,” a self-reinforcing process wherepapers by persistent authors become required scholarly bound-ary markers. Authors of papers in this region are more persistentthan those in any other quadrant (SI Appendix, Fig. S15). Con-versely, papers in the low-citations/high-influence bin dispropor-tionately credit itinerant authors who contribute to many fields.

We find that norms of citation encourage attention to recentwork. We evaluate this by examining how articles make dynamiccontributions to discourse over time. Although document influ-ence is a static attribute representing a paper’s effect on overalldiscourse, sometimes this influence occurs immediately, while inother cases, it takes time to foment. To establish the dynamics ofdiscursive impact, we measure a document’s topic contributionby estimating how different future topics would have been with-out it (Eq. 10). Fig. 3 illustrates this dynamic contribution for twodifferent kinds of SB papers. Felix Bloch’s 1946 “Nuclear induc-tion” (33) is typical in that it had a small impact on most topicsbut a sizeable one on a few. It is also typical in that it received aspike of citations shortly after publication, which then began todecay. Near the turn of the century, we see a significant spike inboth the article’s citations and its contribution of the Hall effectto the condensed matter physics topic. This is when transistorsin microchips became small enough to be affected by the quan-tum Hall effect and the nuclear magnetic moments described byBloch. A second paper, the top cited in our sample from 1947, isa “pure” SB, which garnered relatively few citations until a sud-den spike in 2004. Philip Wallace’s “The band theory of graphite”(34) described the structure of graphene, which at the time, wasonly observable on an iron film. Theoretically, graphene wasexceptionally strong and “growable” in a block structure, but thetechnology to characterize and isolate it without a substrate wasnot discovered until 2004, which was awarded the 2010 NobelPrize in Physics. Wallace’s paper (34) contributed modestly to

most topics and significantly to a few, but in 2001, its contribu-tion to the materials topic jumps significantly.

With a sample of 500 articles from each percentile of docu-ment influence, we assessed the variance and half-life of citationscompared with topic contribution. Half-life is the number of timesteps after which the score is one-half what it was initially. Doc-uments in higher percentiles of influence have longer half-lives,and in every percentile, citations have lower variance and shorterhalf-lives than topic contribution (SI Appendix). This suggests aconventional “statute of limitations” through which scholars nolonger need to cite older articles with persistent influence, pos-sibly because those ideas have entered popular consciousness orbecause their original authors are no longer around to claim them.

Other articles defy the scientific news cycle not because theyare persistently influential, but because they garner little atten-tion for a long period before a spike of citation attention, notunlike Wallace’s paper (34) discussed above. The SB index (35)identifies such articles as a function of the convexity and time-to-maximum citations over time. An article that is dormant for along time, after which it suddenly receives a burst of citations, hasa high SB score, whereas the typical article, which receives a burstof citations on publication that decay thereafter, receives a lowscore. SB scores correlated four times more strongly with docu-ment influence (r =0.20, p< 0.01) than with citations (r =0.05,p< 0.01), suggesting that papers ahead of their time neverthelessreceive fewer citations than their influence would predict. Suchhigh-influence/low-citation articles violate the standard influencetrajectory.

Finally, high citation/low influence papers contain importantnontextual features, while high-influence/low-citations paperstend to have influence that may not be credited in citations. Forexample, the most cited paper in the ACL-ARC collection intro-duced the Penn Treebank (36), an important resource that accel-erated research in parsing. Nevertheless, it was the resource and

3310 | www.pnas.org/cgi/doi/10.1073/pnas.1719792115 Gerow et al.

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0

SOCI

AL

SCIE

NCE

SA

PPLI

EDPH

YSIC

AL

SCIE

NCE

S

Low / High

High discursive influence- Innovative ideas- Dispersed impact across

topics and time

Low citations- Persistently impactful- Delayed or distant impact

High / High

High discursive influence- Large, broad impact- Boosted by covariates

High citations- Acknowledged by peers- Self-reinforcing

Low / Low

Low discursive influence- Small, narrow impact- Detracted by covariates

Low citations- Ignored by peers- Self-reinforcing

High / Low

Low discursive influence- Non-textual influence- Focused contribution to

established fields (topics)

High citations- Persistent, focused authors- Preferential attachment

Gre

ater

Influ

ence

Influence vs. Citations

More CitationsFig. 2. A framework for scholarly impact: citations vs. discursive influence.

not conceptual innovations that made up its impact (22). Whenwe mapped comment- and reply-type APS publications to theframework of Fig. 2, 54% were in the lower right quadrant, wherethey are highly cited but have low influence. Critical commentsand replies kill existing lines of inquiry rather than birth newones, offering corrections, rebuttals, or direct refutations with-out producing new concepts or terminology. As a result, theyattract citations but do not stimulate discursive emulation. Only6% were in the high influence/low citations region.

Many highly influential but undercited papers tended to beauthored by early innovators whose discursive contributionsbecame pervasive over generations. This is also artificially thecase for articles in the first years of the corpus that channelideas borrowed from documents published prior to our sample.We empirically defined the first year to the year during whichmean document influence was within 2 SDs of the global meanas a burn-in period, up until 1942 for APS. Excluding theseearly papers, high citation/low influence papers are from authorswho contributed innovations that are primarily nontextual innature (data, methods, research-killing refutations, etc.), whilehigh-influence/low-citation papers are from conceptual innova-tors whose language has become assumed within a field.

Scholarly Reach in JSTOR Repository. While physics forms a com-munity with shared publishing habits, research in JSTOR offersa wider sample of academic traditions. After estimating a static,500-topic model and discovering that using 53 topics best char-acterized the corpus, we fit the 53-topic dynamic model to fulltexts in JSTOR and found that citations were modestly corre-lated with document influence (ρ< 0.17, p< 0.01) over all top-ics. The diversity of JSTOR means that our sample includes doc-uments from vastly different citation cultures. The correlationbetween citation counts and influence within individual topicswas varied and weak. When citations were scaled by document–topic mixtures (θd,kCd ), however, the correlations were all sig-nificantly more positive (SI Appendix, Table S6). This suggeststhat, while subjects have different citation habits, our measure ofinfluence is sensitive to the subject variation captured by distincttopics. Using JSTOR’s subject taxonomy, citations were groupedby their distance: zero, one, or two. Most citations are within sub-ject (i.e., distance 0), while 10% have distance 1 and 5% havedistance 2. Both outgoing and incoming citations show a pref-erence for influential papers (Fig. 4 B and C). This shows thatinfluential papers reach farther across disciplines in their refer-ences. Likewise, influential papers are cited from farther away.Influential papers are both more likely to cite work beyond theirsubject and to be cited from other subjects. Moreover, influential

articles that receive citations from distant work have a higherratio of influence to citations than works of more local interest.This suggests that the citation statute of limitations illustratedabove may operate in not only time but also scientific space,releasing scholars from the obligation to cite influential work thatis sufficiently old or topically distant from the influenced work,where the originating authors are not present to claim them.

Predicting citations is notoriously difficult (15, 17), in partbecause many papers are never cited. Looking at which papersreceive any citations (29% in JSTOR), more influential paperswere more likely to be cited outright (Fig. 4A). Document influ-ence is also predictive in citation models: a logit model predict-ing citedness with document influence and publication date, td ,of the formCd > 0∼ Id × td +1 estimated a strong positive effectfor document influence [β(Id)= 0.42,P > |z |=0.000]. Influencealso helps predict actual citation counts. Using logistic link neg-ative binomial regressions of the same form as above, the fullyspecified model estimated similar effects with β(Id)= 0.33 (SIAppendix). Similar models fit to each topic produced compara-ble results. We also found topicwise variation in how citation isrelated to influence for authors. Authors’ marginal impact ontheir documents’ influence was more strongly correlated withtheir citation counts (µAuth ×CAuth) in humanities and social sci-ences (e.g., philosophy, literary theory, and education topics).In mathematical and natural science fields (e.g., cell biology,physical chemistry, and various statistics topics), the correlationwas considerably weaker and in many cases, nonexistent (SIAppendix, Table S7). This suggests that authors in “narra-tively” driven areas are much more likely to have citations con-ferred on them for having influenced the total flow of discoursethan authors from empirical and formal fields, who may bemore likely credited for more specific contributions.

The relationship between influence and citations in JSTOR istopic-specific, but some documents contribute to a variety of top-ics. We sampled some of the most cited documents in each per-centile of the influence distribution that were published after theburn-in cutoff (1930 for JSTOR) (SI Appendix, Fig. S9). In thelowest percentile, the top-cited document was 1939’s “A systemfor marking turtles for future identification” by Fred R. Cagle(37). The paper, which lives up to its title, remains highly cited inecology and zoology research. By explaining a technique, “A sys-tem for marking turtles for future identification” offers little inthe way of topic contribution: 9 topics were significantly changedby its publication, while the remaining 44 go unchanged. Cagle’s

Fig. 3. Topic contributions (Eq. 10) and citations for Felix Bloch’s “Nuclearinduction” (33) (Upper) and Philip Wallace’s “The band theory of graphite”(34) (Lower). Both papers featured the typical contribution profile wherethey affect change in a few topics, which diminishes slowly over time. Eachpaper also exhibits a late spike in citations matched by a coincident spike incontribution to a specific topic (labeled).

Gerow et al. PNAS | March 27, 2018 | vol. 115 | no. 13 | 3311

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0

BA

C

Fig. 4. (A) Violin plot of document influence (y axis; kernel density esti-mate bandwidth = 0.1) for cited and uncited documents in JSTOR groupedby decade; 2010 is omitted as incomplete. (B and C) Histograms and densityestimates of (B) incoming and (C) outgoing citations among JSTOR docu-ments grouped by distance in the subject tree.

paper (37) typifies low-influence/high-citation papers that maketechnical offerings adopted by future research, but that do notincite discursive shifts. In the 50th percentile, 1965’s “Popula-tion fluctuations and clutch-size in the great tit” by C. M. Perrins(38) was the most cited (504 times within sample). Perrins’ dis-cursive contribution is spread over a range of topics and time.It immediately shifted discourse in biology- and ecology-relatedtopics, fields where it continues to be cited today as a landmarkpaper. After publication, topics in statistics, group behavior, sex-ual health, medicine, and psychology adopted some of the con-ceptual terminology and remained changed to the end of thecorpus. These contributions owe much to Perrin’s use of termsabout population dynamics, movement, procreation, and individ-ual variation. Finally, the highest cited paper in the 90th per-centile of influence was Sam Peltzman’s 1976 “Toward a moregeneral theory of regulation,” (39) which had an influence +3.5SD above the mean and 701 citations. Recall that influence isa latent variable fit within the model, while contribution is apost hoc estimate of how different topics would have been with-out a given document. Peltzman’s 1976 paper (39) has a typi-cal contribution profile—significantly influencing a few topics butonly slightly altering most—although it sits near the top of theinfluence distribution. This is because much of its influence wasderived extrinsically: it was published by an eminent researcher(who in 2013, Wired magazine listed as 1 of 28 top scientists with-out a Nobel Prize) in the high-impact Journal of Law and Eco-nomics, and it was published by the University of Chicago Press, aleading publisher of economics research. “Toward a more generaltheory of regulation” exemplifies a configuration to which manyresearchers can relate: good papers are often made more so withcontextual boosts, like authorship, venue, and publisher.

DiscussionOur measure of influence tracks changes in future discourseand explicitly identifies the content and context of previous doc-uments that affect these discursive shifts. Document influenceprovides a direct measure of impact that allows us to disentan-gle many dimensions of influence across domains of science andscholarship. The model also estimates lasting contributions totopics over time and helps discover influential but undercitedwork. Our measures not only help predict citations better thanprevious similar models, but they provide an explanation of whatdrives influence. Most importantly, the model reveals how dis-cursive innovations are adopted and credited. Alongside cita-tions, discursive influence brings us closer toward a full-spectrumestimate of scholarly impact.

Assessing scholarly influence is a retrospective task, and it canbe distorted by conflating citations with impact. Not only doesour method enable previously impossible analyses of discursiveinfluence, it could help authors decide what to cite and why.On the one hand, citations provide a simplified, censored, and

synoptic trace of acknowledged influence—an important traceof impact. On the other hand, contributing change to scholarlydiscourse is another measurable kind of influence that can beexplained over time, across individual subjects, and with respectto authors, institutions, and publication venues, all of which con-tribute to the complex evolution of scholarship.

Materials and MethodsThe APS collection contains 509,007 abstracts from 1913 to 2015 coded withtype (article, review, commentary, etc.), journal, authors, and their institu-tional affiliations. To avoid spurious metadata, only papers with an authorfound twice and an affiliation that occurred three times were retained.This resulted in 74,459 covariates, τ , over 251,382 documents dating from1918 to 2015. Stop words, infrequent words, and statistically uninterestingwords (by term frequency–inverse document frequency) were discarded. Thefinal vocabulary contained 15,312 tokens. The model was fit with 37 topics,a value chosen by assessing topic use in a static, Bayesian nonparametricmodel with 500 topics (SI Appendix). Labels were given by three researcherswith doctoral degrees in physics.

The JSTOR collection consists of over 2 million documents from 1894to 2014; 28,861 covariates were coded in τ representing authorship, jour-nal/venue, publisher, and discipline. Documents were excluded if they didnot have at least one author with three or more documents or if they wereclassified in disciplines with a 20-y gap, representing subject instability ordeath (e.g., railroad science). The final sample contained 428,034 full-textarticles. The vocabulary was processed similarly to APS and resulted in 20,155tokens. A model with 53 topics was estimated, with the number of top-ics selected by fitting a 500-topic static model. Topics were labeled by theauthors and assisted by Google Scholar. Discussion of the data, model speci-fication, a closer inspection of the resulting topics, and the labeling processare in SI Appendix.

The Generative Model. We assume that influence is drawn from a Gaussian,the mean of which is given by a projection on τt,d :

`d,k ∼N (µkτd ,σ2`I), [1]

where µ is a K× S matrix of topic-specific coefficients. We assume that µ isdrawn from a centered Gaussian of specified variance:

µk ∼N (0,σ2µI). [2]

The generative process for each time slice t is as follows.

1) Draw topics βt+1|βt , (w, `, z)t,1:Dt∼N (βt + exp(−βt)

∑d `d

∑n wd,nzd,n,

σ2I)2) Draw coefficients µk ∼N (0,σ2

µI)3) For each document d at time t:

a) Draw θt,d ∼Dir(α)b) For each word wt,d,n

i) Draw zt,d,n∼Mult(θt,d)ii) Draw wt,d,n∼Mult(π(βt,zt,d,n

))

c) Draw `t,d ∼N (µτt,d ,σ2`I),

where π(x) maps the multinomial parameters x to their mean.

Approximate Inference. The model has latent variables for words, docu-ments, time slices, topics, and covariate coefficients. Collapsed Gibbs sam-pling and direct expectation maximization are precluded by nonconjugacyin the topic parameters {β1, · · · , βT}. Instead, the model estimates varia-tional parameters that minimize the Kullback–Leibler (KL) divergence to thetrue posterior. Here, the topic parameters are a Gaussian chain governed bythe variational parameters {βk,1, · · · , βk,T} that describe their means. Thelatent variable µ is also fit by variational estimation to the approximate µ.The variational distribution for a document’s influence is given by the Gaus-sian of the mean influence and specified variance. This variational distribu-tion, q(β, `, θ, z,µ|β, ˆ, γ,φ, µ), is

K∏k=1

q(βk,1, · · · , βk,T |βk,1, · · · , βk,T )×K∏

k=1

q(µk|µk)×

T∏t=1

( Dt∏d=1

q(θt,d|γt,d)q(`t,d|ˆt,d)×Nt,d∏n=1

q(zt,d,n|φt,d,n)

). [3]

The simplified objective is fit by expectation maximization. Two terms ofthe evidence lower bound are related to `, which requires expectation- and

3312 | www.pnas.org/cgi/doi/10.1073/pnas.1719792115 Gerow et al.

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0

SOCI

AL

SCIE

NCE

SA

PPLI

EDPH

YSIC

AL

SCIE

NCE

S

maximization-step updates that incorporate the projection in Eq. 1. Wedefine X = Diag(exp(−βt,k))(wt ◦ zt,k) and ∆βt,k = βt+1,k − βt,k. With Scovariates, τd is an S-length vector, and τ is an S×D matrix coding observedcovariates across D. Thus, µ is a K× S matrix, the values of which will con-verge to an estimate of each covariate’s effect on `. The lower bound ofˆ

t,d,k is given by

Lˆt,k

=1

σ2Eq

[XT

∆βt,k

]ˆ

t,k −1

2σ2Eq

[XTX

]ˆ2

t,k −1

2σ2d

(ˆt,k − µkτt)2. [4]

This provides the E-step update for influence:

ˆt,k←

(Eq

[XTX

]+σ2

σ2d

I)−1(E

q

[XT

∆βt,k

]+ µkτt

). [5]

The lower bound of µ then is given by

Lµk=∑

t

∑d

∑k

−1

2σ2`

(ˆt,d,k − µkτt,d)2−∑

k

µ2k

2σ2µ

. [6]

This yields the M-step update for the coefficients:

µTk←

(∑t

∑d

τt,dτTt,d +

σ2`

σ2µ

I)−1∑

t

∑d

τt,dˆ

t,d,k. [7]

µ is initialized by random draws from a centered, multivariate Gaussian,and its maximum likelihood estimation is done by variational expectationmaximization.

Model-Derived Metrics. Document influence is the sum topic-proportionalinfluence:

Id =∑

k

θd,kˆ

t,d,k. [8]

This offers a plausible interpretation: a document’s influence is proportionalto how much it changes the topics from which it draws.

Whereas ˆd,k is a static feature, the topic contribution of a document over

time is also accessible. An estimate of document contribution to a topic k attime t′ can be computed by simulating the topic without document d:

β(−d)t′ ,k =

∥∥∥βt′ ,k − (wdzd,kˆ

d,k)∥∥∥

1. [9]

This provides a conservative estimate, because it overlooks topic drift. Thecontribution can then be measured as the divergence between the topicwith and without d:

Contribution(d, t′) = KL(βt′ ,k, β(−d)

t′ ,k

). [10]

We define author persistence using the inverse independent entropy overtheir documents’ topic mixtures and their prolificness in terms of documentsand timespan. Given an author’s set of documents A, we define persistence as

Persistence(A) =

(1−

H(‖∑

d∈A θd‖1

)dHeK

)· (|A|+ ∆t(A)), [11]

where H(p) computes the entropy of probability distribution p as∑

i pi log(pi)and dHeK is the maximum possible entropy for a K vector. The first term–inverse entropy over an author’s total document–topic mixture–is scaled upby the number of documents |A| and the number of years over which theywere published, ∆t(A). This measure is unbounded: authors can be evermore persistent given more time and documents in the same topics.

ACKNOWLEDGMENTS. A.G. was supported by a fellowship from the IzaakWalton Killam Foundation. A.G. and J.A.E. were supported by a grant fromthe Templeton Foundation (to the Metaknowledge Research Network).J.B.-G. is supported by National Science Foundation (NSF) Grant NCSE-1422492. J.A.E. is also supported by Air Force Office of Scientific ResearchGrant FA9550-15-1-0162 and NSF Grant SBE1158803.

1. Hirsch JE (2007) Does the h index have predictive power? Proc Natl Acad Sci USA104:19193–19198.

2. Bergstrom CT, West JD, Wiseman MA (2008) The eigenfactor metrics. J Neurosci28:11433–11434.

3. Fersht A (2009) The most influential journals: Impact factor and eigenfactor. Proc NatlAcad Sci USA 106:6883–6884.

4. Price DdS (1976) A general theory of bibliometric and other cumulative advantageprocesses. J Am Soc Inf Sci 27:292–306.

5. Wang M, Yu G, Yu D (2008) Measuring the preferential attachment mechanism incitation networks. Phys A 387:4692–4698.

6. Newman MEJ (2004) Coauthorship networks and patterns of scientific collaboration.Proc Natl Acad Sci USA 101:5200–5205.

7. Alberts B (2013) Impact factor distortions. Science 340:787–787.8. Wilhite AW, Fong EA (2012) Coercive citation in academic publishing. Science

335:542–543.9. Wenneras C, Wold A (2001) Nepotism and sexism in peer-review. Women, Science,

and Technology, eds Wyer M, Barbercheck M, Cookmeyer D, Ozturk H, Wayne M(Routledge, London), pp 46–52.

10. Bartneck C, Kokkelmans S (2010) Detecting h-index manipulation through self-citation analysis. Scientometrics 87:85–98.

11. Zhivotovsky L, Krutovsky K (2008) Self-citation can inflate h-index. Scientometrics77:373–375.

12. Adam D (2002) Citation analysis: The counting house. Nature 415:726–729.13. Uzzi B, Mukherjee S, Stringer M, Jones B (2013) Atypical combinations and scientific

impact. Science 342:468–472.14. Whalen R et al. (2016) Citation distance: Measuring changes in scientific search strate-

gies. Companion to the 25th International World Wide Web Conference (WWW2016) (International World Wide Web Conferences Steering Committee, Republic andCanton of Geneva, Switzerland), pp 419–423.

15. Stringer MJ, Sales-Pardo M, Amaral LAN (2008) Effectiveness of journal rankingschemes as a tool for locating information. PLoS One 3:e1683.

16. MacRoberts MH, MacRoberts BR (2010) Problems of citation analysis: A study ofuncited and seldom-cited influences. J Assoc Inf Sci Technol 61:1–12.

17. Wang D, Song C, Barabasi AL (2013) Quantifying long-term scientific impact. Science342:127–132.

18. Althouse BM, West JD, Bergstrom CT, Bergstrom T (2009) Differences in impact factoracross fields and over time. J Am Soc Inf Sci Technol 60:27–34.

19. Harzing AW, Alakangas S, Adams D (2014) hIa: An individual annual h-index toaccommodate disciplinary and career length differences. Scientometrics 99:811–821.

20. Blei DM, Lafferty J (2006) Dynamic topic models. Proceedings of the 23rd Interna-tional Conference on Machine Learning (ICML 2006) (International Machine LearningSociety, La Jolla, CA), pp 113–120.

21. Wang X, McCallum A (2006) Topics over time: A non-Markov continuous-timemodel of topical trends. Proceedings of the 12th ACM International Conference on

Knowledge Discovery and Data Mining (SIGKDD 2006) (ACM Press, New York), pp424–433.

22. Gerrish S, Blei DM (2010) A language-based approach to measuring scholarlyimpact. Proceedings of the 27th International Conference on Machine Learn-ing (ICML 2010) (International Machine Learning Society, La Jolla, CA), pp 375–382.

23. Bird S et al. (2008) The ACL Anthology Reference Corpus: A reference dataset for bibli-ographic research in computational linguistics. Proceedings of the Sixth InternationalConference on Language Resources and Evaluation (LREC 2008) (European LanguageResources Association, Paris), pp 1755–1759.

24. Blei DM, Ng AY, Jordan MI (2003) Latent Dirichlet allocation. J Mach Learn Res 3:993–1022.

25. Blei DM (2012) Probabilistic topic models. Commun ACM 55:77–84.26. Griffiths TL, Steyvers M (2004) Finding scientific topics. Proc Natl Acad Sci USA

101:5228–5235.27. Chang J, Gerrish S, Wang C, Boyd-graber JL, Blei DM (2009) Reading tea leaves: How

humans interpret topic models. Advances in Neural Information Processing Systems,eds Bengio Y, Schuurmans D, Lafferty J, Williams C, Culotta A (Neural InformationProcessing Systems Foundation, La Jolla, CA), Vol 22, pp 288–296.

28. Wang C, Blei D, Heckerman D (2008) Continuous time dynamic topic models. Proceed-ings of the 24th Conference on Uncertainty in Artificial Intelligence (UAI 2008) (AUAIPress, Corvallis, OR), pp 579–586.

29. Shwed U, Bearman PS (2010) The temporal structure of scientific consensus formation.Am Sociol Rev 75:817–840.

30. Wallach HM, Murray I, Salakhutdinov R, Mimno D (2009) Evaluation methods fortopic models. Proceedings of the 26th International Conference on Machine Learning(ICML 2009) (Sage Publications, Los Angeles), pp 1105–1112.

31. Orbanz P, Teh YW (2011) Modern Bayesian nonparametrics. Advances in NeuralInformation Processing 24, eds Shawe-Taylor J, Zemel RS, Bartlett PL, Pereira F,Weinberger KQ (Neural Information Processing Systems Foundation, La Jolla, CA).

32. Orbanz P, Teh YW (2011) Bayesian nonparametric models. Encyclopedia of MachineLearning, eds Sammut C, Webb GI (Springer, Berlin), pp 81–89.

33. Bloch F (1946) Nuclear induction. Phys Rev 70:460–474.34. Wallace PR (1947) The band theory of graphite. Phys Rev 71:622–634.35. Ke Q, Ferrara E, Radicchi F, Flammini A (2015) Defining and identifying sleeping beau-

ties in science. Proc Natl Acad Sci USA 112:7426–7431.36. Marcus MP, Marcinkiewicz MA, Santorini B (1993) Building a large annotated corpus

of English: The Penn treebank. Comput Linguist 19:313–330.37. Cagle FR (1939) A System of marking turtles for future identification. Copeia

1939:170–173.38. Perrins CM (1965) Population fluctuations and clutch-size in the great tit, Parus major

L. J Anim Ecol 34:601–647.39. Peltzman S (1976) Toward a more general theory of regulation. J Law Econ 19:211–

240.

Gerow et al. PNAS | March 27, 2018 | vol. 115 | no. 13 | 3313

Dow

nloa

ded

by g

uest

on

Nov

embe

r 29

, 202

0