Group-based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims”

Group-Based Trajectory Modeling (GBTM) of Citations inScholarly Literature: Dynamic Qualities of “Transient”and “Sticky Knowledge Claims”

Susanne E. BaumgartnerAmsterdam School of Communication Research (ASCoR), University of Amsterdam, Kloveniersburgwal 48,1012 CX Amsterdam, The Netherlands. E-mail: [email protected]

Loet LeydesdorffAmsterdam School of Communication Research (ASCoR), University of Amsterdam, Kloveniersburgwal 48,1012 CX Amsterdam, The Netherlands. E-mail: [email protected]

Group-based trajectory modeling (GBTM) is applied tothe citation curves of articles in six journals and to allcitable items in a single field of science (virology, 24journals) to distinguish among the developmentaltrajectories in subpopulations. Can citation patterns ofhighly-cited papers be distinguished in an early phaseas “fast-breaking” papers? Can “late bloomers” or“sleeping beauties” be identified? Most interesting, wefind differences between “sticky knowledge claims” thatcontinue to be cited more than 10 years after publicationand “transient knowledge claims” that show a decaypattern after reaching a peak within a few years. Onlypapers following the trajectory of a “sticky knowledgeclaim” can be expected to have a sustained impact.These findings raise questions about indicators of“excellence” that use aggregated citation rates after 2 or3 years (e.g., impact factors). Because aggregated cita-tion curves can also be composites of the two patterns,fifth-order polynomials (with four bending points) areneeded to capture citation curves precisely. For the jour-nals under study, the most frequently cited groups werefurthermore much smaller than 10%. Although GBTMhas proved a useful method for investigating differencesamong citation trajectories, the methodology does notallow us to define a percentage of highly cited papersinductively across different fields and journals. Usingmultinomial logistic regression, we conclude that pre-dictor variables such as journal names, number ofauthors, etc., do not affect the stickiness of knowledgeclaims in terms of citations but only the levels of aggre-gated citations (which are field-specific).

Introduction

Group-based trajectory modeling (GBTM; Nagin, 2005)provides nonparametric statistics for distinguishing thedevelopmental trajectories of subpopulations in sets. GBTMis based on using mixed models for the prediction of differ-ent trajectories in the data. This technique was first devel-oped in fields such as criminology and clinical research inorder to distinguish in an early stage, for example, young-sters who would be inclined to criminal behavior in a laterstage of development (Nagin & Odgers, 2010a) or to predictthe further development of symptoms and interventions inthe clinic over time (Nagin & Odgers, 2010b). As opposed tostandard growth curve modeling, the group-based approachprovides statistics for distinguishing among clusters of tra-jectories within a population (Andruff, Carraro, Thompson,Gaudreau, & Louvet, 2009; Nagin, 2005).

In this study, we explore GBTM by applying it to citationpatterns in scientific literature. Citation patterns and citationwindows can be expected to vary among fields of science(Price, 1970), among journals, and among given covariatessuch as document types or numbers of authors and pages(Bornmann, Schier, Marx, & Daniel, 2012; Garfield, 1979).In addition to the well-known impact factor, the ScienceCitation Index (SCI)1 provides a number of journal indica-tors, such as the immediacy factor, the cited half-life, and soon, to trace such differences in development at the journallevel over time. Furthermore, Thomson Reuters, the currentowner of the SCI, provides ScienceWatch as an additionalservice: ScienceWatch lists fast-breaking papers at

Received March 18, 2013; revised April 29, 2013; accepted April 29, 2013

© 2013 ASIS&T • Published online 20 November 2013 in Wiley OnlineLibrary (wileyonlinelibrary.com). DOI: 10.1002/asi.23009

1We use “SCI” as a shorthand for the comparable databases for thesocial sciences (SSCI) and the arts & humanities (A&HCI). In this study,we use data from the SCI-Expanded version at the Web of Science (WoS).

JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY, 65(4):797–811, 2014

mailto:[email protected]

mailto:[email protected]

http://wileyonlinelibrary.com

http://archive.sciencewatch.com/dr/fbp/. Papers that areimmediately cited frequently can be considered as highlyrelevant and potentially ground-breaking (Ponomarev,Williams, Hackett, Schnell, & Haak, 2013).

Citation patterns tend to be heavily skewed and are there-fore far from normally distributed (Seglen, 1992). In recentyears, the use of arithmetic averages (such as implied in theimpact factors of journals) has been abandoned in favorof nonparametric statistics (Bornmann & Mutz, 2010;Leydesdorff, Bornmann, Mutz, & Opthof, 2011; Waltmanet al., 2012). Percentile distributions of citations can be usedfor calculating an “integrated impact indicator” (Leydesdorff& Bornmann, 2012), and, increasingly, consensus hasemerged for considering the top 10% as an “excellence indi-cator” (Bornmann, de Moya-Anegón, & Leydesdorff, 2012;Waltman et al., 2012; cf. National Science Board, 2012).

However, the delineation of this top 10% is again depen-dent on the citation time window, which may vary acrossjournals, fields, and document types. The cutoff at the90th percentile is inspired by administrative standards(Bornmann & Mutz, 2010; National Science Board, 2012)rather than being based on empirical evidence. The NSF, forexample, distinguishes six classes (top 1%, 5%, 10%, 25%,50%, and the remainder). However, it may also be that three(low, medium, high) or four classes (including a top group ofmost-highly cited outliers or a never-cited group) are suffi-cient. This article explores whether GBTM can provide anempirical solution to this problem. Is GBTM able to delin-eate empirically specific excellence and quality classesamong citations in different journals and fields?

In a recent study, Ponomarev et al. (2013) focused onanother indicator of ground-breaking research, “break-through papers.” Breakthrough papers are identified by theseauthors at a high citation threshold (0.1% of the most fre-quently cited articles; cf. Ponomarev, in preparation) andthus as having a strong impact on the field. The citationtrajectories of breakthrough papers as well as “excellent”papers are expected to show high levels of citations from thebeginning; that is, they are immediately recognized as majorcontributions to the field. Citation rates of “excellent” papersare expected to remain high for a few years and then decline(Aversa, 1985; Price, 1976).

In contrast to these breakthrough papers, it has also beenargued that the citations to “late-bloomers” (Merton, 1988) or“sleeping beauties” (Van Raan, 2004) emerge only graduallyor after some years. These papers would follow a citationtrajectory of no or few citations in the first years after publi-cation, with a strong increase in citations after a few years.However, it is still unclear whether such late-bloomers areonly exceptional cases that can be recollected from narrativesbut are perhaps indicated idiosyncratically (Burrell, 2005). Isit possible to identify empirically a meaningful group of theselate bloomers? GBTM may be a useful method for identify-ing typical citation trajectories as well as detecting moreunique patterns such as breakthrough papers and late bloom-ers. In summary, GBTM may be of importance for the studyof citation trajectories in the following ways:

1. By using GBTM, one is able to identify the typical shapeof the development of citations over time. (After howmany years do citations peak? When is the typical declinein citations? Is there a strong fluctuation or stability overtime?)

2. GBTM makes it possible to identify subgroups of papersthat follow specific citation trajectories (e.g., break-through papers or sleeping beauties).

3. GBTM can perhaps be used to identify excellent andbreakthrough papers at an early stage.

4. GBTM allows us to compare citation trajectories acrossjournals, disciplines, and scientific fields.

5. Using the statistical distinction between subpopulations,one can further ask whether external variables or covari-ates (such as document or author characteristics) deter-mine the likelihood that papers will follow a specificcitation trajectory.

We explored GBTM by applying the technique to thecitation curves of six journals in different research fields aswell as to citations in one entire research field (virology)over a period of 16 years (1996–2011). We chose oursamples to generate an overlap with three (of the 11)papers studied as breakthrough papers by Ponomarevet al. (2013); these papers were published in 1996 in Cell,Nature, and Science, respectively. The other journals werechosen to explore potential differences and similaritiesacross fields.

Data

We focus on six journals in different fields, using onlyarticles published in 1996: the Journal of the AmericanSociety for Information Science (JASIS), the Journal of theAmerican Chemical Society (JACS), Cell, Gene, Science,and Nature. Moreover, we chose one research field, virol-ogy, including 24 journals. The choice of these six journalsfrom among the 6,120 journals contained in the JournalCitation Reports for 1996, and the choice of virology as afield, may seem somewhat arbitrary. As noted, the inclusionof Cell, Science, and Nature was motivated by the possibilityto compare our findings with those of Ponomarev et al.(2013).

In addition, we also chose JASIS, JACS, and Gene asjournals. JASIS is a leading journal in library and informa-tion science and provides us with familiar ground so that wecan interpret aggregated patterns in terms of the underlyingarticles. The routine is explicated using JASIS as our leadexample for the explanation. In 1996, JASIST was stillnamed the Journal of the American Society for InformationScience (JASIS), without the additional “and Technology”which was added only in 2001. The 1996 volume (vol. 47)contained 169 papers, of which we use only the 79 that areindicated as “research articles.”

One of us compared JASIST and JACS (Journal ofthe American Chemical Society) in previous studies(Leydesdorff, 1991; Leydesdorff & Bensman, 2006). JACSprovides us with a leading journal in one of the natural

798 JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—April 2014DOI: 10.1002/asi

http://archive.sciencewatch.com/dr/fbp/

sciences, whereas Cell and Gene are typically biomedical.To broaden the focus of the study further, we investigatedcitation trajectories for the multidisciplinary journals Natureand Science, which also have a slight focus on the biomedi-cal sciences.2 Citation trajectories for multidisciplinary jour-nals can be expected to differ from those for journals thatfocus on one specific research field.

Third, we extend the study to a research field (virology),operationalized as a Web of Science (WoS) SubjectCategory. Virology was chosen for pragmatic reasons: Weexpected this field to be highly cited, sufficiently diverse,and relatively small. We retrieved the citable items in the 24journals subsumed under this category in 1996: 110 reviewarticles and 161 letters in addition to the 3,958 articles. Onemight expect reviews to be (significantly?) more frequentlycited than research articles,3 and letters in specialist journalsless frequently cited than the average article (see Figure 3 ofLeydesdorff, 2008, p. 280).

In the final step of the analysis, using mutinominalregression analysis, we investigate whether the type of pub-lication as well as other covariates (i.e., number of authors,number of references, page numbers, and journal name) canpredict specific citation trajectories. In sum, the selectedjournals allow us to compare similarities and differences incitation trajectories from different fields. At this stage, suchan exploratory approach seemed more informative thandrawing a random sample from the journal domain. Table 1provides an overview of the journals and numbers of articlesin each journal. Of course, GBTM can be applied to all otherjournals and fields. This study is meant both as an explor-ative example of applying GBTM to citation curves and as acritical assessment of the usefulness of GBTM in citationanalysis.

Methods

Data were downloaded from the WoS interface in thesecond week of January, 2013. Because citations to the 2012volumes cannot be added until much later in 2013, we usecitation data for the period 1996–2011, that is, 16 years.As noted, 1996 was chosen to facilitate comparison withPonomarev et al.’s (2013) study of breakthrough articleswhile at the same time keeping similar timelines across thesets in order to maximize the possibilities for comparisonsamong the cases.

GBTM has been developed as a subroutine in both SASand Stata (Jones, Nagin, & Roeder, 2001). In this study, weuse SAS 9.2 and the corresponding implementation PROCTRAJ, freely available at http://www.andrew.cmu.edu/user/bjones. Among the three available models, the zero-inflatedPoisson (ZIP) model is most appropriate for citation countdata because one can expect more zeros than under thePoisson assumption (Lambert, 1992; cf. Hausman, Hall, &Griliches, 1984).4 The SAS syntax of the model is discussedin technical detail in the Appendix.

Model selection is pursued in three steps. First, we step-wise increase the number of groups in the model specifica-tion. The Bayesian information criterion (BIC) is used as atest statistic for selecting the number of groups that bestrepresents the heterogeneity among the trajectories. TheBIC is known for penalizing overfitting by introducing addi-tional parameters. The selection of the model with thelargest BIC is recommended,5 but model selection shouldeventually be based also on domain knowledge and reason-able judgment (Nagin, 2005; pp. 74–77). Furthermore,group sizes should be reasonably large (>5%). In the firstround, models with progressively more groups are testeduntil the model fit can no longer be improved.

After identifying the number of groups, different shapesfor the trajectories (linear, quadratic, cubic, etc.) can be testedin a second step. As the default, we assume that a citationcurve can be expected to bend twice over a longer period oftime, namely, first rising to an apex of citations after 2 or 3years (depending on the field of science) and then falling backagain in the decline phase to an asymptotic approach to zerocitations in the long run. We therefore defined all groups asfollowing a cubic shape in the first step of the model fittingprocess. The shapes of the curves, however, can be adaptedsubsequently to alternatives that best fit the various groups.

2Using 13 categories, the journal list compiled for the U.S. Science andEngineering Indicators series (National Science Board, 2012) by Patent-Boards™, classifies Science, Nature, and PNAS as biomedical journals.

3“In the JCR system, any article containing more than 100 references iscoded as a review. Articles in ‘review’ sections of research or clinicaljournals are also coded as reviews, as are articles whose titles containthe word ‘review’ or ‘overview’” (http://thomsonreuters.com/products_services/science/free/essays/impact_factor/, retrieved on February 14,2013).

4Rotolo and Messeni Petruzzelli (2013) provide arguments for why thenegative binomial estimation is more appropriate for modeling citation datathan assuming a Poisson distribution.

5BIC is calculated as: BIC = log(L) − 0.5k log(N), where L is the valueof the model’s maximum likelihood, N is the sample size, and k refers to thenumber of parameters in the model. To compare two models with differentnumbers of groups, the following estimate of the log Bayes factor is used:2loge(B10) ∼ 2(ΔBIC) (Andruff et al., 2009; Jones et al., 2001; Nagin,2005). To compute ΔBIC, the BIC value of the simpler model is subtractedfrom the more complex model, and this value is thereafter multiplied bytwo. In accordance with recommendations of Jones et al. (2001), an esti-mated log Bayes factor larger than five is considered as strong evidence forthe more complex model.

TABLE 1. Descriptive statistics of the sets under study.

1996 sets Articles

JASIST 169 79JACS 2,263 2,142Cell 466 346Gene 760 747Nature 3,104 873Science 2,791 1,064Virology 3,958 articles

110 reviews161 letters

(24 journals) 4,569 4,229

JOURNAL OF THE ASSOCIATION FOR INFORMATION SCIENCE AND TECHNOLOGY—April 2014 799DOI: 10.1002/asi

http://www.andrew.cmu.edu/user/bjones

http://www.andrew.cmu.edu/user/bjones

http://thomsonreuters.com/products_services/science/free/essays/impact_factor/

http://thomsonreuters.com/products_services/science/free/essays/impact_factor/

Once the ideal number of groups and shapes has beenidentified, in the third step, model adequacy can be tested byusing the average posterior probabilities (APP) of groupmembership. The APP measures the likelihood for eachscientific article to belong to its assigned group. Nagin(2005) recommends that the APP should exceed a minimumof .70 for each group. An APP above .70 indicates that, onaverage, research articles are well assigned to their groups.In the graphs, 95% confidence intervals can be provided toshow that confidence intervals of the identified groups donot overlap at specific time points.

Results

Citation Trajectories of JASIS Articles Published in 1996

For the 79 articles published in JASIS in 1996, we testedmodels from one to seven groups: The BIC values of thesemodels were −2,318.69, −1,773.22, −1,638.90, −1,619.49,−1,608.57, −1,605.32, and −1,619.20. The six-group solu-tion with all shapes defined as cubic therefore provided uswith the best fit (BIC = −1,605.32, log Bayes factor = 6.68).Because the six-group solution differs from the five-groupsolution only by distinguishing more subsets among theinfrequently cited papers, we chose the five-group solutionfor presentation. The APPs for the five groups range from.92 to 1.00, indicating that the research articles match excel-lently with their assigned groups.

Figure 1 shows the citation trajectories for these fivegroups during the 16-year time period. The five groups canbe interpreted as follows:

1. A first (and largest) group consists of 35.5% of the papers(n = 28) which are almost never cited, that is, with anaverage citation rate below unity. Over time, these papersapproach zero citations.

2. A second group of 30.4% of the papers (n = 24) is citedinfrequently (approximately once per year).

3. The third group of 23.0% of the papers (n = 18.2) is citedmoderately (fewer than four times). The citations in thisgroup also seem to decline more slowly over time.

4. The fourth group (6.0%, n = 4.7) consists of papers thatone could perhaps call “sleeping beauties.” These papersare only infrequently cited in the first years after publica-tion but their citation rates increase to an average ofalmost six citations in 2011.

5. Four papers (5.1%) are most frequently cited in thismodel; not surprisingly, these were also the most highlycited papers in the set of 79, with cumulative citation ratesranging from 78 to 132.

Note that the percentages are weighted in terms of theAPPs of group membership, so the numbers per group donot have to add up to whole counts. Some cases cannot beattributed unambiguously to one group or another.

Interestingly, the five-group solution shows a sleeping-beauty pattern for group 4. This pattern, however, was foundonly in the five-group model. In comparison, in the three- orfour-group solution, a top group of five papers (6.3%)emerges, and the other papers previously attributed to thesleeping-beauty group are assigned to the moderatelycited group.

Figure 2 provides the three-group solution in terms ofmost highly cited, medium cited, and rarely cited papers.These three groups obviously have to be distinguished, in

FIG. 1. Five groups distinguished in terms of average citation rates among 1,517 citations to 79 articles from JASIS 1996, during the period 1996–2011.Solid lines provide estimates; 95% confidence intervals are indicated with dotted lines. [Color figure can be viewed in the online issue, which is availableat wileyonlinelibrary.com.]


our opinion, because they are different already at the inter-cept and nonoverlapping in terms of confidence intervalsfrom the first year after publication.

In summary, the five-group solution plotted in Figure 1 ismore precise, but Figure 2 provides an unambiguous sepa-ration into three groups. The highly cited group in this(relatively small) set contained four or five of 79 papers(5.1% or 6.3%, respectively). The five-group model alsodistinguished a sleeping-beauty trajectory. Given the smallsample, however, it seems premature to infer to the existenceof this class of sleeping beauties. The percentage of mostfrequently cited papers was considerably lower than 10% inall models.

Journal of the American Chemical Society (JACS)

In the following sections, we present the models for JACS,Cell, Gene, Nature, and Science. Because the ZIP method ofGBTM is sensitive to outliers, we found convergence failuresin some of the cases. Citation distributions are extreme at bothends. The ZIPmethod accounts for the large numbers of zeros(noncitations) in the tails, but extreme values can also beexpected at the high end. In the case of JACS, for example, thethree most frequently cited papers (of 2,142) were cited2,969, 2,277, and 1,594 times, with numbers four and fivefollowing at much lower levels, with only 783 and 670aggregated citations. Under such conditions, GBTM fails toconverge; we decided to consider these (three) outliers asanother group to be excluded from GBTM.

The remaining 2,139 papers were included in the GBTManalysis. The BIC values in this case continued to increasewhen more groups were added until, with 10 or moregroups, the program failed to converge. Thus, the BIC value

could not be used as a criterion for distinguishing thenumber of groups in this case. In such cases, Nagin (2005, p.74 ff.) recommends using domain knowledge to determinethe number of groups.

Comparing the different models with one another, wefound that, when more groups are added, the model distin-guishes mainly among the least cited papers. However, inthe case of citation trajectories, one is most interested in themore frequently cited papers. Therefore, in the model selec-tion process, we stepwise added more groups until no furthermeaningful distinction among the more frequently citedpapers could be found.

After inspection of the possible models, we consideredthe seven-group solution as most informative (Figure 3).This model allows us to show that, from the top to thebottom, the first two groups (group 7 with 1.3% of thepapers and group 6 containing 4.1%) are different in termsof continuing an initially similar (steep) increase in thenumber of citations. The curve of group 6 reaches a peaklevel after a few years and declines thereafter. The topmostgroup, however, reaches a high level of citations and remainsthere. Although perhaps overcharging the terminology, onecould say that these papers have become “citation classics”within this domain (and over a long period of 16 years).

The next two groups (group 4 with 7.5% and group 5with 7.2%) differ similarly in terms of whether the citationcurve bends back to asymptotically approaching zero orremains at a plateau through the entire period. Thus, we findthis distinction both among the most frequently cited papersand the moderately cited papers. Perhaps the absolute levelof citedness can be considered as indicative of the intellec-tual fine structure in this field in terms of specialties, forexample, organic, inorganic, and physical chemistry

FIG. 2. Three trajectories distinguished in terms of average citation rates among 1,517 citations to 79 articles from JASIS 1996, during the period1996–2011. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


(Leydesdorff & Bensman, 2006; cf. Leydesdorff, 1991),with different average citation rates (Garfield, 1979). Both inthe highly cited and in the moderately cited groups,however, a further distinction can be made betweencitation patterns that last for more than 10 years and citationpatterns that decay. We propose to name this difference incitation patterns as “sticky” vs. “transient” knowledgeclaims.

For reasons of presentation, we did not add the confi-dence intervals for all groups in Figure 3. Inspection of themodel depicted in Figure 3 made us aware that citationcurves cannot properly be considered as following third-order polynomials but that fifth-order polynomials would bemore appropriate. With fifth-order polynomials, the curvesfit almost perfectly, with an explained variance of R2 > .95for all seven trajectories. This is shown in Figure 4. Usingfifth-order polynomials in GBTM increases the BIC furtherto −72,323 as against −74,546 for third-order ones. TheAPPs range in this model between .93 and 1.00, indicatingthat the papers can be matched almost precisely into theseseven groups.

Note that the size of the groups changed slightly whenusing fifth-order polynomials compared with the third-orderpolynomial groups. For example, the virtually noncitedgroup of papers is 26.31% in the case of using fifth-orderpolynomials and 26.22% with third-order polynomials.Further increases in the number of groups improved the BICvalues in this case, but again this affects the grouping mainlyof less-cited articles. In other words, it seems not possible toderive an optimal number of groups without making a quali-tative judgment. In summary, we found the following:

1. A number of subpopulations with different average levelsof citations

2. Within the highly cited and moderately cited groups, afurther distinction between articles that have longer termvalue (“sticky knowledge claims”) and articles that typi-cally function as references at the research front for onlya few years (“transient knowledge claims”; Price, 1970;further study of the less-cited groups also reveals thisdistinction at lower absolute levels

3. A differentiated structure in the lower three subgroupscovering 80.0% of the articles: 26.3% almost uncited;32.5% cited incidentally; and 21.2% with a tendency toremain at the level of four citations per year, which there-fore can be expected also to contain sticky knowledgeclaims, but at a much lower level.

Cell and Gene

Among the 346 articles published in Cell in 1996, the twomost frequently cited papers belong to the group of outlierswith 3,204 and 2,389 total citations; the third and fourthmost highly cited papers had 1,848 and 1,830 total citationsduring this same period. For Cell, a three-group model fittedthe data best (BIC = −27,372.01). The BICs for the four- andfive-group model were −27,392.45 and BIC = −27,412.90,respectively. Figure 5 shows these three groups6; the paperswere perfectly assigned to the groups, all APPs being 1.00.

As noted, we chose the journal Cell because Ponomarevet al. (2013) included Hicke and Riezman (1996) as one oftheir set of “fast-breaking papers” in this year. However, thisarticle is not part of the most highly cited group, but asshown in Figure 5, bends back in the second year after

6The default starting values provided by SAS failed to reveal anadequate model. Therefore, starting values were specified in this case (seehttp://www.andrew.cmu.edu/user/bjones/example.htm for more informa-tion on this procedure).

FIG. 3. Seven trajectories of 2,139 articles published in JACS with publication year 1996. [Color figure can be viewed in the online issue, which is availableat wileyonlinelibrary.com.]


http://www.andrew.cmu.edu/user/bjones/example.htm

publication (1998) to a lower citation level, so that itbecomes unambiguously (with a posterior probability of1.00) part of the intermediate group. Not surprisingly, thefifth-order polynomials again fit with a very high level ofprecision (R2 > .95) for all three groups. (Note that the fit iseven R2 > 0.81 for the single case of the fast-breakingpaper.).

In contrast to JACS, we did not find a group of papersfollowing a sticky citation pattern for Cell. Because Cell is abiomedical journal, it may be argued that the research front

in the biomedical sciences moves faster than in other naturalsciences. Perhaps citations decline faster in biomedicalscience than in the natural sciences, making a sticky citationpattern less likely.

To test this assumption, we included an additional journalfrom the biomedical sciences in the analysis, Gene. Gene isin many respects comparable to Cell but is somewhat morespecialized. Although citation rates were lower on average inthis journal than in Cell, the most frequently cited groupof papers (2.61%) convincingly shows a high degree of

FIG. 4. Seven trajectories as in Figure 3, but using fifth-order polynomials. [Color figure can be viewed in the online issue, which is available atwileyonlinelibrary.com].

FIG. 5. Three significant trajectories on the basis of 344 articles published in Cell in 1996; fifth-order polynomials. The fast-breaking paper studied byPonomarev et al. (2013) is added as a dashed line. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


stickiness in their citation patterns (Figure 6). This indicatesthat, also in the biomedical sciences, sticky citation patternscan occur.

Nature

Among the 873 articles published in Nature with publi-cation year 1996, nine had to be removed as outliers in orderto find a converging solution with GBTM. Four groups withfourth-order polynomials fitted the data best.7 The BIC forthis model is −60,734.15; all APPs are 1.00.

In this case, the fast-breaking paper of Nussenzweig et al.(1996; see Ponomarev et al., 2013) is attributed unambigu-ously (APP = 1.00) to the medium-range group (group 2); itpeaks at 52 citations in 1999. Although this paper was fast-breaking immediately following its publication, GBTMshows that it does not belong to the most highly cited papersin Nature of this publication year. Furthermore, the declinein citations to this paper is rather striking, in contrast topapers that remain cited at a higher level throughout theseyears. In terms of cumulative total citations, it ranks only174th among the set of 873 articles in the same journal(Nature) and with the same publication year (1996).

Comparison between Figures 5 and 7 (and also Figure 1)shows us the differences among fits when using differentorders of polynomials. Polynomials with an order lower thanfive are not able to fit to the citation distributions because ofthe specific shape of the peak in the first few years. Thefit with fourth-order polynomials shown in Figure 7 isimproved compared with the fit with third-order polynomials

in Figure 1, but the fit with fifth-order polynomials (inFigure 5) is precise. We discuss this issue in more detail in theDiscussion.

Science

Among the 1,064 articles published in Science in 1996,17 outliers had to be removed before we were able to find aconverging solution. We used fifth-order polynomials to fitthe remaining 1,047 articles. A five-group model emerged asthe best fitting model. The groups are shown in Figure 8.

In this case, the fast-breaking paper included in the setof Ponomarev et al. (2013) belongs to the group of outliersamong the reference set. (We included this group inFigure 8; as noted, it cannot be included into GBTM.) At itspeak in 2002, this fast-breaking paper (Altman et al., 1996)was the most frequently cited one in the set, with 292 cita-tions, but it exhibited the expected decline in citation rates inthe years thereafter. Four other outliers had much highercitation rates in subsequent years (with approximately 600citations per year for the highest ranking one). These fourcan be considered as sticky knowledge claims, whereas thefast-breaking one followed a transient pattern.

Figure 8 shows the excellent fits to the fifth-order poly-nomials including a fit of R2 > .95 for the single case of thefast-breaking paper. Interestingly, the overall decline in thetails of the distributions is less steep than in the case ofNature. Although Nature and Science are typically consid-ered similar journals, these citation patterns may indicatethat there are also differences. One reason for this differencein citation trajectories for the most highly cited papers maybe that Science is less oriented toward the biomedicalsciences than Nature.7Starting values had to be specified in order to find this optimal solution.

FIG. 6. Citation trajectories of 746 cited articles published in Gene during 1996. Only a single outlier had to be removed among the 747 articles publishedin Gene in 1996. GBTM with fifth-order polynomials failed to converge, but the fourth-order ones did. We added the trend lines with fifth-order polynomialsto this figure in Excel. [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


Virology

Can GBTM also be applied to citation trajectories for awhole research field? We expanded the analysis to theresearch field virology, represented by 24 journals. In addi-tion to articles, we included also reviews and letters asdocument types.

The choice of the number of groups remained a bit arbi-trary also in this case, but the two top groups stabilize aftersix groups are distinguished. The six-group solution is

depicted in Figure 9. The two top groups consist of 1.18%(n = 35.8) and 4.33% (n = 183.1) of the papers, and, asbefore, these sets are much smaller than the top 10% of theExcellence Indicator. However, one can also reason that thetop 10% would at least include all these excellent papers(Waltman et al., 2012).

The six-group solution provided us also with an oppor-tunity to compare these empirical results with the normativeframework of the NSF (e.g., Bornmann & Mutz, 2010;National Science Board, 2012), which uses a scheme of top

FIG. 7. Four citation trajectories of 864 articles published in Nature during 1996; fourth-order polynomials. [Color figure can be viewed in the online issue,which is available at wileyonlinelibrary.com.]

FIG. 8. Citation trajectories of 1,064 articles published in Science with publication year 1996. [Color figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]


1%, 5%, 10%, 25%, 50%, and bottom 50% in the rankings.The group sizes found using GBTM do not differ much fromthis distinction (as shown in Figure 9). With the discreteclasses defined by Bornmann and Mutz (2010), a strongoverlap between our groups was found: r = 0.87 (p < .05), ρ= 1.00 (p < .01). If the top 5% of the NSF includes the top1%, etc., using aggregation (National Science Board, 2012),our aggregated classes would be 1.2%, 5.5%, 17.7%, 41.5%,66.8%, and 100.0%. The normative and empirical distribu-tions of the classes are then virtually identical, with r = 0.98(p < .01), ρ = 1.00 (p < .01).

If further groups are added to the model, we againfind sticky and transient knowledge claims. Figure 10shows the model with 10 groups. This fine structuremakes clear that, both among the top-level papers0.38% (n = 16 papers, group 10) and among the fifthgroup in the middle range (3.01% or n = 334.2), articleswith sticky knowledge claims can be distinguishedfrom transient knowledge claims. Similarly, but at alower level, group 7 can be considered as exemplifyingsticky knowledge claims, whereas group 8 shows transientones.

FIG. 9. Six citation trajectories of 4,229 articles, reviews, and letters published in the WoS Subject Category “virology” with publication year 1996 (BIC= −107,949). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]

FIG. 10. Ten citation trajectories among 4,229 articles, reviews, and letters published in the WoS Subject Category “virology” with publication year 1996(BIC = −104,793). [Color figure can be viewed in the online issue, which is available at wileyonlinelibrary.com.]


In comparison with groups 8 and 9, the transient trajec-tories of groups 7 and 10, respectively, peak earlier than thesticky knowledge claims. Whereas the sticky trajectoriespeaked only after 4 years, the transient groups had peakedalready after 2 years. This suggests that typical indicators ofexcellence that take only the first 2 years after publicationinto account fail to distinguish between these two potentialpathways. In contrast with papers that follow a transientpattern, papers following sticky trajectories may have amore sustained influence on the research field and maytherefore be better indicators of excellence independently ofthe absolute levels of the citations within each category.

Covariates

The previous sections have shown that GBTM allows usto distinguish between different citation patterns over time.Most importantly, we saw not only that there are differentlevels of citations (in terms of total numbers of citations) butthat one can distinguish also between sticky and transientknowledge claims within each level. In this section, weaddress the question of whether specific covariates canpredict which trajectories specific papers can be expected tofollow. We show this by using the six groups distinguishedfor the virology papers in the previous section (see Figure 9)as well as using the 10 groups as presented in Figure 10.

As predictor variables we used document type (article vs.review vs. letter), number of authors, number of references,number of pages, and journal name (virology is representedby 24 different journals). The variables at the interval scale(number of authors [NAU], number of references [NREF],and number of pages [NPG]; cf. Bornmann et al., 2012)

were used as independent variables in a multinominal logis-tic regression with group membership as the dependent vari-able. In the first regression, group membership was based onthe six-group model shown in Figure 9; the sixth group wasdefined as the reference group in the multinominal logisticregression.8 The results are provided in Table 2.

The NAU significantly predicted whether a documentbelongs to the sixth group in comparison with almost allother groups (p < .05; only the difference between group 5and group 6 was not significant at p = .06). The more authorsan article had, the more likely the article belonged to thehighest cited group. Furthermore, the number of referencessignificantly differentiated the groups with lower levels ofcitations (groups 1 to 3) from the highest group. The morereferences an article had, the more likely it was to belong tothe highest-citation group in comparison with the threelowest citation groups. However, the NREF did not signifi-cantly differentiate among the more frequently cited groups4 to 6. The NPG of a document did not significantly predictgroup membership.

In a second multinominal logistic regression, the differ-entiation between 10 groups was used as dependentvariable, with group 10 as reference group (see Figure 10).In this case, the covariates no longer significantly predictedgroup membership, except that the NREF among the lessfrequently cited papers is significantly different from that ofthe most frequently cited group. In summary, these analysesshow that the covariates were not relevant for the distinction

8The equation for multinominal logistic regression is:

logPr( )Pr( )

group

groupa b X b X b X

61 1 2 2 3 3= + + +

TABLE 2. Parameter estimates of the multinominal logistic regression.

B Std. error Sig. Exp(B)

95% Conf. interval for Exp(B)

Lower bound Upper bound

Group 1 Intercept 4.68 0.25 0.00NPG 0.03 0.03 0.40 1.03 0.97 1.09NREF –0.03 0.01 0.00 0.98 0.97 0.99NAU –0.08 0.02 0.00 0.92 0.89 0.96

Group 2 Intercept 6.07 0.27 0.00NPG 0.00 0.03 0.95 1.00 0.94 1.06NREF –0.06 0.01 0.00 0.95 0.94 0.96NAU –0.19 0.02 0.00 0.83 0.79 0.86

Group 3 Intercept 3.95 0.25 0.00NPG –0.01 0.03 0.76 0.99 0.94 1.05NREF –0.01 0.01 0.03 0.99 0.98 1.00NAU –0.05 0.02 0.00 0.95 0.92 0.98

Group 4 Intercept 2.94 0.24 0.00NPG –0.02 0.03 0.53 0.98 0.93 1.04NREF 0.00 0.01 0.41 1.00 0.99 1.01NAU –0.03 0.02 0.04 0.97 0.94 1.00

Group 5 Intercept 1.90 0.27 0.00NPG –0.03 0.03 0.39 0.97 0.91 1.04NREF 0.00 0.01 0.82 1.00 0.99 1.01NAU –0.04 0.02 0.06 0.96 0.93 1.00

Note. The reference category is group 6.


between transient and sticky knowledge claims as presentedin Figure 10, but they are predictors for the levels and aggre-gates of citations (Bornmann, Schier, et al., 2012), as is thecase in the six-group model.

For the two categorical variables, “journal name” and“document type,” we conducted additional χ2 tests. Thesetests show that group membership to the six or ten trajectorygroups significantly depended on the journal in which anarticle was published, χ2 = 1,384.93, p < .001 for the case ofsix groups, and χ2 = 1715.49, p < .001 for 10 groups. Forexample, 40 of the 50 papers in the top (sixth) group werepublished in the Journal of Virology. However, VoprosyVirusologii contributes with 78 papers and a total of 98citations to the least-cited group and with two papers to thesecond lowest group (15 and 11 citations, respectively).Other non-English-language journals (Bulletin de l’InstitutPasteur and Zentralblatt für Bakteriologie) remain in thelower three groups, but journals such as Acta Virologica andClinical and Diagnostic Virology are also not cited abovethese levels. These journals thus seem to serve nichemarkets.

The χ2 test for document type showed that significantlymore reviews belonged to the most highly cited groups incomparison with letters and research articles, χ2 = 231.20, p< .001 for the six groups, and χ2 = 242.54, p < .001 for the10 groups. Letters were more likely to be attributed to thelower cited groups than the most frequently cited ones.These results are in line with expectations.

Comparable to discriminant analysis, but in this caseincluding non-normally distributed and categorical distribu-tions, multinomial regression analysis also allows us to gen-erate classification tables that cross-table predicted andobserved group membership. Table 3 shows how well thecovariates predict group membership for the six-group andthe 10-group models (Table 3).

Not surprisingly, total “times cited” provides an almostperfect prediction of 95.4% in the case of six groups, but thisis much less the case when 10 groups are distinguishedbecause, as shown above, in this case similar citation ratescan indicate very different (sticky vs. transient) citation pat-terns. The predictive value of journal name is rather strong,with 41.0% for the six-group solution and 32.4% for the

10-group model. The additional predictive value of the othercovariates is rather low. Adding these covariates (documenttypes, numbers of references and coauthors) to the journalnames as predictors improves the quality of the predictiononly from 41.0% to 44.2% in the case of six groups and from32.4% to 34.5% in the case of 10 groups.

In summary, this analysis shows that some of the covari-ates can predict the number of citations in the aggregate butdo not allow us to distinguish between transient and stickyknowledge claims. Because the aggregated citation rates areoften taken over the last few years for assessment purposes,the indicators tend to focus on transient knowledge claims.

Discussion

Before we turn to our conclusions, let us first criticallydiscuss the usefulness of GBTM as a routine for identifyingcitation trajectories. The findings presented in this studyshow that GBTM can be applied successfully to citationtrajectories. Although GBTM has previously been used onlyto measure the development of individual behavior overtime, the present paper shows that it may also be appliedvery well to the citation trajectories of documents. Theanalysis advances our knowledge of citation behavior byallowing us to describe different subgroups of articles thatfollow specific citation pathways over time. Instead oflooking only at cumulative citations, GBTM allows us todelineate specific citation pathways.

However, the method also has limitations. One majorproblem remains the empirical specification of the numberof relevant groups. In addition to using statistical param-eters, it was also necessary to use more subjective assess-ments to select the best models. Another problem withGBTM is that the program sometimes fails to converge forcomplex models. Only by omitting outliers or by definingstarting values could this problem be solved. Thus, the mainuse of GBTM remains its heuristic value: one can explorethe data and become informed about the number of groupsthat should at least be specified. Therefore, GBTM can atpresent not yet be routinized instrumentally to classify largesets of data (e.g., for automatic application to the thousandsof different journals in the database).

Although we had expected that a third-order polynomialwould fit the trajectories of citations, these polynomialsshowed a poor fit. Therefore, we exploratively tested higherorder polynomials. The analysis showed that fifth-orderpolynomials provided an excellent fit to the data. It seemedthat these shapes adequately described the citation curvesover the 16-year period. Surprisingly, the fifth-order poly-nomials fit to curves on all levels, indicating that the shapesof the trajectories are in this respect similar for infrequentlyand frequently cited articles.

Although higher order polynomials will always lead to abetter fit than lower order ones, and fifth-order polynomialsare extremely flexible so that the fit will easily be good, wewere able to specify this effect in terms of the extremelyrapid increase of citation curves in the first few years that

TABLE 3. Covariates as independent predictors of group membership inthe “virology” set (n = 4,229; 24 journals; articles, reviews, and letters).

Predictor variable

% Correctly predictedgroup membership

(six groups)

% Correctly predictedgroup membership

(10 groups)

Journals 41.0 32.4Document types 35.0 29.8No. of references 37.1 32.5No. of coauthors 34.4 30.1No. of pages 35.5 30.6Times cited 95.4 77.2


leads to a sharp peak that cannot similarly be fitted other-wise. The fit of the third-order polynomial misses this spe-cific characteristic of the citation curve. The (analytical)difference between a plateau and a declining phase and theirpossible mixtures in empirical cases make it meaningful toaccount for more bending points in the curve than two.

One disadvantage of the excellent fits with fifth-orderpolynomials (four bending points) is that significance van-ishes as a relevant criterion, because the 95% confidenceintervals become extremely narrow almost independently ofthe choice of the number of groups distinguished. Futureresearch may also investigate whether the fifth-order poly-nomials also fit the data well for shorter or longer timeperiods. For example, fewer bending points may be neededwhen using shorter time periods. Fitting the data to polyno-mials as is pursued when using GBTM, however, makes thismethod inappropriate for prediction beyond the time intervalunder study: the order of the polynomials is likely to deter-mine whether the curves extrapolate to either zero or infinityin a relatively limited number of time steps (cf. Ponomarev,in preparation).

Conclusions

We have explored the use of GBTM for distinguishingamong the citation curves of differently cited documents.This study advanced our knowledge about citation curves inseveral respects. Most important, the analysis revealed twodifferent citation pathways, which we named sticky andtransient knowledge claims. Papers that follow a sticky-knowledge citation trajectory continue to be cited through-out the years. These papers show a citation peak beyond 3 or4 years after publication, but the subsequent decline is lesssteep, and these articles can still be highly cited after morethan 10 years.

Papers that follow a transient knowledge trajectory showa typical early peak in citations, followed by a steep decline.After a couple of years, these papers are no longer fre-quently cited. These papers can be expected to fulfill ashort-term function at the research front. The distinctionbetween sticky and transient knowledge pathways was mostapparent in the case of JASIS, JACS, and Gene (seeFigures 1, 3, 4, 6) and in the case of virology (seeFigure 10). Sticky and transient knowledge claims can bedistinguished using GBTM only if sufficient groups are dis-tinguished. Within the virology set, for example, the two(analytically different) mechanisms remained entangledwhen six groups were distinguished but became apparentwhen 10 groups were declared.

Although one would expect stickiness to lead cumula-tively to high citation rates in the long run, the focus on highcitedness in the first 2 or 3 years induced by policy andmanagement incentives has increasingly led to definitions ofexcellence in terms of transient knowledge claims. The sci-ences are different in terms of the extent to which a researchfront sets the agenda (Price, 1970), and some journals mayfunction differently from others. It seems to us that this

raises a number of followup questions for further research,such as whether dynamic features of citation curves shouldbe introduced in performance and excellence measures. Thefailure of early prediction in terms of fast-breaking papersthat was shown in the case of the papers in Nature and Cellshows that the quality of a paper in terms of citation impactcannot be concluded in the years shortly after its publicationeven in the case of a prevailing short-term research front.

In line with expectations, the identified citation groupsdiffered not only in their citation patterns over time but alsoin a few other characteristics. In the case of virology, mul-tinominal logistic regressions as well as χ2 tests showed thatthe most highly cited group also differs from the low-citation groups in terms of the number of references citedwithin these papers and the number of coauthors. Further-more, citation patterns were also dependent on documenttypes (letters vs. research articles vs. reviews) and the jour-nals in which the documents were published (within thesame field). However, these covariates did not differentiatebetween papers following a sticky and transient knowledgetrajectory. This indicates that these specific predictors candifferentiate only between different levels of citations butnot different citation pathways. Future research is needed toinvestigate whether other indicators, for example, institu-tional addresses, might allow us to predict these differences.

In a few cases, GBTM analysis also indicated a sleeping-beauty pattern. This was most apparent in the case of JASIS(see Figure 1, group 4). To a lesser degree, this pattern wasalso found for JACS (see Figure 3, group 5). This lattergroup showed a sticky pattern and at least some character-istics of sleeping beauties. Why these sleeping beautiescould be shown in the case of these two journals cannot beconclusively argued here because of the limited set of jour-nals in this study.

Another finding is that the citation curves over time seemto be more complex than expected. GBTM showed thatfifth-order polynomials precisely matched the citationcurves. The fifth-order polynomial provides this excellent fitto citation curves because, in addition to the two bendingpoints that we expected, convex at the apex and concavewhen bounding back in the decline phase, two more bendingpoints can be expected in a potential plateau phase afterreaching the top. In other words, the citation curve is not anexponential decay curve. The fifth-order polynomial also fitsthe potentially steep increases of the citations in the firstyears.

Finally, this work shows that the most frequently citedgroups were in most cases much smaller than typical excel-lence indicators would predict. The most highly cited groupconsisted of 1% to 6% of the papers for the journals understudy. This indicates that there might be fewer papers thatshould be defined as excellent than the previously discussedtop 10%. GBTM is able to specify empirically the mostfrequently cited groups for each journal, but it is not possiblewith GBTM to define empirically an excellent group thatholds across journals or fields. Journals and fields, andjournals within fields, vary greatly in aggregated citation


behavior, both in absolute terms, that is, as aggregates ateach moment of time, and over time.

In sum, this paper introduces GBTM to citation trajecto-ries. Despite some limitations of the method, GBTM pro-vides us with new insights into the trajectories of citationsover a longer period. Most important, GBTM shows thatcitation curves are more complex and diverse than previ-ously expected. By differentiating between sticky andtransient citation patterns, the findings question typical“excellence” indicators that identify these papers in the firstfew years after publication.

Acknowledgments

We thank Daniel Nagin, Lutz Bornmann, IlyaPonomarev, and two anonymous referees for helpful com-ments on previous versions of the manuscript.

References

Altman, J.D., Moss, P., Goulder, P., Barouch, D.H., McHeyzer-Williams,M.G., Bell, J.I., . . . Davis, M.M. (1996). Phenotypic analysis of antigen-specific T lymphocytes. Science, 274(5284), 94–96.

Andruff, H., Carraro, N., Thompson, A., Gaudreau, P., & Louvet, B. (2009).Latent class growth modelling: a tutorial. Tutorials in QuantitativeMethods for Psychology, 5(1), 11–24.

Aversa, E.S. (1985). Citation patterns of highly cited papers and theirrelationship to literature aging: A study of the working literature. Scien-tometrics, 7(3), 383–389.

Bornmann, L., & Mutz, R. (2010). Further steps towards an ideal method ofmeasuring citation performance: The avoidance of citation (ratio) aver-ages in field-normalization. Journal of Informetrics, 5(1), 228–230.

Bornmann, L., de Moya-Anegón, F., & Leydesdorff, L. (2012). The newexcellence indicator in the World Report of the SCImago InstitutionsRankings 2011. Journal of Informetrics, 6(3), 333–335.

Bornmann, L., Schier, H., Marx, W., & Daniel, H.-D. (2012). What factorsdetermine citation counts of publications in chemistry besides theirquality? Journal of Informetrics, 6(1), 11–18.

Burrell, Q.L. (2005). Are “sleeping beauties” to be expected? Scientomet-rics, 65(3), 381–389.

Garfield, E. (1979). Is citation analysis a legitimate evaluation tool? Scien-tometrics, 1(4), 359–375.

Hausman, J.A., Hall, B.H., & Griliches, Z. (1984). Econometric models forcount data with an application to the patents-R&D relationship. Econo-metrica: Journal of the Econometric Society, 52(4), 902–938.

Hicke, L., & Riezman, H. (1996). Ubiquitination of a yeast plasma mem-brane receptor signals its ligand-stimulated endocytosis. Cell, 84(2),277–287.

Jones, B.L., Nagin, D.S., & Roeder, K. (2001). A SAS procedure based onmixture models for estimating developmental trajectories. SociologicalMethods & Research, 29(3), 374–393.

Lambert, D. (1992). Zero-inflated Poisson regression, with an applicationto defects in manufacturing. Technometrics, 34(1), 1–14.

Leydesdorff, L. (1991). The static and dynamic analysis of network datausing information theory. Social Networks, 13(4), 301–345.

Leydesdorff, L. (2008). Caveats for the use of citation indicators in researchand journal evaluation. Journal of the American Society for InformationScience and Technology, 59(2), 278–287.

Leydesdorff, L., & Bensman, S.J. (2006). Classification and powerlaws:The logarithmic transformation. Journal of the American Society forInformation Science and Technology, 57(11), 1470–1486.

Leydesdorff, L., & Bornmann, L. (2012). Percentile ranks and the inte-grated impact indicator (I3). Journal of the American Society forInformation Science and Technology, 63(9), 1901–1902.

Leydesdorff, L., Bornmann, L., Mutz, R., & Opthof, T. (2011). Turning thetables in citation analysis one more time: Principles for comparing sets ofdocuments. Journal of the American Society for Information Science andTechnology, 62(7), 1370–1381.

Merton, R.K. (1988). The Matthew effect in science, II: Cumulative advan-tage and the symbolism of intellectual property. Isis, 79(4), 606–623.

Nagin, D.S. (2005). Group-based modeling of development. Cambridge,MA: Harvard University Press.

Nagin, D.S., & Odgers, C.L. (2010a). Group-based trajectory modeling(nearly) two decades later. Journal of Quantitative Criminology, 26(4),445–453.

Nagin, D.S., & Odgers, C.L. (2010b). Group-based trajectory modeling inclinical research. Annual Review of Clinical Psychology, 6, 109–138.

National Science Board. (2012). Science and engineering indicators.Washington DC: National Science Foundation. Retrieved from http://www.nsf.gov/statistics/seind12/

Nussenzweig, A., Chen, C., da Costa Soares, V., Sanchez, M., Sokol, K.,Nussenzweig, M.C., & Li, G.C. (1996). Requirement for Ku80 in growthand immunoglobulin V (D) J recombination. Nature, 382(6591), 551–555.

Ponomarev, I. (in preparation). Breakthrough paper indicator 2.0: Cangeographical diversity and interdisciplinarity improve the accuracy ofoutstanding papers prediction.

Ponomarev, I., Williams, D., Hackett, C.J., Schnell, J.D., & Haak, L.L.(2013). Predicting highly cited papers: A method for early detection ofcandidate breakthroughs. Technological Forecasting and Social Change(in press).

de Solla Price, D.J. (1970). Citation measures of hard science, soft science,technology, and nonscience. In C.E. Nelson & D.K. Pollock (Eds.),Communication among scientists and engineers (pp. 3–22). Lexington,MA: Heath.

de Solla Price, D.J. (1976). A general theory of bibliometric and othercumulative advantage processes. Journal of the American Society forInformation Science, 27(5), 292–306.

Rotolo, D., & Messeni Petruzzelli, A. (2013). When does centrality matter?Scientific productivity and the moderating role of research specializationand cross-community ties. Journal of Organizational Behavior (in press).

Seglen, P.O. (1992). The skewness of science. Journal of the AmericanSociety for Information Science, 43(9), 628–638.

Van Raan, A.F.J. (2004). Sleeping beauties in science. Scientometrics,59(3), 467–472.

Waltman, L., Calero-Medina, C., Kosten, J., Noyons, E., Tijssen, R.J.W.,van Eck, N.J., . . . Wouters, P. (2012). The Leiden Ranking 2011/2012:Data collection, indicators, and interpretation. Journal of the AmericanSociety for Information Science and Technology, 63(12), 2419–2432.

AppendixThe syntax in SAS to fit models with different numbers

of groups and shapes can, for example, be formulated asfollows:

PROC TRAJ DATA=off OUTSTAT=OS OUT=OFOUTEST=OE OUTPLOT=OP ITDETAIL;ID id; VAR cit1–cit16; INDEP a1–a16;MODEL ZIP; NGROUPS 5; ORDER 3 3 3 3 3;

IORDER 2;run;%TRAJPLOT(OP,OS,‘Citations vs. Time’,

‘Zero Inflated Poisson Model’,‘Citations’,‘Time’)

The first line specifies where SAS should write the output,such as the file “OF,” containing the posterior-probabilityattributions for all cases. The parameter “ITDETAIL” pro-vides the value of the likelihood at each iteration. The vari-ables involved are declared in the second line: the citation


http://www.nsf.gov/statistics/seind12

http://www.nsf.gov/statistics/seind12

rates for 16 moments in time (cit1–cit16) given the respectivetime indicators (a1–a16) as independent variables.

“MODEL ZIP” in the third line specifies assuming thezero-inflated Poisson distribution; “NGROUPS = 5” asks fordistinguishing five groups; and “ORDER 3 3 3 3 3” specifiesthe initial assumption of cubic equations for the fit ofcitation curves with two bending points. The parameter“IORDER” specifies the (linear or nonlinear) function forthe correction of additional zeros given the assumption of aPoisson distribution9 The last line (%TRAJPLOT) asks SAS

to plot the data using these legends for the axes. The variousother parameters specify output files, such as “OP” contain-ing the time-series data that can also be plotted using, forexample, Excel. (The graphic interface of SAS is underde-veloped.) For a detailed tutorial on the model-fitting processusing SAS, the reader is referred to Andruff et al. (2009) andJones et al. (2001).

9Because we ran into problems with memory requirements using 8 GB,we decided to use the default for correcting the zero-inflation in the Poissondistribution instead of a nonlinear correction (using “iorder 2”). Only in thecase of JASIST, we present findings based on IORDER 2, but the differencebetween this model and the model based on the default value is marginal(behind the decimal point).


Documents

Group-based trajectory modeling (GBTM) of citations in scholarly literature: Dynamic qualities of “transient” and “sticky knowledge claims”