Research Trends Issue 25 December 2011

Research Trends Issue 25 December 2011 Page 01

Powered by Scopus

Research Trends Issue 25 December 2011 Page 02

Welcome to the 25th issue of Research Trends.

Welcome to the 25th issue of Research Trends, in which we focus on bibliometric data interpretation. Bibliometric indicators are derived from large databases, often using advanced methodologies. But the numbers themselves are not what count – the key issue is how they are interpreted. The articles in this 25th issue of Research Trends all illustrate the importance of a valid interpretation and the difficulties encountered in interpreting bibliometric results.

The first article, by Henk Moed, Andrew Plume, M’hamed Aisati and Peter Berkvens, looks at the interpretation of trends in publication counts by country. They refer to an article published by a distinguished researcher in the field, Professor Loet Leydesdorff, who has written a short reply.

Next, Judith Kamalski and Colby Riese show how differences in publication counts between individual researchers may say more about where they are in their scientific careers than about the quality of their research.

Anne-Catherine Le Calvez reports back on a meeting on global university rankings organized by the European University Association (EUA), with a focus on how European universities perceive such rankings.

The next article, by Matthew Richardson, reports on studies at the level of research groups showing how scale may influence their performance, especially the output per individual. Such analyses provide insights into what ‘critical mass’ means for research groups, and how it relates to productivity.

One of the key techniques in the field of bibliometrics, co-citation analysis, is the subject of a study by Gali Halevi. She shows that this technique has not only had an impact on research in the library and information science community, but also on inventions described in patents filed by major technological companies, especially during the past decade.

In the final contribution, Sarah Huggett provides us with some numbers on research articles dealing with the issue of data interpretation.

If you would like to comment on any of the topics covered, please use our feedback facility.

Kind regards,

Henk F. Moed, Editor-in-chief

Powered by Scopus

Research Trends Issue 25 December 2011 Page 03

Letter to the Editor.

Dear Sarah Huggett, dear Editor,

With high interest we read your recent article “Heading for success: or how not tot title your paper”, in Issue 24 (September 2011) of Research Trends. For articles focusing on trends it is important to cover the most recent literature. Therefore we would like to draw your attention to the following.

In our recent contribution to the Journal of Informetrics1, we publish a broad analysis of the occurrence and impact of all non-alphanumeric characters in 650,000 titles of peer-reviewed publications published between 1999 and 2008. In this analysis, we did not limit our investigation to a single field, but sampled publications from all fields, including the social sciences and humanities, in as far they are available in the Web of Science database.

Based on this extensive analysis, we draw the following main conclusions regarding the effect on impact:

1. Inclusion of a non-alphanumeric character has a positive effect on impact.

2. However, this effect is not visible if the most frequent non-alphanumeric characters (the hyphen, colon, comma, left and right parenthesis) are disregarded from the calculation.

3. In specific major fields, the effect can:• bepositive,likeintheoverallcase;• benegative,i.e.includinga

non-alphanumeric character has an adverseeffect;

• beabsent,i.e.thereappearstobenosignificant effect.

Additionally, we found that the relative occurrence of non-alphanumeric characters in titles has not been increasing between 1999 and 2008. We found this result was somewhat of a surprise, because some of the most cited analyses on specific non-alphanumeric characters in titles (see for instance Ref. 2) could give the impression that (at least the colon and question mark) “are on the rise” in science. Although this may be true for the limited scope of both analyses, we conclude that this does not hold for scientific publications in general.

In the discussion of our results, we hypothesize that these results can be explained by the need for an author to confirm to a general format of a title. If an author strays too far from this format, then he or she may run an increased risk that the title is disregarded by their peers, thus reducing the chance to get cited. In fact, such a conclusion is clearly in line with both Srivastava’s and Blencowe’s remarks, from which we gather that they will refrain from using a title that is “special” in any way, which would disturb the message they would want to convey.

Although our conclusions do not disagree with the results of Jamali and Nikzad (2011) nor with your analysis on the Cell publications, we think that our contribution adds important nuance to both results. For an author it is advisable to take a good look at the general format of titles in the field they want to publishing in. So, if that field features a lot of question marks or exclamations marks, then do include such characters in a title. Also, if that field is not used to see colons in titles of normal articles, then it may not advisable to include such a colon (let alone three).

Leiden, October 2011

Reindert (Renald) K. ButerAnthony (Ton) F.J. Van Raan

Powered by Scopus

References:

1. Buter, R.K., & Van Raan, A.F.J. (2011) Non-alphanumeric characters in titles of scientific publications: an analysis of their occurrence and correlation with citation impact. Journal of Informetrics, Vol. 5, pp. 608-617.

2. Dillon, J.T. (1982) Impact of the colon: Preliminary reactions. American Psychologist, Vol. 37, pp. 716.

3. Jamali, H.R. & Nikzad M. (2011) Article title type and its relation with the number of downloads and citations. Scientometrics, Vol. 88, pp. 653-661

Page 05 Is science in your country declining? Or is your country becoming a scientific super power, and how quickly?

The first article looks at the interpretation of trends in publication counts by country.

Page 09 Individual Researcher Assessment: from Newby to Expert

This article shows how differences in publication counts between individual researchers may say more about where they are in their scientific careers than about the quality of their research.

Page 11 Reporting Back: How do European universities perceive the rankings? Global University rankings and their impact

Research Trends reports back on a meeting on global university rankings organized by the European University Association (EUA), with a focus on how European universities perceive such rankings.

Page 13 Two’s company: how scale affects research groups

Studies at the level of research groups show how scale may influence their performance, especially the output per individual. Such analyses provide insights into what ‘critical mass’ means for research groups, and how it relates to productivity.

Page 15

The Beginning of a Beautiful Friendship – Social Sciences Research and Industry Products

Co-citation has not only had an impact on research in the library and information science community, but also on inventions described in patents filed by major technological companies, especially during the past decade. Research Trends looks into this in more detail.

Page 17 Section 6: Did you Know?That there are nearly 3,000 papers focusing on the interpretation of data or results?

Research Trends Issue 25 December 2011 Page 04

Section 1: Country Trends Is science in your country declining? Or is your country becoming a scientific super power, and how quickly? Henk F. Moed, Andrew Plume,

M’hamed Aisati and Peter Berkvens

Analysing longitudinal trends in the publication output of nations has a long tradition in the field of bibliometrics. Derek de Solla Price (1978)1 and Francis Narin (1976)2, two founding fathers of the field, began exploring the utility of this type of bibliometric analysis in the 1970s, and they continue to have a considerable impact both on scientific and research policy debates.

History of analyzing nations

This national-level focus on scientific output is captured in many articles published over the past 25 years: ‘The continuing decline of British science’ (Martin et al., 1986)3;Larecherche française est-elle en bonne santé? (Callon & Leydesdorff, 1987)4; ‘The emergence of China as a leading nation in science’ (Zhou & Leydesdorff, 2006)5;‘Therace for world leadership of science and technology: status and forecasts’ (Shelton & Foland, 2009)6;‘Tippingthebalance:theriseof China as a science superpower (Plume, 20117;TheRoyalSociety,20118)’;and‘IsItalianscience declining?’ (Daraio & Moed, 2011)9.

Publication counting seems so simple. But one has to make a series of methodological decisions that specify precisely how the counting is carried out. These decisions determine the numbers that are generated, and should be taken into account when interpreting the outcomes and drawing conclusions from these figures. Table 1 lists ten crucial methodological factors in this process.

A case study: China versus US

Several recent studies have assessed trends in US publication output compared with China. They differ with respect to many factors listed in Table 1. All of these studies found both a decline in US output and an increase for China during specific sub-intervals within the 2000–2010 time period. They even ‘predict’, by means of extrapolation, the year in which China will surpass the US in total publication output. The extrapolated cross-over years differ among the various studies but range between 2013 and a date in the following decade.

A recent study carried out by Loet Leydesdorff10 compared measures of scientific publication output generated by web versions of Web of Science (WoS) and Scopus. While the WoS analysis showed a steady decline in US output during 2000–2010, the Scopus results suggested that the US had a constant world share of publications during 2004–2009, and increased its share in 2010. A study conducted at Elsevier replicated the findings derived from Scopus’ web version. However, Elsevier’s study also used results derived from a special bibliometric version of Scopus created at Elsevier, one that draws on the same raw data as in the web version but loads it into a different software environment and applies several data-cleaning processes.

Figure 1 shows the outcomes of this comparison. Notably, the results for the US differ considerably between the two Scopus versions. These discrepancies are due to the fact that not all author affiliations contain the name of the country in which the authors’ institutions are located. This is especially true for US affiliations: many indicate the US state, but not the country name. In Chinese publications, such a phenomenon occurs less frequently, possibly because Chinese authors find it important to highlight their country of origin.

In Elsevier’s bibliometric version of Scopus a large fraction of missing country names were added, which increased the measured numberofpublications;however,inthewebversion of Scopus this data cleaning is still ongoing (at present, only missing affiliations from 2010 are added). The process operates backwards in time: by the end of the year additions for the years 2005–2009 will be added. So the increase in US world share in 2010 previously derived from the web version of Scopus is due to more complete capturing of affiliation countries in that year.

Research Trends Issue 25 December 2011 Page 05

Powered by Scopus

Research Trends Issue 25 December 2011 Page 06

References:

1. Price, D.J.D. (1978) Towards a model for science indicators. In Toward a Metric of Science: The Advent of Science Indicators (eds Elkana, Y., Lederberg, J., Merton, R.K., Thackray, A. & Zuckerman, H.) (New York: John Wiley, pp. 69–95).

2. Narin, F. (1976) Evaluative Bibliometrics: The Use of Publication and Citation Analysis in the Evaluation of Scientific Activity. (Washington D.C.: National Science Foundation,).

3. Martin, B.R, Irvine, J., Narin, F. & Sterritt, C. (1987) The continuing decline of British science. Nature, Vol. 330, pp. 123–126.

4. Callon, M. & Leydesdorff, L. (1987). La recherche française est-elle en bonne santé? La Recherche Vol. 18, pp. 412–419.

5. Zhou, P., & Leydesdorff, L. (2006) The emergence of China as a leading nation in science. Research Policy, Vol. 35, pp. 83–104.

6. Shelton, R. D. & Foland, P. (2009) The race for world leadership of science and technology: status and forecasts. Proceedings of the 12th International Conference of the International Society for Scientometrics and Informetrics (eds Larsen, B & Larsen, J.), Volume I, pp. 369–380 (Rio de Janeiro, Brazil, July 14–17, 2009).

7. Plume, A. (2011) Tipping the balance: The rise of China as a science superpower. Research Trends, Issue 22.

8. The Royal Society (2011) Knowledge, Networks and Nations: Global Scientific Collaboration in the 21st Century.

9. Daraio, C. & Moed, H.F. (2011). Is Italian science declining? Research Policy, Vol. 40, pp 1380–1392.

10. Leydesdorff, L. (2011). World shares of publications of the USA, EU-27, and China compared and predicated using the new interface of the Web-of-Science versus Scopus. arXiv:1110.1802v2 [cs.DL].

Years

USA - ScopusBib V

USA - ScopusWeb V

China - ScopusBib V

China - ScopusWeb V

00

00

10

20

30

01 02 03 04 05 06 07 08 09 10

% A

rtic

les

Figure 1 – Scopus Bib V: data from bibliometric versionofScopuscreatedatElsevier;ScopusWebV: The Web version of Scopus. Source: Scopus.

What does this mean?

This case illustrates once more how careful one must be when interpreting bibliometric trend data (even at the level of countries), how outcomes can differ between one database version and another, how affiliation practices can differ among countries, and how these differences can affect both numbers and annual trends.

There is no absolute norm for what constitutes good database coverage. Scopus tends to have a more comprehensive coverage, especially of Chinese journals, while WoS has more selective journal coverage. Each gives a specific view of the US and Chinese output. Both databases give a declining trend for the US and an increasing one for China.

The crossover times are different, and sooner for Scopus than for WoS, but this should be expected from a database that has a more comprehensive coverage of Chinese journals.

Research Trends Issue 25 December 2011 Page 07

Factor Description

Selection of a database Which database does one use in the measurement? Coverage may differ substantially from one database to another.

Different versions of a database Different versions of a database may exist. For instance, several groups have created their own bibliometric versions based on raw data from Scopus or Web of Science, adding information to it, performing data cleaning and so on. Results from such bibliometric versions may differ from those obtained with the web versions of the same databases.

Changes in database coverage Databasecoveragemaychangeovertime;forinstance,newjournalsmaybeaddedfrom a particular year onwards. How does one deal with these changes?

Adequacy of database coverage Does a database cover the publication output of a country and/or in a research field sufficiently well? For instance, databases principally covering journals miss important output in social sciences and humanities (published in books) and in engineering (published in conference proceedings).

Fractional versus integer counting How should one count a paper co-published between a US and a UK author? As one US paper and one UK paper (integer counts)? Or as 0.5 US and 0.5 UK papers (fractional counts)? More sophisticated schemes can also be explored.

Absolute or relative counts Does one analyze the absolute number of published articles, or article shares (for instance, the percentage of papers published from a particular country relative to the total number of articles indexed for the database)?

Time period considered To which time period does the data collection relate? This is especially important when examining longitudinal data. For instance, a country may show an increase in some years, and a steady state or even decline in a subsequent time period.

Document types included in the counts Databases index many types of documents: full research articles, but also shorter letters, reviews, editorials, discussion papers, and more. Which types should be included in the counts?

Publication year vs. database or tape year A paper published at the end of a calendar year (e.g., in December 2010) may be included in the database in the next year (e.g., March 2011). Is such a paper counted as a 2010 or as a 2011 paper?

Country delimitation Papers are assigned to countries according to the geographical location of the institutions of publishing authors. But how precisely is this done? Does the database include the affiliations of all authors? Have variations in country names been taken into account?

Table 1 - Methodological issues in bibliometric analysis of nations.

year - % Chinayear - % China-webyear - % USAyear - % USA-web

R2 Linear = 0.977R2 Linear = 0.713R2 Quadratic 0.824R2 Linear = 0.973

2000 2005 2010 2015 2020

30

Perc

enta

ge o

f Wor

ld S

hare

of A

rtic

les

20

10

0

Figure 1 – Predicted cross-over between the USAandChinabasedonthenewScopusdata;confidence intervals at the 95%-level. (SPSS, v.18.) Sources:Moedetal.(2011)1;theopencirclesarefrom Leydesdorff (2011)2.

Comment by Loet Leydesdorff:

When can the cross-over between China and the USA be expected using Scopus data?

Moed et al.’s article1 is a reaction to a recent paper2 in which I showed that the cross-over between China and the USA would be postponed until after 2020 when using the Science Citation Index-Expanded of Thomson-Reuters. By contrast, a team at Elsevier had argued in a report of the Royal Society, and on the basis of Scopus data, for a possible cross-over as early as 2013 (Refs 3.4).

The new analysis additionally clarifies why the linear fit for the US data remains poor (R2 = 0.71) – it is because of problems with this data.

However, the fit for China is not different from previously reported studies (R2 = 0.97). Using the Science Citation Index (WoS v.5), one can find more precise fits and therefore a higher reliability for the prediction of a cross-over occurring after 2020.

As noted2, these longer-term predictions are unlikely to be valid because of decreasing marginal returns in competitive markets. The metrics are embedded in a long-standing debate which I first entered in 1987 (see Refs 5,6). Given the new data, the prediction in the report of the Royal Society that the cross-over in the Scopus database would take place as early as 2013 can be postponed by approximately two years.

Loet LeydesdorffAmsterdam School of Communication Research, University of Amsterdam,http://www.leydesdorff.net; loet@leydesdorff.net

Research Trends Issue 25 December 2011 Page 08

References:

1. Moed, H. F., Plume, A., Aisati, M, & Berkvens, P. (2011). Is science in your country declining? Or is your country becoming a scientific super power, and how quickly? Research Trends, Issue 25.

2. Leydesdorff, L. (2011). World shares of publications of the USA, EU-27, and China compared and predicated using the new interface of the Web-of-Science versus Scopus. arXiv:1110.1802v2 [cs.DL].

3. The Royal Society (2011) Knowledge, Networks and Nations: Global Scientific Collaboration in the 21st Century.

4. Plume, A. (2011) Tipping the balance: The rise of China as a science superpower. Research Trends, Issue 22.

5. Callon, M. & Leydesdorff, L. (1987). La recherche française est-elle en bonne santé? La Recherche Vol. 18, pp. 412–419.

6. Shelton, R. D., & Leydesdorff, L. (in press). Publish or patent: bibliometric evidence for empirical trade-offs in national funding strategies. Journal of the American Society for Information Science and Technology.

Section 2: Research AssessmentIndividual Researcher Assessment: from Newby to Expert Judith Kamalski and Colby Riese

How do we know whether a particular researcher is promising, or to whom funding should be allocated, or who the best candidate for a certain position is? Research Trends tries to offer some guidance.

A basic, but important, question relates to who is doing the assessment, and why. Is it a line manager, a funding body, or potential collaborators? The evaluator and the evaluated may even be one and the same, as when a researcher sets out to benchmark their own performance. In these scenarios, the goal might be to recruit, promote,orretainaresearcher;toallocatetimeorequipmenttoacertainresearcher;to determine who should receive awards ormoney;ortoassessone’sownposition.Depending on the goal, different aspects of the evaluation of scientific quality may come into play.

In all these cases, one can turn to bibliometric indicators to supplement other methods, such as peer review and interviews. Whether the focus is on productivity, impact, or collaborations, the numbers can shed a light on an individual’s performance. When the researcher in question is established and has published extensively, this should be quite straightforward. But in the early days of a scientific career, this is much harder.

We can distinguish several stages in a researcher’s career. To keep it simple, here we look at three different and somewhat arbitrary career stages: Years 1, 5 and 10.

Year 1: a promising future

Imagine a young and promising researcher in Year 1 of their career, who has not published anything yet – though there’s plenty in the pipeline and lots of exciting ideas. Their time is mostly spent reading and forming ideas for future research. So how can we judge this individual’s performance? Bibliometric analysis is not going to be helpful here, so we might consider looking at their examination results or peer-review comments. Other factors might include their networking activities: are they members of a scientific network? Do they contribute to the network’s discussions? Have they run any workshops or given conferences presentations?

Year 5: underway to the next stage

By now the researcher has published a few articles, and is slowly building a reputation in the field. Their time is mostly spent conducting experiments, networking, and writing up articles. Here, metrics can be more useful in assessing performance than in the earlier stages, but traditional metrics based on averages will not provide an accurate measure because of the small number of publications and citations involved. More immediate metrics could be provided by looking at usage, or downloads, of the researcher’s articles (e.g., how many times have their publications been viewed in a certain database, such as Scopus?). Another relevant aspect could be collaboration: has the researcher published with other reputable researchers in different institutions and countries?

Year 10: established and independent

By the time a researcher gets to this stage of their career, the track record is sufficient for metrics such as the h-index to provide a meaningful measure of output. But one could also look at public presence: how does this person contribute to conferences and events in the relevant subject area, or as a keynote speaker, or in the media? In some fields, patent data may also be relevant.

Let the data do the talking

Research Trends conducted a quick analysis in SciVal Strata (for previous analyses by Research Trends using Strata, see here), a new tool by Elsevier intended to provide a visualization of the activity of a researcher in different stages of their career. Researchers themselves are able to assess their own complete, scientific impact, and also view themselves as part of a team. Strata is not constrained by one particular metric, as a single performance measure is certainly not sufficient to represent an individual researcher’s performance accurately or fairly.

In Figure 1, we see an anonymous young researcher after a few years in science. In Figure 2, we see the experienced, established researcher. While this is just one of many possible ways of looking at performance, it is clear to see how the two profiles differ. The young researcher has only just started to publish, and has not received any citations to date. The established researcher has papers almost every year: the older ones have all been cited, and among the more recent ones there are only a few uncited papers.

Research Trends Issue 25 December 2011 Page 09

Powered by Scopus

Take-away lessons for individual assessment

For every stage in a researcher’s career, and for every goal that the person doing the assessment has in mind, there are appropriate tools and measurements. But it is important to bear in mind that consideration must also be given to non-bibliometric indicators that have value for a particular assessor or institution, and which can provide extra information that might tip the balance one way or the other in an overall assessment.

Research Trends Issue 25 December 2011 Page 10

Benchmark Influence Collaboration

Cited / Uncited Documents

Citation Indices | Document Output | Citations Received | Cited / Uncited Documents

Cited

Options Chart view Table view

i

Publication year

Num

ber o

f doc

umen

ts

1996

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

1.5 1.5

0

0.5

1

2011

Uncited

Benchmark Influence Collaboration

Cited / Uncited Documents

Citation Indices | Document Output | Citations Received | Cited / Uncited Documents

Cited

Options Chart view Table view

i

Publication year

Num

ber o

f doc

umen

ts

1996

1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 20110

10 10

0

5

2011

Uncited

Figure 1 – Profile for a young researcher. Only one paper has been published and this is yet to be cited. Source: SciVal Strata, Scopus data.

Figure 2 – Profile for an established researcher. Dark-colored bars represent cited documents, and lighter-colored bars denote uncited documents. Source: SciVal Strata, Scopus data.

Some quotes by researchers on measuring performance:

“ At the start of my scientific career, I measured my performance by looking at the respect I gained from my colleagues. Later, my measure of performance became the public significance of my work as expressed at local and international conferences. And in the final stage of my career,

I aim at obtaining results which I can compare with the best achievements in my area of research”

” In my opinion the ‘research performance’ of an individual is very much defined by the influence of his/her ideas over the colleagues in his/her scientific field”

Research Trends Issue 25 December 2011 Page 11

Section 3: Reporting BackHow do European universities perceive the rankings?Global University rankings and their impact – report commissioned by the European University Association Anne-Catherine Le Calvez

On June 15, 2011, the European University Association (EUA) made public the results of the report ‘Global University rankings and their impact’. This report, led by Professor Andrejs Rauhvargers, provides a comparative analysis of the methodologies used in the most popular rankings*. The presentation of the report’s results was followed by a panel discussion with university leaders and higher education experts** about the impact of ranking on universities. The report does not intend to rank their various rankings but to analyze the methodologies and indicate the current developments of alternatives to measure university quality and performance in all its dimensions and complexity.

Useful rankings

The authors of the report recognize that rankings are here to stay, given their high level of acceptance by various stakeholders. The report acknowledges the positive aspects of the rankings for universities: they draw the attention of governments to highereducationandresearch;theyimproveaccountabilityandmanagementmethods;and they demonstrate the importance of collecting reliable data. Regarding the robustness of the data on the output, both Web of Science and Scopus were mentioned as reliable databases as far as the sciences and medicine are concerned.

Main findings and criticisms

Going through the comparison of the various methodologies, the report details what is actually measured, how the scores for indicators are measured, and how the final scores are calculated – and therefore what the results actually mean.

The first criticism of university rankings is that they tend to principally measure research activities and not teaching. Moreover, the ‘unintended consequences’ of the rankings are clear, with more and more institutions tending to modify their strategy in order to improve their position in the rankings instead of focusing on their main missions.

For some ranking systems, lack of transparency is a major concern, and the QS World University Ranking in particular was criticized for not being sufficiently transparent.

The report also reveals the subjectivity in the proxies chosen and in the weight attached to each, which leads to composite scores that reflect the ranking provider’s concept of quality (for example, it may be decided that a given indicator may count for 25% or 50% of overall assessment score, yet this choice reflects a subjective assessment of what is important for a high-quality institute). In addition, indicator scores are not absolute but relative measures, which can complicate comparisons of indicator scores. For example, if the indicator is number of students per faculty, what does a score of, say, 23 mean? That there are 23 students per faculty member? Or does it mean that this institute has 23% of the students per faculty compared with institutes with the highest number of students/faculty? Moreover, considering simple counts or relative values is not neutral. As an example, the Academic Ranking of World Universities ranking does not take into consideration the size of the institutions.

Other indicators measuring teaching quality are perceived as strongly questionable, far more so than the ones measuring research. Moreover, the EUA report describes how differences in the way academics publish and cite each other in different fields can create a strong bias in rankings. As such, attempts have been made to normalize across disciplines, and the field-normalization in the Leiden Ranking is a highly regarded example of this effort.

Powered by Scopus

Research Trends Issue 25 December 2011 Page 12

Recommendations

The EUA report makes several recommendations for ranking-makers, including the need to mention what the ranking is for, and for whom it is intended. Among the suggestions to improve the rankings, the following received the greatest attention from the audience:

1) Include non-journal publications properly, including books, which are especially important for social sciences and the arts andhumanities;

2) Address language issues (is an abstract available in English, as local language versionsareoftenlessvisible?);

3) Include more universities: currently the rankings assess only 1–3% of the 17,000 existinguniversitiesworldwide;

4) Take into consideration the teaching mission with relevant indicators.

Which ranking, which evaluation tool for which purpose?

Going further, the panel discussion’s participants recommend going beyond the rankings and analysing in detail what information is needed by institutions to assess the diversity of research activities, and to take strategic decisions and implement those choices. Citation analysis or any other single indicator is obviously not sufficient to make decisions on a well-informed basis. The discussion should be to determine what the best tools are depending on the question.

Expectations about the U-Multirank project are high, considering its aim to show the various missions of the universities, far away from a league table, helping students to make choices.

As a conclusion, Jean-Marc Rapp, President of EUA, outlined the next steps in the EUA’s agenda: to analyze both the desired and un-intended consequences of the ranking systems, and comparing the different ways of assessing universities (ranking/rating, benchmarking, quality assessment and so on). This proves again how crucial those evaluation matters are for universities, who are looking for advice about how to make the most informed choices.

The original report

http://www.eua.be/pubs/Global_University_Rankings_and_Their_Impact.pdf

Notes

*The EUA looked at the following rankings: Shanghai Academic Ranking of World Universities(ARWU);TimesHigherEducationWorld University Ranking (in cooperation with Quacquarelli Symonds until 2009, andThomsonReutersfrom2010);World’sBestUniversitiesRanking(QS);GlobalUniversitiesranking(Reitor);HEEACTRankings(Taiwan);EUUniversity-basedResearchAssessment(AUBRWorkingGroup,EU);LeidenRanking;CHEUniversity/Excellencerankings;U-Multirank;U-Mapclassification;and Webometrics.

**Participants involved in the meeting: 1) Presentation of the report: Jan Truszczynski, Director General for Education & Culture, EuropeanCommission;AllanPäll, Vice-Chairperson, European Student’s Union;GeroFederkeil,VicePresident(VP)International Observatory on Academic Rankings and Excellence. 2) Panel discussion: Chaired by Professor Ellen Hazelkorn, VP Research&Enterprise,Dublin,Ireland;Professor Jean-Pierre Finance, President UniversitéHenriPoincaré;SirHowardNewby, Vice-Chancellor, University of Liverpool, UnitedKingdom;JensOddershede,Rector,University of Southern Denmark, Chairman of Universities Denmark.

Curriculum Vitae: Andrejs Rauvargers

Andrejs Rauhvargers was born 1952 in Riga, Latvia, and has a Ph.D. in Chemistry from the University of Latvia. He is Secretary General of the Latvian Rectors’ Conference and Professor at the Faculty of Education at the University of Latvia. He has also served as Deputy State Secretary at Latvian Ministry of Education, where he participated in developing legislation for higher education and was closely involved in the establishment of the higher education quality assurance system in Latvia and its coordination with neighboring countries Estonia and Lithuania. He was also responsible for establishing a system for academic recognition in Latvia.

Internationally, Rauhvargers is a member of the Bologna Follow-Up Group and since 2005 has chaired the working group studying the progress in the 46 ‘Bologna’ countries and preparing the Bologna

Stocktaking reports published in 2007 and 2009. He served as president of the European Network of Academic Recognition Centres (ENIC) from 1997 to 2001 and as President of the Intergovernmental Committee of the Lisbon Recognition Convention from 2001 to 2008.

He is the author of several other major reports and a number of publications on various aspects of international higher education, both national and international, and has participated in and managed several higher education reform projects in Croatia, Latvia, Lithuania, Montenegro and Poland. He has been an invited speaker in his field in more than 20 countries.

In 2006 Andrejs Rauhvargers was awarded the European Association for International Education’s Constance Meldrum Award for innovation, leadership, and inspiration in international higher education.

Source: http://www.eua.be/about/who-we-are/secretariat/Andrejs-Rauhvargers.aspx

Research Trends Issue 25 December 2011 Page 13

Section 4:Behind the dataTwo’s company: how scale affects research groups Matthew Richardson

In the 1880s, French agricultural engineer Max Ringelmann carried out a series of experiments exploring how much people put into task in individual and group settings. To do so, Ringelmann had male students pull on a rope and measured the force exerted, first individually and then when they were part of a team. Ringelmann found that while the overall performance increased as students were added to the team, the average performance exerted per worker, or individual performance, decreased linearly with each additional worker1. In other words, if each individual had put their effort into their own work, the result would have been greater than when they worked as a team. The effect would come to be studied by social psychologists throughout the twentieth century looking at group performance, but, until 1986, the earliest known source of information about Ringelmann’s studies was a 1927 paper by the German industrial psychologist Walther Moede.

In 1974, Alan Ingham and colleagues performed similar rope-pulling experiments in an attempt to verify the existence of the Ringelmann effect2. Their findings showed individual performance to decrease significantly with the first few additional co-workers, up to a group of three workers – beyond which additional workers did not significantly decrease individual performance. Ingham et al. described two possible causes for the decrease in individual performance: “It remained unclear whether group members pulled less hard because of incoordination or because of losses in motivation”2. Controlling for incoordination in a further experiment in which an individual pulled on the rope – but in some cases believed he was part of a group – the study found a very similar pattern of decreased performance, showing that loss of motivation occurred when co-workers (whether perceived or real) were added to the team.

Ringelmann in research

The Ringelmann effect has real implications for scientific research. Scientific researchers are not simply pulling a rope in unison – but if individual performance decreases as additional researchers come to work on a particular problem, this makes large-scale research projects an unattractive prospect not only for individuals, but also for funding bodies. As Andrea Bonaccorsi and Cinzia Daraio put it, “pressure on public budgets in almost all industrialised countries has led governments to pursue (or at least declare they pursue) efficiency in the allocation and management of resources in the public research sector. The increasing societal demand for accountability and transparency of science also makes it important to demonstrate that public funding follows clear rules”3.

Ton van Raan has investigated the relationship between a variety of bibliometric indicators of size and research quality, at the level of the research group4. Van Raan states that “[t]he research group is the most important working floor entity in science, as clearly shown by the internal structure of universitiesandresearchinstitutes”;however,van Raan goes on to say that obtaining data at the level of a research group is far more difficult than for individual authors, institutes or even for whole countries: this is because research groups are not captured in the bibliographic fields attached to papers, such as author names or institutional addresses.

Powered by Scopus

Research Trends Issue 25 December 2011 Page 14

Sizing up subject fields

When Ralph Kenna and Bertrand Berche started to investigate the relationship between the size of a research group and the performance of those groups, they turned to the UK’s Research Assessment Exercise (RAE) as a source of data. The RAE captures data regarding not only the quality of research groups, but their field of research and size. In a rapid chain of papers, Kenna and Berche have compared the sizes of research groups in various fields with the quality assigned to those groups’ research5–9. Their findings have shown that in every field, there exists a critical mass for increased productivity, with an upper and lower boundary. The lower boundary relates to the classical notion of critical mass, “loosely described as the minimum size a research team must attainforittobeviableinthelongerterm”;between the lower and upper boundary for critical mass, “the overall strength of research teams tends to rise quadratically with increasingsize”;beyondtheupperboundary,“research quality levels out”.

The basic implication is that “this levelling off refutes arguments which advocate ever increasing concentration of research support into a few large institutions”, and their research shows optimal research group sizes in a number of disciplines8. Lately, Kenna and Berche have even used their approach to develop a method of normalizing quality between different research disciplines9.

While Kenna and Berche have developed a way to analyze the effect research group size on its overall performance, issues remain regarding how research groups can be assessed. Currently, data supplied for national research assessment programs seemtobethebestoption,whereavailable;however, tools such as SciVal Strata are starting to address the problem, opening up ways of assessing research groups by citation analysis. (For more information on this approach, the reader is referred to the article by Judith Kamalski and Colby Riese in this issue.)

References:

1. Kravitz, D.A. & Martin, B. (1986) Ringelmann rediscovered: the original article. Journal of Personality and Social Psychology, Vol. 50, No. 5, pp. 936–941.

2. Ingham, A.G. et al. (1974) The Ringelmann effect: studies of group size and group performance. Journal of Experimental Social Psychology, Vol. 10, pp. 371–384.

3. Bonaccorsi, A. & Daraio, C. (2005) Exploring size and agglomeration effects on public research productivity. Scientometrics, Vol. 63, No. 1, pp. 87–120.

4. van Raan, A.F.J. (2008) Scaling rules in the science system: influence of field-specific citations characteristics on the impact of research groups. Journal of the American Society for Information Science and Technology, Vol. 59, No. 4, pp. 565–576.

5. Kenna, R. & Berche, B. (2010) The extensive nature of group quality. Europhysics Letters, Vol. 90, DOI: 10.1209/0295-5075/90/58002.

6. Kenna, R. & Berche, B. (2011) Critical mass and the dependency of research quality on group size. Scientometrics, Vol. 86, pp. 527–540.

7. Kenna, R. & Berche, B. (2011) Statistics of statisticians: critical mass of statistics and operational research groups in the UK. arXiv:1102.4914v2.

8. Kenna, R. & Berche, B. (2011) Concentration versus dispersion of research resources: a contribution to the debate. arXiv:1006.3701v1.

9. Kenna, R. & Berche, B. (2011) Normalization of peer-evaluation measures of group research quality across academic disciplines. Research Evaluation, Vol. 20, No. 2, pp. 107–116.

Research Trends Issue 25 December 2011 Page 15

Section 5:Behind the dataThe Beginning of a Beautiful Friendship – Social Sciences Research and Industry Products

Gali Halevi

Co-citation analysis morphs into patents

If you think the term ‘co-citation analysis’ is strictly related to bibliometrics, and lays within the realm of Library and Information Science, think again!

The term was coined in 1973 when Henry Small described this methodology in his renowned article ‘Co-citation in the scientific literature: a new measure of the relationship between two documents’, which was cited hundreds of times in the following years. Co-citation analysis has been used for numerous discoveries such as the emergence of author networks in a particular field of investigation, and mapping the emergence of a scientific field1–3. Small’s technique of using co-citations to analyze a body of research in order to discover scientific trends and collaborations (among other things) was in the most part used in thesocialsciences;theninthelate1990stheterm suddenly started appearing not only in academic discussions but also in patent literature, and the approach has been used by industry pioneers such as Xerox, IBM, Microsoft, and more recently Google.

Technology catches up

How did co-citation analysis, which was conceived as a way to analyze relationships between articles, morph into a product apparatus? Looking at the time frame when patents using the term appear (see Figure 1), it seems that computational development at that time might give us an answer. The late 1990s saw a surge in different types of electronic databases which enabled indexing and retrieval of documents. These years also saw an increase in both co-citation analysis research publications and patents alike, including some interesting surges in 2009 and 2010, when semantic information retrieval and web page rankings took center stage in web-based applications. The first patent that uses co-citation analysis as a method is a Xerox patent that uses co-citation analysis to generate clusters of documents in a database4.

This patent (US Patent 6038574) describes a co-citation analysis method in which hyperlinks in web pages are viewed as references, and the relationships found are used to help create a web page index. Following the Xerox patent application, a series of patents mentioning co-citation analysis in either the method, claims, or references to prior publications appear through the patent literature featuring prominent technological companies (see Figure 2).

Looking at some of the patents one can see some interesting applications of co-citation analysis: for example Google Patent (WO2008134373A1), which uses “co-citation analysis, or anchor text analysis (e.g., analysis of text in or near links to the multimedia events) to determine related multimedia events” (Description ofEmbodiment;0035).IBMappliedco-citation analysis to link analysis on the Web (US7792827B2), and Microsoft used co-citation analysis in their development of latent semantic analysis (US20070239431A1).

Technology? Transferred!

This type of relationship between academic research and product development is a phenomenon that we have learned to expect in areas of medicine or engineering, for example. In these areas, we can more easily find a direct link between theoretical models and their growth into tangible products. What this analysis shows is that social science research can go through a similar evolution, but the cycle through which it develops takes longertodeploy;inthiscasemorethan 20 years.

Powered by Scopus

References:

1. Culnan, M. J. (1987) Mapping the intellectual structure of MIS, 1980-1985: a co-citation analysis. MIS Quarterly: Management Information Systems, Vol. 11, pp. 341–350.

2. Chen, C. (1999) Visualising semantic spaces and author co-citation networks in digital libraries. Information Processing and Management, Vol. 35, pp. 401–420.

3. Zhao, D., & Strotmann, A. (2011) Intellectual structure of stem cell research: a comprehensive author co-citation analysis of a highly collaborative and multidisciplinary field. Scientometrics, Vol. 87, 115–131.

4. US Patent 6038574 (2000) Method and apparatus for clustering a collection of linked documents using co-citation analysis.

Research Trends Issue 25 December 2011 Page 16

14

Num

ber

of A

rtic

les/

Pate

nts

Year

12

10

8

6

1973

4

21

0

1977

1979

1980

1981

1982

1983

1984

1985

1986

1987

1988

1989

1990

1991

1992

1993

1994

1995

1996

1997

1998

1999

2000

2001

2002

2003

2004

2005

2006

2007

2008

2009

2010

2011

1 1

4

3

2

1

4

1 1

6

7

3

4

9

7

11

13

1

Articles

11

1

9

6

5

4

7

1

3

2

4

7

9

1

7

4

11

7

13

1

11

9

12

0 0

1

2

5

Patents

Normalized Assignee & Assignee Name

Nor

mal

ized

Ass

igne

e

Google Inc

AT&T Inc

Ford Motor Co

NEC Corp of America

NEC Corp

IBM

LexisNexis Group

Individual

Microsoft Corp

Xerox Corp

Number of Occurences

0 1 2 3 4 5 6 7 8 9 10 1 12 13 14 15 16 17 18 19 20 21 22

Figure 1 – Occurrence of the term ‘co-citation analysis’ in peer reviewed articles versus patents. Sources: TotalPatent and Scopus.

Figure 2 – Co-citation patents assignees. Source: TotalPatent.

Section 6:Did You Know?…That there are nearly 3,000 papers focusing on the interpretation of data or results?

Sarah Huggett

Research can be the primary focus of a scientist’s career, but finishing an experiment does not mean that the task is over: once the data have been compiled, they must be analyzed and interpreted for a research project to be complete. This is a crucial part of a researcher’s workflow, as confirmed by the relative abundance of literature on this rather precise topic. Indeed, a quick Scopus search for (interpret* and (data or result*)) in the title of scholarly papers published in journals between 1996 and 2010 retrieves 2,855 results, revealing that there have been no less than 160 papers focusing on this fairly specific subject published every year. This corpus shows growth of 2.7% per annum between 1996 and 2010, which is comparable to the growth of scientific literature as a whole. It may be deduced from these results that scientific data and their interpretations are closely intertwined, which can explain why they are growing hand in hand.

Research Trends Issue 25 December 2011 Page 17

Powered by Scopus

Research Trends Issue 25 December 2011 Page 18

Research Trends:Editorial BoardHenk MoedJudith KamalskiIris KisjesAndrew PlumeSarah HuggettMatthew RichardsonGali Halevi

You can find more information onwww.researchtrends.com or contact us at research@elsevier.com

Powered by Scopus