FROM PUBLICATION TO IMPACT: INTRODUCTION TO CITATION ANALYSIS

Professor Anne-Wil Harzing

Middlesex University

www.harzing.com

PRESENTATION OUTLINE !  Impact analysis: why?

!  Publication is NOT the end result of research

!  What is impact and how to measure it?

!  Why care about citation analysis?

!  How to get cited? !  Remember the four P’s of publishing? You need to publish before you can get cited

and you won’t be cited if your work isn’t any good

!  However, you can improve your chances of being cited through: Communication, Collaboration, Care (more details to follow)

!  Where to find citation data and how to use them? !  Why use Google Scholar as a complement to ISI data?

!  Citation search and analysis through Google Scholar/Publish or Perish

!  How to make your case for research impact?

WHAT IS IMPACT? EMERALD IMPACT MATRIX !  Knowledge (academic):

!  implications for research !  citations

!  Practice: !  implications for practice !  actual application of research in industry and business, patents

!  Teaching: !  research-led teaching !  provision of case studies and examples

!  Public policy: !  implications for policy !  actual influence on public policy

!  Society: !  implications for society !  actual change of norms & values, e.g. environmental impact, social responsibility

HOW TO MEASURE IMPACT? !  Non-academic aspects of impact are difficult to operationalise & measure

!  Alt-metric.com free bookmarklet: tracks tweets, blogs, and other social media !  Does this constitute impact? Does it work reliably (it didn’t for me)?

!  Impactstory (aka Total Impact) !  Collection creation fairly time-consuming, import from Google Scholar Citations possible

!  Impact report not very meaningful, web interface instable and jumps around

!  Other efforts: e.g. readermeter, crowdometer, !  All in beta, timed out/closed, (very) incomplete; enthusiastic starters, but no follow-through

!  Aimed at one-off searches, usually need additional work (author ID, GSC profile, twitter account)

!  Could be incorporated into academic impact, e.g. Journal of Consumer Research only publishes work that they expect to have societal and practical impact

!  In universities the main emphasis remains on academic impact !  Citations are still seen as the main metric for academic impact

!  Presentation focuses on citation analysis of Google Scholar data through Publish or Perish !  Google Scholar could be argued to include some of the other forms of impact, because of its

broader coverage than traditional commercial databases such as ISI and Scopus

!  Publish or Perish is designed so that even the most inexperienced users can get good results and a clear and comprehensive overview quickly for a variety of uses

CITATION ANALYSIS: WHY CARE?

!  Why publish if nobody cites your work? !  Not publishing is like being mute, not being cited is a lot like talking without anybody

listening

!  Okay, your work might still be read by students, managers, or academics who do not publish !  But: academic research should also contribute to academic discourse

!  Government research assessments and academic promotions increasingly focus on citation impact

!  Why would you want to know if your work is cited? !  To prepare for confirmation/promotion/yearly performance appraisal !  To know who is building on your work

!  They might be future collaborators

!  It is exciting to see how others are using your research, you might get new ideas through it

!  To get an ego boost, to know someone has (presumably) read your work !

HOW TO GET CITED? MY OWN TAKE (1)

!  First of all Competence (aka as Performance in the four P’s of publishing), you won’t be cited if your work isn’t any good. !  However, you can improve your chances of being cited through the three remaining c’s

!  Communicate (they can only cite your paper if they know about it) !  Personal website, the best thing I have ever done

!  Put your papers online Online papers, full list of publications

!  SSRN, Academia.edu, ResearchGate and LinkedIn are great alternatives if you don’t have your own web site

!  Make sure your papers are found if someone searches for a topic relating to your research in Google

!  As an example my research interests: Language in international business; Headquarters subsidiary relationships; Transfer of management practices

!  Conferences, attend & talk to people !  volunteer for PDWs, as discussant, session chair, committee member

!  Email, ask for papers and send yours in return !  Don’t be shy to send your papers, most academics appreciate it

EXAMPLE EMAIL

!  I don’t know if you remember me, but I met you at Victoria University (NZ) when you came to speak many years ago. At the time I was a PhD student researching multilingualism and the role of English in banks in Luxembourg.

!  Now, I’m delighted to share with you what I have just published on YouTube.

It’s a short 20 minute illustration (doodle) on some aspects of my PhD. It’s a video that could be used as a teaching resource. I’ve attached a free e-print from the 2013 journal article that goes with it and there are other publications found in the YouTube description. I hope you enjoy it as much as I did in producing it!

!  https://www.youtube.com/watch?v=1O1yE9ylqZo !  http://www.tandfonline.com/eprint/VQn6deUXGpAkvGBDDgr7/full !  Signature

[Picture included]

HOW TO GET CITED? MY OWN TAKE (2)

!  Collaborate (not just because it makes doing research more fun) !  It often leads to better quality research

!  Complementary skills (theory development, empirical research, analysis, practical implications)

!  Always someone to read your paper critically

!  More motivation to finish your papers

!  Co-authored papers are cited more (because each author has their own network)

!  Your collaborators will cite you in their other projects

!  Care (this more generally makes our profession a nicer place to be) !  For your own reputation, it is your most valuable asset

!  Nobody wants to use and cite the work of someone they don’t respect

!  For others; help wherever you can !  Keep the promises you make at conferences

!  Alert collaborators and academic friends to useful information & congratulate them on their achievements

!  Thank others for their help!

HOW TO GET CITED? WHAT CAN UNIVERSITIES DO? !  Create a research culture

!  Invite (international) academic visitors

!  Get involved in (international) collaborations

!  Run seminar series, even if they are very informal

!  Be considerate !  Acknowledge that especially for areas such as HRM and accounting publishing in local journals

that attract fewer citations is important for knowledge transfer to practice

!  Acknowledge that not all topics are easy to publish in top North American journals (but don’t discard that option just because you have Australian/British/European data, see Carol Kulik’s 2005 Journal of Management editorial)

!  Celebrate !  Achievements (in all forms), but don’t make others feel like a failure !  Diversity in topics, research methods, writing styles

!  Do not engage in head-to-head “competition” with North American academics, we can only lose !  Build on your strengths, but do change the 10-30% of your approach you need to adjust “to be heard”

WHERE TO FIND CITATION DATA? GOOGLE SCHOLAR TO COMPLEMENT ISI

!  ISI has long been the “gold standard” of impact measurement

!  Recently, alternatives have become available !  Scopus (general),

!  Citeseer (computer science),

!  RePEc (Economics)

!  Google Scholar is the only alternative that is: !  Not focused on a single discipline only

!  Freely available to anyone with an Internet connection

!  Suitable for citation analysis (but only with Publish or Perish as an interface)

GS TO COMPLEMENT ISI WHY? (1)

!  Google Scholar gives a more comprehensive citation count

!  WoS General Search is limited to ISI-listed journals !  Citations to books, book chapters, dissertations, theses, working papers, reports,

conference papers, and journal articles published in non-ISI journals are not included

!  Of the journals included in my JQL (generally only high-quality journals), ISI coverage runs from 30%-43% for Finance & Accounting, Management & Marketing to 73%-80% for Economics, MIS, Mgt Science/OR/Ops

!  WoS Cited Reference Search !  Does include citations to non-ISI publications. However, it only includes citations

from journals that are ISI-listed

!  Counts citations to non-ISI publications only towards the first author

!  Co-authorship is increasingly common and hence ISI misses many citations

GS TO COMPLEMENT ISI: WHY? (2) !  In contrast to GS, ISI seem to have difficulty in dealing with non-Anglo name variants

!  Until mid-1990s of citations to “Baden-Fuller” have been entered as “Badenfuller”, citations to “van Raan” have been entered as “Vanraan”

!  Names with diacritics (e.g. Köhler) used to give an error message !  Now they resolve to both Koehler and Kohler (Köhler still doesn’t show up) which makes it hard to

disambiguate authors

!  A search for names in non-Roman scripts still gives an error message

!  WoS has very limited coverage of non-English sources !  e.g. one French Accounting academic has 30 cites in ISI, but >1000 in GS as most of citations are

in French journals !  Non-English coverage is particularly important for disciplines with a strong local content such as

accounting and industrial relations

!  WoS has very poor aggregation of minor variations of the same title/name !  “Stray” citations are very common !  A Harzing, AW Harzing, A-W Harzing, AWK Harzing, AK Harzing !  Books suffer from frequent typos in data entry: Culture’s consequences has more than 100

different instances: Clotures Consequence, Clultural Consequenc, Culultures Consequenc, Cult Consequences In

CITATIONS IN DIFFERENT DATA SOURCES: WHAT’S THE DIFFERENCE? (MY OWN CITES)

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

ISI General Search Scopus ISI Cited Reference Google Scholar

1618 2460

3110

9117

H-INDEX: THE NEW ACADEMIC METRIC?

GOOGLE SCHOLAR FOR INDIVIDUALS WHAT MEASURES TO USE?

!  H-index increasingly seen as a convenient summary of quantity & impact and used in many research assessments !  H-index of 10 means 10 papers with at least 10 citations each !  Can be adjusted for co-authorships (hi norm) and for age of paper (hc index) or for

both co-authorship and professional age (hIa = annualised individual h-index)

!  Total citations is probably the fairest way to assess impact for individuals !  Focus on citations per paper might discourage people to publish additional papers !  Can be adjusted by years (Total citations/years active) and number of authors (Sum

of citations for each paper divided by the number of authors) !  Only relevant for more senior researchers; own example, published since 1995:

!  in the first five years I had about 20 ISI citations

!  in the next five years I had about 125 ISI citations

!  in the last five years I had about 2000 ISI citations

GOOGLE SCHOLAR FOR INDIVIDUALS HOW TO TRACK CITATIONS?

!  Publish or Perish !  Designed to make GS a more useful alternative to ISI

!  Designed to empower individual academics by providing citation analysis with a wide range of metrics at a click of the mouse

!  As with ISI: don’t take its results as absolute and think before passing a “verdict”; we are dealing with human beings, not machines!

!  Demonstration of author search

!  Different metrics

!  Merging publications

!  Sorting

!  Can also be used to assess journals, do literature research, etc.

PUBLISH OR PERISH FOR INDIVIDUALS HOW TO PRESENT YOUR CASE (1)?

!  Review various measures of impact (Run search: “A Harzing”) !  Many well-cited pubs (h-index) vs some highly-cited pubs (g-index)

!  Young versus older papers (hc index measures current impact)

!  Single versus co-authored (hI norm corrects for multiple authors)

!  Which of your publications is a star? (Idem) !  Review citations per year for each publication

!  Shows you which of your research topics have more impact

!  What type of publication is it?

NEW METRIC INDIVIDUAL ANNUAL H-INDEX !  Harzing, Alakangas & Adams (2014) propose hIa: Individual annual h-index

!  Corrects the h-index for co-authorship and length of academic career !  The average number of single-author equivalent impactful publications per year, hence a hIa

of more than one shows excellent research performance

!  Provides a more reliable comparison between disciplines (see graph, based on a sample of 150 Associate and Full Professors at the University of Melbourne)

!  For more details see: DOI 10.1007/s11192-013-1208-0

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

0.50

0

5

10

15

20

25

Life Sciences Sciences Engineering Social Sciences

Humanities

h-index (left axis) hIa-index (right axis)

MERGING PUBLICATIONS !  Publications can have multiple versions in Google Scholar, because

!  Academics make mistakes in referencing publications, creating stray citations

!  Google Scholar makes mistakes in parsing publications

!  Publication might appear on websites with different referencing standards

!  Some publications actually have more than one version (e.g. conference/working paper and final version)

!  Merging publications can be a good option !  Simply select the publication and drag it onto the master record with the mouse

!  Please note that in some cases this might lead to inflated citations it the same citing works are duplicated or both conference and journals publication are cited

DIFFERENT IMPACT METRICS

STAR PUBLICATIONS

PUBLISH OR PERISH FOR INDIVIDUALS HOW TO PRESENT YOUR CASE (2)?

!  How does your article score within the journal? !  The top, or one of the top-3/5(%) or top-10(%) scorers in that year? (OS 2003)

!  Play with this for the different data sources

!  6th most cited in Google Scholar

!  4th most cited in Microsoft Academic Search

!  3rd most cited in Scopus

!  2nd most cited in Thomson Reuters ISI

!  The first listed paper from your country/outside North America? (IJHRM 1995)

!  The top-10/20/50 since inception (IBR 1997)

!  The most-cited paper in the journal since inception (ESEP 2008)

TOP 10% MOST CITED

FIRST SINGLE-AUTHORED OR FIRST NON-US ARTICLE

TOP-10/20/50 SINCE INCEPTION

TOP MOST CITED PAPER

ISI OR SCOPUS DATA IN POP !  What if you still want to use ISI of Scopus data

!  Publish or Perish can import both ISI and Scopus records

!  PoP automatically calculates the same citation metrics as for Google Scholar

!  Detailed comparisons between data-bases become possible

!  Procedures !  Perform one or more searches in ISI or Scopus and save in one of the formats

!  http://www.harzing.com/pophelp/importing.htm

!  Import files into Publish or Perish !  This can be done in seconds with a few mouse-clicks

!  You can import as many files as you want in one go

!  Simple visual comparisons can be done in the multi-query centre (see screenshot)

!  For more sophisticated comparisons export statistics to Excel for further processing !  Currently working on a project that compares nearly 150 Melbourne professors over a full

year using Scopus, ISI and Google Scholar data

DATABASE COMPARISONS

MORE INFORMATION? !  Chapter 1: Introduction to citation analysis

!  Chapter 2: Introduction to Publish or Perish

!  Chapter 3: Author searches

!  Chapter 4: Journal searches

!  Chapter 5: General citation search queries

!  Chapter 6: Multi-query centre

!  Chapter 7: Making your case for tenure or promotion !  Chapter 8: How to evaluate other academics?

!  Chapter 9: Tips for deans and other academic administrators

!  Chapter 10: Where to submit your paper? !  Chapter 11: Conducting a literature review

!  Chapter 12: Doing bibliometric research on authors & journals

!  Chapter 13: Evaluating Google Scholar

!  Chapter 14: Evaluating Thomson ISI Web of Science

!  Chapter 15: A Google Scholar h-index for journals

!  Chapter 16: Author citation analysis across disciplines

Now available online: http://www.harzing.com/popbook/index.htm

Any questions or comments?

THE END!