1 Polluted Leftovers: Repository Metrics from the ... · Web Service that Accurately Counts Item Downloads from Institutional Repositories,” Library Hi Tech, vol. 35, no. 1, March

!PollutedLeftovers:RepositoryMetricsfromthePerspectiveofaMostDownloadedItem

[email protected]|[email protected],MontanaStateUniversity|JeffMixter,OCLCResearch|LeilaSterman,MontanaStateUniversity|

4/3/17Coali,onforNetworkedInforma,onmee,ng,Albuquerque,NM

1

MeasuringUpgrant:About

!  IMLS-fundedgrant,2014-2017!  “MeasuringUp:AssessingAccuracyofReportedUseandImpactofDigitalRepositories”!  hVp://scholarworks.montana.edu/xmlui/handle/1/8924!  Partners:MSU,UNM,OCLCResearch,ARL

!  Maindriver!  Improvingaccuracyandconsistencyofrepor[ng

!  Researchdemonstratesunderandover-coun[ngofIRuse!  Solu[on=RAMP(RepositoryAnaly[cs&MetricsPortal)


2

Overview

!  Problemsofestablishingconsistent,reliablemetricsofIns[tu[onalRepository(IR)usage

!  Currentresearchintosystemicover-coun[ngandunder-coun[ng

!  MappingIRlogdatatoGoogleAnaly[csandSearchConsoledataandcharacterizingthedifferences!  Assump[onsunderlyinganaly[csservicescanruleoutlegi[mateuser

interac[ons


3

ProblemsofIRReporting

!  Over-andUndercoun[ng

!  Varietyofanaly[csservicesandmethods!  Logfiles!  Soewareservices(pagetagging)

!  Generalconcerns!  Bots!  Varietyofconfigura[ons==maintenanceandcon[nuityques[onsfora

UL,consistencyandreliabilityacrossIRecosystemmorebroadly


4

DSpaceConfigurationOptions

Op,on DSpaceVersion DSpaceUICompa,bility

Integra,onwithEarlierStats

OtherNotes

GoogleAnaly[cs(GA)

GAUIsinceprev5,DSpacepluginsincev5.“Events”sincev5.

XMLUI,Mirage2themeonly.

NA DSpluginmustbeenabled

Elas[csearch(ES)

Sincev3,deprecatedinv6.

XMLUIonly. LegacySolrdatacanbeconverted.

Mustbeenabled.Defaultsetoffieldsisnotconfigurable.

Solr

Sincev1.6. XMLandJSPUI,somedifferenceinhowfaceteventsarelogged.

Legacysystemstatscanbeconverted.

SameasES,plusdownloads,workflow,andevents.

hVps://wiki.duraspace.org/display/DSDOC5x/Sta[s[cs+and+Metrics


5

ANewReportingModel

PageType Defini[on Examples

CitableContentDownloads

Non-HTMLscholarlycontentthatmaybeformallycitedintheresearchprocess

●  Publica[on(.pdf)●  Presenta[on(.ppt)●  DataSets(.csv)

ItemSummary HTMLpagestohelpuserdecidetodownloadthefullpublica[on

●  Title&Abstract●  ItemMetadata

AncillaryHTMLpagesthatprovidegeneralinforma[onornaviga[on

●  SearchResults●  BrowsebyAuthor●  Sta[s[cs


6

TwoClassesofWebAnalytics

HTML

Analytics Service (SaaS) 1Log Files

2Page Tagging {JavaScript}


7

Non-HTML

SearchAnaly[cs

Does!

Pagetaggingmethodsdonottacknon-HTMLCitableContentDownloads


8

GAAncillaryPVandItemSummaryPVvsCCD

134-dayperiodinSpring2016


9

GAHTMLversusGSCCCDtracking

PageTypeAnaly[cs SearchConsole

Pages Events SearchAnaly[cs

CitableContentDownloads - 26,355 562,933

ItemSummary 284,303 - -

Ancillary 201,793 - -

CCDTrackingImprovement

+2,000%4/3/17Coali,onforNetworkedInforma,onmee,ng,Albuquerque,NM

10

MontanaMethodChallenges

!  Missingnon-GoogledirectlinkCCDevents!  Yahoo!  Bing!  Email!  Facebook&socialmedia

!  GSClimits[meandaccess!  Moving90-daywindow!  Granulardata=programmingskillstoaccessAPI


11

RAMP:RepositoryAnalytics&MetricsPortal

!  ramp.montana.edu

!  Abenchmarkingtool

!  Prototypeapplica[on!  Nolocalinstalla[onorconfigura[onrequired!  Integratedrepor[ngfromGoogleAPIs


12

RAMPIRasofMarch27,2017

•  8IRregistered•  Trackingover250,000digitalitems•  Capturingover20,000CCDperday

thatwerepreviouslyinvisiblethroughGA.


13

RAMP


14

RAMP


15

CitableContent

ClickThroughURL Country Device Posi,on Date

Impressions Clicks

No 0hVp://scholarworks.montana.edu/xmlui/handle/1/9348 hrv DESKTOP 31 3/8/17 1 0

Yes 0hVp://scholarworks.montana.edu/xmlui/bitstream/handle/1/8705/WhitenS0814.pdf;sequence=1 pan MOBILE 6 3/8/17 1 0

Yes 0hVp://scholarworks.montana.edu/xmlui/bitstream/handle/1/3670/31762001131281.pdf;sequence=1 fra DESKTOP 24 3/8/17 1 0

Yes 0hVp://scholarworks.montana.edu/xmlui/bitstream/handle/1/7215/31762101989810.pdf?sequence=1 chn DESKTOP 13 3/8/17 2 0

Yes 0

hVp://scholarworks.montana.edu/xmlui/bitstream/handle/1/11518/15-002_Surface-aVached_cells_biofilms_A1b.pdf?sequence=1 gbr DESKTOP 10 3/8/17 1 0

Yes 0hVp://scholarworks.montana.edu/xmlui/bitstream/1/1091/1/ColemanT1212.pdf kwt MOBILE 3 3/8/17 1 0

No 0hVp://scholarworks.montana.edu/xmlui/handle/1/9049 gbr DESKTOP 9 3/8/17 1 0

No 0hVp://scholarworks.montana.edu/xmlui/handle/1/2567 egy DESKTOP 44 3/8/17 1 0

Yes 0hVp://scholarworks.montana.edu/xmlui/bitstream/handle/1/7546/31762102468723.pdf;sequence=1 twn DESKTOP 14 3/8/17 1 0

No 0hVp://scholarworks.montana.edu/xmlui/handle/1/1854 tur DESKTOP 128 3/8/17 1 0

SampleRowsfromDataSet


16

PollutedLeftovers

!  HowmuchDSpaceSolrdataisnotalsocapturedbyGoogleAnaly[cs(GA)orGoogleSearchConsole(GSC)data?

!  Howmuchofthisac[vityishumanandhowmuchisbot?Whatkindsofhumanac[vityarenotbeingcapturedbythirdpartyservices?

!  ByfocusingonGA/GSCdataarewemissingasignificantpercentageofcitablecontentdownloads?Afrac[on?Afrac[onofafrac[on?


17

!LookingforStoriesintheData


18

ProcessingSolrLogs

!  Querystatscoreforcitablecontentdownloads(CCD)!  BundleName=ORIGINAL!  isBot=False!  Sta[s[csType=view!  Type=0

!  Querysearchcoreformetadata!  search.resourcetype=2(bitstreams)

!  MapCCDlogeventstometadatausingmetadata.search.resourceid=log.owningItem

!  ConsolidateloggedeventsperHandleperday


19

ProcessingGA/GSCData

!  HarvestdatadimensionsviaAPI!  Eventcategory!  Page!  UniqueEvents

!  FilterforCCD!  Pagehas“bitstream”inURL!  Clicks>0

!  AddacolumnforHandlesextractedfromURLs

!  JointoSolrdataoncombinedkeyofHandle_date


20

Overview

Table SolrCCD GoogleCCD

JoinedSolrsta[s[csandsearchcoredata

854,781

GA/GSCdatawithHandles 166,199

SolreventswithcorrespondingGA/GSCevents

356,557 166,199

SolreventswithoutcorrespondingGA/GSCevents

498,224


21

ACloserLookataMostDownloadedItem


22

Overview:12178

Table SolrCCD GoogleCCD

JoinedSolrsta[s[csandsearchcoredata

79,925

GA/GSCwithHandles 462

SolreventswithcorrespondingGA/GSCevents

74,612 462

SolreventswithoutcorrespondingGA/GSCevents

5313


23

ComparisonofLoggedEvents

GA/GSC

Solr


24

JanuaryLoggedEvents

GA/GSC

Solr


25

JanuarySolrUniqueIPandUniqueLat/Long


26

MarchLoggedEvents

GA/GSC

Solr


27

MarchSolrUniqueIP&UniqueLat/Long


28

Considerations

!  Humanvs.BotBehavior!  Whataboutorganiza[ons?

!  ReturningUsers&BookmarkedContent!  Visitsanddevices!  Difficulttolumpeventsbydaybecauseeventsmaybeseparatedby

hours

!  Pageviewsvs.Downloads

!  Accessvs.Use!  Whenisaccessnotuse?


29

Closing

!  Varia[onamongIRplayormsandconfigura[onscomplicatesbenchmarkinginternallyandacrossins[tu[ons

!  Logdataover-countusage!  GoogledoesamuchbeVerjoboffilteringbots

!  Pagetaggingservicesundercountusage!  Itemlevelanalysissuggeststhatundercoun[ngresultsfromassump[ons

whichruleoutlegi[mateuserinterac[ons


30

Publications

Published: Patrick OBrien, Kenning Arlitsch, Jeff Mixter, Jonathan Wheeler, Leila Sterman. “RAMP: Repository Analytics and Metrics Portal: A Prototype Web Service that Accurately Counts Item Downloads from Institutional Repositories,” Library Hi Tech, vol. 35, no. 1, March 2017 Patrick OBrien, Kenning Arlitsch, Leila Sterman, Jeff Mixter, Jonathan Wheeler, and Susan Borda. “Undercounting File Downloads from Institutional Repositories,” Journal of Library Administration, vol. 56, no. 7, 2016 Proposal funded by IMLS: ”Measuring Up: Assessing Accuracy of Reported Use and Impact of Digital Repositories” - scholarworks.montana.edu/xmlui/handle/1/8924


31

!Thankyou!

JonathanWheeler,DataCura[onLibrarian,[email protected],DeanoftheLibrary,[email protected]@kenning_msu


32

Documents

1 Polluted Leftovers: Repository Metrics from the ... · Web Service that Accurately Counts Item Downloads from Institutional Repositories,” Library Hi Tech, vol. 35, no. 1, March