This article was downloaded by: [University of Windsor]On: 16 November 2014, At: 07:16Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registeredoffice: Mortimer House, 37-41 Mortimer Street, London W1T 3JH, UK

Journal of Hospital LibrarianshipPublication details, including instructions for authors andsubscription information:http://www.tandfonline.com/loi/whos20

Measuring Return on Investment in VALibrariesKaren Jemison a , Ed Poletti b , Janet Schneider c , Nancy Clark d &Ron Drew Stone ea VA Palo Alto Health Care System , Palo Alto, CA, USAb Learning Resources Service , Central Arkansas Veterans HealthcareSystem , Little Rock, AR, USAc Library Service , James A. Haley Veterans Hospital , Tampa, FL,USAd Library Network Office , VA Central Office , Washington, DC, USAe Center for Performance and ROI , Birmingham, AL, USAPublished online: 04 Nov 2009.

To cite this article: Karen Jemison , Ed Poletti , Janet Schneider , Nancy Clark & Ron Drew Stone(2009) Measuring Return on Investment in VA Libraries, Journal of Hospital Librarianship, 9:4, 379-390,DOI: 10.1080/15323260903253803

To link to this article: http://dx.doi.org/10.1080/15323260903253803

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the“Content”) contained in the publications on our platform. However, Taylor & Francis,our agents, and our licensors make no representations or warranties whatsoever as tothe accuracy, completeness, or suitability for any purpose of the Content. Any opinionsand views expressed in this publication are the opinions and views of the authors,and are not the views of or endorsed by Taylor & Francis. The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information. Taylor and Francis shall not be liable for any losses, actions, claims,proceedings, demands, costs, expenses, damages, and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with, in relation to or arisingout of the use of the Content.

This article may be used for research, teaching, and private study purposes. Anysubstantial or systematic reproduction, redistribution, reselling, loan, sub-licensing,systematic supply, or distribution in any form to anyone is expressly forbidden. Terms &

Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

379

Journal of Hospital Librarianship, 9:379–390, 2009Copyright © Taylor & Francis Group, LLC ISSN: 1532-3269 print/1532-3277 onlineDOI: 10.1080/15323260903253803

WHOS1532-32691532-3277Journal of Hospital Librarianship, Vol. 9, No. 4, September 2009: pp. 0–0Journal of Hospital Librarianship

Measuring Return on Investment in VA Libraries

Return on Investment in VA LibrariesK. Jemison et al.

KAREN JEMISONVA Palo Alto Health Care System, Palo Alto, CA, USA

ED POLETTILearning Resources Service, Central Arkansas Veterans Healthcare System, Little Rock, AR, USA

JANET SCHNEIDERLibrary Service, James A. Haley Veterans Hospital, Tampa, FL, USA

NANCY CLARKLibrary Network Office, VA Central Office, Washington, DC, USA

RON DREW STONECenter for Performance and ROI, Birmingham, AL, USA

After evaluating available library-targeted return on investment(ROI) tools, VA Library Network Librarians created a workgroup toformulate a tool geared specifically to the Department of VeteransAffairs (VA) hospital libraries. The workgroup devised an approachwith three separate components: a Return on Investment Analysistool that each library could employ; a Library Scorecard thatprovided local and national examples of typical benefits, forlibraries that could not devote time to using the ROI instrument;and a Management Support Report for libraries that wanted amission-based document to use with their local management. Thetools were essentially complete in 18 months, after numerous drafts.

KEYWORDS measurement tools, return on investment, ROI,scorecard

Received July 9, 2009; accepted August 3, 2009.Address correspondence to Karen Jemison, Library Manager, VA Palo Alto Health Care

System, Library (142D), 3801 Miranda Ave., Palo Alto, CA 94304, USA. E-mail: Karen.Jemison@va.gov

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

380 K. Jemison et al.

BACKGROUND

Those interested in applying return on investment (ROI) analysis to alibrary have a rich vein of material to mine. The literature stretches fromthe early 1990s to present day, covering all types of libraries. It is not diffi-cult to absorb the principles that must govern a credible ROI evaluation.The difficulty lies in applying the principles to cerebral decision-making byusers, rather than to the production of widgits. In libraries and educationalendeavors, most ROI literature can be summarized as: “ROI is crucial tomaking your case to financial decision-makers, but it is extremely difficultto assign causality (which then may translate to monetary benefit) to yourmost central functions. Instead, you’ll often have to settle for counting use—circulation, interlibrary loans, and so on—and assigning problematicvalues to them.”

We spent a fair amount of time studying the literature and trying toidentify pieces of VA Library services that could fit within an acceptablebusiness model, and then in finding a basis for the assignment of value.If we could not find comparable commercial charges for a service orproduct, we did not include the latter in our Return worksheet (what thehospital receives from its investment in the library). We ruled out “softdata” so that librarians could be confident in using the instrument withtheir management, and in using the results for their own evaluationefforts.

OBJECTIVES

The primary focus was on the Return on Investment Analysis tool. Centralconcerns were

• Fiscal Credibility with Management. In ROI, it is crucial to maintain aconservative analysis. Understating the costs or overstating the value orbenefits renders the entire ROI Analysis implausible. For example, ifyou assume a “knowledge” benefit to which you have attached a gen-eralized benefit per transaction, management may well see the fallacyof the generalization as well as query your rationale for the value youhave assigned. In our assigned values, we took national figures compa-rable to baseline commercial charges for excellent service, speed, andquality for medical information delivery, and then reduced those by30% to 45%. In our directions showing library managers how to fill outthe Analysis, we made a plea for equivalency: if the services providedfall below excellent, we asked that the multiplier (the value attached toeach transaction) be scaled down to reflect that. Notably high ROIsinvite disbelief and raking examination. A typical business return of

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

Return on Investment in VA Libraries 381

25% is considered reasonable. Returns of over 100% need strong evidence.As a conservative estimate of value, we asked that users eliminateservices under the Return worksheet that their hospital would not payfor through a commercial service. We also recommended that programsnot meaningful to hospital management be excluded, no matter howworthy library management deemed them. However, we captured com-mercial cost data as a value because we wanted to show the benefit ofa professional internal library and staff as opposed to an outsourcedcommercial information broker, and we shared that on the ROI reportto management.

• Equivalent Measurement. In the Investment worksheet (what the hospitalinvests to get library service), we needed to make sure that the same itemswere counted the same way, to achieve a level playing field. Figure 1shows the Investment worksheet entries. In the VA hospital system, afiscal unit called DSS (for Decision Support System) measures exactly thesame costs for each hospital. That meant we could adequately define andretrieve information about typical hospital overhead costs. Discoveringthis and determining what to count and what to ask for took us consider-able time. However, this step was invaluable in eliminating uncertaintiesfor the librarian trying to use the ROI instrument, and in providing parityfor these overhead costs. We also had to account for regional andnational contributions that support the local libraries to get a realisticpicture of the investment VA makes.

FIGURE 1 Worksheet A–The Investment worksheet. Explanatory notes 1 to 16 (not reproducedhere) explain each line. Formulas are embedded, and carry data over to other worksheets.

Return on Investment for Library ServicesAnnual Investment

Note Investment Amount Multiplier (see Notes)

Hospital Cost

1 Local:2 Budget for materials and services $0 $03 Library space (number of square

feet)0 $0.00 $0

4 Staff for Library services:5 Professional staff salaries: $0 1.26 $06 Support staff salaries: $0 1.26 $07 Hospital overhead for the Library:8 EMS cost per square foot $0.00 0 $09 Human Resources cost per

FTEE$0.00 0.00 $0

10 Fiscal cost per FTEE $0.00 0.00 $011 A&MMS cost per FTEE $0.00 0.00 $012 IRM cost per FTEE $0.00 0.00 $013 Regional & National:14 CO Library Program Support $7,687 $7,68715 VISN and regional Library

contracts $0 0.00 $0

16 Total annual investment for Library Service $7,687

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

382 K. Jemison et al.

The need for parity exemplifies one large difficulty with simplified ROIcalculators. To offer ease of use, calculators cover a diversity of libraries andlibrary situations without well-defined or even guaranteed-retrievable com-parable data. The investment portions tend to be sketchy, and the returnsections sometimes include “common sense” transactional values that areonly rough estimates. The figures may not be challenged by a public audi-ence. But if it is an eagle-eyed, cost-cutting hospital director examining thetool, simplified calculators leave a library manager vulnerable to challengeand may compromise his reputation for fiscal astuteness and responsibility.

In the Return worksheet (Figure 2), we looked at the library contribu-tion in several ways. What would it cost the VA library to provide the samelevel of service, for the same amount of activity if it were commercially out-sourced, compared to the current in-house cost? We felt this was a validcomparison for on-demand, technically accurate, up to date, frequently special-ized or complex materials/service, and fast turnaround.

These terms are likely to be compromised, or downwardly negotiatedoutside a private sector contract—such as in a sidebar contract with a uni-versity, public library, or affiliated hospital network. (Their fees would reflectthat lesser product delivery. Many of the multipliers, which now reflect feesof an information broker, would need to be modified.)

In the Return on Investment Analysis we included

• Business Impact. How much clinician time do we save that they typicallyreinvest in productive work—and how much do we save the hospitalwith mediated searching?

• Demonstrated Value. What is our collection’s cost effectiveness per usecompared to the commercial charge for each use? What are the cost sav-ings we offer the hospital with ILL service (our cost versus commercialcost for the item), and what cost avoidance do we offer with ILL (whatjournal subscription and purchase costs do we avoid)?

• Library Use versus Outputs. Use figures (such as circulation and ILL) mea-sure library benefits to the hospital. Output or outcome measures aremore desirable (such as the impact of the information on the individualclinician, the patient or his medical care), but VA does not gather nationalstatistical data that would allow us to assign a dependable monetary valueto outputs. (For example, the averaged cost of a saved outpatient visit isnot available.) This lead us to

• Intangible Benefits. Despite the lack of attached monetary value, these arethe heart and soul of our service. We suggested that the librarian’s finalreport feature these prominently. They are often considered far moreimportant than the tangible monetary benefit. Worksheet C shows a sampleone-page ROI summary for management (from VA Palo Alto Health CareSystem). Some categories are compressed to not exceed one page, oursuggested length for the report.

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

Return on Investment in VA Libraries 383

METHODS

The ROI instrument was developed in consultation with a representativefrom the Center for Performance and ROI, under the aegis of the ROI Coor-dinator in VA Central Office, Washington DC. The “Intangible Benefits”component used a 1-year study in VISN (Veterans Integrated Service Network)

FIGURE 2 Worksheet B–The Return worksheet. Explanatory notes 1 to 17B (not reproducedhere) explain each line. Formulas are embedded, and carry data over to other worksheets.

Return on Investment for Library ServicesAnnual Return

Note Products & Services Received Amount Multiplier (see Notes)

Commercial Cost

1 Direct to Hospital Staff:2 Journal articles used

2A Number of uses from the collection

0 $55 $0

2B Number borrowed through ILL

0 $55 $0

2C Number of unique journal titles borrowed through ILL:

0

3 Books used3A Number of uses from the

collection0 $230 $0

3B Number of books borrowed through ILL

0 $230 $0

3C Number of unique book titles borrowed through ILL:

0

4 Media used:4A Number of uses from the

collection0 $245 $0

4B Number of media borrowed through ILL

0 $245 $0

4C Number of unique media titles borrowed through ILL

0

5 Hospital administrative journal subscriptions cost paid by Library

$0 $0

6 Hospital administrative books cost paid by Library

$0 $0

7 Reference questions & mediated searches:7A Number of brief or simple

questions and searches0 $35 $0

7B Number of complex questions and searches

0 $170 $0

8 Number of database searches by users

0 $25 $0

9 Professional Staff Cost and Contributions:Percentage of time spent in collection development and selection.

0.0% $0 $0

10 Support Staff Cost and Contributions:10A Percentage of time spent in

maintaining print and electronic collection access.

0.0% $0 $0

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

384 K. Jemison et al.

16 (a region with 8 hospitals of varying size and complexity), whichsurveyed health care professionals about the benefits they received frommediated search requests. The Associate Director at VA Palo Alto helped byproviding feedback from a management/fiscal perspective about the tool asit was developed. Standardized costs for commercial services, examples ofreports and data tools, and suggestions for data sources were providedwithin the ROI Analysis. We expect to modify all three products from thiseffort on a regular basis.

FIGURE 2 (Continued).

10B Percentage of time spent in providing interlibrary loan.

0.0% $0 $0

11 Demonstrated value of the Library's materials: Per use:11A Our collection cost

effectiveness#VALUE! 0

11B Commercial cost effectiveness $0 012 Demonstrated Value of Interlibrary Loan Service

12A Annual ILL budget $012B Our cost savings $012C Our cost avoidance $0

Journals 0 $150 Books 0 $230 Media 0 $245

13 Impact of Literature Searches on Clinical Decision MakingIn 98.5% of all searches pertinent information was receivedIn 95.5% of all searches new knowledge was provided49% influenced the advice to patient and family16% affected the choice of tests30% affected the choice of drugs14% influenced the diagnosis5% changed the length of stay30% influenced the choice of treatment30% avoided a hospital admission3% avoided adverse events or complications8% avoided patient mortality49% altered the mode of treatment89% reinforced a mode of treatment95% were judged useful to direct patient care

13A Staff estimated a hospital savings from Librarian's reference/search:

$349 $0

13B Staff estimated a time savings for them from Librarian's reference/search:

$139 $0

14 Search training 0 $0 $015 Direct to Patients & Family:16 Books & media used $0

Annual book budget $0 $0 Annual media budget $0 $0 Donated material, if significant (see note)

$0 $0

17 Reference questions & mediated searches:17A Brief or simple work 0 $35 $017B Complex work 0 $170 $018 Total annual return by Library Service $0

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

Return on Investment in VA Libraries 385

RESULTS

The first three hospitals to test the ROI Analysis came up with ROIs of 210% to450%. Two of the hospitals presented their findings to management, report-ing the results as part of general proactive strategies to remind management

FIGURE 3 Worksheet C– Sample one-page ROI summary for management (from VA PaloAlto Health Care System).

Return on Investment for Medical Library ServicesOur Business Contribution Report for 2008

Cost Benefit Ratio 4.1:1

Return on Investment 308%

LIBRARY IMPACT ACCORDING TO PHYSICIANS: OUR INTANGIBLE BUSINESS BENEFITS

In 95.5% of searches new knowledge was provided

In 98.5% of searches pertinent Information was received

49% of searches influenced the advice to patient and family

16% affected the choice of tests

30% affected the choice of drugs

14% influenced the diagnosis

5% changed the length of stay

30% influenced the choice of treatment

30% avoided a hospital admission

3% avoided adverse events or complications

8% avoided patient mortality

49% altered the mode of treatment

89% reinforced a mode of treatment

95% were judged useful to direct patient care

Investment Hospital CostBudget $477,989Staff $395,844Space and hospital overhead $116,343CO Library Program support & VISN Library contracts $22,702Total annual investment $1,012,877

Return to Staff and Patients Commercial CostJournal articles, books and media used $421,905Administrative material supplied $20,758Database searches by users $2,515,725Reference questions, mediated searches, & search training $169,900

BUSINESS IMPACT MEASURES

Physicians estimated hospital savings from Library searches $759,424Their own reported time savings from Library searches $241,971Total annual return to the HCS $4,129,683

Demonstrated value of the Library's print & electronic collections

Our collection cost effectiveness per use: $10.76Compared to commercial cost per use: $27.91

Demonstrated value of Interlibrary Loan services

Cost savings $191,000

Cost avoidance $133,150Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

386 K. Jemison et al.

that the libraries are well run. At one hospital, the ROI was used in a man-agement briefing and a budget defense that proposed an FTE increase. Thestrategy worked, although in a beautiful example of our central problem,we cannot say how great a role the ROI findings played in management’sdecision.

The ROI instrument includes a worksheet with these and other exam-ples, and the Library Scorecard summarizes the examples. We hope that asmore VAs use the tool, they will share their results to enrich both products.

CONCLUSIONS

The ROI Analysis offers VA libraries a valid, standardized method of measuringROI. Each VA library’s circumstances—in affiliation and consortia agreements,fiscal resources, space, and staffing—make national tools for ROI based onaveraging questionable. Therefore valid local ROI measures are invaluable.

APPENDIX A: SOURCES FOR THE ROI ANALYSIS

Parker-Pope T. Research for hire: firms help patients find latest cures forchronic illnesses. Wall Street Journal (Eastern edition). 2002;Nov 26:D1.

Esparza JM. Showing the money: dollar values as evidence of the library’svalue. Nat Net. 2005;30:23.

Science Direct. (2007). Customer service for clinical online access, Science-Direct fees, July 2007.

Example: $30 per article + locating costs + payment/account costs; com-mercial broker overhead is 100% = $60 per article electronically; $50 forprint with standard delivery, conservatively.

Fortney L. Price history for core clinical journals in medicine and nursing2003–2007. Ebsco Industries; 2007.

Doody’s core titles in the health sciences 2007: list overview and analysis.Doodys; 2006.

Re-imagining our libraries through the eyes of our administrators by thelibrarians of the South Center VA healthcare network (VISN 16).

This study was conducted with data from April 2006 to March 2007.Contributing to this effort were Edward J. Poletti, MLS, AHIP, CentralArkansas Veterans Health Care System, Little Rock, AR; Dixie A.Jones, MLS, AHIP, Overton Brooks VA Medical Center Library,Shreveport, LA; Marvett S. Burns, MLS, VA Gulf Coast VeteransHealth Care System, Biloxi, MS; Lola H. Purvis, MLIS, CLIS, VA

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

Return on Investment in VA Libraries 387

Medical Center, Alexandria, LA; Priscilla L. Stephenson, MSLS, MSEd,AHIP, G.V. “Sonny” Montgomery VA Medical Center, Jackson, MS;Sara Blackwell, MLS, VA Medical Center, Oklahoma City, OK; FeliciaLittle, MLIS, Michael E. DeBakey VA Medical Center, Houston,TX; Dianne Jones, “G.V. “Sonny” Montgomery VA Medical Center,Jackson, MS.

APPENDIX B: LIBRARY SCORECARD

The Scorecard is a snapshot of typical management goals, with examples ofhow VA Libraries meet these standard organizational aims. It is meant tooffer libraries and management a précis about return on investment (ROI)and library achievements for those who are unable to complete a full ROIanalysis. Under some of the categories, we include information we havegathered from VA Libraries that have used our Return on Investment Analysistool. We have also included local examples to give you an idea of what fitsin each category. You need not apply these specific examples to yourScorecard; they are merely illustrative. Access/Availability is closely relatedto Customer Delivery in libraries; you may need to balance them. However,a Scorecard always includes

• Access/Availability• Operations/Administration• Customer Delivery• Outcomes• Return on Investment

so please do not eliminate any of these elements. Your local examples, whenyou choose to include them, will reflect the measures you keep because youknow they are meaningful in your situation, and are an organic reflection ofyour program.

THE VA LIBRARY SCORECARD

Libraries in VA typically provide local desktop access to significant medicaljournals. The Central Office Library Network Program also provides nationaldesktop access to New England Journal of Medicine and ACP PIER Plus.Library savings in multiple use items, as well as reciprocating libraryagreements extend the range of material for patient care and research,while keeping the cost per use well below average commercial costs (seebelow). Interlibrary loan, online research, professional library assistance,

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

388 K. Jemison et al.

and collection development all lead to improved patient care and smart fiscalaccountability. The in-house professional expertise of librarians contributes sig-nificantly to the outcomes below, resting as they do on:

• An in-depth understanding of VA mission and goals, and local manage-ment’s priorities and programmatic planning;

• Far-ranging knowledge of medical, surgical, psychiatric and research liter-ature relevant to the hospital’s complexity and care;

• Managerial understanding about how to evaluate and monitor the fluctu-ating value of the literature, its associated databases, and their costs.

Access/Availability

LOCAL EXAMPLES

Electronic Distribution for Instant Full Text Material: 75% of the journal collection.Percentage of the collection reviewed annually for:

• the level of electronic availability for library clients (the number of timeswe exceeded our level for simultaneous usage for each title, or turn-aways): 100%

• the physical availability (the number of times, and the length of time adatabase or electronic title was up and available to library clients, or waitdata): 100%

Percentage of requests for purchase inside and outside the library which aremet: 100%

Online availability of: the library catalog, primary journal indexesfor medicine, nursing and psychiatry, UpToDate, 268 hand selected jour-nals as above, reference and service request functions and tutorials forproviders.

Immediate access to professional search help during MedicalLibrary open hours and pre-opening assistance ten hours a week fromVA Virtual Library Reference program (participating VA librarians across thecountry accessible on e-mail).

Operations/Administration

BUSINESS IMPACT MEASURES

Cost Effectiveness of the collection: our cost per use in ROI studies fromDallas, Little Rock, Palo Alto and Tampa, was 13% to 46% lower than costper use for commercial services.

Interlibrary Loan annual cost savings and cost avoidance showed arange of $77,745 to $1,083,755.

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

Return on Investment in VA Libraries 389

Librarians’ searches reaped a clinician-estimated hospital savings rang-ing from $159,493 to $1,977,783, and a time savings for clinicians valuedfrom $50,818 to $630,170.

LOCAL EXAMPLES

Percentage of the journal collection reviewed annually for cost per use: 100%Provided 463 mediated searches and 97,124 end-user searches.Answered 1300 reference questions for staff and patients.

Customer Delivery

LOCAL EXAMPLES

The Right Product. Percentage of the collection reviewed annually forclient relevance, quality, and use: 100%

Percentage of borrowing reviewed annually for possible purchase: 100%Added ExamMaster, Images MD, and MKSAP and 14 new journal titles

this year.Speed of Delivery. Average days for delivery* of items we don’t own for

staff: 1.83Percentage of collection journal titles available on the desktop, elec-

tronically in full text: 75%Percentage of requests filled for items we don’t own (ILL): 98%

OUTCOMES

From a year’s survey of providers in 8 hospitals in VISN 16:In 95.5% of librarians’ searches new knowledge was providedIn 98.5% of searches pertinent information was received49% influenced the advice to patient and family16% affected the choice of tests30% affected the choice of drugs14% influenced the diagnosis5% changed the length of stay30% influenced the choice of treatment30% avoided a hospital admission3% avoided adverse events or complications8% avoided patient mortality49% altered the mode of treatment

*Turnaround data can be found using QuickDoc or Docline report.

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14

390 K. Jemison et al.

Return on Investment

ROI studies in Dallas, Little Rock, Palo Alto, Seattle, and Tampa showed arange of 3:1 to 11.5:1 for Cost Benefit and 210% to 1050% for Return onInvestment.

Dow

nloa

ded

by [

Uni

vers

ity o

f W

inds

or]

at 0

7:16

16

Nov

embe

r 20

14