Assessing and Evaluating Services in Libraries and Information Centers Towards Sustained Progress...

Assessing and Evaluating Services in Libraries and Information Centers Towards Sustained Progress and Development

Dennis A. AlonzoDean, College of Education

Director, Special Programs

Head, Curriculum Development Center

University of Southeastern Philippines

Davao City

Key Concepts

Reasons for EvaluationThe Framework of EvaluationIdentifying Performance issues for

EvaluationMethods Performance Measurement for the eLib

Why do we need to Evaluate?

To gather empirical data to inform decisions.

As an internal control mechanism to ensure that the resources are used efficiently and effectively

To convince the funders and the clients that the service is delivering the benefits that were expected when the investment was made

Dennis A. Alonzo 3

Dennis A. Alonzo 4

Big Question in Evaluation

What do you want to know?

Ideal Library

Set clear, tough and meaningful standards Tell users in a clear, straightforward way about

services Consult widely about what services people

need and how services can be improved Make services available to everyone who

needs them Treat all people fairly. Have polite and helpful

staff

Ideal Library

Use resources effectively by budgeting carefully

Continually make improvementsWork with other providers to provide a

better serviceShow that users agree that the services

provided are really good.

Issues in Evaluation

Collect information to facilitate decision making

Justify increasing expenditures Evaluate the quality of services provided Plan for future improvements and directions Identify the extent to which problems can be

solved Identify differing needs of different user

categories

Issues in Evaluation

Plan public relations work and information dissemination

Provide feedbacks to and evaluate contractors

Involve users in management – allows users to rediscover a voice in library management and express views about service priorities

Avoid “questionnaire fatigue”

Focus of Evaluation

Appraisal of strengths and weaknesses?Effectiveness of its educational

services?

Identifying Performance Issues

The way the management structure functions Internal operations relating to information

materials, such as cataloguing and classification

Library/ information services to users New programs of service delivery Alternative possibilities for doing anything The functioning of a total system prior to

planning change

Dennis A. Alonzo 11

Evaluation defined:

Can and should enhance the quality of interventions (policies and programs) designed to solve or ameliorate problems in social and corporate setting (Owen, 2006)

Process of knowledge productionUses rigorous empirical enquiry

Dennis A. Alonzo 12

Logic of Evaluation (Fournier, 1995)

Establishing criteria of worth On what dimensions must the evaluand do well?

Constructing standards How well should the evaluand perform?

Measuring performance and comparing with standards How well the evaluand perform?

Synthesizing and integrating evidence into a judgment of merit or worth What is the worth of the evaluand?

Dennis A. Alonzo 13

Objects of an Evaluation

PoliciesLegislative policiesLarge scale policiesLocal policies

ProgramsProductsPerson/ People

Criteria for Evaluation

SuccessEfficiencyEffectivenessBenefitsCosts – which can be evaluated

independently or in association with any of the above

Dennis A. Alonzo 15

Evaluation Forms and Approaches

Program

Proactive Evaluation

Clarificative Evaluation

Interactive Evaluation

Monitoring Evaluation

Impact Evaluation

Owen, 2006

Dennis A. Alonzo 16

PROACTIVE EVALUATION

Takes place before the program is designed

Assists planners to make decisions about what type of program is needed

Provides input about how best to develop program in advance of the planning stage

Dennis A. Alonzo 17

Typical Issues

Is there a need for a program? What do we know about the problem that the program

will address? What is recognized as best practice in this area? Have there been attempts to find solutions to this

problem? What does the relevant research or conventional

wisdom tell us about this problem? What could find out from external sources to

rejuvenate an existing policy or program?

Dennis A. Alonzo 18

Major Approaches

1. Needs Assessment or Needs Analysis

2. Research Synthesis

3. Meta-analysis

4. Narrative Review

5. Review of Best Practices

Dennis A. Alonzo 19

Evaluation Forms and Approaches

Program

Proactive Evaluation

Clarificative Evaluation

Interactive Evaluation

Monitoring Evaluation

Impact Evaluation

Owen, 2006

Early After Implementation

Dennis A. Alonzo 20

CLARIFICATIVE EVALUATION

Designed to assist stakeholders to conceptualize interventions and improve their coherence, and thus increase the chances that their implementation will lead to the desired outcomes.

Concentrates on making explicit the internal structure and functioning of an interventions

Program logic or theory is developed/ revised

Dennis A. Alonzo 21

Issues to be addressed

What are the intended outcomes of this program and how is the program designed to achieve them?

What are the underlying rationale for this program?

What program structures or elements need to be modified to maximize program potential to achieve the intended outcomes?

Is the program plausible? Which aspects of the program are amenable to a

subsequent monitoring or impact assessment?

Dennis A. Alonzo 22

Approaches:

1. Evaluability Assessment (EA)

2. Program Logic

3. Ex-ante Evaluation

Dennis A. Alonzo 23

Evaluation Forms and Approaches

Program

Proactive Evaluation

Clarificative Evaluation

Interactive Evaluation

Monitoring Evaluation

Impact Evaluation

Owen, 2006

After The Program Design has been

Clarified/ Finalized

Dennis A. Alonzo 24

3. INTERACTIVE EVALUATION

Provides systematic evaluation findings through which local providers can make decisions about the future direction of the program;

Provides assistance in planning and carrying out self-evaluations;

Focuses evaluation on organizational change and improvement, in most cases on a continuous basis; and

Empowers providers and participants.

Dennis A. Alonzo 25

Typical Issues to be addressed

What is this program trying to achieve?How is this program progressing?Is the delivery working?Is it consistent with the program plan?How could the delivery be changed so

as to make it more effective?How could this organization be changed

so as to make it more effective?

Dennis A. Alonzo 26

Approaches

Responsive Evaluation (Stake, 1980)Action ResearchDevelopment EvaluationEmpowerment EvaluationQuality Review

Dennis A. Alonzo 27

Evaluation Forms and Approaches

Program

Proactive Evaluation

Clarificative Evaluation

Interactive Evaluation

Monitoring Evaluation

Impact Evaluation

Owen, 2006

Conducted to determine the

performance of each unit of the program

Dennis A. Alonzo 28

4. MONITORING EVALUATION

Appropriate when a program is well established and ongoing.

Involve the development of a system of regular monitoring of the progress of the program.

Include a rapid response capability (Mangano, 1989) and to provide timely information for organizational leaders (Owen & Lambert, 1998)

Dennis A. Alonzo 29

Typical Issues

Is the program reaching the target population? Is implementation meeting program benchmarks? How is implementation progressing between sites? How is implementation progressing now compared to

a month ago, or a year ago? Are our cost rising or falling? How can we fine-tune this program to make it more

efficient? How can we fine-tune this program to make it more

effective? Is there a site which needs attention to ensure more

effective delivery?

Dennis A. Alonzo 30

Key Approaches

1. Component Analysis

2. Devolved Performance Evaluation

3. Systems Analysis

Dennis A. Alonzo 31

Evaluation Forms and Approaches

Program

Proactive Evaluation

Clarificative Evaluation

Interactive Evaluation

Monitoring Evaluation

Impact Evaluation

Owen, 2006

May be conducted during the early implementation

but mostly done after program phase out

Dennis A. Alonzo 32

IMPACT EVALUATION

Determines the range and extent of outcomes of a program;

Determine whether the program has been implemented as planned and how implementation has affected outcomes;

Provides evidence to funders, senior managers and politicians about the extent to which resources allocated to a program have been spent wisely; and

Informs decision about replication or extension of a program

Dennis A. Alonzo 33

Typical Issues

Has the program been implemented as planned? Has the stated goals of the program been achieved? Have the needs of those served by the program been

achieved? What are the unintended outcomes of the program? Does the implementation strategy lead to the intended

outcomes? How do differences in implementation affect program

outcomes? Is the program more effective fro some participants

than for others? Has the program been cost-effective?

Dennis A. Alonzo 34

Key Approaches

1. Objectives-based (Tyler, 1950)

2. Needs-based (Schriven, 10972)

3. Goal Free

4. Process – outcome

5. Realistic Evaluation

6. Performance Audit

Dennis A. Alonzo 35

Framework for Planning an Evaluation

1. Specifying the EvaluandWhat is the object of the evaluation?What is known about the evaluand?How was it developed?How long has it been in existence?What is the nature of the evaluand:

policy/program/organization/product?Who are the key players in its development

(actual or projected) and its implementation?

Dennis A. Alonzo 36

Framework for Planning an Evaluation

2. Purpose What is the fundamental reason for commissioning

the evaluation? Consistent with evaluation form, the evaluation is

primarily concerned with: Synthesis of information to aid program development; Clarification of the program; Improvement of the implementation of the program; Monitoring program outcomes; Determining program impact.

Dennis A. Alonzo 37

Framework for Planning an Evaluation

3. Clients/ AudiencesTo whom will the findings of the evaluation

be directed? Identify your clients, the primary audience,

and other people who will use the information to make decisions

Dennis A. Alonzo 38

Framework for Planning an Evaluation

4. ResourcesWhat person power and material resources

are available to undertake the evaluation?The resources available determine the

extent of the data management and the range of evaluation findings that can be provided.

Dennis A. Alonzo 39

Framework for Planning an Evaluation

5. Evaluation focus/ fociWhich element(s) of the program will need

to be investigated: program context, program design, program implementation, program outcomes or a combination?

What is the state of the development of the evaluand?

Dennis A. Alonzo 40

Framework for Planning an Evaluation

6. Evaluation issues and key questions Identify the issues to be addressed Questions lead the direction of the evaluation

7. Data management Identify data collection strategy and analysis Is sampling important? Is anything known about this from other sources? How ill the data be collected? How will the data be analyzed to adress the key

evaluation questions?

Dennis A. Alonzo 41

Framework for Planning an Evaluation

8. Dissemination of FindingsWhat strategies for reporting will be used?When will reporting take place?What kind of information will be included

(findings, conclusions, judgments, recommendations)?

Who will make the recommendations?

Dennis A. Alonzo 42

Framework for Planning an Evaluation

9. Codes of BehaviorWhat are the ethical conditions which

underlie the evaluation report?

10. Budget and TimelineGiven the resources, what will be achieved

at key time-points during the evaluation?

Performance Measurement for the eLib

Access to electronic journal Word processing packages Excel and other statistical packages Demonstration software Internet use Bibliographic software Digitized books and journals Electronic information database OPACs Networked CD-ROMS on local area networks Full text outputs via bibliographic searching Web based training packages

Performance Indicators for eLib

Informative contentReliabilityValidityAppropriatenessPracticabilityComparability

Performance Issues

Skills levelReal vs browsingRecreational useProvision of unwanted/ unanticipated

servicesQueuing/ booking/ walkoutsRemote logging in/ problems withProblems of outputting data

Performance Issues

No define service periodQuality and reliability of internet dataNon-useChanges over timeDistributed resourcesProblems with the library’s controlThe service oriented culturePCs vs Macs

Proposed list of performance indicators

Percentage of target population reached by elib services

Number of log-ins to elib services per capita per month

Number of remote log-ins to elib services per capita per month

Number of electronic documents delivered per capita per month

Cost per log in per elib service

Proposed list of performance indicators

Reference enquiries submitted electronically per month

Library computer work station use rate Number or library work station per capita Rejected log-ins as a percentage of total log

ins Systems availability Mean waiting time for access to library

computer workstations IT expenditure as percentage of total library

expenditures