Evaluating library assessment capabilities

A measure of effectiveness

Simon Hart

University of Otago Library

http://www.matarikinetwork.com

Benchmarking

“…the process of identifying best practices and learning

from others.

…enables a better understanding of practice, process or

performance and provides insights into how improvements

might be made.”

Jackson, N. (2001) Benchmarking in UK HE: an overview

Quality Assurance in Education, 9 (4), 218 – 235.

If we enable and support the

academic endeavor

how do we measure our

effectiveness?

Two key problems in academic

library assessment:

the failure of libraries to select measures that

demonstrate a contribution to the strategic plan of the

institution

the selection of performance measures that are not

meaningful to campus administration.

Matthews, J. R. (2007). Library assessment in higher education.

Westport CT: Libraries Unlimited

How do we measure our effectiveness?

How do we assess our assessment?

How could we frame our assessment to

identify an improvement path?

What should a mature approach to

library assessment look like?

Dartmouth 2013

Otago 2014

Queens 2015

Durham 2016

Detail the processes used to gather data that

informed the development of the project.

Detail the performance measures used to

evaluate the success of the project.

Outline plans you have for the ongoing evaluation

of the effectiveness of the projects outcomes.

Initial

Repeatable

Defined

Quantitatively

managed

Continuous

improvement

5

4

3

2

1

optimized

measured

confirmed

documented

ad hoc

Capacity Maturity Model

Paulk, M. C., Curtis, B., Chrissis, M. B. and Weber, C. V. (1993). Capability maturity model,

version 1.1, Institute for Software Research Paper 7.

8 benchmarks

1-5 scale of capability

2 phased application

Becker, J., Knackstedt, R., & Pöppelbuß, D. W. I. J. (2009). Developing

maturity models for IT management. Business & Information Systems

Engineering, 1(3), 213-222.

Procedure model for developing maturity models

Need identified

Process built in

Comparison made

Design & test model

Frankie Wilson : The Quality Maturity Model

http://www.qualitymaturitymodel.org.uk/

Frankie Wilson : The Quality Maturity Model

Wilson, F. (2013). The quality maturity model: assessing organisational quality

culture in academic libraries. Doctoral dissertation, Brunel University, School of

Information Systems, Computing and Mathematics.

Bakkalbasi, N., Sundre, D. L., & Fulcher, K. F. (2012). Assessing assessment: A

framework to evaluate assessment practices and progress for library collections and

services. Proceedings of the 2012 Library Assessment Conference. (pp. 533-537).

Stages of Assessment cycle

Assessment Capability Maturity Model

Assessment Capability Maturity Model

Stages of the Assessment Cycle

Leve

ls o

f ca

pab

ility

Methods & data collections: Gathering feedback

Continuous Assessment of methods and data reuse is used to improve feedback gathering exercises

Managed Data from defined feedback methods is structured for use and reuse.

DefinedA range of repeatable methods are used to gather feedback

Repeatable Feedback is proactively sought from sub-set of clients for specific purposes

Ad hoc Feedback is client initiated and not collated

Amos, H. & Hart, S. (2013). International collaboration for quality: A case

study of fostering collaboration across an international network of

University Libraries through an activity based benchmarking project,

Proceedings of the 34th Annual IATUL conference.

Hart, S. & Amos, H. (2014). The development of performance measures

through an activity based benchmarking project across an international

network of academic libraries. Performance Measurement and Metrics,

15(1/2), 58-66.

Hart, S. & Amos, H. (in press). Building on what works: towards a library

assessment capability maturity model. Proceedings of the 11th

Northumbria International Conference on Performance Measurement in

Libraries and Information Services.

ReferencesAmos, H. & Hart, S. (2013). International collaboration for quality: A case study of fostering collaboration across an international network of University Libraries through an activity based benchmarking project, Proceedings of the 34th Annual IATUL conference. Retrieved from http://docs.lib.purdue.edu/iatul/2013/papers/48/.

Bakkalbasi, N., Sundre, D. L., & Fulcher, K. F. (2012). Assessing assessment: A framework to evaluate assessment practices and progress for library collections and services. In S. Hiller, M. Kyrillidou, A. Pappalardo, J. Self, A. Yeager (Eds.) Proceedings of the 2012 Library Assessment Conference. (pp. 533-537). Charlottesville, VA: Association of Research Libraries.

Becker, J., Knackstedt, R., & Pöppelbuß, D. W. I. J. (2009). Developing maturity models for IT management. Business & Information Systems Engineering, 1(3), 213-222.

Hart, S. & Amos, H. (2014). The development of performance measures through an activity based benchmarking project across an international network of academic libraries. Performance Measurement and Metrics, 15(1/2), 58-66.

Hevner A.R., March S. T., Park J., Ram S. (2004). Design science in information systems research. MIS Quarterly 28(1):75–105.

Jackson, N. (2001). Benchmarking in UK HE: an overview. Quality Assurance in Education, 9(4), 218-235.

Matthews, J. R. (2007). The Evaluation and measurement of library services. Westport, CT: Libraries Unlimited.

Paulk, M. C., Curtis, B., Chrissis, M. B. and Weber, C. V. (1993). Capability maturity model, version 1.1, Institute for Software Research Paper 7. Retrieved from http://repository.cmu.edu/sei/171.

Sankey , M. (2014). Benchmarks for Technology Enhanced learning: a report on the refresh project 2014 and Inter-Institutional Benchmarking Summit. Retrieved from http://www.acode.edu.au/mod/resource/view.php?id=269.

Wilson, F. & Town, J. S. (2006), Benchmarking and library quality maturity, Performance Measurement and Metrics, 7(2) 75–82.

Wilson, F. (2013). The quality maturity model: assessing organisational quality culture in academic libraries. Doctoral dissertation, Brunel University, School of Information Systems, Computing and Mathematics.

Wilson, F. (2015). The Quality Maturity Model: your roadmap to a culture of quality. Library Management, 36(3), 258-267.