Enhancing Quality of Library Instruction Programs through Delayed Assessment 6th Northumbria International Conference on Performance Measures in Libraries

Enhancing Quality of Library Instruction Programs through Delayed Assessment

6th Northumbria International Conference on Performance Measures in Libraries and Information Services, Durham, UK. August 22-25, 2005 Gabrielle Wong

Reference LibrarianHong Kong University of Science and Technology

PM6, Durham, 23 Aug. 05/Gabrielle Wong, HKUST Library 2

Presentation Outline

1. Introduction2. Background of the Survey3. Survey Design and the

Questionnaire4. Sampling5. Key Findings6. Future Assessments


Introduction

A survey was conducted by HKUST Library to assess the enduring impact of its library instruction program

Questionnaires were sent 4-8 weeks after library classes in Fall 2004

The survey provides data that demonstrates the value of the program, and helps librarians to better understand the characteristics and needs of users

It was a valuable learning process for HKUST librarians; this experience serves as the basis for future assessment endeavour


HKUST

Founded in 1991 One of the three

research universities in Hong Kong

Undergraduates: 5,500; Postgraduates: 3,000; Faculty: 440

4 Schools: Science, Engineering, Business, Humanities & Social Sciences


Library Instruction at HKUST Library Serves the whole university community:

students, staff and faculty Over 200 library sessions every year,

reaching 4,000 attendees One-shot sessions, 50-80 minutes Wide variety: database workshops, course-

specific library sessions, topic/subject specific instruction (e.g. job hunting, microforms, web searching, research workshops for postgraduates)


Background of the SurveyEvaluation Practice since 1996 Evaluation forms were distributed at the

end of sessions Measured users' satisfaction Instructors' summary for future

improvement Generally, over 80% attendees agreed

that the class was worth recommending to their peers

Voluntary evaluation Results were not analyzed systematically


Background of the SurveyThe Needs for a Formal Assessment Do the library sessions help the

attendees in their research and studies?

What do they think after a period of time, during which they are expected to apply the skills for practical purposes?

How can we do better?


Background of the Survey The University Librarian, Dr. Samson

Soong, recommended the Reference Department to launch a study to answer these questions

The survey aimed to measure the enduring impact of the library program, and produce figures that he may use to demonstrate such value to the university administration

It should also expose the strengths and weaknesses of the program, thus facilitating improvements


Survey Design: Major Considerations first attempt of such kind simple, straightforward and effective cover as many classes as possible

(programmatic) librarians' workload demand on users' side assess the enduring/long term valueMethodology adopted: a delayed

perception survey


Survey Design: Delayed Assessment The survey was administered 4-8

weeks after the sessions The purpose of the delay period was to

allow attendees to apply the skills on their assignments and projects

Skills may be reinforced, proved insufficient or simply forgotten

A mean to assess the enduring value of the classes


Survey Design: The Questionnaire 2 pages, 14 questions, should take less

than 5 minutes to complete composed mostly of structured

questions rather than open questions to encourage feedback

provided options for open comments whenever possible


Survey Design: The QuestionnairePerception Questions: Q.2, Assessing impact: how the class

helped/changed the attendees Q.3, Specific skills learned Q.4, Skill retention Q.5, Overall class rating Q.6. Overall instructor rating Q.7 Suggestions for improvement


Survey Design: The QuestionnaireOther Questions: Library use habit

information search confidence no. of library classes attended previously research frequency

Demographic status place of permanent residence gender


Sampling

Selection criteria: class size > 15 if it was a course-specific session, there

should have been assignment or project associated with it

No intention to mimic the university population

25 sessions selected: 15 Course-Specific, 10 Open Workshops


About Class Types

Course-Specific (CS) Open Workshop (OW) library session serves the

information need of the associated credit course, e.g. FINA221

library session was "compulsory"

all students enrolled in the course were to be surveyed

well-defined group, we could use print and anonymous questionnaire distributed by the professors

high response rate

library session opens for all interested users to register by email

attendees were naturally more motivated

composed of a higher proportion of PGs and staff, therefore a more mature group

we have no choice but to survey them with emails

lost anonymity much lower response rate


Responses

688 questionnaires sent out Received (Print): 388 (90.7%) Received (E-mail): 78 (30.0%) Total: 466 (67.7%) 89% of respondents attended the

library sessions 11% had not attended the sessions, but

answered the demographic and library use habit questions


Findings: Impact

Question 2, rate on a 4-point scale ("1" Strongly Disagree to "4" Strongly Agree) :

As a result of the class, Ia. learned about sources to find needed information.b. learned about search methods to find needed

information.c. learned how to find needed information more

quickly.d. was more confident in researching information.e. have increased my interest in using the Library.


Findings: Impact

Mean Score

Agree % Disagree %

a. sources 3.15 95.6 4.4

b. search method

3.12 93.0 7.0

c. efficiency 3.04 85.3 14.7

d. confidence

2.85 74.1 25.9

e. interest 2.68 61.2 38.8


Findings: Impact

The differences of these mean scores are statistically significant (p<.05).

a sources

b search method

c efficiency

d confidence

e interest

CS – mean

3.09 3.06 2.98 2.80 2.59

Agree % 95.2 91.8 84.7 72.2 57.9

Disagree %

4.8 8.2 15.3 27.8 42.1

OW – mean

3.52 3.50 3.38 3.16 3.21

Agree % 98.3 100 98.1 86.0 81.0

Disagree %

1.7 0 10.9 14.0 19.0


Findings: Skills LearntQuestion 3:Please indicate the skills in which this class has helped you (you may check more than one item):

a. Choose the best databases for my information needb. Form better search strategiesc. Select more appropriate keywords to searchd. Export or save my results more effectivelye. Find more relevant internet resourcesf. Use printed resources more effectivelyg. Others


Findings: Skills Learnt Average no. of choices per respondent:

2.06 Top 3 choices:

Form better search strategies Find more relevant internet resources Select more appropriate keywords to

search


Findings: Skills Retention

CS OW All

Yes 233 39 272

No 107 18 125

Valid % 68.5% 68.4% 68.5%

Question 4: Have you continued to use the skills that you learned in this class?


Findings: Skills RetentionWritten Comments – "Yes"

Continued to use the skills learned to find information - general remarks 17 - specified "finding more relevant information" 5 - specified efficiency 5 - specified particular information sources 10 37

Used the skills to find better Internet resources or use search engines more effectively 11

Used the library catalog 5

Used databases 40

Applied the search skills learned 27

Used the skills learned to achieve particular goals (projects, assignments, etc.) 64

Used the subject guide learned in class 1


Findings: Skills RetentionWritten Comments – "No"

No chance or need to apply the skills 22

Forgot what they learned in the classes 8

No time 7

Preferred or were satisfied with using the Internet 3

Low interest in using the Library resources 4

Considered the class not useful 2


Findings: Overall RatingsQuestion 5 and 6:

5.What is your OVERALL rating of the class:6.What is your OVERALL rating of the instructor:

Very goodGoodAbove averageAverageBelow averageBadVery bad


Findings: Overall Ratings

Overall Class Rating

0.6% 0.6%

5.1%

25.2%

36.0%

27.8%

4.8%

0.0% 0.0% 0.0%

3.5%

15.8%

52.6%

28.1%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

Very Bad Bad Below Average Average Above Average Good Very Good

CS OW


Findings: Overall Ratings Overall Instructor Rating

0.0%0.9%

3.5%

25.6%

29.1%

31.7%

9.3%

0.0% 0.0% 0.0% 0.0%

21.1%

45.6%

33.3%

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

Very Bad Bad Below Average Average Above Average Good Very Good

CS OW


Findings: Overall Ratings

To benchmark with the grading scale in the University's credit course evaluation:

Very bad=0, Bad=16.7 Below average=33.3, Average=50, Above average=66.7 Good=83.3, Very good=100

Mean Scores CS OW All University average scores in the same semester

Class Score 66.34 84.21 68.82 70.2-75.1

Instructor Score

69.19 85.38 71.49 72.4-78.5


Findings: Suggestions for ImprovementQuestion 7:How could this class be improved? (you may check more than

one item) a. I already find the class satisfactory b. Shorter time c. Longer time d. Cover fewer topics, please specify e. Cover more topics, please specify f. Cover some topics in more detail, please specify g. Cover some topics in less detail, please specify h. Instructor should speak more slowly i. Instructor should speak more clearly j. Offer students more hands-on practice k. Have a smaller class l. Others, please specify


Findings: Suggestions for Improvement Average no. of choices per respondent: 1.48 Top Three Choices:

CS OW

Shorter time(31%)

Satisfactory(55%)

Satisfactory(27%)

Shorter time(20%)

Hands-on(22%)

Hands-on(18%)


Findings: Suggestions for ImprovementWritten Comments: Fewer topics: 2 commented on the flood of

information in the class More topics: 7 comments, 4 called for more

coverage on databases More details: 14 comments on various

issues: 4 about search skills, 2 called for more examples, 2 for databases

Others: 15 various comments, 6 asked for more and better handouts


Other Observations

Class size: Bigger classes gave lower class ratings,

instructor ratings and lower skill retention Information search confidence:

Respondents with higher confidence in information search rated higher in these three key indicators


What the Findings show Us It demonstrates positive value of the Library's

bibliographic program by yielding these figures: After a period of time, over 85% of users remain

affirmative that the class was useful Almost 70% of users retain the skills learnt The overall ratings are comparable to credit courses

in the University It tells us what users want:

Smaller class More hands-on Better class handouts


What we Learnt from the Survey There are significant differences between

the Open Workshop Group and the Course-Specific Group

Why users did not use the skills: did not see the need/did not have the chance forgot what they learnt

Users with higher confidence in information search are more receptive and appreciative of library instruction

Class size does matter


Where do We Go from Here? Nurture a culture of assessment at HKUST

Library Next assessment in 2006 Administer a comparable immediate

evaluation attempting to quantify the "delay" effect

Two separate schemes for CS and OW? Focus on the outcomes of one particular

class instead of a programmatic assessment? Consider testing attendees' library skills

instead of measuring perception?

Thank you!

Documents

Enhancing Quality of Library Instruction Programs through Delayed Assessment 6th Northumbria International Conference on Performance Measures in Libraries