Information Literacy AssessmentChallenges & Opportunities for One-shot Library Instruction
Presented by Diana Myers HyattOctober 23, 2012
Introduction
Examine assessment process andbest practices for assessing LI sessionsPlanning for learning and assessment, from the program to the session levelSpecific assessment tools suitable for sessions, their strengths and weaknessesSpecific examples of incorporating assessmentHow can we leverage limited resources for maximum learning impact?
ChallengesChallenges of assessing in one-shot instruction sessionsLimited class time. How do you effectively measure
learning in such a short time? Librarian staffing and workloadLimited librarian training in teaching and
assessmentLimited support from faculty/college administrationUCR trends? Budget cuts?
Why assess? Improve instructional programs and teaching
methods to better meet learning goals. Improve student learning. Gives them a chance to
review learning goals, see their progress and what they still need to learn.
Part of accountability, meeting department and institutional learning goals.
Demonstrate value of library with evidence.
What is assessment? Assessment to improve learning:
How well are students learning what we intend them to learn? Measurement of student learning based on established learning outcomes.
Continuous, cyclical process of incremental improvement and change
Assessment Primer: How to do assessment, University of Connecticut www.assessment.uconn.edu/primer/how1.html
Assessment Best Practices (ACRL)Program evaluation: Develops a process for program planning, evaluation, and revision. Measures the progress of meeting the program’s goals and objectives Integrates with course and curriculum assessment, institutional evaluations
and regional/professional accreditation initiatives. Uses appropriate assessment/evaluation method for relevant purposes, for
example formative and summative and/or short-term and longitudinal.
Student outcomes: Acknowledge differences in learning and teaching styles in the outcome
measures. Multiple methods: needs assessment, pre-tests, post-tests, portfolio
assessment, oral defense, quizzes, direct observation, peer and self review, and experience.
Focus on learner performance, knowledge acquisition, and attitude appraisal.
Assess the learners’ process and product. Include learner-, peer-, and self-evaluation.
Characteristics of Programs of Information Literacy that Illustrate Best Practices: A Guidelinehttp://www.ala.org/acrl/standards/characteristics
Assessment Best Practices
Assessment program should: be continuous: drive change and innovation in an
ongoing, iterative process, pinpointing areas for improvement and development
be learner-centered: focus on student learning rather than on us
measure what students are learning as accurately and directly as possible
learning goals should be flexible enough to be adaptable to specific courses, integrating current program goals (implicit and explicit) with ACRL standards
When possible, allow for immediate teaching improvement by informing teaching librarian of current student skill levels through assessment opportunities prior to the workshop
Strategic plan for learning and assessment Moving from the general level to the specific (session level) Library develops main learning goals integrating university
and library missions, ACRL standards and input from librarians and faculty
From these goals, articulate specific learning outcomes which can be assessed and measured
◦ target groups, scaffolding (e.g. first-year English, graduate art students)
◦ Delivery vehicles (LI sessions, online tutorials, embedded teaching)
Determine which specific learning outcomes are best addressed in LI sessions.
Mapping learning outcomes to specific courses
Oakleaf, M. (2009). WRITING INFORMATION LITERACY ASSESSMENT PLANS: A guide to best practice. Communications in Information Literacy, 3(2), 80-90.
Learning plan for LI sessionsLearning outcomes for specific LI sessions based on:Library-level goals for information literacyLearning goals of the specific course and
professor. Instructor collaboration, course syllabus and library curriculum map
Students’ current course assignments and IL mastery level. Assignment description, pre-session assessment (assignment or pre-test)
Assessment component of learning plan
A good plan gives teaching librarians time to focus on teaching and engaging students, because learning objectives and assessment tools for each workshop are clear.
Articulate what specific results will show that students have mastered a learning outcome.
How will assessment measures be gathered and reported? How will the gathered data be reported and shared?
Assessment plan is reviewed on a regular basis. New assessment methods should be tested before
using on a widespread basis.
Assessment planning for LI sessions Rubric-based assessment plan combining learning outcomes paired
with appropriate assessment tools and scoring. Done collaboratively within the library.
Learning outcome Assessment instrument
Acceptable performance criteria
Explores general information sources to increase familiarity with the topic
Learning activity: Find and cite one appropriate article from online subject encyclopedia
Appropriate source, reason given, proper citation format
Assessment resources developed (test banks, learning activities with worksheets, etc.)
How to measure success. What scores will be considered adequate?
Teaching librarian now has appropriate learning outcomes and ready-to-use assessment tools that can be easily adapted for specific LI sessions.
Types of assessment instruments
Technique Potential Strength Potential Weakness
Locally developed multiple choice tests
Can provide direct evidence of student mastery of learning objectives
Appropriate mixes of items allow faculty to address various types of learning objectives
Can be centrally developed and then adapted for specific workshops
Easily scored and tabulated
Reliability and validity generally are unknown
Creating effective exams requires time and skill
Traditional testing methods may not provide authentic measurement
Embedded assignment and activities
Can provide direct evidence of student mastery of learning objectives
Involves active learning, possibly more retention
Can provide authentic assessment of learning objectives
Data collection is unobtrusive to students
Requires time to develop and coordinate and score, need simple grading rubrics
Requires faculty trust that the program will be assessed, not individual teachers
Reliability and validity generally are unknown
Norms generally are not available
Summary of assessment techniques
University of Connecticut Assessment Primer, Assessment Primer, http://assessment.uconn.edu/primer/how1.html
Summary of assessment techniques
Technique Potential Strength Potential Weakness
Surveys, one-minute papers, 1-2-3 mode, Survey monkey, clicker technology, Poll Anywhere
Are flexible in format and can include questions about many issues
Can be administered to large groups of respondents
Can easily assess the views of various stakeholders
Questions generally have a clear relationship to the objectives being assessed
Can be conducted relatively quickly and easily
Responses to closed-ended questions are easy to tabulate and to report in tables or graphs
Open-ended questions allow faculty to uncover unanticipated results
Can be used to track opinions across time to explore trends
Polls can be used for quick pre-tests with instant feedback, encourage active participation
Provide indirect evidence about student learning
Validity depends on the quality of the questions and response options (avoid guessing)
What people say they do or know may be inconsistent with what they actually do or know
Open-ended responses can be difficult and time-consuming to analyze
Embedded performance-based assessment Our assessment methods should support problem-based
and other active learning scenarios rather than straight lectures.
Combining active learning activities with assessment Could include collaborative learning exercises, creating
3-item bibliography (encyclopedia article, book, journal article), other focused library activities with worksheets, concept maps or other concrete product that can be assessed -- better learning using constructivist methods.
Rubric for quantitative measurements. Or multiple choice post-test afterwards for easier assessment.
Trade-off: more complex learning, but harder to measure in class.
LI session evaluation by students3-2-1 model using Google
forms. More structured than one-minute paper.
Indirect measure, but can reveal important information. How well are responses aligned with our teaching goals?
Gives students a chance to reflect on and articulate what they’ve learned for better retention
Although open-ended questions are harder to compile for statistical purposes, they give better feedback to librarians to improve instruction.
Provides an opportunity for personal follow-up with students who want it by providing space for questions and email info.
Nardine, J. and Meier, C. (2010), "Assessing the one-shot instruction session: Leveraging technology for optimum results", Virginia Libraries, Vol. 56 No. 3, pp. 25-8
Assessment outside workshops
Library/Faculty collaboration on class assignments integrating IL learning outcomes.
Target key classes and assignments such as English composition, introductory classes. ◦ bibliographies, annotated bibliographies◦ research blogs/journals on their research
experience - very qualitative but could be eye-opening.
◦ shared assignments with linked learning outcomes◦ students complete online IL module with tutorials,
activities and assessments.
Evaluating our assessment methods
Is the method accurately measuring the learning outcomes it’s supposed to?
Is it valuable? Does it improve our teaching?
Does it put reasonable demands on our staff and resources for the value it brings?
Comparison of multiple forms of evidence for the same learning outcome (including class assignments)
Assessment method examplesACRL standard 1.1: The information literate student defines
and articulates the need for information. Small group activity: hypothetical information need situation
(buying a car, taking a trip), articulate the information need and specifics of what information is needed, also brainstorm possible sources (friends, consumer reports, review or travel websites, travel guides). Assessment: state general information need, 3 questions that need to be answered, 3 possible sources in Google Docs
Multiple choice post-test using Google form: what is the most important first step in library research? ◦ What is the most important first step in library research?
a) identify key authors on research problem.b) know research problemc) Locate key articles on research problemd) Locate key books on research problem
Give them a broad topic such as “commuters and Internet crime”. Have them come up with a good research question and identify keywords for first stage of research. Could use Google spreadsheet or Google form. Could use concept map or Wordle for keyword brainstorming.
Sample use of Google Docs for learning activity
Shannon R. Simpson, Google Spreadsheets and real-time assessment: Instant feedback for library instruction, Coll. res. libr. news October 2012 73:528-549.
Standard 1.1c Explores general information sources to increase familiarity with the topic.Performance based exercise: Given a topic by librarian, find an appropriate encyclopedia article that gives background information on aspect of the topic and create citation. Answers cut and pasted into Google spreadsheet or fill out Google Docs form.
Reliability and Validity of Assessment Methods
Reliability Consistent responses over time. Variation in scores is actual difference
in what is being measured, such as knowledge Students have adequate time to take assessment, guessing is
minimized Consistent methods for all sessions (e.g. same post-test questions).
Importance of collaboration and buy-in in developing assessment methods.
Must have consistent scoring or rating scale, so easy to score and guidelines for scoring are clear
Validity The assessment tool measures what we want to measure? Do our results correlate with other proven measures that assess the
same learning outcome? Does our method accurately predict future student performance?
Conclusion
No one best way to assess student learning. Use a variety of methods, direct and indirect.
Trade-offs between more thorough assessments and time needed to do them. Have to prioritize learning outcomes and make sure they are realistic given time and staff constraints.
Developing a “learning bank” of active learning activities and specific assessment tools, such as a test bank, can give teaching librarians more time to focus on improving their teaching methods.
Program-level learning plan helps us see the big picture. Learning objectives in one LI session are scaffolded with other sessions. Complement other learning vehicles such as class assignments, tours.
Use assessment methods that involve a learning activity. Learning and assessment together.
“Good teaching is inseparable from good assessing. The question, therefore, is not whether to evaluate students, but how to measure performance in ways that will enrich learning, rather than restrict it.”
Ernest L. Boyer, The Basic School: A Community for Learning (New York: The Carnegie Foundation for the Advancement of Teaching, 1995), 104.