Phase I: Overview

A Standardized Approach to Assessment:A Standardized Approach to Assessment:

Results from a Pilot Study of Library Instruction Sessions at San José State University Results from a Pilot Study of Library Instruction Sessions at San José State University

Shannon M. StaleySocial Sciences LibrarianShannon M. StaleySocial Sciences Librarian

Purpose of Study

To Develop and Test an assessment tool that:

• Facilitates the development of pre and post achievement tests

• Links survey questions to learning outcomes and ACRL standards

• Delivers real time reports and statistical analysis

Timeline:An Overview

• Review of the Literature – Summer 2008

• Create assessment tool – Fall 2008

• Garner input from library faculty / Refine tool - 12/08

• Develop multiple choice questions – 12/08

• Standardize questions – 1/09

• Pilot test assessment tool – 1/09

• Enact actual study – Spring 2009

Methodology:Develop Multiple-Choice Questions

• Investigated questions from already existing tutorials / survey instruments for ideas

• Created 20 multiple-choice questions based on social sciences 100W learning outcomes

• Each question presented four answers, with only one correct answer

• Each question had an added option for “not sure”

Methodology:Standardization of Survey Multiple Choice Questions

• Multiple choice questions were emphasized to allow for immediate computation of results

• The co-occurrence of LibQual during Spring 2009 provides for complimentary qualitative data

• Questions were reviewed by library faculty to ensure they reflected learning outcomes / ACRL standards

• Questions and learning outcomes were emailed to all teaching faculty in the Psychology department for review

• Questions were reviewed by experts at the Center for Assessment under the Office of Institutional Research

• Cognitive interviews with 5 students further addressed the clarity of survey questions

Methodology:Pilot Study

Conducted with students in a PSYC 139 course to

determine:

• Average length of time to complete surveys

• Overall impact on what could be covered in a single instruction session

• Any technical glitches with survey interface, login, usability etc.

• If survey questions were clear and comprehensible to students


Conducted with students in PSYC 139 to

determine:

• Average length of time to complete surveys - 5 minutes (10 minutes total for pre and post survey completion)

• Overall impact on what could be covered in a single instruction session





determine:


• Overall impact on what could be covered in a single instruction session - no major impact with tighter planning



sstaley

Tighter planning involved explaining to the instructor ahead of time to encourage students to be prompt. Starting the session on time with those in attendance. Making introductions, providing contact information on the white board prior to the nitty gritty - this allowed those finishing up their pre survey to not miss out on anything important. Provided students with a course outline and handouts after they completed the post survey. Wanted them to rely on their own recall of what occurred during the session.



determine:



• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos




determine:



• Any technical glitches with survey interface, login, usability etc. - no major problems. Logins for the post test could not contain typos

• If survey questions were clear and comprehensible to students – no confusion was reported

Methodology:Planning and Instruction

• Coordinated with instructor beforehand in encouraging students to show up early

• Scripted instructions were read to students prior to taking the pre test survey

• Lecture with embedded exercises; last 15 minutes reserved for getting started on course assignments

• Course outline and handouts were distributed to students after completion of their post test surveys

Methodology:Actual Study – Spring 2009

• Conducted with students in 5 PSYC 100W courses (83 students total)

• Survey participation was voluntary

• Participants who completed the pre-test survey but failed to complete the post-test survey were automatically eliminated from the data pool

• 11 multiple-choice survey questions covering 3 ACRL standards and 8 learning objectives

• 6 background questions on the pre-test survey – they were not repeated on the post-test

Methodology:Pre-Test Survey Background Questions

Methodology:PSYC 100W Multiple-Choice Survey Questions

Data AnalysisA Bird’s Eye View



Data AnalysisStatistical Significance

• Paired TTEST of student mean scores

• Excel formula computes statistical significance in the report generation feature of the assessment tool

Data AnalysisAssessment Tool Sample Report

Data AnalysisFuture Enhancements

• Master spreadsheet that incorporates all reports

• More granular TTEST analysis of individual questions

• Statistical analysis of how background factors (class level, research experience, instruction experience) affect scores

Benefits

• Assessment data will inform pedagogical directions and enhance the educational impact on SJSU students

• Data can be shared with department faculty and campus administrators to demonstrate instructional efficacy

• Data can also be used to bolster documentation during the accreditation process

• Librarians can publish findings from their own sessions, particularly with a more rigorous statistical model

• Librarians can include assessment reports in their

dossiers

Technology

Phase I: Overview