Rutgers University Libraries Page 1

Learning Assessment Report, 2011/12

Introduction

More than 20,000 Rutgers students participate in the 1,000 library instruction classes taught by library

faculty each year. Over half of these classes are for undergraduate students. The Rutgers University

Libraries support for student learning also includes reference desk services, email and chat reference,

one-on-one research consultations, web tutorials, LibGuides developed for specific courses, online

videos, a “How do I…?” web series, and a user focused website. Rutgers faculty can access a variety of

instructional support services and materials from the Libraries website, including: consultation for

developing library research components for classes or library workshops; lists of instruction coordinators

and subject specialist librarians available for teaching; access to the RUL Instruction Repository (Sakai

site) collection of reusable library handouts, tutorials, resources, customized class guides, etc.; samples

of reusable library research assignments; and various other learning tools.

The Libraries do not yet have a system wide learning assessment plan. This report shares an existing

framework and outlines how we will develop our plan during the coming academic year. We are well-

positioned to develop a plan because we have recently appointed a new associate university librarian

for research and instructional services and completed a study of information literacy that featured

interviews with teaching faculty partners. The university call for action, embodied by the requirement

that we report, and the learning assessment work our academic colleagues are doing brings welcome

attention to our Middle States recommendation for a university wide information literacy assessment

plan.

Learning Goals

The libraries ascribe to the information literacy competency standards developed by the Association of

College and Research Libraries, the premier division of the American Library Association for libraries

serving higher education.1 The standards are supported by a number of higher education organizations,

including the Middle States Commission on Higher Education, and discipline specific standards have

been approved by a select number of professional associations.2 The five ACRL standards specify that

the information literate student:

1. Determines the nature and extent of the information needed.

2. Accesses needed information effectively and efficiently.

1 Information Literacy Competency Standards for Higher Education (ACRL, 2000). Viewed

4/30/12,http://www.ala.org/acrl/standards/informationliteracycompetency#stan. 2 Discipline Specific Information Literacy Standards (ACRL, various). Viewed 5/22/12,

http://www.ala.org/acrl/issues/infolit/resources/inaction/disciplines.

Rutgers University Libraries Page 2

3. Evaluates information and its sources critically and incorporates selected information into his or

her knowledge base and value system.

4. Uses information effectively, individually or as a member of a group, to accomplish a specific

purpose.

5. Understands many of the economic, legal, and social issues surrounding the use of information

and accesses and uses information ethically and legally.

The Rutgers University Libraries Information Literacy Learning Goals and Outcome Expectations were

developed during the 2008 Middle States accreditation and are included in Section VI of the self study as

Table 6.9 and as Table 1 below. They are also posted to the Libraries website at:

http://www.libraries.rutgers.edu/rul/lib_instruct/RUL_Information_Literacy_Goals_and_Outcome_Expe

ctations.pdf.

Rutgers University Libraries Page 3

Table 1. Rutgers University Libraries Information Literacy Goals and Outcome Expectations*

General Goal Department Goals and Objectives

Students will determine the nature and extent of the information needed.

Define and articulate the need for information.

Identify key concepts and terms that describe the information need.

Identify a variety of types and formats of potential sources for information.

Know how information is formally and informally produced, organized and disseminated.

Recognize that knowledge can be organized into disciplines that influence the way information is accessed.

Identify the value and differences of potential resources in a variety of formats (e.g., multimedia, database, website, data set, audio/visual, books).

Reevaluate the nature and extent of the information need.

Review the initial information need to clarify, revise, or refine the question.

Students will access needed information effectively and efficiently.

Select the most appropriate investigative methods or information retrieval systems for accessing the needed information.

Identify appropriate investigative methods (e.g. laboratory experiment, simulation, fieldwork).

Construct and implement effectively-designed search strategies.

Develop a research plan appropriate to the investigative method.

Identify keywords, synonyms and related terms for the information needed.

Select controlled vocabulary specific to the discipline or information retrieval source.

Construct a search strategy using appropriate commands for the information retrieval system selected (e.g., Boolean operators, truncations, and proximity for search engines; internal organizers such as indexes for books).

Retrieve information online or in person using a variety of methods.

Use specialized online or in person services available at the institution to retrieve information needed (e.g., interlibrary loan/document delivery, professional associations, institutional research offices, communication resources, experts and practitioners).

Refine the search strategy if necessary.

Rutgers University Libraries Page 4

*From the Middle States Reaccreditation (2008) self-study report, “Educational Change at Rutgers: An Institutional Self-Study” (http://middlestates.rutgers.edu/report_final.shtml), Section VI: Assessment of Student Learning (http://middlestates.rutgers.edu/pdf/sectionVI.pdf), pages 38-39.

Assess the quantity, quality, and relevance of the search result to determine whether alternative information retrieval systems or investigative methods should be utilized.

Identify gaps in the information retrieved and determine if the search strategy should be revised.

Repeat the search using the revised strategy as necessary.

Extract, record, and manage the information and its sources.

Differentiate among the types of sources cited and understand elements and correct syntax of a citation for a wide range of resources.

Record all pertinent citation information for future reference.

Students will evaluate information and its sources critically and incorporate selected information into their knowledge base and value system.

Articulate and apply initial criteria for evaluating both the information and its sources.

Determine whether the initial query should be revised.

Determine if original information need has been satisfied or if additional information is needed.

Review search strategy and incorporate additional concepts as necessary.

Review information retrieval sources used and expand to include others as needed.

Students will, individually or as members of a group, use information effectively to accomplish a specific purpose.

[This standard is primarily the province of the teaching faculty. Librarians can assist with this effort, but not address it independently with existing programs.]

Students will understand many of the economic, legal, and social issues surrounding the use of information and will access and use information ethically and legally.

Understand many of the ethical, legal and socio-economic issues surrounding information and information technology.

Demonstrate an understanding of intellectual property, copyright, and fair use of copyrighted material.

Acknowledge the use of information sources in communicating the product or performance.

Select an appropriate documentation style and use it consistently to cite sources.

Rutgers University Libraries Page 5

The Libraries goals and outcome expectations align with the hierarchy of mission statements and

learning goals at the university as follows:

Table 2. Alignment of Mission

University Libraries

As the sole comprehensive public research university in the state’s system of higher education, Rutgers, The State University of New Jersey, has the threefold mission of providing for the instructional needs of New Jersey’s citizens through its undergraduate, graduate, and continuing education programs; conducting the cutting-edge research that contributes to the medical, environmental, social and cultural well-being of the state, as well as aiding the economy and the state’s businesses and industries; and performing public service in support of the needs of the citizens of the state and its local, county, and state governments.

The Rutgers University Libraries support and enrich the instructional, research, and public service missions of the University through the stewardship of scholarly information and the delivery of information services.

Table 3. Alignment of Learning Goals

Rutgers University Libraries

Determine the nature and extent of the information needed

Access needed information effectively and efficiently

Evaluate information and its sources critically and incorporate selected information into their knowledge base and value system

Individually, or as members of a group, use information effectively to accomplish a specific task

Understand many of the economic, legal, and social issues surrounding the use of information and accesses and uses information ethically and legally

University Level

Students will develop their skills in gathering, accessing, analyzing, and interpreting information, in part through using the tools of modem computer technology

Decanal Level Example

SAS-NB

Evaluate and critically assess sources and use the conventions of attribution and citation correctly

Employ current technologies to access information, to conduct research, and to communicate findings

Analyze and critically assess information from traditional and emergent technologies

Department Level Examples

Art History, SAS-NB

Majors and Minors in Art History will be able to recognize and understand the fundamental interpretive methods engaged by art historians and employ the appropriate technologies for conducting research in the history of art, including print and electronic resources

Chemistry and Chemical Biology, SAS-NB

…Students at the upper level apply what they have learned to problems that require the evaluation of the scientific literature and the design of studies to test hypotheses

Rutgers University Libraries Page 6

Course Level Examples

Expository Writing 01:355:101 (SAS-NB)

Evaluate and critically assess sources and use the conventions of attribution and citation correctly

Analyze and synthesize information and ideas from multiple sources to generate new insights

English Composition 102, NCAS/UC-Newark

…Students are also asked to produce writing that draws effectively upon research o Evaluate the usefulness of secondary sources o Learn to evaluate Internet resources receptively but skeptically, alert to issues of

responsibility, authority, and documentation o Participate in a librarian’s tour of the Rutgers Library system, and produce at least one piece

of writing that draws upon Rutgers Library resources o Consider both the ethical and rhetorical dimensions of plagiarism o Learn to critically evaluate Internet resources

Composition 102, Writing Program at Rutgers-Camden

Construct an annotated bibliography of secondary sources

Evaluate secondary sources and differentiate between reliable and unreliable sources

Locate and use sources in the library independently

Included in Course Syllabi/Synopsis/Expanded Description

Some library faculty members include learning goals or objectives with their instruction materials.

Examples of learning goals included in course-specific LibGuides are:

English 102 (Newark)/ Lorraine Hansberry’s Raisin in the Sun: http://libguides.rutgers.edu/raisininthesun

Earth & Environmental Sciences Graduate Students Retreat 2012: http://libguides.rutgers.edu/EESretreat2012 Latino Studies: http://libguides.rutgers.edu/latino

Library Research for Graduate Students in Nursing: http://libguides.rutgers.edu/nursing_grad

Where or How the Learning Goals are Met

Our learning goals are met in the classroom, in sessions taught by library or teaching faculty, as well as

through the various online and support services described in the introduction to this report.

Assessment Plan, Structure, and Process

The Plan

We will begin to develop our assessment plan during the summer and start adding assessment activities

into our instruction in the fall semester. It will be based on our learning goals and collaboratively

developed and supported rubrics. All library faculty members will ascribe to the goals and select rubrics

appropriate to each learning activity in which they participate. For individual sessions, they will work

with teaching faculty to select appropriate measures; for common library activities, such as a general

Rutgers University Libraries Page 7

tutorial, the group developing the activity will select appropriate measures. A call for outcomes and

responses will precede filing of the annual report to the Executive Council on Assessment.

Structure

The Libraries instruction program is led by the associate university librarian for research and

instructional services (aul/ris). Dr. Melissa Just was recently appointed to this position and will be

leading our information literacy initiatives.

Overall assessment initiatives are coordinated and supported by an office for planning and

organizational research consisting of an associate university librarian (aul/por) and a project

coordinator.

Other individuals and groups in the Libraries share an interest in learning assessment, including the user

services council, campus instruction coordinators, and the library faculty planning and coordinating

committee. In addition, individual library faculty members have instructional responsibilities in specific

disciplinary areas, collaborate on teaching large introductory courses, especially for the various writing

programs, and may conduct their own research.

Process

This current learning assessment plan proposal was developed at a meeting of the aul/por and campus

instruction coordinators and has been reviewed by the aul/ris and the user services council. It will be

shared with all library faculty members by email and posted to our internal website.

Library faculty who teach will participate in development and implementation of the assessment plan.

Oversight and assistance will be provided by the user services council and the associate university

librarians for research and instructional services and planning and organizational research. Groundwork

has already been done by an information literacy planning task force charged by the planning and

coordinating committee. The task force report3 summarizes our current effort and next steps, which

include assessment.

The Tools/Measures

Librarians develop library instruction sessions in collaboration with teaching faculty, and student success

is incorporated into overall coursework and measured by grades assigned by and other tools/measures

selected by teaching faculty. As a result, library faculty members have not adopted a uniform approach

to learning assessment. We propose adoption of the following elements of a learning assessment plan,

to begin during the 2013 academic year:

Because Middle States has recognized that the integrated approach for information literacy is

desirable,4 librarians will present faculty with our information literacy learning goals when

3 Information Literacy Planning Task Force Report (Rutgers University Libraries, 2010). Viewed June 4, 2012.

http://www.libraries.rutgers.edu/rul/staff/groups/fac_coordinating_com/reports/info_lit_planning.pdf. 4 “…information literacy should be integrated into the general education core and students’ major fields of study,

and its relationships across the curriculum should be transparent to the student,” Developing Research &

Rutgers University Libraries Page 8

collaborating on teaching sessions and regularly request that they incorporate the learning goals

into their teaching and include a relevant measure in their assessment.

Librarians will use our information literacy learning goals to develop statements of learning

expectations for their overall teaching areas or for individual classes, as appropriate, and share

them with teaching faculty and students.

Occasionally students complete a library research assignment that the librarian is able to grade,

or at least review, and librarians will regularly seek this opportunity. Samples are provided on

this web page: http://www.libraries.rutgers.edu/rul/lib_instruct/samples.shtml. Direct

measures may include citation analysis.

Librarians will regularly and formally ask teaching faculty for feedback.

The Libraries will also review results from SERU, especially those questions having to do with

library research skills; regularly review quantitative and qualitative data from the Libraries

online Customer Satisfaction Survey; and may conduct targeted email surveys of teaching

faculty, interviews, or focus groups.

The Libraries will consider again the possibility of administering the ETS iSkills™ Assessment or

the Standardized Assessment of Information Literacy Skills (SAILS). Based on past experience,

however, we are not at all hopeful that a sufficient number of survey participants would take

part without a university mandate. The final report about our 2005 participation in Project SAILS

is available on the Libraries website:

http://www.libraries.rutgers.edu/rul/staff/pub_serv/reports/SAILS_final_report.pdf. The

primary finding was that “The average student at Rutgers University performed on all

standards at about the same level as the average student from all institutions combined.”5

This version of the instrument we used was Phase III of the IMLS-funded development

project.

A better approach to a standardized review with benchmarking would be university

implementation of the information literacy module under development for the National

Survey of Student Engagement (NSSE). The module will be beta tested in 2013 and become

available as an optional module in 2014. Using the module biennially would be extremely

beneficial to the Libraries and the students we teach as well as teaching faculty setting

program or course goals and standards.

Communication Skills: Guidelines for Information Literacy in the Curriculum (MSCHE, 2003), p. 18. Viewed 4/9/12,

http://www.msche.org/publications/Developing-Skills080111151714.pdf.

5 Project SAILS Final Report (Rutgers University Libraries, 2005). Page 3. Viewed June 4, 2012.

http://www.libraries.rutgers.edu/rul/staff/pub_serv/reports/SAILS_final_report.pdf.

Rutgers University Libraries Page 9

Benchmarks/Standards

Standards articulated by the Association of College and Research Libraries (ACRL) are the foundation for

our learning goals. We are fortunate that the ACRL standards align with those set by Middle States.

Table 4. Standards Comparison

Middle States ACRL

Determine the nature and extent of needed information

Determines the nature and extent of the information needed.

Access information effectively and efficiently Accesses needed information effectively and efficiently

Evaluate critically the sources and content of information

Evaluates information and its sources critically and incorporates selected information into his or her knowledge base and value system

Incorporate selected information in the learner’s knowledge base and value system

Use information effectively to accomplish a specific purpose

Uses information effectively, individually or as a member of a group, to accomplish a specific purpose

Understand the economic, legal and social issues surrounding the use of information and information technology

Understands many of the economic, legal, and social issues surrounding the use of information and accesses and uses information ethically and legally

Observe laws, regulations, and institutional policies related to the access and use of information

Benchmarking with other institutions will be possible using the information literacy module being

developed for the National Survey of Student Engagement (NSSE), as described in the preceding section.

Internal benchmarks will flow naturally from our Department Goals and Objectives (see Table 1), and rubrics will be set in the 2012/13 academic year. We will begin our deliberations with Information Literacy Progression Standards for Use in New Jersey College and Universities,6 developed collaboratively by leading statewide library associations and endorsed by the NJ State College Council of Academic Vice Presidents and provosts at the senior public colleges and universities. We will also consider the Information Literacy VALUE Rubric,7 developed by the Association of American Colleges and Universities and review rubrics from other institutions available from the IMLS-funded RAILS (Rubric Assessment of Information Literacy Skills) project.8

For classroom teaching, our success in this area will depend on close collaboration with teaching faculty.

While we are committed to integrating information literacy into the overall curriculum, we will also

determine how we can establish measures for other modes of information literacy delivery, such as our

tutorials.

6 Progression Standards for Information Literacy (NJLA, VALE, CJRLC, 2009). Viewed 4/30/12,

http://njla.pbworks.com/w/page/12189896/Progression%20Standards%20for%20Information%20Literacy. 7 Information Literacy VALUE Rubric (AAC&U, 2009?). Viewed 6/1/12,

http://assessment.aas.duke.edu/documents/InformationLiteracy.pdf. 8 RAILS: Rubric Assessment of Information Literacy Skills. Viewed 6/4/2012. http://railsontrack.info/.

Rutgers University Libraries Page 10

Assessment Implementation and Results

Assessment results will not be available until 2012/13. However, some highlights of current assessment

activities are available.

Study

Following 2008 Middle States recommendations about information literacy, an Information Literacy

Implementation Advisory Committee (ILIAC) of library faculty members was charged to develop a

greater role for library faculty responsiveness to student needs for information literacy. The group has

examined our instruction program and interviewed teaching faculty whose classes have information

literacy as a learning goal and who are receiving basic instruction from a library faculty member. A draft

of the ILIAC report has been submitted to and endorsed by the aul/ris and the User Services Council. Dr.

Just will next convene a task force to look at how we should move forward with instruction as a library

system. ILIAC members Interviewed 22 teaching faculty members “found both predictable variations in

approach based upon discipline and widespread agreement about the skills and information students

must have—and sometimes lack--in order to be competent researchers and communicators of

research.” The interviews also resulted in “many suggestions for ways in which we can improve our

program and ways in which we can partner with [teaching faculty].” A final report is expected shortly.

MAAP

The Libraries are participating as a pilot unit in the Mission Assessment, Alignment, and Planning effort.

Brent Ruben and Susan Lawrence met with the University Librarian’s Cabinet, and Kathleen Immordino

is providing research advice to the aul/por and project coordinator. We have mined data from our

online Customer Satisfaction Survey, SERU, and other internal data sources for undergraduate

indicators of satisfaction, importance, and use and have posted charts to our website at:

http://www.libraries.rutgers.edu/rul/staff/planning/maap/maap.shtml. We plan to roll this information

out more aggressively with a news story and placement on our website.

Surveys

SERU

Results from the 2011 SERU survey show that 71% of students study in a library

sometimes/frequently/usually or always. 38% study in one of the library facilities frequently/usually or

always. Students’ opinions of their library research skills when they started at Rutgers and when they

took the survey show improvement for all skill ratings: excellent (13.6% from 6.0%), very good (31.2%

from 13.3%), good (33.2% from 32.2, fair (16.8% from 32.9%), poor (4.0% from 12.8%), or very poor

(1.2% from 2.9%). 91.2% of students were somewhat satisfied/satisfied/very satisfied with the

accessibility of library materials, while 47.1% were satisfied/very satisfied. Having access to a world class

library collection was rated as very important/essential by 35.2% of students, while 88.6% rated such

access as somewhat important/important/very important/essential. 90.7% of students were somewhat

satisfied/satisfied/very satisfied with accessibility of library staff, while 47.0% were satisfied/very

satisfied.

The OIRAP SERU analysis from 2009 provides benchmarking with other participating institutions. Rutgers

students rated their library research skills as better than students at most other institutions. Of the 6

Rutgers University Libraries Page 11

institutions participating, Rutgers ranked 2nd in the very good/excellent current ability level for library

research skills. Furthermore, progress is shown with 15.0% judging that they had very good/excellent

library research skills when they started at Rutgers rising to 42.8% at the time of the survey. This percent

is the 2nd highest in the comparison group of 6. Likewise, those judging that they had poor/very poor

skills dropped from 17.3% to 5.3%, also the 2nd highest in the group. 57.4% of students were

satisfied/very satisfied with accessibility of library staff, a rank of 5; while 61.7% were satisfied/very

satisfied with availability of library materials, a rank of 6.

The 2009 analysis also included analysis by school and discipline. Having that analysis for later years

would be helpful since we could distribute it for further analysis and action to individual library faculty

according to their discipline liaison responsibilities.

Customer Satisfaction Survey

Our online Customer Satisfaction Survey, which we opened October 1, 2009, is our primary assessment

tool. The survey asks two questions having to do with instruction and rates performance and

importance on a seven point scale. Results from July 1, 2011 through June 7, 2012 are:

Performance Importance

I have used the following Library services in the past year… (choose all that apply)

Attended instructional sessions 5.68 4.96

My satisfaction with and the importance to me of the following services provided by this Library…

Instructional Services 4.93 4.77

Note that the opportunity to rate satisfaction and importance is offered only to survey respondents who

indicate that they used the service.

For undergraduate students, satisfaction and importance receive substantially different scores for the

following services: collections, online catalog, and electronic resources, indicating that these areas need

work. In each area, importance outweighs satisfaction. Services where satisfaction and importance are

ranked closely are interlibrary loan, remote access, and website.

Graduate students rank all services as more important than undergraduates, and importance always

outranks satisfaction. The difference between satisfaction and importance is larger for collections,

online catalog, electronic resources, and interlibrary loan than for remote access and website.

52.9% of graduate students and 48.4% of undergraduate students agree/strongly agree that locating

information is simple and easy, while 62.4% of graduate students and 56.6% of undergraduate students

agree/strongly agree that they are often successful in finding the information they are seeking. When

somewhat agree is added, the percentages rise to 83.7%, 78.6%, 86.4%, and 84.1%, respectively.

Both the in-depth and regular versions of the survey offer many opportunities for respondents to give

comments. Various groups and individuals in the Libraries have examined satisfaction/importance and

comment reports, which are regularly posted to a Sakai site that anyone in the Libraries can join. To

Rutgers University Libraries Page 12

date, no further examination of instruction has been undertaken, although that will change with new

initiatives that will be undertaken by Dr. Just and as a result of the ILIAC report.

Individual Library Faculty Work

To prepare this report, we issued a call for individual work in learning assessment and received three

responses out of a teaching corps of approximately 50 library faculty members. One of these responses

represents a substantial body of work that has resulted in direct involvement with student learning on

both the graduate and undergraduate levels through a close partnership with teaching faculty,

embedding in courses, and changes in response to assessment results. See Appendix A.

Response to Assessment Results: “Closing the Loop” Activities

For this section of the report, we issued a call for individual work in “closing the loop” activities and

received three responses in addition to the work reported in Appendix A. One librarian reported using

clickers during instruction to assess the understanding of all of the students and not just the vocal ones.

As a result, she started teaching additional skills during classes, including how to read citations. This

same faculty member reported that after bringing back a group of students later in the semester to

assess their progress, she realized they were not comfortable using databases that were not

demonstrated during the instruction session. In later classes, she started demonstrating more databases

and made sure each student was provided with a subject specific database based on their research

topic. Another library faculty member reported that students in an online course were not reading the

set of expectations he had written for them. They were included in the eCollege template as well as on

words about the final paper, the reading list, and the splash page of every unit. The faculty member was

criticized for not communicating and now asks students to agree to an affirmative statement that they

have read and agree with the expectations. The third faculty member reports using one-minute papers

at each class session to inform what would be taught at the next session. She and a colleague are

planning to do an assessment on an exercise they have proposed for the fall 2012 Business Forum

classes in Newark. The proposal has been accepted by the Business School, and more than 1,000

students will participate.

As long as we are primarily doing level one assessment (reaction), the best we can do is alter the course

content to address stated deficiencies. Dr. Just plans to move us to level two assessment where we are

able to measure learning and change our instruction based on actual deficiencies.

Successful Improvement

The development of an instruction program across the library system will allow us to report on and

make improvement. The small amount of individual work that includes improvement is indicated in the

“closing the loop” paragraph above. We accept the requirement to make successful improvement as a

challenge that we will address this coming year.

Maintenance/Updating Process

The plan we will develop this coming year will include a maintenance/updating process. Some upcoming

triggers that will provide input for us to update our assessment process are the instruction task force

that Dr. Just will appoint, the library strategic plan, and the new president’s promised strategic plan.

Rutgers University Libraries Page 13

Conclusion

Clearly the Libraries have much work to do. We need a coherent instruction program that supports and

is responsive to student and faculty learning goals, and we need the means to continually assess and

improve. We have the foundations in place and look forward to reporting great successes in 2013.

Contact: Jeanne Boyle Associate University Librarian for Planning and Organizational Research Rutgers University Libraries

jeboyle@rulmail.rutgers.edu