2003 Annual NSDL PI SurveyA Collaborative Formative

Evaluation

Purpose of Presentation

Describe the community-supported formative evaluation efforts to evaluate the NSDL’s distributed library and community organizational processes

Elicit feedback on the results and recommendations from the 2003 Annual NSDL PI Survey

Timeline of the Evaluation Effort

Dec. 2001: Evaluation Working Group Meeting articulated four questions to be addressed in a pilot study

February 2002: Evaluation Workshop developed a work plan to answer the four questions. An online survey would be used to identify: How are the distributed library building and community governance processes working?

Feb.-July 2002: Survey developed and delivered. Preliminary survey data reported at JCDL.

Dec. 2002: Final survey data presented to the community during the EIEC meeting. Committee agreed that an NSDL-wide implementation of the survey should be undertaken

Apr.-June 2003: Survey revised, tested for validity and usability June-July 2003: Survey delivered Oct. 2003: Survey analyzed and presented at NSDL Annual

Meeting

Survey Details

Primary Question: How are the distributed library building and community governance processes working?

Developed measures in four areas to answer this question: NSDL communications infrastructure Extent and success of NSDL projects’ collaboration Projects’ participation level in NSDL organizational structures Information needs and motivations factors for NSDL community Note: The questions and scales have evolved with each iteration,

but the four areas being evaluated have remained the same. Questions sought to identify:

Levels of use, effectiveness, and extent of participation Information needs and motivations Barriers to use, effectiveness, participation and suggestions for

improvement Types of scales: Likert, multiple choice, open-ended

Survey Details (continued)

Primary audience: NSDL PIs (though all project members were encouraged to contribute to survey completion)

Distribution: survey available online four weeks during June-July 2003; notification and reminders sent to All-Projects and PI listservs

Verification: one survey per project was included in final analysis; used NSF award number to verify results, then disassociated the award number from further analysis

Examples of survey questions->

Summary of Key Findings from the Four Areas

1. Communication Infrastructure: The NSDL Annual Meeting is still the best place for projects to establish communication and identify potential collaborators. Other communication tools are perceived as not supporting projects accomplishing their goals.

2. Collaboration: 80% of respondents indicated that they had collaborated with a median of 2 other NSDL projects since the beginning of 2003. 58% of respondents incorporated something they learned through a collaborative relationship into their project’s design and use. Respondents overwhelmingly identified lack of time and being unclear about other projects’ goals and activities as barriers to further collaboration.

2. Organizational Structures: A majority of the respondents had not interacted with NSDL organizational structures since the beginning of 2003. 36% of respondents had a mid-level rating on the overall perceived effectiveness of NSDL groups. “I'm sure all this activity is necessary but it hasn't trickled down to the individual projects.”

4.a. Information Needs: A majority of all respondents were highly aware of the standards and technologies needed to integrate projects into NSDL. 62% of respondents indicated that they had implemented some part of their project’s evaluation plan in the past year. 100% of respondents reported that they had disseminated information about their projects through an average of 8 publications or presentation from June 2002 to June 2003.

4.b. Motivation: 85% of respondents indicated that it was important to participate in NSDL communication, organizational groups and library building activities because they wanted to help establish a National Science Digital Library

Results, Recommendations & Questions

Individual projects are not aware of whose expectations they should meet, what level of participation is expected or what the outcomes of participation should be when using the NSDL communications infrastructure, pursing collaborative opportunities and participating in NSDL organizational structures. Recommendation: Establish a process for defining and

communicating shared priorities and expectations. Recommendation: Provide targeted “entry points” for project or

individual participation The motivation to expend extra time and resources on

participation, collaboration and communication is often redirected by deadlines imposed by a two-year funding cycle. Recommendation: Identify tangible benefits for projects to

participate, aside from building NSDL. Question: Is it necessary that every project participate in order for

NSDL to be built?

Results, Recommendations & Questions

Respondents identified what type of support they would like from the Core Integration; however, this could be a moving target because projects are at different stages of development. Recommendation: Identify specifically what projects need in

terms of technical support. Recommendation: Define and communicate what levels of

support can be realistically expected from the Core Integration team.

Recommendation: Leverage the expertise of NSDL projects. Respondents indicated that they were unsure about how

collaboration was being defined and what were the expected results of collaborative efforts. Recommendation: Establish a process to define and measure

collaboration in a distributed environment. Question: How much emphasis should be placed on

collaborative achievements as factor in the success of NSDL?

Results, Recommendations & Questions

Projects are not sure how to identify others working on similar topics, outside the NSDL Annual Meeting. They expressed the need for a “human” connector to point them to the right collaborator; and, the need for direction about how to be involved in organizational structures. Question: How could the current NSDL organizational

structure adapt to meet the demand for connections on a more granular level than now exists?

The current communications infrastructure is not meeting projects’ information needs and the multitude of communications channels present too many choices when deciding what to read regularly and where to look for information to support project work. Recommendation: Establish one resource that is

continually maintained and updated with basic technical information and other information relevant to project development.

Design & Delivery Issues

Audience: Pilot - everyone working on a project – result was conflicting responses from

one project. 2003 - one response per project, though everyone working 20 hrs/wk for

past 3 mos. could contribute opinions. This is easier to validate, verify and analyze, but are we missing out?

Moving target topics… Ur-question remains the same: How are the distributed library building and

community governance processes working? four issues to evaluate remain the same: NSDL communications

infrastructure; the extent and success of NSDL projects’ collaboration; the projects’ level of participation in the NSDL organizational structures; and, the information needs and motivations of the community.

definitions, issues, tools have changed (i.e., governance -> organizational structure; CommPortal ->re-designed)

Design and Delivery Issues (continued)

Reliability, Validity, Sampling Pilot - Didn’t do. 2003 - extensive review and comments from EIE Committee members and

the Core Integration Team; expert review and validity testing with NSDL PIs; lastly, think-aloud usability testing

Distribution: Pilot – used proprietary software for distribution and analysis. 2003 – CI (Casey & Alex) hand built a form. There has to be a better way.

Timing Distribute when the maximum number of respondents are near their

computers; both the Pilot and 2003 surveys were delivered in early summer

Coordinate with other NSDL-wide data-gathering efforts Re-Use (of instrument and of data)

Design so the same basic instrument can be used regularly – cost and time savings

Pilot: sparked the idea of creating an “Evaluation” category for the Collaboration Finder and for creating the NSDL Progress Report

2003: Provided data for the NSDL Progress Report

Process Issues & Questions

Clarify the purposes and timing of the Annual PI Survey relative to other data-gathering activities occurring across NSDL: 1) surveys from other standing committees; 2) the Collaboration Finder; and, 3) NSF reporting requirements.

The Annual PI Survey has functioned well for gathering data about perceptions and beliefs about organizational aspects of NSDL. Does it function well as a tool for identifying projects’ immediate technical or informational needs?

The results from this Annual PI Survey were used in the NSDL Progress Report. Re-purposing survey data reflects a larger organizational need for both formative evaluation and for data to support outreach efforts. How should the Annual PI Survey respond to shifting NSDL organizational needs and goals?