Assessing K-12 Service-Learning Impacts and Quality

Shelley H. Billig, Ph.D.RMC Research Corporation

Overview

• Evaluation Toolkit• Logic Models and Theory of

Change• Measuring Impact

– Academic impact– Civic impact– Personal/social impacts– Others

• Measuring Quality

Evaluation Toolkit

• How-To Guide For Traditional Evaluation

• Overview

• Steps for Planning and Implementation

• Worksheets

• Detailed Guidance

Session Experiences

• “Dip in” for various aspects of the evaluation and assessment process.

Logic Models

• Shows how and why the program will work;

• Connects program activities to program outcomes through outputs:– e.g., activity (input) Increased

student engagement in schools (output) Increased student academic achievement (outcome).

TX K-12 Learn and Serve Logic Model

Partner Contributions

LEA and campus investment Contributions by community

partners

Support for Service-Learning

Training/technical assistance Recognition of service-learning

as improvement strategy Teacher buy-in

Service-Learning Experience

Frequency and type

of project, implementation quality, linkage to subject matter, effectiveness of partnerships

Number and type of students participating, intensity and duration of participation

Student Outcomes

Academic performance Civic knowledge, skills, dispositions,

and behavior S.T.A.R.S. outcomes (e.g.,

leadership skills, ethic of service, value of lifelong citizenship)

School engagement, school and community attachment

Context (e.g., student, K-12 campus, and community characteristics, support for service-learning, accountability pressures)

Efforts to Promote, Sustain, and Institutionalize Service-Learning Programming

Partner/Community Outcomes

Benefits for K-12 campuses, LEAs, community partners

Social capital/Community capacity

Try It!

• Start with your activities – what will you do?

• Specify your goals – what will you accomplish?

• Identify those factors that will influence results, both positively and negatively. Include demographics, context, implementation issues

Logic Model Use

• Once the logic model has been created with stakeholders, review it.

• Use the logic model to design or revamp the evaluation.

• Use the constructs identified in the logic model to guide the development of measures (instrumentation).

Logic Model Resources

• University of Wisconsin-Extension Program Development and Evaluation, www.uwex.edu/ces/pdande

• W.K. Kellogg Foundation, www.wkkf.org – Evaluation Handbook, 1998– Logic Model Development Guide,

2001

Assessing Academic Impact

– Where is service-learning likely to have the greatest academic impact?

• Knowledge areas (general and specific)• Types of skills (general and specific)• Moderators

– What measures are available?

Academic Skills

• Areas of likely impact– Higher order thinking skills such as problem solving,

analysis, transfer of learning– Specific knowledge related to service activities, e.g.,

measurement, writing, synthesizing materials, research

• What are the available measures?– Rubrics– Test scores– Writing scores– Problem solving essays– Measures of academic engagement– Absenteeism– Surveys– More!

Assessing Civic Impacts

• Where is service-learning likely to have the greatest civic impact?– Knowledge?– Skills?– Dispositions?– Attachment to community?

• Moderators

• What measures are available?

Civic Impacts

• Areas of likely impact– Knowledge of community needs– Skills for planning and implementing projects– Taking responsibility– Ethic of service

• Available measures– Surveys– Focus groups/interviews– Hours of service– Actual changes

Assessing Social/Personal Impacts

– Where is service-learning likely to have the greatest impact on social/personal areas?

• Knowledge areas (general and specific)• Types of skills (general and specific)• Moderators

– What measures are available?

Social/Personal Outcomes

• Areas of Likely Impact– Development of character traits such as

responsibility, trust, and caring– Respect for diversity– More bonds with adults– More protective factors such as resilience– Greater self-efficacy

• Measures– Surveys– Focus groups/interviews– Rubrics– Self-assessments

Sample Measures of Outcome Areas

• Examine the samples

• What is the best use of each? By whom and for what purpose?

• What do you like and dislike about each?

Measuring Moderators

Moderators are characteristics that influence outcomes.

Examples: Instructor experience; program design; program quality.

Need to identify and measure those factors like to serve as moderators.

What moderators are likely to be in play in your settings?

• Consider demographics.

• Consider program design.

• Consider quality.

• Consider instructor characteristics like experience, levels of training, and expertise.

• Consider partner characteristics.

Characteristics of Good Assessments

• Valid

• Reliable

• Objective

• Useful

• Items measure only one construct at a time (not double barrelled…)

• Related to the logic model or assessment questions

Open Discussion

• Questions and Answers

Resources

• National Service-Learning Clearinghouse www.servicelearning.org

• Billig, S. H., & Waterman, A. S., eds. (2003). Studying service-learning: Innovations in education research methodology. Mahwah, NJ: Lawrence Erlbaum Associates. Available at www.erlbaum.com

• Bringle, R. G., Phillips, M. A., & Hudson, M. (2004). The measure of service learning: Research scales to assess student experiences. Washington, DC: American Psychological Association. Available at www.apa.org/books