Simple Evaluation Templates: Gathering Data for

Program Improvement & Impact Reporting

Camping & Environmental Education Institute2/25/15

Jeff Buckley, Jennifer Cantwell,

Casey Mull, Whitney Cherry, Julie Dillard, Nicole Crawson

Topics we will cover…..

• What and Why Do You Evaluate? • Survey Templates• Impact Statements• Data Tabulation Templates • Tools for creating Surveys• Questions

You will leave with….• Resources for developing effective survey

questions• Survey Templates• Resources/tools for collecting and analyzing

data• Sample impact statements

Disclaimers

• Adapted from previous presentations by Jeff Buckley, Jenna Daniel & Casey Mull.

• Materials informed by best practices from Extension Professionals at University of Florida.

• Resources developed in collaboration with Nick Fuhrman, Ph.D., Associate Professor & Graduate Coordinator, Dept. of Agricultural Leadership, Education, & Communication, UGA.

• Consult with your state’s Evaluation Specialist for additional, more advanced, guidance on Program Evaluation.

• We welcome your constructive feedback as we work to provide useful resources in the area of Program Development.

Why is it important?• As a result of shrinking state budgets, the need to

effectively document program impacts is more critical than ever.

• Many states do not have a full time Evaluation Specialist.• County Agents and State Specialists must develop their

own evaluation instruments.• Many County Agents and State Specialists may not feel

qualified or confident enough to develop their own instruments and/or to tabulate and report data.

Evaluation Templates

Purse Strings and Heart Strings Qualitative Data – Domain Analysis Post-test Only Retrospective Post then Pre Quiz-like, post-test only Audience Poll Documentation Form

What and Why?• Why do you want to gather data?• What types of programs to you provide?• What do you want to know?• What will you do with the data?• Who will you share data with? • What are your goals and objectives?

Things to consider• Who is your audience?– Process Evaluation – Internal Use– Outcome Evaluation – External Audience

• How will you share the results?– Impact Statement– News Article– Journal Article – Be sure to check with IRB!– Other

Institutional Review Board (IRB)

Research oversight committee charged with ensuring that human subjects research is conducted in compliance with the applicable federal, state, and institutional policies and procedures.

When is IRB-approval needed?• When using results for:– Research– You plan to generalize the information learned – Scholarly article or report

• Not required for:– Evaluation– Internal use – program improvement, etc. – Reports to: Funders, stakeholders, media, impact

statements, news articles, etc.

Types of Outcomes • Short-term– New knowledge, increased skills, changes in

attitude or values• Intermediate– Modified behavior

• Long-term– Improved condition, altered status

What can our surveys measure?

• Short-term Outcomes – Knowledge• Increase in knowledge• Demonstration of knowledge

– Attitude or Values (behavioral intent)• As a result of intervention, I plan to….

– Skills

Camp & EE Surveys

Let’s take the survey!

Let’s look at the results!

Camp Surveys - Impact ExamplesAfter participating in the

4-H Health is Our Pledge (HOP) Class at 4-H Summer Camp participants indicated they….

% of youth

Learned new things they can do to be healthier. 92%Have gotten more exercise at 4-H camp than they normally do. 80%Plan to drink less soft drinks and more water or milk in the future. 75%

Participants correctly identified best practices for developing a healthy lifestyle 83% of the time.

Because of participation in the Herpetology Class at 4-H Summer Camp participants indicated they…. % of youth

Learned new things about reptiles and amphibians. 84%

Would like to learn more about herpetology and other science-related subjects.

88%

Camp & EE Surveys

• What are we measuring?• How can we share the data?• Other examples:–Herpetology – Survey –Tidelands – Survey & Data Tabulation

Spreadsheet

Educational Club Meeting Surveys

• Survey Examples– Career Awareness & Exploration: Government and

Public Administration field– States of Matter: Physical vs. Chemical Changes

• Data Tabulation Spreadsheets• Impact Statements– Different type – 1 or more constructs, combining

questions

Project Achievement Surveys

• Survey Examples– Junior / Senior Project Achievement– This survey also includes Likert questions,

constructs• Data Tabulation Spreadsheets – in progress!• Impact Statements – in progress!

A little more advanced….

Constructs:• Multiple questions designed to measure the

same concept• Helps: – identify poorly worded questions– Validity and reliability (negatively worded

questions)

Examples of Constructs (draft stage)

Junior/Senior Project Achievement Survey:• JOBS – better prepared to apply for jobs• MASTERY – correctly identify best practices in public

speaking• CONFIDENCE – indicate increase in confidence in public

speaking• GENEROSITY/BELONGING – project achievement gave me

the opportunity to value and practice service for others and to be part of a safe and inclusive community

Likert Style Templates

• Standard Likert– Survey– Data Tabulation Spreadsheet

• Retrospective Post-Then-Pre– Survey– Data Tabulation Spreadsheet

Likert Style Templates

• The percentage of program participants who agreed that they were confident in their ability to develop measurable objectives increased from 38% before to 100% after the program

• Overall respondents indicated they agreed they are more likely to get involved in their community as a result of the GPK Leadership Adventure weekend.

Points to Consider when Writing Likert Style Questions/Statements

• Consist of a statement and a rating scale.• Need to conform to one rating scale.• One data point per question.• Is what you’re measuring important to the

intended audience of the evaluation?• Good for older youth and adults.

Resources for Writing Questions

• Verbs for Writing Measurable Objectives

• Sample Rating Scales for Likert Questions

QUESTIONS??

To Access Templates, go to…www.georgia4h.org/evaluationresources

For more information contact:University of Georgia, State 4-H Office

Jeff Buckley, jbuckley@uga.eduJennifer Cantwell, jecantw@uga.eduState 4-H Office – 706-542-4444

For more information contact:University of Florida Extension

Whitney Cherry, cherryw@ufl.eduJulie Dillard, juliepd@ufl.eduNicole Crawson, ncrawson@ufl.edu