Embed Size (px)
Citation preview
Simple Survey Resources: Templates, Tabulation &
ImpactJeff Buckley, Jenna Daniel,
& Casey Mull
Overview• Situation• Need Being Met• Audience (Agents & Associates)• Basic Overview of the Training• How Can This Be Utilized in Your State?
Situation
• As a result of shrinking state budgets, the need to effectively document program impact is more critical than ever.
• Many states do not have a full time Evaluation Specialist.• County Agents and State Specialists must develop their
own evaluation instruments.• Many County Agents and State Specialists may not feel
qualified or confident enough to develop their own instruments and/or to tabulate and report data.
Need Being Met
• A 4-H program evaluation training was created
• Implemented by Georgia 4-H State Specialists & CAES Faculty
• Piloted at the 2012 Winter School– 4-H Agents & Associates
Simple Survey Training• Training Format:
– 90 minute session– Interactive– 4-H Agents and Associates – Participants created their own program
evaluation
Simple Survey Training• Why is it Important to Gather Data?– Who is your audience?– Process Evaluation – Internal Use– Outcome Evaluation – External Audience
• How will you share the results?– Impact Statement– News Article– Journal Article– Other
Simple Survey Training• What Can a Survey Measure?– Knowledge– Attitude– Behavior
• Designing Measurable Objectives– Refer to the handout “Verbs for Writing Measurable
Objectives.”– Measurable objectives will translate into more
effective constructs and questions.
Simple Survey Training• Consider…– What you want your participants to know.– How you want their attitude to change.– What sort of behavior changes you want to see.
• Creating A Construct– Pick two outcomes/constructs and develop
three “questions” each, OR– Pick three outcomes/constructs and develop
two “questions” each.– (The template has space for six “questions”.)
Simple Survey Training• Creating A Construct
– The food in the cafeteria was hot.– The dining area was clean and comfortable.– There were a variety of fruits and vegetables.– Beverage selections included healthy options.– Vegetables were not over-cooked.– The servers were on task and pleasant.
Simple Survey Training• Likert Style Questions
– These templates are designed for use with Likert style questions.
– Pick the rating scale that works best for your objectives/program.
– This is the most challenging part of developing your survey. Allow time. Ask for feedback.
Simple Survey Training• Likert Style Questions
– Consist of a statement and a rating scale.– Need to conform to one rating scale.– Measurable verbs that fit the appropriate cognitive
domain.– One data point per question.– Is what you’re measuring important to the intended
audience of the evaluation?
Simple Survey Training• Instrument Templates
– Traditional– Retrospective Postthenpre– 6 quantitative items– 1 qualitative item– Feedback on the program
Simple Survey Training• Tabulation
– Excel Spreadsheet– Directions– Worksheet for both instrument templates– “Locked” format
Now You’re Ready to Gather Data!
Now You’re Ready to Gather Data!
Now You’re Ready to Gather Data!
Results• Presentations Conducted– Galaxy– Georgia 4-H Winter Conference– Georgia Association of 4-H Agents Conference– Southern Region Volunteer Conference
Results• Impact
– The percentage of program participants who stated that they could…
– develop measurable objectives increased from 60% before to 100% after the program.
– design evaluation instruments for their programs increased from 69% before to 91% after.
– are likely to develop their own evaluation tools in the future increased from 46% to 89/%.
– A domain analysis of the qualitative revealed the following themes…– Participants were very glad to have the templates.– Participants plan to use the tools and knowledge they gained to
gather data.– Participants plan to use the data they collect to develop higher
quality impact statements.
Results• Extension Agent Quotes & Impact Statements– With a mean score of 3.1 (SD=.994), overall
respondents from Question 5 indicated they "agreed" they are more likely to get involved in their community as a result of the GPK Leadership Adventure weekend.
– Retrospective post then pre: [In Question 2], 84.6% of respondents indicated an increase in their confidence concerning their ability to “design or construct a map of the distribution of major world languages”.
Simple Surveys in Florida 4-H• Tabulation of Data– Dropping the data in the sheets is so simple and saves a
ton of time. It is much easier than manually tabulating learning gains
– Data is easy to interpret from the sheets
• How Data Has Been Used– Data has been used with stake holders to secure program
support and has been used in annual reports of accomplishment.
– Data will be used in presentations at professional conferences to show outcomes and projected impacts of programming.
• 5.
Training Evaluation
• Training Program Evaluation Exercise– Confidence in ability– Knowledge gain– Training evaluation
How Can You Utilize this Program?
• Resources:– www.georgia4h.org/evaluationresources
• State Meetings • Webinars• New Agent Training
Questions?
For more information contact:University of Georgia, State 4-H Office
Jeff Buckley, [email protected] Daniel, [email protected]
Casey Mull, [email protected] 4-H Office – 706-542-4444
