Embed Size (px)
DESCRIPTION
A description of how to initiate program evaluation from the perspective of ICE standards
Citation preview
You Can Do Program Evaluation
November 19, 2010
Stephen B. Johnson, PhDDiane M. Talley, MAJames A. Penny, PhD
Castle Worldwide
NOCA 11005.3 The certificate provider shall conduct periodic program evaluations to assess program quality and effectiveness and implement future improvements. ANSI6.2.10.1 The evaluation shall measure the quality, effectiveness, and value of the certificate program against stated program performance objectives.6.2.10.4 The evaluation shall include mechanisms to monitor and identify regularly the need for changes to the program’s purpose, score, or learning outcomes …. Program evaluation standardDefinition of evaluation: Systematic investigation of the worth of something to gain direction for future improvement (Joint Commission - Program Evaluation Standards page 3).
Why are we here?
Plato and Aristotle in The School of Athens fresco, by Raphael.
ICE 2010 Conference Atlanta Georgia
Simplified program evaluation model
1. Identify a problem.2. Generate and implement alternatives to relieve
the stress.3. Evaluate those alternatives.4. Adopt those most likely to reduce problem.
ICE 2010 Conference Atlanta Georgia
Good program evaluation is a tool1. Improve your program2. Illuminate/explain how/why a program works
3. Challenge assumptions4. Avoid the doldrums5. For certificate programs – focuses on the learner
6. Takes program design from:•Trial and error•Based on what is known/assumed •Heuristic design (planned trial and error)
7. To a planned program design that might surprise
ICE 2010 Conference Atlanta Georgia
The Wright model
ICE 2010 Conference Atlanta Georgia
The Wright flyer
ICE 2010 Conference Atlanta Georgia
Wright wind tunnel
ICE 2010 Conference Atlanta Georgia
General Model of Program Evaluation
ICE 2010 Conference Atlanta Georgia
Questions about evaluation
Why evaluate? Purpose? Who will do it? Where do the questions come from? What questions to ask? Importance (and number) of questions? Methods to use? How will the results be used/implemented? Time and budget? What is the fall back if something breaks?
ICE 2010 Conference Atlanta Georgia
Evaluating education/training programs
Appraise the worth of educational undertaking, against a desired standard, for the purpose of decision making typically around changing elements of the program (Popham, Scriven et. al.)
Focus on learner behavior/outcomes
MOST concerned with post-instruction outcomes directly related to the learner
ICE 2010 Conference Atlanta Georgia
Novice errors
A single evaluation – inevitably flawed Too many questions
Expensive, inconclusive, problems with implementing solutions
Choose easy but trivial issues The change is not worth the cost The identified problem may actually be a symptom – the problem may be deeper
ICE 2010 Conference Atlanta Georgia
Going deep
ICE 2010 Conference Atlanta Georgia
Hubble deep field image
ICE 2010 Conference Atlanta Georgia
Our advice
1. Start with 1 or 2 practical problems.2. Be aware of confusing the symptom with the
problem.3. Plan for the long term (with flexibility).4. Get deeper over time.
Will most likely lead to innovative solutions and program outcomes including different clients, products, or services
5. Remember the Wright model.
ICE 2010 Conference Atlanta Georgia
Questions?
Stephen B. Johnson [email protected] M. Talley [email protected] A. Penny [email protected]
919.572.6880www.castleworldwide.com
