Upload
kevin-elliott
View
223
Download
3
Tags:
Embed Size (px)
Citation preview
Step3: Defining evaluative criteria
To build a criterion list, consider the following procedures: A needs assessment Logic model of linking the evaluand to the needs An assesment of other relevant values, such as
process, outcomes, and cost A strategy to organize your criterion checklist
Make sure that you go into the evaluation with a well-thought-out plan so that you know what you need to know, where to get that information, and how you are going to put it together when you write up your report.
Needs assessment
Needs that we identify become the outcome criteria we use for the evaluation
The data collected during the needs assessment phase can be used as baseline data for comparison if we wish to track change in certain outcome variables.
Needs assessment
Understand the true needs of your evaluation end users (consumers or impactees)
Who are your end users? They are the person or entity who buys or users a product or
service, enroll in a training program,etc. Upstream stakeholder (i.e. People on upper level of the structure –
manager, designer) Immediate recipients (i.e. People who directly consume your product or
service – consumer, trainee) Downstream consumers (i.e. People who indirectly involved in your
evaluation)
Needs assessment
Rules to follow during the evaluation Systematic (step-by-step, thorough) Objective (free from bias) Transparent (easy to repeat and follow)
Understanding needs Needs vs. Wants
Difference and why A need is something without which unsatisfactory functioning occurs.
Different kinds of needs Context dependence Conscious needs vs. Unconsious needs
Needs we know and needs we do not know Met needs vs. Unmet needs
Building a factory (increase job, but create pollution) Performance needs vs. Instrumental needs
„need to do“ something for satisfactory functioning (actual problems) vs. Proposed solutions
Access email vs. Lightweight laptop Most of the case, performance needs is considered, but not the instrumental
needs
Needs assessment method
Two phases: Identifying and documenting performance needs Investigating the underlying causes of
performance needs
Identifying performance needs
Steps Examples
Document the extent of the presenting need by gathering evidence
- Ask police for teen drug arrests
Find out more about individuals - Is the drug use more prevalent among boys or girls
Look for unnoticed performance needs - e.g. By combining of open-ended inquiry
-Have there also been problems with violence, or other crimes- asking parents, teachers or community members
Come out the clear picture of the needs
- Lay out the evidence
Identify the performance needs for an increase in drug abuse by teenagers
Investigating the cause of the needs
Using logic model
Training program
Improved skillsImproved
performance
If we implement thisWe will address this underlying need
Which should solve our performance problem
Exercise
If we implement this We will address this underlying need
Which should solve our performance problem if that was the true underlying need
Training program Unskilled employee
Education and communication
Improved performance
Performance incentives Unmotivated staff
Job enrichment Unchallenged job
Provision of resources Lack of resources
Flexible scheduling Home and work conflict
Organizational climate intervention
Bad working enviornment
More to do
Asking potential participants about their needs is just the tip of the iceberg
Many important information has to be gathered in some other way Working with experienced upstream impactees
Working with project officer for grant writing training program
Identifying correct cause by asking open-end inquiry to downstream and upstream impactees
Poor involvement of grant writing Lack of motivation – increase incentive Lack of confidence – build up self confidence
Other relevant criteria
Think about other criteria based on Process evaluation Outcome evaluation Comparative cost-effectiveness Exportability Like: legal, ethical, authenticity, scentific, economic, historical
requirements, etc.
Being diagnostic Collecting facts
Step3: output report
Needs assessment Identify consumers or impactees (e.g. Table3.2) Identify different needs (e.g. Table3.3)
Logic model (e.g. Exhibit3.6 and Exhibit3.7) An assessment of other relavent values with
the consideration of process, outcome and cost (e.g. Table3.4)
Organizing your criteria see step4 output report
Step4: Organizing criteria and indentifying sources of evidence
Rolling design unanticipated criteria may add to the list Existing criteria may be modified
Never draw a conclusion based on a single piece of evidence Try to view the same issue from different angles through:
Different types of data (both qualitative and quantitative) Multiple sources of information (e.g., existing document,
observations, input from more than one group of stakeholders)
Triangulation: using different ways or data to verify the conclusionStep-by-Step: start the small scope of data collecting, extend it slowly later on.
Checkpoints
When organizing your criteria, always keep the followings in mind: Process
How good are the evaluand‘s content and implementation Outcomes
How good are the impacts on immediate recipients and other impactees
Comparative Cost-Effectives How costly is it? Excessive, quite high, acceptable or reasonable
Exportability How can we extend this to other settings?
The process evaluation checkpoint
Process evaluation (Table 4.1, 4.2) Content
What the evaluand consists of, i.e., basic components or design)
Implementation How well or efficiently the evaluand was implemented
or delivered to the consumers who needed it Other features
Any other features that make the program good or bad which are not covered by the first two and are not outcomes or cost-related criteria
The process evaluation checkpoint
Now feed into these three categories with all the process-relevant ingredients we generated from: the needs assessment and other relevant needs (e.g.
Table3.3, Table 3.4, Exhibit3.6) Add additional consideration if necessary
category Subcategories and criteria Source of evdience
Content evaluation
Implementation evlauation
Others (if necessary)
The outcome evaluation checkpoint
What is outcome Things that happen as a result of the program Outcomes can affect anyone listed as consumers
How to do Based on logic model in step3 (e.g. Exhibit3.6 and Exhibit3.7) Organize them into subcategories See Table4.3 (D-p60)
category Subcategories and criteria Source of evdience
Knowledge, skill and attitude gain
Application of knowledge, skill and attitudes
The comparative cost-effectiveness checkpoint
Any evaluation has to take cost into account What are costs?
Money Time Effort Space Opportunity costs
i.e. If our evaluation use this resources, then other department cannot use them for some time, which generates cost.
Cost cube (Scriven, 1991)
Type of costs Money, resources, time, opportunities
Costs When Preparation, implementation, maintenance,
evaluation Costs to whom
Participants, community organizations, workshop provider, others
D-p61
Cost cube table
Preparation Implementation Maintenance Evaluation
Money
Resources
Time
Opportunity
Costs to Whom
Comparison with competitors
•Find out the most important costs for your evaluation•Compare your cost with other competitors
The exportability checkpoint
What elements of the evaluand (i.e., innovative design or approach) might make it potentially valuable or a significant contribution or advance in another setting Think about examples