Upload
olivia-horn
View
217
Download
2
Embed Size (px)
Citation preview
Merit determination
Keep in mind of what the evaluation is. Evaluation is the systematic determination of the
quality or value of something (Scriven, 1991). Today we will discuss how to determin the
importance of evaluand components or dimensions
Merit determination
It is the process of setting „standards“ (definitions of what performance should constitute „satisfactory“, „good“, etc.) and applying those standards to descriptive data to draw explicitly evaluative conclusions about performance on a particular dimension or component.
Two steps
Step 1: defining what constitutes poor, adequate, good, very good and excellent performances on a particular dimension (or component)
Step 2. using this definition to convert empirical evidence into evaluative conclusions (e.g. something explicit about quality or value)
Determing merit
Decscriptive facts about
performance
Quality or value
determination guide
Evaluative conclusions
Using single quantitaive measure
In a simple case, performance is measured on a single quantitative dimension The quality or value determination guide is just a
set of cutoffs E.g., >90%=A/excellent, 80%-89%=B/good, 70%-
79%=C/adequate E.g., satisfactory/unsatisfactory
Difficulty: where to put the cutoff score and how to compare with different systems?
Exercise
School grading system USA: A(>90%), B(80-90%), C(70-79%), D(60-69%),
F(<60%) New Zealand: A(>80%), B(65-79%), C(50-64%), D(35-
49%), F(<35%). Does that mean in New Zealand, it is easier to get A? Can this way of grading ensure objectivity and
consistency of grading across courses? If yes, why? If no, why not?
What happens in your country? Form a group to discuss
Using qualitative or multiple measures
Using a single measure is not generally good practice.
When using multiple measures, it is tricky on how to merge them together to come out with conclusion.
See table 8.1
Experience
Do not try to go for high precision It is perfectly appropriate to give an answer that
still has a certain amount of fuzziness or uncertainty associated with it.
Please do not oversell the precision of your work Providing a well-supported broad-brush answer
to an important question is not a bad idea.
Rubric
Rubric is a tool that provides an evaluative description of what performance or quality „looks like“.
It has two levels: Grading rubric is used to determin absolute
quality or value (e.g., Table8.2) Ranking rubric is used to determin relative quality
or value
Rubric for absolute value
Rubric for “grading” is based on: Discussion with domain expert Discussion with upstream stakeholders Existing rules (scope of duties) or literatures Evaluand expectations (needs assessment) Evaluation context (job market, current situation)
Sample grading rubric 1
Table 8.3 Using Table 8.3 to grade Table 8.1 – what is
grade for this master program? Why? Do it by yourself first? Form a pair and discuss your point
Sample grading rubric 2
Table 8.4 provides you a better grading rubric Scope Performance indicators Ranking (1-5) Discuss
How do they develope rank baesd on performance indicators
Exercise
Refine Table 8.3 Identify scopes Identify performance indicators Refine the ranking according to identified
indicators. Form a group and discuss
Exercise
Develop a grading rubric for your evaluation project Take Table 8.4 as example
Please include scope, performance indicators and ranking description
Using rubric for determining „relative“ merit
Relative metric is important for experiment that uses a control or comparison group. Student scores are interpreted by comparison
with other similar schools
It simily tell us how the person or program did relative to peers or competitors.
Using rubric for determining „relevative“ merit
Score falls in Grade assigned
Top 10% A
Next 20% B
Next 50% C
Next 15% D
Next 5% F
“Grading on the curve”: instructors rank students into different percentage
E.g.: GRE, SAT, GMAT, IQ
Significance
Statistical significance: Any observed difference (or statistical relationship)
is unlikely to be due to chance Practical significance:
Real impact on people‘s life E.g., the difference has a noticeable and nontrivial effect
on functioning or performance
When determining the merit of a particular outcome, we should taken both significance into consideration
Relative merit
Using comparison to determine relative merit Benchmark process, outcome, and cost criteria
against what has been achieved elsewhere (e.g. by other evaluands of a similar setting).
Benchmarking
It is a systematic study of one or more other organizations‘ systems, processes, and outcomes to identify ideas for improving organizational effectiveness.
It refers to a process of gathering comparison data about what organizations in similar or related industries are achieving (e.g. About process, outcomes, and costs). Quantiative data Qualitative data (observation of processes)
Exercise
Grading Table 8.8 according to the rubric in Table 8.7 What is your grading Why is it? How can you improve table 8.7? Form a group to discuss
Exercise
Take the grantsmanworkshop, draw the absolute and relative rubrics to grade this training program Absolute rubric (see Table 8.4) Relative rubric (see Table 8.7) Write down half page Form a group to discuss
Exercise
Draw the absolute and relative rubric for your evaluation project Work together with your project team Absolute rubric (see Table 8.4) Relative rubric (see Table 8.7)