S519: Evaluation of Information Systems
Analyzing data:
Synthesis
D-Ch9
Last week
What are merit determination? What is absolute merit? How to draw rubric? What is relative merit? How to draw rubric?
Synthesis methodology
It is a tool to allow us to draw overall evaluative conclusions from multiple findings about a single evaluand.
Synthesis is the process of combining a set of ratings or performances on several components or dimensions into an overall rating.
Synthesis methodology
KEC Box 11: Overall Significance What are the main areas where the evaluand is
doing well Is this the most cost-effective use of the available
resources to address the identified needs without excessive adverse impact?
Synthesis methodology
Merit determination To develop the rubrics To use rubrics to summarize the multiple findings A kind of synthesis method
Rubrics are one of the simplest methods to blend data. But when data is a bit more complex, it is difficult to use
a rubric as the only tool Data are not equally important or reliable Different nuances and combinations (such as Table8.3)
It is not
It is not meta-analysis A special statistical techniques to give a weighted
average of effect sizes across multiple studies – for quantitative studies
It is not literature review A judgment from a reviewer’s point of view.
It is
A systematic way of taking into account the pluses and minuses uncovered when the evaluation team draws the evaluative conclusions.
Some synthesis is always necessary, whether the evaluation is formative or summative
It combines the data we collected and uses comparison to place the evaluand’s performance in a wider context.
Keep in mind
Doing poorly on some minimal important criteria
Doing poorly on some crucial criteria
Are very different!
Costs and comparisons in synthesis
Evaluation is Not just: Did the value of the outcomes outweight
the value of the resource it took to achieve them? But: Is this evaluand the best possible use of
available resources to achieve outcomes of the greatest possible value?
Evaluation
Sythesis for „ranking“ If it is „ranking“ (relative) evaluation:
Consider each alternative and make explicit comparisons
Synthesis for „grading“ If it is „grading“ (absolute) evaluation:
Consider different context settings and provide better interpretation of merit
Qualitative or quantitative
Quantitative synthesis Using numerical weights
Qualitative synthesis Using qualitative labels
Synthesis for “grading”
The primary evaluation question is for absolut quality or value How well did the evaluand perform on this
dimension? How effective, valuable, or meritorious is the
evaluand overall? Is this component worth the resources put into it?
Quantitative weighting example with „bars“
Case: Personnel evaluation in a small accounting firm 13 defined tasks (e.g., telephone, reception, data
entry, etc.) Each employee has responsibiltiy for 4-6 tasks
Evaluation: Importance weighting (through the voting of the
selected stakeholders) In-depth discussion with business owners Derive the importance metric and bars
Quantitative weighting example with „bars“
Evaluation Define the levels of importance: 3 to 5 levels work well in most case Do not go to too many levels (why? Is this useful?) For example
task 1. minor task (1) 2. normal-priority task (2) 3. high-priority task (3) 4. extremely high-priority task (4)
Quantitative weighting example with „bars“
Evaluation Setting up rubrics for each 13 tasks Normally 4-6 level is sufficient
Example: Performance Rubric 1. Totally unacceptable performance (1) 2. Medicore (substandard) performance (2) 3. Good performance (expected level) (3) 4. Performance that exceeded expectations (4) 5. All-around excellent performance (5)
Synthesis – draw the overall conclusion See Exhibit 9.2 (p158)
Exercise
Peronal evaluation in a small accouting firm
Tasks Importance Score for Alice
Telephone 1 2
Data entry 2 3
Tax data management
4 1
Client support 4 5
Reporting 3 3
Communicating 1 3
How about Alice according to Exhibit 9.2?
Exerice
How about Alice Mean
= 1*2+2*3+4*1+4*5+3*3+1*3/(1+2+4+4+3+1)=44/15=2.93
She has the good performance?
Qualitative weighting example 1 (with no „bars“)
Case: a school-based health program evaluation It contains 9 different components: nutrition education, mental
health services, safer sex, legal service and others. How to evaluate these system in low-budget and short period of
time whether they are meeting important needs of the students and their families
Evaluation: Interview Student surveys
School health system evaluation
Survey question design: Two quantitative questions
How useful was the program to you? (4-point response scale: not at all useful, somewhat useful, useful, very useful)
How satisfied were you with the program? One qualitative question (open-end)?
What other changes or events, good or bad, have happened to you or someone you know because of receiving the service?
School health system evaluation
Survey result about nutrition system shows in Table 9.1
Look at table 9.1, think about: How can you draw a conclusion from this result
about the nutrition system? Is it good or bad? Pay attention of N in Table 9.1
School health system evaluation
Setting the importance for these three questions (1-strongest data, 3=weakest data) 1. Ratings of usefulness (directly related to needs) 2. Responses to the open-ended question 3. Satisfaction ratings
Creating rubrics for each question Table 9.2 for question 1 and question 2 Table 9.3 for open-ended question
School health system evaluation
How to grade the nutrition system based on the first two quantitative questions: Based on Table 9.1, come out with the rubric as
Table 9.2 Why 90% is select, 70%-90%.. how to draw Table 9.2 from Table 9.1 and collected
data?
School health system evaluation
Table 9.3 Rubric for converting data from qualitative
evaluation - open-ended responses into merit ratings
Is that a good way to do this? Are you happy with this table? If not, how do you want to improve it?
School health system evaluation
Synthesis to draw overall conclusion Step-by-step Start with the strongest data (question 1) Blend with open-ended comments Finally take the satisfaction ratings into account
See table 9.4 for the whole process
School health system evaluation
How to draw final conclusion?
Usefulness ratings
Satisfaction ratings
Open-ended comments
Final coclusion:
Merit of the nutrition program
See table 9.4Discuss how to apply this to your group project
Using quantitative ratings to draw the suggested results and using qualitative ratings to find the positive or negative facts to re-adjust the results
Exercise
Form your group project Discuss this case (p157-166) How to apply this to your group project?
Qualitative (nonnumerical) weighting example 2
Bar A minimum level of performance on a specific dimension Performance below this cannot be compensated for by much
better performance on other dimensions (see Exhibit 9.2)
Hard hurdle (also referred as global bars) Overall passing requirement for an evaluand as a whole (see
Exhibit 9.2)
Soft hurdle Overall requirement for entry into a high rating category Place a limit on the maximum rating (e.g., I want all As for my
classes)
Qualitative (nonnumerical) weighting example 2
Case: Evaluation of the learning capacity of a small biotechnology start-up company „biosleep“.
Evaluation 27 subdimensions of organizational learning capacity
(see table 9.5) Data collection: survey and interview Rubric: similar as Table8.2 Importance is built by using strategy 6 in Chapter 7
Using program theory and evidence of causal linkages (p118-125)
Biosleep
Evaluation Synthesis
Pack the ratings on the subdimensions into 8 main dimensions
Combine the ratings on these 8 main dimensions to draw an overall conclusion
Biosleep
Dimension by dimension Layer by layer
Sub-dimnention1
Sub-dimnention2
Sub-dimnention3
Sub-dimnention4
Dimnention1
Dimnention2
Overall rating
Biosleep
Synthesis Subdimensions Dimensions
Using Table 9.6 to draw conclusions of dimentions based on subdimensions
Using Table 9.6 to judge Table 9.5 and come out the result as Exhibit 9.4
Dimensions overall evaluation Based on Table 9.7 (created based on literature review, What is your conclusion for the evaluation of Biosleep?
And why?
Exericse
Form your group project Discuss on how are you going to grade your
evaluation? Which example you would like to follow? How to develop rubric for dimension and overall?
Recommended