23
S519: Evaluation of Information Systems Evaluation Criteria Ch3+4

S519: Evaluation of Information Systems Evaluation Criteria Ch3+4

Embed Size (px)

Citation preview

S519: Evaluation of Information Systems

Evaluation Criteria

Ch3+4

Step3: Defining evaluative criteria

To build a criterion list, consider the following procedures: A needs assessment Logic model of linking the evaluand to the needs An assesment of other relevant values, such as

process, outcomes, and cost A strategy to organize your criterion checklist

Make sure that you go into the evaluation with a well-thought-out plan so that you know what you need to know, where to get that information, and how you are going to put it together when you write up your report.

Needs assessment

Needs that we identify become the outcome criteria we use for the evaluation

The data collected during the needs assessment phase can be used as baseline data for comparison if we wish to track change in certain outcome variables.

Needs assessment

Understand the true needs of your evaluation end users (consumers or impactees)

Who are your end users? They are the person or entity who buys or users a product or

service, enroll in a training program,etc. Upstream stakeholder (i.e. People on upper level of the structure –

manager, designer) Immediate recipients (i.e. People who directly consume your product or

service – consumer, trainee) Downstream consumers (i.e. People who indirectly involved in your

evaluation)

Needs assessment

Rules to follow during the evaluation Systematic (step-by-step, thorough) Objective (free from bias) Transparent (easy to repeat and follow)

Understanding needs Needs vs. Wants

Difference and why A need is something without which unsatisfactory functioning occurs.

Different kinds of needs Context dependence Conscious needs vs. Unconsious needs

Needs we know and needs we do not know Met needs vs. Unmet needs

Building a factory (increase job, but create pollution) Performance needs vs. Instrumental needs

„need to do“ something for satisfactory functioning (actual problems) vs. Proposed solutions

Access email vs. Lightweight laptop Most of the case, performance needs is considered, but not the instrumental

needs

Needs assessment method

Two phases: Identifying and documenting performance needs Investigating the underlying causes of

performance needs

Identifying performance needs

Steps Examples

Document the extent of the presenting need by gathering evidence

- Ask police for teen drug arrests

Find out more about individuals - Is the drug use more prevalent among boys or girls

Look for unnoticed performance needs - e.g. By combining of open-ended inquiry

-Have there also been problems with violence, or other crimes- asking parents, teachers or community members

Come out the clear picture of the needs

- Lay out the evidence

Identify the performance needs for an increase in drug abuse by teenagers

Investigating the cause of the needs

Using logic model

Training program

Improved skillsImproved

performance

If we implement thisWe will address this underlying need

Which should solve our performance problem

Exercise

If we implement this We will address this underlying need

Which should solve our performance problem if that was the true underlying need

Training program Unskilled employee

Education and communication

Improved performance

Performance incentives Unmotivated staff

Job enrichment Unchallenged job

Provision of resources Lack of resources

Flexible scheduling Home and work conflict

Organizational climate intervention

Bad working enviornment

More to do

Asking potential participants about their needs is just the tip of the iceberg

Many important information has to be gathered in some other way Working with experienced upstream impactees

Working with project officer for grant writing training program

Identifying correct cause by asking open-end inquiry to downstream and upstream impactees

Poor involvement of grant writing Lack of motivation – increase incentive Lack of confidence – build up self confidence

Other relevant criteria

Think about other criteria based on Process evaluation Outcome evaluation Comparative cost-effectiveness Exportability Like: legal, ethical, authenticity, scentific, economic, historical

requirements, etc.

Being diagnostic Collecting facts

Step3: output report

Needs assessment Identify consumers or impactees (e.g. Table3.2) Identify different needs (e.g. Table3.3)

Logic model (e.g. Exhibit3.6 and Exhibit3.7) An assessment of other relavent values with

the consideration of process, outcome and cost (e.g. Table3.4)

Organizing your criteria see step4 output report

Step4: Organizing criteria and indentifying sources of evidence

Rolling design unanticipated criteria may add to the list Existing criteria may be modified

Never draw a conclusion based on a single piece of evidence Try to view the same issue from different angles through:

Different types of data (both qualitative and quantitative) Multiple sources of information (e.g., existing document,

observations, input from more than one group of stakeholders)

Triangulation: using different ways or data to verify the conclusionStep-by-Step: start the small scope of data collecting, extend it slowly later on.

Checkpoints

When organizing your criteria, always keep the followings in mind: Process

How good are the evaluand‘s content and implementation Outcomes

How good are the impacts on immediate recipients and other impactees

Comparative Cost-Effectives How costly is it? Excessive, quite high, acceptable or reasonable

Exportability How can we extend this to other settings?

The process evaluation checkpoint

Process evaluation (Table 4.1, 4.2) Content

What the evaluand consists of, i.e., basic components or design)

Implementation How well or efficiently the evaluand was implemented

or delivered to the consumers who needed it Other features

Any other features that make the program good or bad which are not covered by the first two and are not outcomes or cost-related criteria

The process evaluation checkpoint

Now feed into these three categories with all the process-relevant ingredients we generated from: the needs assessment and other relevant needs (e.g.

Table3.3, Table 3.4, Exhibit3.6) Add additional consideration if necessary

category Subcategories and criteria Source of evdience

Content evaluation

Implementation evlauation

Others (if necessary)

The outcome evaluation checkpoint

What is outcome Things that happen as a result of the program Outcomes can affect anyone listed as consumers

How to do Based on logic model in step3 (e.g. Exhibit3.6 and Exhibit3.7) Organize them into subcategories See Table4.3 (D-p60)

category Subcategories and criteria Source of evdience

Knowledge, skill and attitude gain

Application of knowledge, skill and attitudes

The comparative cost-effectiveness checkpoint

Any evaluation has to take cost into account What are costs?

Money Time Effort Space Opportunity costs

i.e. If our evaluation use this resources, then other department cannot use them for some time, which generates cost.

Cost cube (Scriven, 1991)

Type of costs Money, resources, time, opportunities

Costs When Preparation, implementation, maintenance,

evaluation Costs to whom

Participants, community organizations, workshop provider, others

D-p61

Cost cube table

Preparation Implementation Maintenance Evaluation

Money

Resources

Time

Opportunity

Costs to Whom

Comparison with competitors

•Find out the most important costs for your evaluation•Compare your cost with other competitors

The exportability checkpoint

What elements of the evaluand (i.e., innovative design or approach) might make it potentially valuable or a significant contribution or advance in another setting Think about examples

Step4: Output report

Checkpoints for Process (e.g., Table4.1) Outcomes (e.g., Table4.3) Comparative Cost-Effectives (e.g., cost cube

table, Exhibit4.2) Exportability

Short summary of potential areas for exportability