Analyzing data: Rank

What is synthesis methodology?, why do we need that?

What is synthesis for grading? Quantitative Qualitative How to merge all the conclusions to get

the final grade?

Review

S519

What are „ranking“ evaluations? Examples? Difference comparing with „grading“

evaluation?

Synthesizing for „ranking“

S519

Qualitative Qualitative weight and sum (QWS)

Quantitative Numerical weight and sum (NWS)

Qualitative and quantitative

S519

It is a quantitative synthesis method for summing evaluand performance across multiple criteria.

It includes Assign numerical importance weight and a

numerical performance score to each criteria (dimension)

Multiply weights by performance scores Sum these products The summing result represents the overall

merit of the evaluand

Numerical Weight and Sum (NWS)

S519

It fits for There are only a small number of criteria There is some other mechanism for taking bars

into account (why) There is defensible needs-based strategy for

ascribing weights.

Numerical Weight and Sum (NWS)

S519

A comparative evaluation on three different interventions for training managers

A mountain retreat featuring interactive sessions with multiple world-class management gurus

An in-house training and mentoring program run by human resources,

A set of videos and latest book on management from management guru Peter Drucker

Training program evaluation

S519

Needs assessment for this evaluation Bear in mind that this is a comparison evaluation How do you want to compare these programs, what are

the key features of the programs Identify the dimension of merit (Process,

Outcomes and Cost) Decide the importance of the merit (giving

weights to merits, based on needs?)

See Table 9.8

Training program evaluation

S519

Next steps Data collection (what are your experiences for

your project data collection?) Data analysis

Rate their performance based on pre-defined ratings: excellent, very good, good, fair, or poor) (see Table 9.9 for this example)

Convert weights into numbers (see Table 9.10) Convert ratings into numbers (see Table 9.10)

Synthesis step (how? See Table 9.11)

How to interpret Table 9.11

Training program evaluation

S519

Do it by your own hand: Converting Table9.9 to Table9.10 (defining your own

numeric value for importance and grading scales) and try to find out which program is the best comparing with others.

If suddenly, the cost criteria become extremely important, will this change the final result? Work on your own Form the pair and discussion

Pros and cons for NWS?

Exercise

S519

It is non-numerical synthesis methodology for summing the performances of an evaluand on multiple criteria to determine overall merit.

It is a ranking method for determining the relative merit of two or more evaluands

It is not suitable for grading It fits for

Personnel selection, products/service/proposal selection

Qualitative Weight and Sum (QWS)

S519

Step1: Determine importance in terms of maximum possible value How (see Chapter 7, six strategies) Table 9.12 (compare with Table 9.8)

Step2: Set bars Bar is the cut point between acceptable

and unacceptable criteria. Such as: Too expensive to afford Too long away from their work

QWS

S519

Step3: Create value determination rubrics Rubrics are level-based (see Chapter 8)

Description on each level, how to deal with bar? Unacceptableno noticeable valuemarginally

valuablevaluableextremely valuable Such as what performance would look like at each level

Each dimension can have its own rubrics or each group of dimensions can have their own rubrics

Each group of questions can have their own rubrics Synthesis step can have its own rubrics

Example: Rubric for rating finanical cost of training (see table 9.14)

QWS

S519

Step4: Check equivalence of value levels across dimensions The validity of the QWS method is highly

dependent on ensuring the rough equivalence on the value levels defined for each dimension

For example, whether table 9.14 and table 9.15 have the roughly equivalent value levels

How to do that? Put them into a matrix. See table 9.16

QWS

S519

Step5: rate value of actual performance on each dimension Rating table 9.9 according to rubric

(table9.16) See Table 9.17

Step6: tally the number of ratings at each level and look for a clear winner For each program, how many symbols

they got? Throw out programs with unacceptable

ratings, see whether there is a clear winner?

QWS

S519

Step7: refocus Delete the rows with similar score (see

table9.18) Count how many symbols each of them

got Can we find the clear winner?

Yes or no? Why? How should we go further?

QWS

S519