Utility Analysis& Professional Qualifications in Test Administration

BPS Level A and Level B (intermediate) courses

Assessment in the Workplace

Overview What is Utility Analysis (UA) ? Traditional UA framework : SDy and costs of

selection Practical Example Issues in UA Alternative UA framework : Boudreau et al (1997) Level A and Level B (Intermediate) M21 Seen Paper

What is Utility Analysis ?‘a family of theories and measures designed to describe, predict

and/or explain what determines the usefulness or desirability of decision options and to examine how information affects decisions’ (Boudreau, 1991)

Can be applied to selection processes, training, compensation, performance assessment, and internal staffing

Evolved to provide tools for better describing and communicating impact of HRM and Occ. Psy. interventions on organisational goals.

Traditional UA framework for utility analysis of new selection technique

Top-down

hiring

Applicant Ability to Performance Value of

Pool Predict Intervention

New

Intervention

Fundamental Processes in UA

The relationship between predictors and criteria (represented by ‘r’)

The nature of the criteria, represented by SDy

The nature of the selection process, represented by Z

The nature of the implementation process, represented by C.

‘The expected increase in output is directly proportional to the predictive validity of the test’ (Brogden, 1946)

Brogden’s equation :Yearly saving

per employee = (SDy x rv x Z) - [costs of selection]

selected (£/$,etc)

where :

SDy = standard deviation of value in performance

rv = predictive validity coefficient of the test

Z = mean standard score of all applicants (selected and rejected) on the test

Costs of Selection = administration costs per selected employee multiplied by the number of employees selected

Hence, for the amount saved for N selected employees, staying with the company for T years :

Total savings = [N x T x (SDy x rv x Z )] - (costs of selection)

How much more is a good worker worth than a poor worker (i.e. SDy)

Schmitt and Hunter (1982) : 2 assumptions

The value of a worker to a company is directly proportional to that worker’s performance (output) when the worker’s performance is measured on some criterion relevant to the company.

If performance is normally distributed, then the value of performance should also be normally distributed.

When SDy is large vs when SDy is small

If SDy is large (high) : then employees vary a lot in how much they are worth to the company ...

If SDy small (low) : then investment in selection is of less value, as there is less discrimination possible amongst workers

( >>> random selection ?!)

The costs of selection

[ N.B. costs include costs of assessing those who are rejected as well as those that are selected ]

Costs must include all costs (i.e. labour time, materials, selection programme development costs, etc.)

If the cost of assessing each applicant is C and SR (selection ratio) is the proportion of applicants selected :

Costs per selected applicant = C / SR.

Costs of selection = (costs per selected applicant) x (number of applicants selected)

Examples of Utility Analysis Results

Schmitt & Hunter’s estimates of predicted savings due to large scale implementation of improved selection methods :

Use of Computer Aptitude Battery by US Federal Government : $64,000 per selectee, over 9 years.

In US Federal Government selection, replacement of ability tests and work sample tests by less valid predictors would lead to losses of $3.12 billion (job tryout) to $15.8 billion (age) per year.

General improvements in the Philadelphia Police Force (5000 employees) : $18 million/year

General improvements in selection - US National Economy $87-$152 billion/year.

Practical ExerciseOld Selection Test

SDy = $30,000, rv = 0.2, Z = 1.0

Costs per selected employee = £1,000

10 people (N = 10) selected who stay on average 4 years (T = 4)

New Selection Test

SDy = $30,000, rv = 0.4, Z = 1.0, N = 10, T = 4

Costs per selected employee = £2,000

Calculate the total savings (larger figure = more saved) for the ‘old’ test and the ‘new’ test.

Should the old or the new test be used ?

Special Issues To what extent can the validity reported for a test in a manual be

applied to your selection context (Validity Generalisation & synthetic validity)

‘Real life’ issues not considered by ‘traditional’ utility analysis• Inflation (selection costs need to be adjusted)

• Variable costs of employees

• Taxation (of additional profits)

• Lost investment elsewhere

• Number of good candidates available?

• Do not overlook the old selection system (typically assume validity is zero).

• The real ‘worth’ of improved selection

Boudreau et al (1997)

Actual Hiring

Method

Applicant Ability to

Pool Predict

current aspects of VALUE

interventions new performance OF

intervention(s) INTERVENTION

Boudreau et al (1997)

Multivariate validity over time (the ‘Black Box’ of validity)

• temporal stability of validity

• multidimensional criteria (the nature of SDy) Using employee movement to represent multiple criteria The interaction between multiple predictors and multiple

criteria The ‘Black Box’ of Z Is maximising organisational performance (££££’s) always

an organisation’s main objective ?

Professional Qualifications in Test Administration

BPS ‘Certificates of Competence in Occupational Testing’

BPS website - for level A and level B (int.) ‘General Information Packs’ (http://www.bps.org.uk)

Is it worth me taking these additional courses? Course dates and costs - see MSc noticeboards Course content & structure

• Level A : Ability tests, aptitude tests, interest inventories

• Level B (Intermediate) : Hogan’s Personality Inventory (HPI)

• Both 3 days (9am-5pm), with pre- & post-course work