Assessment Centers and Situational Judgment Tests - …annex.ipacweb.org/library/conf/10/joiner.pdf · Session Title: Assessment Centers and Situational Judgment Tests Speaker Information:

Session Title:

Assessment Centers and Situational Judgment Tests

Speaker Information:

Dennis A. Joiner has specialized in supervisory and management assessment for employee selection, promotion and career development since 1977. He has developed assessment centers, oral examinations and various types of technical job knowledge and situational judgment tests for states, cities, counties, special districts and private sector organizations throughout the United States. He has authored articles and chapters on various aspects of assessment, was a member of the International Task Force on Assessment Center Guidelines which produced the 1989 and 2008 assessment center guidelines and has made numerous presentations at local, regional, national and international conferences. Dennis is an IPAC member, IPMA member, past-president of the IPMA Assessment Council, a founder, past president and current life member of the Personnel Testing Council of Northern California (PTC-NC), and was the 2003 recipient of the Stephen E. Bemis Memorial Award for outstanding contributions to the assessment profession.

Contact Information:

Dennis A. Joiner & Associates 4975 Daru Way, Fair Oaks, CA 95628 (916) 967-7795 [email protected]

Brief Presentation Summary:

This session will presented information on the current use of assessment centers and situational judgment tests for selecting supervisors and managers in the public sector. In September 2009 new “Guidelines and Ethical Considerations for Assessment Center Operations” were published in the International Journal of Selection and Assessment. Participants of this session received a copy of the new Guidelines and a presentation on the changes from the prior Guidelines. Several examples of efficient assessment center processes utilizing both live and video-based technologies were described. Also, many public agencies have begun using Situational Judgment Tests to select supervisors and managers for public safety and non-public safety jobs. These tests were also described as part of this session as will their relationship to assessment centers. The following pages provide a more detailed summary of the information presented through this session.

mailto:[email protected]

Page 2 of 6

ASSESSMENT CENTERS AND SITUATIONAL JUDGMENT TESTS1

Dennis A. Joiner Dennis A. Joiner & Associates Two very effective and well respected types of tests for selecting supervisors and managers are Assessment Centers (AC) and Situational Judgment Tests (SJT). This article takes a brief look at each of these types of tests with an emphasis on current trends. Assessment Centers The most significant recent development with ACs was the publication of new AC guidelines, officially titled “The Guidelines and Ethical Considerations for Assessment Center Operations” (International Task Force on Assessment Centers, 2009). This fifth edition of the AC guidelines continues to define what an AC is and what is not an AC in order to preserve the essential elements of the AC process which are believed to enhance and ensure the predictive accuracy of the process. According to the AC guidelines, “an assessment center consists of a standardized evaluation of behavior based on multiple inputs. Several trained observers and techniques are used. Judgments about behavior are made, in major part, from specifically developed assessment simulations.” The use of simulations that allow candidates to construct and demonstrate their response to job relevant situations is probably the most important core concept and what sets ACs apart from other types of testing processes. In addition to minor updates and revisions to most sections of the guidelines, the new AC guidelines include major changes in the section on Assessor Training (training content and length of training) and new sections on “Assessment centers for different purposes” and “Conducting assessment centers across cultural contexts.” 2 ______________ 1

This paper was adapted in large part from an article that appeared in the March 2010 issue of the PTC/MW Quarterly Newsletter, Vol. VI, No.1, pp 13-15 (Personnel Testing Council-Metropolitan Washington DC, www.PTCMW.org). That article provided summary of a presentation by the author titled, “Assessment Centers and Management and Supervisory Tests,” presented at the jointly sponsored IPAC/IPMA-HR International Training Conference held in Nashville, TN, September 12-16, 2009.

2

A complete copy of the new AC Guidelines can be obtained by e-mailing a request to the author at [email protected].

http://www.ptcmw.org/

mailto:[email protected]

Page 3 of 6

Most of the actual changes to ACs in recent years have been increases in the use of technology within the process (use of computers, use of the internet and use of video). A recent article in Public Personnel Management (Gowing, Morris, Adler and Gold, 2008) provides examples of 1. a telephone-based assessment program with multiple recorded phone based role plays; 2. a web-based “day in the life” assessment program with phone calls, e-mails, instant messages, voicemail, in-person talks; and 3. a video- AC consisting of 3 job simulations administered in a short time frame and scored later. Video-based ACs are the most common variation to the traditional live ACs. Video technology introduces a number of advantages to the assessment process. Video can be used to provide standardized content for candidates to respond to including video shot in the field that provides material that would otherwise be difficult to recreate for each candidate during the assessment process (e.g. major emergency incidents). Also, video is often used to capture the candidate’s responses for later scoring and to create a record of candidate performance that can be used to enhance candidate feedback and career development. Some video-based programs use video for both the stimulus material and for recording the candidates’ responses for later scoring. In these “100 percent video AC programs,” candidates view material (e.g. actors playing roles and asking questions) and then respond to the image on the video monitor during timed response periods. The candidate’s responses are captured by cameras positioned on or near the video monitor. Video assessment programs such as this result in substantial improvements in the standardization of the assessment process as compared to live programs. Another advantage is that the testing process can be completed much faster as long as the organization has multiple video testing rooms, equipment and qualified staff to run the exercises and equipment. The main disadvantages of the video assessment approach are: 1) the loss of true interpersonal interaction in the exercises which this author has found to consistently result in loss of information and lower assessment scores; and 2) an increased probability of equipment problems the more electronic equipment you add to the process. The lesson here is to build in lots of redundancy in the form of back-up cameras, extra stimulus DVDs, video monitors, camera operators, etc.

Guidelines and Ethical Considerations for Assessment Center Operations Major Changes

More guidance on Assessor Training content and the required

length of Assessor Training further clarified

Differences among Assessment Centers used for different

purposes: Selection vs. Diagnosis vs. Development

Global Assessment Center Practices: Multi-Cultural Issues and

Cross Cultural Issues

Page 4 of 6

Situational Judgment Tests In part due to the economic downturn effecting many organizations, due in part to the higher cost of conducting ACs and due in large part to their many attributes as tests, there has been an increase in the use of situational judgment tests. SJTs can be designed to measure many of the same leadership qualities ACs are used to evaluate. SJTs are sometimes referred to as written simulation tests or low-fidelity simulations. In this type of test candidates are given a series of situations. The situations are usually in written form consisting of one short paragraph. An alternate approach is to provide the situations/scenarios via video. Following each situation there are a number of possible courses of action (usually four choices). Candidates are asked to choose the best course of action and often also the worst course of action from the list of choices. Commonly the SJT instructions ask the candidates to put themselves into the situation and choose the action they would most likely take and the action they would least likely take in response to the scenario. Scoring of these SJTs can be accomplished by providing one point for each correct response or the various choices for each scenario item can be weighted. For example, the weighted response scoring approach can provide more points for choosing the responses keyed as the best choices and less points (or a reduction of points) for choosing the worst choice as the action the candidate would “most likely take” and/or for choosing the (pre-identified) best choice as the action the candidate would “least likely take.” Another variation of the SJT, often referred to as a “multiple-choice inbasket,” consists of providing the candidate with the situations, scenarios or issues via separate written items and then having the candidate respond to the same types of multiple choice questions as described above.

Video Assessment Center Advantages

Great for standardizing the exam process – Fairness

Great for preventing cheating – Exam Security

Lots of questions and answers and “presentation skills” observable

Video Assessment Center Disadvantages

No interpersonal interaction or human connection between candidates

and actors on the video or with the assessors

Lower scores achieved by candidates across all exercises

Possible equipment problems – Must build in redundancy

Common Variations of SJTs

Multiple-choice paper and pencil test

Video presented scenarios, multiple-choice response

Multiple-choice “Inbasket” - paper or computer

presented stimulus and questions

Page 5 of 6

The research on SJTs tends to be quite positive. In a Journal of Applied Psychology article authored by Michael A. McDaniel and others (2001), a meta-analysis based on 102 coefficients and 10,640 people resulted in an estimated population validity coefficient of .34. This research indicates that SJTs are good predictors of job performance and compares favorably with the validity coefficients cited for ACs. Sometimes SJTs are used as part of an AC and sometimes as a cost-efficient way to reduce the size of the candidate group moving on to the AC. In my research over the last few years I have found the correlation between supervisory and management SJT total scores and the AC total scores to range from .21 to .47 with small candidate groups of 13 to 20 (no correction for sample size or restriction of range). I have also found SJT scores correlate very well with supervisory and management AC exercises (above .30) and not at all or even negatively with police and fire emergency exercises. So, it appears that the SJTs are measuring what they are intended to measure - supervisory and management leadership skills. The consistently positive correlation between SJTs and ACs appears to support using SJTs to reduce the size of the candidate group that moves on to the AC. The SJT correlation with job performance lends some support for replacing ACs with SJTs. My first caution here would be to consider the primary differences between the two types of test processes. An AC allows candidates to provide a much more comprehensive response to scenarios/exercises within which the candidate must develop and implement his/her response. SJTs can cover a lot more content (as many as 50 or more job relevant scenarios); however, the candidate only needs to recognize and choose the best (and worst) response from the choices provided. A second caution would be to consider the content coverage carefully. As I have found with the lack of correlation with the SJTs and emergency exercises, you need to consider whether the

Why SJTs are used for supervisory and managerial positions:

Low-cost, efficient approach to assessing leadership, human

relations, supervisory and managerial competencies

Easily administered to any size candidate group

No reading list or candidate study time required (Quick List)

Require no human ratings and no hand scoring (save time and

costs associated with using interview or assessment panels)

Positive candidate feedback and high candidate acceptance

Custom keyed to the culture and needs of the organization

Good validity with low or no adverse impact as compared to

many other types of written tests

Page 6 of 6

SJT is covering all of the important content and contexts you want to evaluate in your testing process. If not, you may be able to add some of that content to your SJT or you may decide to supplement the SJT with one or more additional test components. Conclusion ACs and SJTs can both be very valuable testing components for assessing supervisory and management skills. They can be used together or used alone depending on the content you need to cover, the skills you need to assess and the many practical constraints you operate within. References Gowing, M.K., Morris, D.M., Adler, S. & Gold, M. (2008). The Next Generation of Leadership Assessments: Some Case Studies. Public Personnel Management, 37(4), 435-455. International Task Force on Assessment Center Guidelines (2009). Guidelines and Ethical Considerations for Assessment Center Operations: International Journal of Selection and Assessment, 63(3), 243-253. McDaniel, M.A., Morgeson, F.P., Finnegan, E.B., Campion, M.A. & Braverman, E.P. (2001). Use of situational judgment tests to predict job performance: A clarification of the literature. Journal of Applied Psychology, 86, 730-740. McDaniel, M.A. & Nguyen, N.T. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9, 103-113.

D E N N I S A . J O I N E R

A S S E S S M E N T S P E C I A L I S T

D E N N I S A . J O I N E R & A S S O C I A T E S

Assessment Centers and Situational Judgment Tests

Session Overview

This session will present information on the current use of assessment centers and situational judgment tests for selecting supervisors and managers in the public sector.

Main Topics:

Guidelines and Ethical Considerations for Assessment Center Operations

Some examples of efficient assessment processes

Using Situational Judgment Tests


The current Guidelines were endorsed on September 24, 2008 at the 34th International Congress on Assessment Center Methods in Washington, DC.

The Guidelines were published in the International Journal of Selection and Assessment, Vol. 17, No. 3, September 2009


History and Purpose

Why the 1st AC “Standards” were developed in 1975

Essential elements of Assessment Centers identified

Standards enhanced in 1979: more on assessor training and additional information on validation


Based on the 1975 and 1979 Standards: an Assessment Center includes:

Standardized Evaluation of Behavior by Multiple Trained Assessors using Multiple Assessment Techniques

At least one Job Simulation Exercise (usually more)

An Overall Evaluation Session separate from behavior observation with Final Results based on a Pooling of Information from Assessors and Techniques

Dimensions, attributes, characteristics or qualities evaluated are based on analysis of job relevant behaviors


1989 Major Changes

Name Change to “Guidelines” vs. “Standards”

Consensus/Clinical or Statistical Scoring allowed

What should be assessed: dimensions, attributes, characteristics, aptitudes, skills, abilities, knowledge, tasks or exercises?


2000 Main Changes

What should be assessed: “dimensions” or “competencies”

Integration of more technology advances into AC processes –computers, audio and video recording of behavior

AC must include “at least one, and usually several, job-related simulations that require the assessee to demonstrate a constructed behavioral response.”

More delineation on Assessor Training . . .

Use of Computers in ACs

Candidates - Computer Use for exercises

Assessors - Rating Form Templates/Checklists

Staff - Spreadsheets for Mechanical Scoring

Use of Video

Stimuli for Candidate Response

Video of Candidates’ Responses Scored Later

Total Video Programs


2008 Major Changes

More guidance on Assessor Training content and the required length of Assessor Training further clarified

Differences among Assessment Centers used for different purposes: Selection vs. Diagnosis vs. Development

Global Assessment Center Practices: Multi-Cultural Issues and Cross Cultural Issues

Efficient Assessment Processes

Typical Public Sector Assessment Center

6 Assessors trained for one day

8-10 Candidates (per day of assessment)

Three or four Job Simulation Exercises

Possibly 1 additional day for Assessors to integrate or finalize all scores

More than 40 candidates, add more Assessors and more training or have Assessors specialize in one exercise and drop integration day(s)


The Next Generation of Leadership Assessments: Some Case StudiesAn Article by Marilyn K. Gowing, PhD, David M. Morris, PhD, JD, Seymour Adler, Ph D, and Mitchell Gold, PhD, in Public Personnel Management, Vol. 37 No. 4 Winter 2008, pages 435-455

This recent article, provides a brief overview of leadership theory and research, some background on the traditional assessment center process, and then describes three different “streamlined” approaches to leadership assessment.


The Next Generation of Leadership Assessments: three approaches to assessment . . .

Telephone Assessment Program (TAP)90 minute to 3 hour long “live simulations”: multiple recorded phone-based role plays

LEADeR (a web based assessment program)3.5 hour long “day in the life”: phone calls, e-mails, IMs, VM, and in-person talks

Video-based Assessment CentersTypically 3 job-simulation exercises administered in short timeframe and scored later

Good article, good overview of 3 new AC models. Available at IPMA-HR.org


Example of a Video AC with 72 to 80 Candidates

Three exercises based on current Job Analysis

Same 1½ days for All Candidates to complete AC

Same 6 Assessors evaluate each candidate (later)

Same 2 Assessors specialize in each exercise using the same standards for all candidates

Same Questions asked same way (on video monitor)

Same exact time allowed for answering each question

All responses on videotape: 2 cameras in 8-10 rooms


Video Assessment Center Advantages & Disadvantages

Great for standardizing the exam process – Fairness

Great for preventing cheating – Exam Security

Lots of questions and answers and “presentation skills” observable

No interpersonal interaction or human connection between candidates and video actors or assessors

Lower scores achieved across all exercises

Equipment Problems – Must build in redundancy

Part Two of Presentation:Situational Judgment Tests

What are Situational Judgment Tests (SJTs)?

Why are SJTs used?

Variations in SJT formats

Situational Judgment Test Item - Example

You have noticed that there is a lack of cooperation and trust among the employees in the organization you manage. This counter productive behavior is adversely affecting both the quality and the quantity of the work produced by the employees. You would . . .

A. Inform the employees that failure to cooperate and work harmoniously will result in disciplinary action.

B. Bring in someone to do some team building with the employees.

C. Meet with the employees to discuss your observations andidentify the cause of the interpersonal problems.

D. Change some of the work assignments in order to attain a more cohesive and cooperative work group.

1. Most Likely _____ 2. Least Likely _____

Situational Judgment Tests

Why SJTs are used for supervisory and managerial positions:

Low-cost, efficient, standardized approach to assessing human relations, leadership, supervisory and managerial competencies

Easily administered to any size candidate group

No reading list or candidate study time required (Quick List)

Require no human ratings and no hand scoring (save time and costs associated with using interview or assessment panels)

Result in positive candidate feedback and high candidate acceptance

Custom keyed to the culture and needs of your organization

Good validity with low or no adverse impact as compared to many other types of written tests

Situational Judgment Tests

Common Variations of SJTs

Multiple-choice paper and pencil test

Video presented scenarios, multiple-choice response

Multiple-choice “Inbasket” - paper or computer presented stimulus and questions

Situational Judgment Test Spinoff

The “Multiple Situations” Exercise:

Some Situations Require a Written Response

Some Situations Require an Oral Response with Preparation Time

Some Situations Require an Immediate Oral Response without Preparation Time

You Can Also Incorporate a Multiple-Choice Format SJT into your Assessment Center as a Weighted Component

Situational Judgment Test Spinoff

The Multiple Situations Exercise: Pros and Cons

Advantage = Covers Multiple Types of Critical Job Tasks through the use of Multiple Scenarios

Disadvantage = Not a True “Simulation” of the Job since Candidates Describe What They “Would” Do vs. Demonstration of How They Would Do It

Issue is Breadth of Exercise vs. Depth

Session Summary Thoughts

An Assessment Center is a Structured Evaluation of Job Relevant Worker Characteristics/Dimensions

All of the Assessment Center / Job Simulation Technology and Methods are available for improving your Selection/Promotion and Career Development programs

Just don’t call it an “Assessment Center” if it is not!

Situational Judgment Tests can also be used to improve the accuracy of your selection of Supervisors and Managers.

For More Information Contact:

Dennis A. Joiner

Assessment Specialist

Dennis A. Joiner & Associates

4975 Daru Way

Fair Oaks, CA 95628

USA

Phone: (916) 967-7795

Email: [email protected]

Documents

Assessment Centers and Situational Judgment Tests - …annex.ipacweb.org/library/conf/10/joiner.pdf · Session Title: Assessment Centers and Situational Judgment Tests Speaker Information: