View
1
Download
0
Category
Preview:
Citation preview
EEI’s SO/PD II Employment Test Battery
Midwest Energy AssociationElectric Operations Technical & Leadership Summit
May 13 – 15, 2015
Background
EEI’s System Operator/Power Dispatching employment test battery has been in use since the 1980’s
Over time, a number of changes have occurred in how the jobs are performed and staffed:
New technologies
Increased job demands competing for employees’ attention
Impact of increased regulatory compliance requirements (FERC, SERC, NERC, etc.)
Changing applicant pools and increased hiring projections
In 2010, plans were initiated to develop a computer-based test to address the above changes
Benefits
Capability to measure job candidates’ multitasking ability and the impact of stress on job performance;
Inclusion of recent methodologies that are likely to have less adverse impact against protected classes of job applicants;
Computer-based test battery that better reflects the job environment than a paper-based test battery and facilitates test administration and scoring; and
Inclusion of Gas Controller jobs for combination electric and gas companies.
Intended Jobs
The new test battery will be used for hiring in three SO/PD functional areas
Distribution – monitoring and troubleshooting the distribution system including the dispatching of field crews
Transmission – monitoring, regulating, and troubleshooting the transmission system including substations
Generation/Balancing Authority – monitoring and regulating the energy supply to meet load and interconnect sales commitments
It will also be validated for use in hiring Natural Gas Controllers
SO/PD II Project Phases
Three main phases:
Job Analysis – included a review of job documents, development of a task list, job observation, and survey to identify the Knowledge, Skills, and Abilities (KSAs) critical to the job
Test Development – based on the information collected in the job analysis phase, three new tests were developed and two were chosen from EEI’s existing library
Validation – evaluate the relationship between scores on the test and performance on the job
Phase 1: Job Analysis
Detailed analysis of three SO/PD functional areas as well as Natural Gas Controller
Reviewed hundreds of job descriptions
Conducted job observations at multiple sites
Conducted interviews with job incumbents
Developed a comprehensive task list with input from over 30 SO/PD & NGC supervisors, managers, trainers, and incumbents
Administered industry-wide Job Analysis Questionnaire to over 1,400 incumbents and supervisors to determine which tasks are critical to success and what characteristics are required to perform them
Phase 2: Test Development
EEI and contractor evaluated what types of selection tests would be most effective at measuring the identified KSAOs
Two new tests were developed and two existing EEI tests were selected for inclusion
Analytic Thinking Skills Test (ATS)
The test contains three sections: Argument, Problem Solving, and Logic-based Reasoning.
Argument – consists of passages that present an argument and some evidence supporting the argument followed by a question that asks the test taker to draw a conclusion based on the evidence.
Problem-Solving – consists of passages followed by a set of facts that the test taker must use to determine the answer.
Logic-based Reasoning – consists of passages that present some facts about a specific topic followed by several statements that represent inferences one might draw. The test taker must decide if the statement is True, False, or Indeterminable.
Analytic Thinking Skills Test (ATS)
Example Argument Question
B is the correct answer
Our IT department has noticed an increase in the number of computer viruses. Please be careful about opening e-mails from anyone you do not know.
The argument above depends on which of the following assumptions?
A. Computer viruses being transmitted through e-mails are particularly damaging.
B. E-mails sent from unknown parties are more likely to contain computer viruses.
C. Employees are more likely to receive e-mails from unknown parties than from people they know.
D. The company’s virus protection software is not effective against viruses transmitted through e-mails.
E. Computer viruses are only transmitted through e-mails.
Analytic Thinking Skills Test (ATS)
Example Problem-Solving Question
B is the correct Answer
A project must be completed by performing tasks in order. There are five tasks: Task1, Task2, Task3, Task4, Task5.
• Task1 and Task4 must be completed consecutively, but only after completing Task5.• Task3 cannot be completed before Task1.• Task4 depends on the completion of Task2.
Based on the above information, which of the following is a correct ordering of the project?
A. Task2, Task5, Task1, Task3, Task4B. Task5, Task2, Task4, Task1, Task3C. Task1, Task2, Task3, Task4, Task5D. Task2, Task1, Task4, Task5, Task3E. Task3, Task5, Task1, Task4, Task2
Analytic Thinking Skills Test (ATS)
Example Logic-Based Reasoning Question
“Indeterminable” is the correct answer
Explosives are substances or devices capable of producing a volume of rapidly expanding gases that exert a sudden pressure on their surroundings. Chemical explosives are the most commonly used, although there are mechanical and nuclear explosives. All mechanical explosives are devices in which a physical reaction is produced, such as that caused by overloading a container with compressed air. While nuclear explosives are by far the most powerful, all nuclear explosives have been restricted to military weapons.
From the information above, it can be validly concluded that:
All mechanical explosives have been restricted to military weapons. (T/F/I)
Multitasking Simulation (MTS)
Test takers must complete four separate tasks, each presented in a separate quadrant of the screen simultaneously
Memorization task – candidates must memorize a series of short alphanumeric codes and then recognize them when prompted.
Basic math skills task – candidates solve addition problems.
Visual monitoring task – candidates must monitor a needle on a gauge as it moves from the center to the left or right and stop it before it reaches the bottom of the gauge.
Listening task – candidates periodically hear low and high tones and must click a button to identify the type of tone.
Existing EEI Tests
The results of the job analysis were reviewed to determine any additional target areas for assessment
Basic math and verbal ability were identified as needed at entry
Existing tests in EEI’s library were reviewed and Math Usage and Reading Comprehension were identified for inclusion in the test battery
Phase 3: Validation Study
About 800 incumbents took the new test battery and their job performance was rated by their supervisors
Data were analyzed to evaluate the effectiveness of the test battery in predicting job performance
The relationship between scores on the new test battery and job performance were found to be statistically significant indicating that the SO/PD II test battery is an effective predictor of job success.
Implementation
The SO/PD II test battery was rolled out on April 30th, 2015 for companies to begin using in their hiring process
Practice tests are available to companies by subscription through EEI
For more information about the test or to discuss using it in your company, contact Amanda Allen at EEI (aallen@eei.orgor 202-508-5153).
Recommended