Seven Steps for Developing a Valid Paper-and-Pencil Written Promotional Examination Using Content Validation Stacy L. Bell, M.S. Executive Vice President

Seven Steps for Developing a Valid Paper-and-Pencil Written Promotional

Examination Using Content Validation

Stacy L. Bell, M.S.Executive Vice PresidentFire & Police Selection, Inc.August 23, 2007

Presented at: Fire Rescue International

August 2007

Copyright © 2007 Fire & Police Selection, Inc. www.fpsi.com

Presentation Overview

The presentation will:

Focus on requirements of the 1978 Federal Testing Guidelines for developing a job-related written test.

Focus exclusively on job knowledge written tests used for promotion.

Focus on the steps necessary to use content validation to support the job knowledge written test.

The presentation will not:

Provide a comprehensive review of litigated cases involving job knowledge written tests used for promotion.

Provide legal advice on handling a litigation.

Provide a prescription/recipe strategy that can be used for all circumstances.


Proceed with Caution…

Based on the size and type of your employer, the

following agencies or people can initiate litigation

against your testing practices:

Equal Employment Opportunity Commission (EEOC) Department of Justice (DOJ) Office of Federal Contract Compliance Programs

(OFCCP) (under DOL) Private Plaintiff Attorney


Examples of Litigated Promotional Written Test Processes

Bouman v. Block, 940 F2d 1211 (9th Cir. 1991).

Brown v. City of Chicago, 8 F.Supp.2d 1095 (N.D.I11. 1998).

Hearn v. City of Jackson, Miss. 110 Fed. 424 (5th Cir. 2004).

Isabel v. City of Memphis, F.Supp.2d, 2003 (6th Cir. 2003).

Paige v. State of California, 102 F.3d 1035, 1040 (9th Cir. 1996).


Civil Rights Litigation—See 4D of Uniform Guidelines & Civil Rights Act of 1991

Amends Section 703 of the 1964 Civil Rights Act (Title VII)(k)(1)(A). Anunlawful employment practice based on disparate impact is establishedunder this title only if: A(i) a complaining party demonstrates that a respondent uses a

particular employment practice that causes a disparate impact on the basis of race, color, religion, sex, or national origin, and the respondent fails to demonstrate that the challenged practice is job-related for the position in question and consistent with business necessity; OR,

A(ii) the complaining party makes the demonstration described in subparagraph (C) with respect to an alternate employment practice, and the respondent refuses to adopt such alternative employment practice.


Right Test…Wrong Use

Brown v. City of Chicago, 8 F.Supp.2d 1095 (N.D.I11. 1998). The City of Chicago police promotional written test used for

lieutenant resulted in gross disparate impact against minorities.

An alternative employment practice was approved whereby the City suggested combining merit promotions with the rank-order promotions. The City of Chicago failed to incorporate this alternative employment practice (AEP) into its practices.

Despite the overwhelming need for more minority representation, the City made promotions based on a rank-order list.


Successfully Upheld Promotional Written Test Challenges

Hearn v. City of Jackson, Miss. 110 Fed.Apprx. 424 (5th Cir. 2004).

Black police officers brought action under Title VII alleging that selection of individuals for sergeant position discriminated against black officers.

Plaintiffs argued that all applicants should have been allowed to proceed through all three stages of the recruitment process rather than be cut from the written test which had adverse impact.

Court ruled that the City supported the appropriateness of the written test through a job analysis and that the written test was a legitimate and cost effective method to screen out candidates on the basis of weakness in technical and legal knowledge required on the job.

Isabel v. City of Memphis, F.Supp.2d, 2003 (6th Cir. 2003).

Sergeants in the Memphis Police Department brought action under Title VII alleging that selection of individuals for lieutenant position discriminated against Blacks.

City had negotiated to use a 70% cutoff for the test which had disparate impact against Blacks.

Consultant recommended using the SEM to reduce the 70% cutoff which resulted in no disparate impact against Blacks.

Court ruled in favor of the City for using a job-related and valid written test and for lowering the cutoff score using the SEM to reduce the disparate impact against Blacks.


Disclaimer

The steps outlined in this presentation are based on the requirements outlinedby the Uniform Guidelines (1978), the Principles, and the Standards.

The proposed model is not intended to be a one-size-fits-all model, but rather ageneric template which could be employed in an ideal setting.

While we cannot guarantee that by following these steps you will avoidlitigation at all cost, implementing the practices outlined in this presentationwill greatly increase your likelihood of success in the event of a challenge toyour promotional written testing process.


Step #1: Conduct a Job Analysis

Step #2: Develop a Selection Plan

Step #3: Identify Test Plan Goals

Step #4: Develop the Test Content

Step #5: Validate the Test

Step #6: Compile the Test

Step #7: Post-Administration Analyses

7 Steps for Content Validation of a Job Knowledge Written Test for Promotions


Step 1: Conduct a Job Analysis

1-2 page job descriptions are ALMOST NEVER SUFFICIENT for showing validation under the Guidelines…unless (at a bare minimum!): Importance ratings SME input Operationally defined KSAs Duty/KSA linkages

In 90% of circumstances where validity is required, we find that new, updated

job analyses need to be developed.

A house is only as strong as the foundation.


What is in a Job?


The General Process

The JobTasks

or Duties

KSAs

Job

/Desk A

ud

its

SMEs Review and Provide Ratings


Who are “SMEs”?


Rating of Job Tasks/Duties

Task/Duty


Rating KSAPCs


Closing the Loop

The Job

Tasks or

DutiesKSAs

Assessment questions are developed or

selected based on their relationship

to KSAs

SMEs Link Those KSAs that are Important AND

Required at Entry to each Important and Frequently

Performed Job Task


Creating a Uniform Guidelines Style Job Analysis

According to the Guidelines, Job Duties include at least:

Frequency (15B3, 14D4) This duty is (Select one option from below) by me or other active (xxxx) in my department.

1 not performed2 performed less than yearly3 usually performed quarterly to yearly4 usually performed monthly to quarterly5 usually performed daily to weekly6 usually performed several times a day

Importance (14B2, 14C1,2,4; 14D2,3; 15B3, 15C3,4,5; 15D3) Competent performance of this duty is (Select one option from below) for the job of (xxxx) in

my department.1 not required2 not important – Trivial or minor significance to the performance of the job.3 important – Helpful, and/or meaningful to the performance of the job.4 critical – Necessary for the performance of the job.5 extremely critical – Necessary for the performance or the job with more extreme

consequences



According to the Guidelines, Job Duties include at least:

Differentiating “Best Worker” Ratings (14C9) (Content Validity only) Above-minimum performance of this duty makes (Select one option

from below) in the overall job performance for the job of (xxxx) in my department.

1 little or no difference

2 some difference

3 a significant difference

4 a very significant difference



According to the Guidelines, KSAPCs include at least:

Frequency (15B3, 14D4) This KSAPC is (Select one option from below) by me or other active (xxxx) in my

department.1 not performed2 performed less than yearly3 usually performed quarterly to yearly4 usually performed monthly to quarterly5 usually performed daily to weekly6 usually performed several times a day

Importance (14B2, 14C1,2,4; 14D2,3; 15B3, 15C3,4,5; 15D3) This KSAPC is (Select one option from below) for the job of (xxxx) in my department.

1 not required2 not important – Trivial or minor significance to the performance of the job.3 important – Helpful, and/or meaningful to the performance of the job.4 critical – Necessary for the performance of the job.5 extremely critical – Necessary for the performance or the job with more extreme

consequences



According to the Guidelines, KSAPCs include at least:

Differentiating “Best Worker” Ratings (14C9) (Content Validity only) Above-minimum performance of this KSAPC makes (Select one option from below) in the

overall job performance for the job of (xxxx) in my department.1 little or no difference2 some difference3 a significant difference4 a very significant difference

When Needed (5F; 14C1) Use one of the following ratings to indicate when this KSAPC is needed for the (xxxx)

position in your department.1 This knowledge, skill, ability, or personal characteristic is fully developed on the job.2 This knowledge, skill, ability, or personal characteristic is generally required at entry,

but is developed on the job (that is, it is more fully developed or learned through job training or job experience).

3 This knowledge, skill, ability, or personal characteristic is fully required at entry (that is, a firefighter in my department is required to have already learned or developed this before they start the job).



Mastery Level Scale

This level of job knowledge held must be at a (Select one option from below) level for successful job performance.

1—Low: none or only a few general concepts or specifics available in memory in none or only a few circumstances without referencing materials or asking questions.

2—Familiarity: have some general concepts and some specifics available in memory in some circumstances without referencing materials or asking questions.

3—Working knowledge: have most general concepts and most specifics available in memory in most circumstances without referencing materials or asking questions.

4—Mastery: have almost all general concepts and almost all specifics available in memory in almost all circumstances without referencing materials or asking questions.

The data from these ratings are useful for choosing the job knowledges that should be included in a writtenjob knowledge test. We suggest only including job knowledges that have average ratings of 3.0 or higher onwritten job knowledge tests. See Section 14C(4) of the Guidelines for specific requirements aboutmeasuring job knowledge in a testing process.



Duty/KSA Linkages (14C4, Content Validity only)

This KSAPC is ________ to the performance of this duty.

1 not important

2 of minor importance

3 important

4 of major importance

5 critically important


Content Validity Essentials

Demonstrated by data that the content of a selection procedure is representativeof important aspects of performance on the job

The content of the “test” is representative of the content of the job.

Cannot alone support inferences about mental processes.

Cannot measure traits (such as intelligence, aptitude, personality, commonsense, judgment, leadership, spatial ability).

Not appropriate for those KSAs that will be expected to be learned on the job.


Examples of a Strong and Weak KSAPC

Example of a weak KSAPC: Knowledge of ventilation practices.

Example of a strong KSAPC: Knowledge of ventilation practices and techniques to release

contained heat, smoke, and gases in order to enter a building. Includes application of appropriate fire suppression techniques and equipment, e.g., manual and power tools and ventilation fans.


Step 2: Develop a Selection Plan

Review the KSAPCs from the job analysis

Select only those KSAPCs that meet the following criteria: Required at the time of hire—Needed on day one Important or critical (necessary) for the performance of the job Mastery level of knowledge (e.g., knowledge must be memorized

rather than looked up)

Of the KSAPCs that meet this criteria, determine the best way to measure

the KSAPC (e.g., on a written test, in a physical ability test, in a structured

oral interview, etc.).

Concentrate only on those items that are best measured with a written test.


Sample KSAPC Measurement Survey

Type of Selection Device Used to Measure KSAPCKSAPC WT PAT AC SIP

Knowledge of hose practices and techniques to supply water in proper quantity to the fire combat area with adequate pressure. X

Verbally communicate in English to explain fire fighting techniques and procedures to others and to receive information and instruction. X X

Effective teamwork techniques, including skills to accept, understand and follow orders, to work and live harmoniously with co-workers and to consistently follow through on routine or specially assigned duties. X X

Knowledge of building design, construction and material characteristics including such aspects as fire walls and partitions, materials used, and roof types. X

Strength sufficient to lift, load, and unload patient and/or victim on gurney. X


Step 3: Identify Test Plan Goals

Identify the test sources. Review relevant job-related materials and discuss the

target job in considerable detail with the incumbents. This will provide you with an initial understanding of the type of work performed and help you focus your attention on job specific information for the job class under analysis.

Review the knowledges that meet the necessary criteria and determine which source and/or textbooks are best suited to measure the various knowledges.

Ensure that the sources do not contradict one another in content.


Preparatory Materials Offered to Candidates?

Ensure that materials are:

Distributed to all candidates taking the test At the same time

Current

Specific


Preparatory Sessions Offered to Candidates?

Use of mini-academies has been shown to increase overall testPerformance and reduce adverse impact.

Be cautious of administering too many mini-academies as researchsuggests conducting multiple mini-academies may actually result inincreasing adverse impact.

Try to schedule a mini-academy at a location that is geographicallyconvenient to all candidates and is offered at a reasonable time of day.

Invite all candidates to attend the mini-academy and provide plenty ofnotice of the date and time.


Identify the Number of Test Items

Include enough items to ensure high test reliability. A number of factors impact test reliability, but perhaps the single most

important factor is the number of test items per relevant KSAPC in the test overall.

Consider using SME input to determine internal weights for the writtentest.

SMEs to be provided with the list of KSAPCs to be measured and asked to assign 100 points among the KSAPC.

Ensure adequate sampling of KSAPCs. A sufficient number of items should be developed to effectively measure

each KSAPC at the desired level. Note that some KSAPCs will require more items than others for making a

“sufficiently deep” assessment of the levels held by the candidates.Ensure proportional sampling of KSAPCs.

The test should be internally weighted in a way that insures a robust measurement of the relevant KSAPCs.


Determine the Type of Test Items

Following the determination of the length of the test and the number of items tobe derived from each source, a test plan was developed. The use of a processby-content matrix ensures adequate sampling of job knowledge content areasand problem-solving processes. Problem-solving areas involve the following

A. Knowledge of terminology. B. Understanding of principles. C. Application of knowledge to new situations

While knowledge of terminology is important, the understanding and applicationof principles are considered to be of primary importance. This is reflected in therecommendation that a majority of the items should involve the application ofknowledge and understanding of principles. Furthermore, not all documents areequally well suited for each of the three problem-solving processes. Therefore,the manner in which each is sampled takes this into consideration.


Process-by-Content Matrix

PROCESS-BY-CONTENT MATRIX—POLICE SERGEANT

SOURCE DEF PRINC APP TOTAL 1. Essentials of Modern Police Work 4 10 20 34 2. Community Policing 3 7 13 23 3. Rules of Evidence 3 10 17 30 4. Department Rules & Regulations 1 3 6 10 5. State Criminal Code 4 5 9 18 6. State Vehicle Code 4 6 10 20 7. City Ordinances 2 2 6 10 8. Performance Appraisal Guidelines/ 0 1 1 2

Employee Ratings 9. Labor Agreement with the City 0 1 2 3

Total 21 45 84

150


Consider Bloom’s Taxonomy

The use of Bloom’s Taxonomy (1956) is very helpful when writing

items intended to measure knowledges at various levels.

SMEs should consider how the knowledge is applied on the job

e.g., factual recall, application, analysis, etc.

Often times the higher levels of Bloom’s are best measured in a

testing format other than a written test (e.g., an interview, an

essay, etc.).


Consider Bloom’s Taxonomy


Step 4: Develop the Test Content

Select a diverse panel of 4-10 SMEs (use a minimum of one year experience).

Review the selection plan to ensure that the SMEs understand the parameters.

Have SMEs sign a “Confidentiality Form.”

If SMEs are developing test items, train them on item-writing.

SMEs to write sample items to be reviewed by other SMEs on the panel.

SMEs to write test items using the “Validation Survey” form.

SMEs to review and edit other SMEs items.

Ensure proper grammar, style, and consistency. Ensure that the selection plan and test plan requirements are met. Ensure that all items meet the criteria set forth on the “Validation Survey” form.

Create the final test version for the panel of SMEs to review.


Step 5: Validate the Test

Using the “Validation Survey” form, SMEs will assign various ratings to the items in thetest bank including:

Ratings on the quality of each test item Ratings on the job-relatedness of each test item Ratings regarding the appropriate level of each test item (i.e., difficulty) Ratings to ensure that the test items are based on current information Ratings to determine if the item measures an aspect of job knowledge that

must be memorized Ratings to determine the consequence of error if the applicants does not

possess the knowledge required to answer the itemIdentify an appropriate time limit.

A common rule-of-thumb used by practitioners to determine a written test cutoff time is to allow one minute per test item plus thirty additional minutes (e.g., a 150-item test would yield a three hour time limit).

A reasonable time limit would allow for 95% of the candidates to complete the test within the time limit.


#1 #2 #3 #4 #5 #6 #7 #81. Does the question STEM:

A. Relate to a skill, ability, or personal characteristic in the job analysis? Y/N Y N Y N Y N Y N Y N Y N Y N Y NB. Relate to a job duty on the job analysis? Y/N Y N Y N Y N Y N Y N Y N Y N Y NC. PROVIDE ENOUGH CONTENT to simulate a response? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

2. Are the DISTRACTORS: A. Similar in difficulty? Y/N Y N Y N Y N Y N Y N Y N Y N Y N B. Distinct? Y/N Y N Y N Y N Y N Y N Y N Y N Y N C. Incorrect, yet plausible? Y/N Y N Y N Y N Y N Y N Y N Y N Y N D. Similar in length? Y/N Y N Y N Y N Y N Y N Y N Y N Y N E. Correctly matching to the stem? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

3. Is the key CORRECT IN ALL CIRCUMSTANCES? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

4. Does the question PROVIDE CLUES to other questions? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

5. Is the question UNNECESSARILY DIFFICULT or COMPLEX? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

6. What percent of MINIMALLY-QUALIFIED APPLICANTS would you expect to answer the question correctly?

7. Is the question FAIR to all groups of people? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

8. Is the question BASED ON CURRENT INFORMATION? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

#'s fromJob Anal.

#'s fromJob Anal.

11. Is the question a GOOD MEASURE of the knowledge being assessed? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

12. Is the knowledge measured NECESSARY THE FIRST DAY on the job? Y/N Y N Y N Y N Y N Y N Y N Y N Y N

13. How important is it that the knowledge tested be MEMORIZED?0 - NOT NECESSARY--can be looked up without impacting job performance1 - IMPORTANT--negative job impact is LIKELY if it had to be looked up2 - ESSENTIAL--negative job impact is MOST LIKELY if it had to be looked up

14. How serious are the CONSEQUENCES if the applicant does not possess the knowledge required to answer this question correctly?

0 - LITTLE or NO consequences1 - MODERATE consequences2 - SEVERE consequences

10. What JOB KNOWLEDGE is being measured by the question?

9. What JOB DUTY is represented by the item?

0-2

0-2

%

Name: Test Question Number (please fill-in the question number below)


Conclusion: UGESP, Standards, Principles, and Court Cases

UNIFORM GUIDELINES (EEOC, 1978)

STANDARDS (APA, 1999)

PRINCIPLES (SIOP, 1987

and 2002 draft)

COURT CASES

Validation Survey Compliance

Validation Survey Questions #1-5 Chp. 1, p. 11 2002 Draft, p. 8

(1), (2)

Validation Survey Question #6 Sect. 5H, 14C7, 15C7

Chp. 14, p. 157 2002 Draft, pp. 73, 81

(1), (2)

Validation Survey Questions #8-10 Sect. 14C4-5, 15C4-5

Standard 14.9-10

(1), (2)

Validation Survey Questions #11-14 Sect. 14C1, 5F Standard 14.17 2002 Draft pp. 16, 79

(1), (2)

(1) U.S. v. South Carolina

(2) Contreras v. City of Los Angeles


Step 6: Compile the Test

Analyze the “Angoff” ratings identified by SMEs.

Ensure inter-rater reliability amongst the raters.

Discard raters whose ratings are statistically different from

other raters.

Calculate the overall difficulty level of the test.


Step 7: Post-Administration Analyses

Conduct an item-level analysis to: Identify point biserials (quality/effectiveness) Identify item difficulty (effectiveness/utility) Identify differential item functioning (fairness)

Conduct a test-level analysis to: Identify descriptive test statistics Analyze psychometric statistics

Ensure test reliability

Calculate “Conditional” Standard Error of Measurement

Documents

Seven Steps for Developing a Valid Paper-and-Pencil Written Promotional Examination Using Content Validation Stacy L. Bell, M.S. Executive Vice President