chapter 10 Assessing via Tests 1

III Choosing the Right MethodChapter 10 Assessing Via Tests

Paper & Pencil

Work Sample

Situational Judgment (SJT)

Computer Adaptive

chapter 10 Assessing via Tests 2

Terminology

Test

“..objective and standardized procedure for measuring a psychological construct using a sample of behavior”

What are some examples?

Construct

“…fairly well developed idea of a trait.”

Most KSAOs

Kerlinger’s definitions

Give an example of a construct and operational definition using Kerlinger’s definitions –that’s IO related

chapter 10 Assessing via Tests 3

More terms

Standardization

“…controlling conditions and procedures so scores among different people are comparable.”

What makes an interview a test rather than just an interview?

Does a test require a passive or active response (behavior) or can it be either?

chapter 10 Assessing via Tests 4

NORM-REFERENCED AND DOMAIN-REFERENCED TESTING

What’ the difference between the two?

(domain-referenced is also call criterion-referenced)

What are some examples (or situations) where the purpose would require one or the other? (thought question)

Hint: when criticality of performance is important or

where characteristics of the applicant pool may vary geographically or over time.

Which type requires statistical (correlational) and which type requires judgmental validation?

chapter 10 Assessing via Tests 5

TRADITIONAL COGNITIVE TESTS

Discrimination and generalization

i.e. recognize or discover relationships

Know, perceive, remember, understand, cognitive manipulation

Problem solving, evaluation of ideas

Compare and contrast

IQ, GMA, Cognitive ability

Also can be considered aptitude and/ or achievement

e.g. WPT with NFL players

(do you think scores are related to performance?)

chapter 10 Assessing via Tests 6

Cognitive Ability

• Off the shelf

WPT

WGCTA

DAT (Differential aptitude Test)

What is a job you would use each of them for?

What are the relative advantages and disadvantages for developing homemade tests vs. commercially available ones?

chapter 10 Assessing via Tests 7

PERFORMANCE TESTS

How can they be both criteria and / or predictors? Give examples.

Work Samples and Simulations “standardized abstraction of the work”

Give hypothetical examples of a high and low fidelity simulation

Developing Work Samples to measure “proficiency”

Would all or some of the tasks be measures for (1) selection v. (2) criterion?

Situational Judgments Can be either multiple choice or video (higher fidelity)

With incremental validity over GMA, Personality, job experience (Chan & Schmitt, 02)

chapter 10 Assessing via Tests 8

Non-Cognitive Performance

Physical Abilities

See recent PTC newsletter on legal issues

Sometimes its difficult to set valid cut scores

Fitness Testing

a two edged sword for setting standards and not doing so. What is the legal conundrum here?

Should fitness testing be required for some jobs on a daily basis?

Should drug testing be performed for some jobs and not other?

Sensory & Psychomotor Proficiencies

chapter 10 Assessing via Tests 9

COMPUTERIZED TESTING

What is a stenographer? Where did they go?

What are some psychometric problems with mass testing across different geographical locations that CAT can help with?

How can IRT help?

Computerized Adaptive Testing (CAT)

What’s the difference between linear testing and branching algorithms?

chapter 10 Assessing via Tests 10

ISSUES AND CONTROVERSIES

Setting Cut Scores –rarely recommended but often necessary:

Civil service

Licenses & certification

Cyclical hiring (needs forecasting estimates, e.g. teachers)

Sequential (for multiple hurdle)

Predicted Yield Model (Thorndike, ‘49)

Fluctuation of candidate qualifications varies

availability of openings varies

Depends upon accurate forecasting (history and research)

What are some consideration ?

Regression-Based Methods (see fig 10.2 p254)

chapter 10 Assessing via Tests 11

Issues and Challenges

Translations of Psychomotor Instruments

Cross cultural testing a challenge:

Different approaches to testing

Test administration problems

Score equivalence

Considerations:

Use IRT to establish item equivalence

Develop “global” measures (Schmitt, Kihm & Robie, 2000)

What is this global thing?

chapter 10 Assessing via Tests 12

Tests and Controversy

Do we need licensing tests to establish credentials?

Do we need educational proficiency exams?

We will always assess; either

With tests (more objective)

With subjective judgment