Dealing with Large Applicant PoolsDr. Frank Olmos and Joshua KahnLos Angeles County Office of Education

PTC-SC Luncheon PresentationMonterey Park, CA, May 29, 2013

Let’s do the math…

2

High unemploymentX

- Reduced Public Sector Budgets

Internet Recruiting Broad Outreach

+ Online Applications Low-effort applying Reduced Self-filtering

= Large Applicant Pools

= Workload Problem for many merit systems

Strategies to reduce pool

Limit exposure▪ Limit posting sites (no Monsters)▪ Paper bulletins only▪ Shorten posting period▪ Do not accept “interest cards”

Limit application window▪ Open for one day or hour▪ Posting cut-off at x-number of applicants

(first come first serve for testing slots, self-schedule)

Increase burden to apply▪ Paper app, hand pickup and deliver▪ Charge $ per application ($5 is legal)▪ Increase requirements for complete app

(proof of diploma, degree, transcripts) Random selection

▪ Lottery

3

Will reduce your pool, but… At best, not improve on

the average level of ability Possibly skew toward

lower end

None are “merit” based Negative impact on

image as public employer affecting other recruitments

Increase protests, appeals

How will you defend this?

The opportunity

Large applicant pools increase the potential utility of a valid selection procedure.

Utility: The expected organizational gain from your selection procedure

▪ Gain #1: Success rate of those hired▪ Gain #2: Monetary value of higher performance

Validity: the ability of your selection procedure to identify top talent

Selection Ratio: job openings (n) divided by the number of job applicants (N). Lower Ratio = Higher Selectivity

4

Large versus small pools

5

The likelihood of recruiting exceptional candidates increases with large pools.

large small

Exam Score

Num

ber

Large versus small pools

6

Large pools enable identifying pass points.

Put test validity to work

7

Job

Per

form

ance

Exam Score, Current Employees

Large pools enable high pass points.

Selected mean= 36

Overall mean = 26

Challenge

How to be efficient without sacrificing quality?

1.Analyze where large pools take the most effort

2.Utilize most efficient job-related screening tools first

8

Job-Related Strategies

1. Self-Selection

2. Efficient first stage screening

3. Broad-based testing

9

1. Focused Recruitment

2. Clear Job Descriptions and

Requirements

3. Realistic Job Previews10

Self-selection

Self-selection:Focused Recruitment

Occupation-related websites▪ Dice.com, OhSoEZ.com

Social Networking websites▪ Professional bulletin boards and discussion groups▪ LinkedIn Groups

Regular community-oriented networking Old fashioned paper bulletins and local media

distribution Career preparatory schools and training

organizations PTA and booster organizations Internal email, intranet, word-of-mouth by

employees11

Self-selection:Clear Requirements and Expectations Clear duty statements and minimum

qualifications▪ “Any combination of education and experience…” ??

Realistic job preview Clear statement of rewards and challenges of

the job in bulletin and other job description information.

Web-based preview/orientation▪ Paraeducator, Special Education▪ 3 hour, on-line workshop, must be completed as part

of the application process▪ Uses Adobe Connect12

1. Automated objective written tests

2. Un-proctored Internet Testing

3. Auto-scored supplemental

questionnaires

13

Efficient first-stage testing

Efficient first-stage testing:Objectively Scored Testing

Scantron (mark-sense answer sheets) or NCS automated scoring and score upload.

Work horse of selection procedures, and still among most efficient.

Test BEFORE screening for minimum qualifications

▪ Review minimums for only those who pass the test.

14

Efficient first-stage testing:Un-proctored Internet Testing (UIT) Basic skills testing (reading, writing, math,

reasoning, etc.)▪ Offered by all major test publishing and consulting firms▪ Register or apply on-line▪ Computer administered and timed▪ May be computer adaptive testing▪ Candidate cheating and impersonation remain issues▪ Those who pass are invited in for verification testing and

subsequent exam parts (performance, orals, etc.) Completion of online workshop post-testing

▪ Use with on-line preview/orientation with a multiple choice exam.

Online empirically-keyed personality testing, situational judgment, biodata, and occupational preference/suitability

▪ Less prone to faking and cheating15

Efficient first-stage testing:Supplemental Questionnaires

1. Part of the application process▪ Could be just a few questions focused on MQs▪ Could be automated tally of years experience and level

of education▪ Potentially useful but blunt instrument▪ Prone to candidate inflation due to “interpretation”

2. Based on job analysis to identify areas to assess▪ Type and level of duties and related tasks▪ Skills and facets of the skills▪ Each area is broken down into individual questionnaire

sections and items▪ Should require some form of verification

16

Example Items:Skills (MS Office)

To what extent have you performed the following EXCEL/spreadsheet tasks? 1. Set up rows and columns of data2. Cut and paste data from one spreadsheet into another 3. Write formulas in cells that reference more than one worksheet4. Create pivot tables 5. Create graphs 6. Use built-in statistical functions7. Write macros to automate routine functions

To what extent have you performed the following WORD/word processing tasks?1. Formatted documents with section breaks2. Formatted documents in “column” format 3. Created and formatted tables 4. Inserted pictures and graphs 5. Created locked forms with "form fields" 6. Used "track changes" feature in document editing7. Used mail merge to insert Excel source data into templates

17

Supplemental questionnairesPoints per response (example)

0 = I have no background in this0 = I know what this is but I have not

done it1 = I have assisted others (or received

training) in this, but not done it independently

2 = I have done this independently but not frequently

4 = I have done this as a regular part of my job18

Supplemental questionnaires Supporting Examples

Describe an accomplishment that illustrates your skill level in conducting job classification work. (Please limit your response to 200 words.)

Describe an example of a task you performed that illustrates your skill level with Excel/spreadsheets. (Please limit your response to 100 words.)

Or for skills, actual testing to verify (similar to UIT)19

Supplemental questionnaires Two-Stage Pass Points

Stage 11. Calculate MAR – Sum of Minimally Acceptable Responses

for each item.2. Set pass point based on self-rated scores. (If based on

applicant flow management, pass 10-20% more than you actually need.)

Stage 23. For those that exceed the pass point, have SMEs compare

the supporting examples to the self ratings.4. SMEs apply a weight of 0, .5, or 1.0 to the area of self

rating. Where:▪ 1.0 = example is consistent with the self ratings▪ .5 = questionable support for the self ratings▪ 0.0 = lacking support for the self ratings

5. Apply the corrected scores to the pass point to determine who actually has passed the hurdle.

20

Broad-based testing

Core test battery for as many classifications as possible

Score banking and certification to different exam plans

Different weights and cut-off scores for different classifications based upon relevancy and level needed

College-Board Model (SAT, GRE, etc.)

21

Broad-based testing:What problem are we solving?

Many related classifications Differ on specific job content and levels of responsibility Many obstacles to class consolidation - logistical, political,

fiscal

Each class has a unique examination Variable content and quality, not consistently updated Many of the same candidates 2-3 month recruitment & examination cycle time

No overall competency structure across the classes Lack of core competencies and competency progression Pass point paradoxes (lower pass points for higher level

jobs based on selection ratios)22

Broad-based testing:Purpose and Objectives

Improve testing reliability and validity

Improve coherence of clerical examination plans

Reduce redundant testing Increase recruitment efficiency Reduce time to place candidates on

eligibility lists23

Broad-based testing:Recruitment

Recruit for broad-based test

Annual calendar of assessment (e.g., quarterly)

Establishes a pre-tested pool of potential candidates for the relevant classes

For most job classifications, limited or no public posting for job specific recruitments

Invitations to apply for specific jobs are sent to pre-tested pool based on test cut-off score

24

Method 1:

General

Population

Recruit To Take BBC

PRE-TESTEDBBC POOL

Recruit for Job B

Recruit for Job A

Recruit for Job C

Broad-based testing:Recruitment

Recruit for a classification or class job family

Build a pre-tested pool from each recruitment process

Candidates apply for each recruitment process

Allow non-tested candidates to participate in each process & exams

Pre-tested candidates are informed of their status based on test cut-off scores

25

Method 2:

Broad-based testing – Clerical & Secretarial Classifications: Competencies Assessed - Tests

1. Sequencing and ordering speed and accuracy (timed)

2. Checking and comparing speed and accuracy (timed)

3. Computational speed and accuracy (timed)4. Following Instructions (MC)5. English Usage & Grammar (MC)6. Data Entry speed and accuracy (timed, OPAC)7. Microsoft Word Skills (Performance, OPAC)

26

Broad-based testing: Applicable BBC Classifications

1. Clerk2. Intermediate Clerk3. Senior Clerk4. School Clerk5. Senior School Clerk6. Data Control Clerk7. Senior Data Control

Clerk8. Typist Clerk9. Intermediate Typist

Clerk10. Temporary Office

Worker11. Senior Typist Clerk12. Department Assistant,

Dance13. Department Assistant,

Music

14. Department Assistant, Theater

15. Department Assistant, Visual Arts

16. Receptionist17. Reader18. Information Resources

Specialist19. Media Dispatching Clerk20. Secretary21. Division Secretary22. Legal Secretary23. School Administrative

Secretary24. Senior Division Secretary25. Executive Legal Secretary26. Executive Assistant27

Broad-based testing:Broad-Based Administrative (BBA)

ADMINISTRATIVE ANALYST ADMINISTRATIVE

ASSISTANT ADMINISTRATIVE AIDE ASSISTANT

ADMINISTRATIVE ANALYST RESOURCE &

DEVELOPMENT ANALYST PROJECT COORDINATOR COMPENSATION ANALYST RESEARCH ANALYST LEGISLATIVE ANALYST

BUDGET ANALYST HEAD START PROGRAM

RESULTS SPECIALIST HUMAN RESOURCES AIDE HUMAN RESOURCES

ANALYST HS PROGRAM

DEVELOPMENT SPECIALIST ASSISTANT HUMAN

RESOURCES ANALYST LABOR RELATIONS

SPECIALIST

28

Broad-based testing:Broad-Based Services(BBS)

CUSTODIAN SENIOR CUSTODIAN MAINTENANCE WORKER SENIOR MAINTENANCE WORKER DELIVERY DRIVER REPROGRAPHICS WORKER UTILITY WORKER

29

The Take Back Message

30

Job-Related Strategies

Self-Selection (Pre-Application Stage) Focused Recruitment Clear Job Descriptions and Requirements Realistic Job Previews

Efficient first stage screening (Pre-Invite Stage) Automated objective written tests Un-proctored Internet Testing Auto-scored supplemental questionnaires

Broad-based testing (Efficient Testing Administration) One Test for Multiple Classifications Score Bank Candidate Scores Improve reliability, validity, and efficiency

31

What are you doing?

32