Crowd Sourcing Crowd Sourcing TestsTests

Warren Bobrow, Ph.D.Warren Bobrow, Ph.D.

All About Performance, LLCAll About Performance, LLC

warren@allaboutperformancewarren@allaboutperformance.biz.biz

What is Crowd Sourcing?What is Crowd Sourcing?

Obtaining services or content by Obtaining services or content by asking for input from an online asking for input from an online community.community. Software codeSoftware code KickStarterKickStarter Marketing CampaignsMarketing Campaigns Product Development Product Development

What Are the Potential What Are the Potential ApplicationsApplications

Test QuestionsTest Questions

Survey ItemsSurvey Items

Recruitment IdeasRecruitment Ideas

Test DevelopmentTest Development

Where appropriate for content valid Where appropriate for content valid teststests No in-house SMEsNo in-house SMEs Frequently changing contentFrequently changing content Unique jobs where there is not published Unique jobs where there is not published

contentcontent

In reality, it’s a very scaled up version In reality, it’s a very scaled up version of item banking from previous examsof item banking from previous exams

ConcernsConcerns Test SecurityTest Security

Risks are the same as with unproctored tests.Risks are the same as with unproctored tests. Perhaps less so if the test is adaptive.Perhaps less so if the test is adaptive.

Quality of the ItemsQuality of the Items Is the online expert community better or Is the online expert community better or

worse at writing items than your SMEs?worse at writing items than your SMEs? NormingNorming

Averages likely to be lower, but better item Averages likely to be lower, but better item statistics due to larger samples.statistics due to larger samples.

StandardizationStandardization Likely to be adaptiveLikely to be adaptive

ApplicationApplication

Virtual software companyVirtual software company Low number of SMEs and Low number of SMEs and

geographically spread outgeographically spread out

Growing fairly rapidlyGrowing fairly rapidly

Need for programmers and Need for programmers and professional staff (marketing, etc)professional staff (marketing, etc)

OutcomesOutcomes

CostCost It’s FREE (for now)It’s FREE (for now) Integrates into ATSIntegrates into ATS

Candidate ReactionCandidate Reaction Meets the face validity testMeets the face validity test Still some dissatisfaction as they items Still some dissatisfaction as they items

are multiple choiceare multiple choice

Test StatisticsTest Statistics

Tests are normedTests are normed

Not a presumption of a normal curve.Not a presumption of a normal curve.

Master = 780-800Expert = 700-779

Proficient = 450-699Familiar = 300-449Beginner = < 300

Observed StatisticsObserved Statistics

Type of TestType of Test MeanMean SDSD Median Median IntercorrIntercorr

ITIT 598.2598.2 162.6162.6 .32.32

OfficeOffice 639.1639.1 134.1134.1 .11.11

•Scores are above the midpoint of the middle range (450 to 699), which indicates that there are some non-experts in the crowd.

•Items have large SDs, indicating that there is good variance.

•Not all tests are given in combination with others, but the intercorrelations appear to be low-positive.

OutcomesOutcomes

Too early for criterion related study.Too early for criterion related study.

The automation and integration with the ATS The automation and integration with the ATS is a huge time saver for a company with a is a huge time saver for a company with a small HR staff.small HR staff.

Have not lost any candidates to the process.Have not lost any candidates to the process.

Used in combination with personality/aptitude Used in combination with personality/aptitude testing and structured interviews.testing and structured interviews.

What Does the Future What Does the Future Hold?Hold?

Unlikely that this will take over Unlikely that this will take over content validation testing.content validation testing. Not enough control over the process for Not enough control over the process for

man organizations.man organizations.

Likely to be adopted by tech startups Likely to be adopted by tech startups and small firmsand small firms Meets an important needMeets an important need Is a good cultural fitIs a good cultural fit

Q&AQ&A