Liz Burns, Juniper NetworksJennifer Semko, Baker & McKenzie

Jack Terry, NBEORachel Schoenig, ACT

Dennis Maynes, Caveon Test Security

Lessons Learned from Using Statistics to Invalidate Scores

June 19, 2014

Caveon Webinar Series presents:

1

Agenda for Today

• Question/Answer Format

• Please use the chat window to ask questions

• Conclusions

2

To Kick Off the Discussion…

Let’s chat about why an organization would use statistics to invalidate scores.

3

Q1

If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you should do?

5

Q1 - Answer

1. Don’t second guess yourself. Remain confident.

2. Be prepared to provide legal counsel with a high level summary.

3. Bring your counsel on board in a timely manner.

If a candidate hires legal counsel and contests a score invalidation, what are the three most important things you should do?

6

Q2

In your opinion, which statistics provide the most credible information concerning potential test fraud?

7

Q2 - Answer

• Old/New or EVT* Items• Trojan Horse Items• Value

• Consistent measurement• Identify candidates who benefit• Can be automated• Easy to explain/understand

In your opinion, which statistics provide the most credible information concerning potential test fraud?

*EVT = Embedded Verification Test8

Q3

When and how do you believe one should recommend score invalidations to your board or governing body?

9

Q3 - Answer

When? When you have a good faith basis for questioning the validity of a score

How? Provide a full and frank summary; give your Board the tools they need

When and how do you believe one should recommend score invalidations to your board or governing body?

10

Q4

Describe at least three things that should be avoided when using statistics to invalidate scores?

11

Q4 - Answer

1. Using complicated statistical explanations as rationale

2. Using complex charts or graphs to illustrate your point

3. Using the word ‘cheat’, ‘cheater’, or other emotionally-charged, accusatory language

Describe at least three things that should be avoided when using statistics to invalidate scores?

12

Q5

Based on your experience, what are pros and cons to using statistics to invalidate scores?

13

Q5 - AnswerBased on your experience, what are pros and cons to using

statistics to invalidate scores?

Pros• Statistics are powerful and

illustrative.• Statistics can be applied

consistently.• Statistics can provide you with a

“targeted action” approach.

14

Q5 - AnswerBased on your experience, what are pro’s and con’s to using

statistics to invalidate scores?

Cons• Staying ahead of breaches to your

program is challenging.• Statistics are not a “silver bullet.”• Exam security requires a

comprehensive plan that is constantly evolving.

15

Q6Describe a time when you were successful in using statistics to invalidate a test score (or scores). What factors contributed to yoursuccess? Have you ever used only statistics to successfully invalidate a score (i.e., there was no physical or other corroborating evidence of cheating besides the statistical results)?

16

Q6 - AnswerDescribe a time when you were successful in using

statistics to invalidate a test score.

• Studied similarity data, performed analysis

• Reviewed seating charts• Identified index values >7.5• Created a report and referred to

board

17

Q7

Describe a time when you had statistical evidence of aberration, but were unsuccessful in using that evidence to invalidate the score (e.g., you lost a legal dispute or you decided not to pursue the case further for some reason). Explain why you were not successful.

18

Q7 - AnswerDescribe a time when you had statistical evidence of aberration, but

were unsuccessful in using that evidence to invalidate the score. What happened?

• Lack of corroborating evidence• Unable to convey the compelling

nature of the statistical evidence

19

Q8If a program has a solid testing agreement in place with proctors and candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?

20

Q8 - AnswerIf a program has a solid testing agreement in place with proctors and

candidates, how much time, effort, and money do you estimate is required to invalidate one test score based on statistics?

This varies based on your organization and processes followed.

• Two examples from our panelists• Example 1: Three primary areas of

expense• Example 2: Internal, automated process

21

Q9

What internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.

22

Q9 - AnswerWhat internal challenges and obstacles did you confront in instituting your invalidation program? Were they operational? Legal? Political? Please describe.

• Political – Invalidated scores belonged to customers and partners. Stakeholders had to know what to expect and why this was beneficial.

• Legal – Is process legally defensible? Do we have all the right documentation in place?

• Operational – Continually evolving and staying on top of technologies is key.

23

Q10

Have you instituted any KPIs or other measurements around your use of statistical analyses and invalidations to measure program impact and success? What are they?

26

Q10 - AnswerHave you instituted any KPIs or other measurements around your use of

statistical analyses and invalidations to measure program impact and success?

• Metrics re: cases opened, closed, cancel rates

• Metrics re: resolution options selected by examinee

• KPIs to measure effectiveness of program

27

Q11

What legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?

28

Q11 - AnswerWhat legal agreements or legal foundation need to be in place in order to consider invalidating based on statistics?

• Clear, comprehensive, signed candidate agreements

• Ethics policies in place• Have a plan, be comprehensive• Court will look to your agreements

29

Audience Questions

30

Audience Questions

“We see a lot of exam takers who score very high in very short amounts of time. We know that it would be impossible to even read the questions in the short time spans, but it’s hard to prove cheating.

Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?”

31

Audience Question - Answer

• Some reading rates exceed 1,000 wpm (trained speed reading)

• Average adult reading rates are 250-300 wpm on non-technical content

• Reading rates should be verified• High-speed, erratic reading strongly

indicates braindump usage

Is there a guideline on the amount of time taken per question that we can use to take these cheaters down?

32

Conclusions

“I would urge us to reframe our concernsabout test data integrity not as cheatingconcerns, but as a validity issue.”

“I would first strongly caution everyoneNOT to do any analyses at all until firstdeveloping and adopting policies andprocedures about what to DO with theresults of any analyses.”

– Gregory Cizek, NCSA, 2013

http://www.caveon.com/tilsa-test-security-guidebook-next-steps/33

Conclusions“Perhaps the worst situation one could be in wouldbe a situation where analyses have been conductedand it must be admitted publically that nothing hasbeen done with the results.”

“It is important to treat each similarly situated casethe same way, and a coherent, comprehensive setof policies and procedures, uniformly applied, isessential.”

– Gregory Cizek, NCSA, 2013

http://www.caveon.com/tilsa-test-security-guidebook-next-steps/34

Thank you!

- Follow Caveon on twitter @caveon- Check out our blog…www.caveon.com/blog- LinkedIn Group – “Caveon Test Security”

Liz Burns, Juniper NetworksRachel Schoenig, ACT

Jennifer Semko, Baker & McKenzieDr. Jack Terry, NBEO

Dennis Maynes, Caveon Test Security

35