1 Examination Accreditation Program Exam Security: Preventing Exam Espionage Saturday, October 1...

1

Examination Accreditation Program

Exam Security:Preventing Exam Espionage

Saturday, October 1Baltimore – Maryland - USA

2

Exam Security: Introducing the Presenters

During Test Development

Lawrence J. Fabrey, Ph.D.

During Test AdministrationDave Moore

After the TestChris Beer

Tadas Dabsys

3

Benefits of ARELLO-Accreditation

Independent Evaluation Confidence and Protection Defensible Standards Reasonable Consistency in Quality Facilitates License Recognition

www.ARELLO.org (ARELLO Guidelines for Accreditation)

4

ARELLO Additional Resources

About Exam Development An Accreditation Overview ARELLO® Matters to Consider in Exam RFP ARELLO® Top Ten Questions and Complaints on

Licensing Exams ARELLO® Supreme Court Case - Exam Validity ARELLO® Examination Accreditation Program

- White Paper on Determining Minimum Passing Scores

5

Security During Examination Development

• Physical Environment • Electronic Access • Staffing • Inventory Storage• Subject Matter Experts• Other Functions

• Translation, Printing, Artwork, etc.

6

Development – Physical Access

• Secure Room/Area • Vault – Key Access • Limited Authorized Access• Chain-of-Custody• Shredding – including notes/drafts

7

Development – Electronic Access

• Passwords & Screensavers• Selective Access Permission • Specific Time Period• Read-only, non-print, non-export• Webinars

8

Development - Staffing

• Staff • Confidentiality Agreements • Conflict of Interest • Audits

9

Development - SME’s and Other Functions

• Subject Matter Experts • Translators• Artwork• Supplies for Practical Exams• Printers• Couriers

10

Development – Inventory and Distribution

• Printing (Serial #’s, Shrink wrap)• Shipping (Traceable, Armed Courier)• Electronic Transmission• Web Access

11

Security During Exam Administration

• Paper & Pencil • Computer-based Testing• Practical Exams• Open-book Exams• Special Accommodations • Facilities & Staff• Candidates

12

Administration – Paper & Pencil

• Printing • Shipping Receipt and Inventory• Storage• Contingency Plan• Return Shipment & Inventory

13

Administration - CBT

• CBT Centers • Restricted Login Access• Restricted Time Periods• Restricted Printing• Cameras, DVR’s

14

Administration – Special Administrations

• Practical Exams • Raters (SME) – Conflict of Interest• Supplies

15

Administration – Open-book

• Open-book Exams • Notes• Highlight/Underline• Post-its/Dividers• Screening

16

Administration - Facilities

• Limited Access• Spacing Candidates• Monitor Restroom Breaks• Secure Storage - Supplements• Special Accommodations

17

Administration - Staffing

• Conflict of Interest• Self-regulating• Restricted Access (Email, Internet)• Audits• Training – Active Proctoring• Termination

18

Administration - Special

• Special Accommodations• Interpreters

19

Administration - Candidates

• Candidates• Identification• Roster Signature• Fingerprint Login• Clothing/Purses• Activity/Mannerisms

20

Available through SkyMall and Web

21

Caught on Camera

Smart phone

Notes

22

I know I will be Caught

23

24

Show Me the Money!

25

Security After the Test

• Data Forensics• Monitoring Websites• Item Updating• Facility Screening• Audits• Incident Reporting

26

After – Data Forensics - Items

p-value: Percentage of candidates who answer an item correctly

pbis: Correlation between performance on an item and performance on the overall exam

option stats: Percentage of candidates who answer each option correctly

option pbis: Correlation between performance on the options and the overall exam

27

After – Data Forensics - Items

p-value: Too high, too low

pbis: Too low

options:Distractors with no responses

Distractors with more responses than key

O pbis: Distractors higher than key

Omits Above expectations

28

After – Data Forensics - Candidates

Suspicious Candidate Activity

Test for significant differences in pass rates Better performance on harder items High scores in low test times Suspicious improvement in scores

29

After – Monitoring Websites

• Cram Schools• Sample Exams• False Advertising• Social Networking

30

After – Item Updating

• Monitor Item Statistics• Replace Items• Add New Items

31

After – Facility Screening

• Look for Notes in Testing Station• Screen Restrooms and Hallways

32

After - Audits

• Test Center Audits• “Secret Shopper” Programs• Inventory• Continuing Education for Proctors

33

After – Incident Reporting

• Documentation• Evidence• Reporting to Client

34

Examples

• Cheating Incidents

35

ARELLO Accredited Providers

Applied Measurement Professionalshttp://www.goamp.com

Pearson VUEhttp://www.pearsonvue.com

PSI Services LLChttp://psionline.com

36

QUESTIONS?COMMENTS!

Fabulous Prize Giveaway