Responding to Questions About Testing Programs: Ask the Experts

Panelists• Chuck Friedman, Session Coordinator-

ModeratorProgram Director, Professional Examination

Service (ProExam)• Ida Darragh Director of Testing, North American Registry of

Midwives• Susan Davis-Becker

Senior Psychometrician, Alpine Testing Solutions

• Jodi Herold Psychometric Consultant, University of Toronto

2

Additional Presenters

• YOU………………….conference participants

3

Which of the following best characterizes the organization with

which you work?

A. Umbrella agency or state licensing Boards

B. Professional associationC. National certifying organizationD. Vendor or corporation

4

Topics to be Addressed1. Failing candidates: retake issues 2. Challenging exam questions3. Use of multiple assessment modalities4. Translation5. General: Challenging issues facing test sponsors6. Practice Analysis issues7. Special Accommodations: ADA8. Computer Based Testing: Continuous or Window9. YOUR Additional Questions

Should our program limit the number or frequency (wait time) of

examination retakes for failing candidates?

But first, let’s find out what we are doing!

Which of these options best describes your program’s policy on testing

retakes?

A. No restrictions on retakesB. Limited number retakesC. Remediation required before retakingD. Wait time between retakesE. Combination of the above

7

Rationale for Allowing Retakes

• Common assumption that any single assessment is subject to a variety of measurement errors;

• Especially in high stakes testing (licensure), it is common to allow a “reasonable” number of attempts to pass

• NCCA Standards indicate that the certification program must provide a rationale for the retesting policy for failing candidates

Considerations in Policies for Retesting

• Item Exposure: must have equivalent alternate forms

• Size of exam: length of exam, depth of item bank

Should Candidates be Allowed to Challenge the Content of an

Exam?

10

Do you allow candidates to challenge the content (question/answers) of your exam?

A. YesB. No

Exam Challenge

• Responding to candidate concerns over content (clarity, fairness, appropriateness)

• Options – No challenge allowed– Challenge based on recall of test content– Challenge based on a secure review of

exam form

Exam Challenge: Benefits

• Potentially identify errors not detected during development & reduce/eliminate impact on pass/fail decisions

• Show responsiveness, keep the conversation between candidates and organization

Exam Challenge: Drawbacks• Must maintain an active process and

be responsive• Must develop and defend an action

plan if an error is identified– Do you apply it to just this candidate?

Future candidates? Past candidates? – Are you able to change/re-publish the

exam?

Multiple Assessment Types

• Should our testing program use multiple assessment types?

• What are the pros and cons to using multiple or alternative testing types?

Which of the following best describes your organization’s testing mandate?

A. Entry-to-PracticeB. Ensuring Continuing CompetenceC. Pre-service training course content

masteryD. Continuing Ed course content masteryE. Combination of the above

16

Typical Assessment TypesA. Written or online MCQB. Written or online exam with innovative

item types (e.g., matching, rank order)C. Simulation station-type (e.g., OSCE)D. PortfolioE. Behaviour Based Interview and/or

Chart Stimulated RecallF. Site visitsG. Other?

Does your program currently use multiple assessment types?

A. YesB. NoC. Not sureD. Sometimes yes, sometimes no

18

Multiple Modalities: Considerations

• Simultaneous or Sequential?– E.g., pass written to advance to OSCE

• Multiple pass-fail decisions– Each decision has error associated

• Implications of failure– Credential not issued– Remediation– Allowed to retake component not passed

Multiple Modalities: Considerations

• Performance-Based Assessments– Simulations– Workplace-based assessments– BBI/CSR

• Costly to develop and administer• Rater Effects• Assessment of competencies not feasibly

assessed with written format exams• Reliability “versus” Validity

Multiple Modalities: Summary• Practice Profile

– What proportion of competencies can be assessed with written? Alternative modalities?

• Budget / PersonPower– Development– Maintenance

• Use of Results– Impact on applicants/registrants– Validity of Decisions

Should our exam be translated?

• Issues in language translation for certification exams:

– Purpose– Costs– Defensibility

Purpose of Translated Exams

• To better accommodate candidates within the country of origin whose native language is another; or

• To take your certification global; to make it appropriate in other countries

(Cultural context; expectation of those who hire your certificants)

Which statement below best describes your program’s translation options?

A. Exam is offered only in one language

B. Exam is offered in more than one language but only in the country of origin

C. Exam is offered in more than one country and in the local language

24

Criteria for Translators

According to the Guide to the Translation of Legal Materials, National Center for State Courts, 2011, a translator should have:

• A sound knowledge of the source language• Above average writing ability in the target

language• Reasonable familiarity in the subject

matter

NCCA Standards

• When forms are to be translated into another language, the process must be designed to ensure that content is equivalent.

• When examinations are adapted across languages, certification programs must demonstrate that results obtained from adapted and source versions are comparable.

Adaptation Definition NCCA Standards

• The process by which a segment of text is converted to another language and/or cultural context, preserving equivalence of meaning, level of difficulty, and conceptual complexity from the original version to the new (adapted) version.

Overview of Issues

• Purpose: need clarity within the organization and in published materials for candidates and stakeholders

• Costs: translators, reviewers, editors• Ongoing review of results:

outcomes, defensibility

In recent years, what issues have been most challenging to test sponsors? What are potential

solutions?

29

Current Testing Challenges

Which challenge do you see as most relevant to your program?

A. Maintaining Security of contentB. Technology: item types, tablet testing,

badgesC. Requirements for maintaining

credential (CEUs, retesting)

Security• Test content is expensive to develop,

but given high stakes associated with test scores, can be valuable to steal

• Advise programs to – Evaluate general and specific risks– Determine risk mitigation strategies– Actively monitor security through item

analysis, candidate monitoring, and web content.

Technology• Examples

– New item types– Flexible administration mode– Scoring on demand

• Evaluate options carefully– What value will this add to our program?– What are the costs?– What are the risks?

Credential Maintenance• After credential is awarded, what is

next?• Options

– Nothing, credential is maintained unless removed for cause

– Requirement for CEUs based on number, topics, etc.

– Requirement for retesting (either same or different test)

Job Analysis

• What is the "right" amount (N or %) that need to respond to a Practice analysis survey to be "enough"?

What is the “right” % of respondents to practice/job analysis survey?

A. 0 - 25%B. 26 - 50%C. 51 - 75%D. 76- 99 %E. Only 100% will do

35

Sampling Strategy• Census or targeted

– If targeted, sampling frame• Which parameters to include in sampling

frame• Can also track additional parameters post-

hoc

• Key parameters to track with demographic questions– ALWAYS put demographics at

BEGINNING

Demographic Parameters• You and your advisory committee

decide– Geographic region– Areas of practice / specialization– Location of practice (e.g. facility type)– Age / time since credential or licensure– Gender– OTHERS specific to your situation

Census Survey• ALL Registrants

– How long to keep survey open?– Frequency of reminders

• Demographic parameters of interest– Know your distributions in population– Tracking mid-survey– ?Targeted reminders to groups with low

representation?– Demographic differences in non-completion?

Sampling Frame - TARGETS

  Region 1 Region 2 Region 3 Region 4 Region 5

Practice Area %

Practice Area 1 6.5% 21.2% 4.3% 2.5% 1.4% 36%Practice Area 2 1.3% 4.1% 0.8% 0.5% 0.3% 7%Practice Area 3 2.5% 8.3% 1.7% 1.0% 0.6% 14%Practice Area 4 5.0% 16.5% 3.4% 2.0% 1.1% 28%Practice Area 5 1.1% 3.5% 0.7% 0.4% 0.2% 6%Practice Area 6 1.6% 5.3% 1.1% 0.6% 0.4% 9%

Geographic Region % 18.0% 59.0% 12.0% 7.0% 4.0%

 

Sampling – Target vs Actual

 

Practice Area

TARGET %

Practice Area

ACTUAL %

Practice Area 1 36% 54%Practice Area 2 7% 10%Practice Area 3 14% 10%Practice Area 4 28% 15%Practice Area 5 6% 2%Practice Area 6 9% 11%

 

Geographic Region

TARGET %

Geographic Region

ACTUAL %

Region 1 18.0% 25.0%Region 2 59.0% 40.0%Region 3 12.0% 15.0%Region 4 7.0% 8.0%Region 5 4.0% 12.0%

Sampling Frame - ACTUAL

  Region 1 Region 2 Region 3 Region 4 Region 5

Practice Area %

Practice Area 1 13.5% 21.6% 8.1% 4.3% 6.5% 54%Practice Area 2 2.5% 4.0% 1.5% 0.8% 1.2% 10%Practice Area 3 2.5% 4.0% 1.5% 0.8% 1.2% 10%Practice Area 4 3.8% 6.0% 2.3% 1.2% 1.8% 15%Practice Area 5 0.5% 0.8% 0.3% 0.2% 0.2% 2%Practice Area 6 2.8% 4.4% 1.7% 0.9% 1.3% 11%

Geographic Region % 25.0% 40.0% 15.0% 8.0% 12.0%

 

Job Analysis - Summary

• What is the "right" amount (N or %) that need to respond to a Practice analysis survey to be "enough"?

• It’s an art AND a science• Keep your advisory committee

CLOSE AT ALL TIMES

Special Testing AccommodationsWhat are the most common; most

unusualHow to evaluate requestsWhat about test anxiety?

But, first, let’s find out what we are doing!

What is the most frequent testing accommodation granted by your

program

A. Extra timeB. Reader or audio program, large print C. Separate testing roomD. Paper/pencil if normally by computerE. Frequent breaks for food or water

44

General Requirements for Special Accommodations

• Documentation of disability by a professional (physician, psychologist, educational specialist), including type of accommodation needed to assure equal opportunity to demonstrate knowledge and/or skill

• Previous accommodation granted by educational institution or standardized test programs

Standards for Special Accommodations

• Procedures for requesting accommodations for candidates with a disability should be stated clearly and be publicly available.

• Proper documentation must be provided

Standards

• Any accommodation provided should be reasonable and not compromise the fundamental nature of assessment or the validity of the certification decision.

• Certification programs should not reveal on score reports or certificates that any accommodation was provided during the administration of the examination.

And, finally

• Refusals for accommodation should be given in writing, and include a description of the appeals process

Example from the LSATThese guidelines seek to establish that an individual has: (1) been diagnosed with a mental or physical impairment; (2) is substantially limited, compared to most people, in a major life activity relevant to taking the LSAT as a result of this physical or mental impairment; and (3) requires testing accommodations to address his or her specific functional limitations (with a rationale and objective basis for the testing accommodations requested).

Is “test anxiety” a diagnosis?

• The ADA does not identify or recognize “test anxiety” as a disability that requires special accommodation.

• Another view: test anxiety can affect concentration, memory, and reasoning, and could cause lower test scores that do not reflect the candidate’s true ability. With a documented medical assessment and/or history of accommodation, the program could offer testing in a separate room or untimed testing opportunities.

Should our program offer CBT tests “on-demand” or in

“windows”?

51

Testing Schedule

How is your exam administered?A. On-demand (always available)B. Windows (available at specified times)C. One-two days a year

On-Demand Administration

• Candidates can take the exam on their own schedule– Could potentially increase volume

• Flexible exam development and publishing schedule

• Increased security risks

Window Administration

• Candidates must plan to take the test during one of the pre-specified windows

• Exam development and publishing schedule should be set around administration plans

• Helpful with small or start-up programs

• Increased security options

Contact Information

• Ida Darragh testing@narm.org

• Susan Davis-Becker susan.davisbecker@alpinetesting.com

• Chuck Friedman cfriedman@proexam.org

• Jodi Herold jodi.herold@utoronto.ca

55