CAS EXAMINATION PROCESS 2006 CAS Annual Meeting – Session C4 San Francisco CAS Exam Committee presents: Steve Armstrong, FCAS Nasser Hadidi, FCAS Derek

CAS EXAMINATION CAS EXAMINATION PROCESSPROCESS

20062006 CAS Annual Meeting – Session C4CAS Annual Meeting – Session C4

San FranciscoSan Francisco

CAS Exam Committee presents:CAS Exam Committee presents:

Steve Armstrong, FCAS Steve Armstrong, FCAS

Nasser Hadidi, FCAS Nasser Hadidi, FCAS

Derek Jones, FCAS Derek Jones, FCAS

Daniel Roth, FCAS Daniel Roth, FCAS

2

Audit Findings (2001)Audit Findings (2001)

The CAS Does Many Things Well:The CAS Does Many Things Well:

• Good Communication with CandidatesGood Communication with Candidates

• Sound Procedures for Maintaining Sound Procedures for Maintaining Confidential InformationConfidential Information

• Exams are Administered with Exams are Administered with Appropriate Controls and Standardized Appropriate Controls and Standardized ProceduresProcedures

3

Audit FindingsAudit Findings

Several Areas for Improvement:Several Areas for Improvement:

• Need Better Link Between Learning Need Better Link Between Learning Objectives and Exams/ReadingsObjectives and Exams/Readings

• Learning Objectives and Exam Blueprints Learning Objectives and Exam Blueprints Should be PublishedShould be Published

• Need Better Training of Item WritersNeed Better Training of Item Writers

• Need to Consider Alternative Processes for Need to Consider Alternative Processes for Selecting Pass MarksSelecting Pass Marks

4

Major MilestonesMajor Milestones 2001 – Chauncey Began Facilitating Meetings 2001 – Chauncey Began Facilitating Meetings

to Write Learning Objectivesto Write Learning Objectives 2001 – Pass Mark Panels Pilot2001 – Pass Mark Panels Pilot 2002 – Item Writer Training Pilot2002 – Item Writer Training Pilot 2003 – Executive Council Agrees to Fund 2003 – Executive Council Agrees to Fund

Item Writer Training and Pass Mark Panels as Item Writer Training and Pass Mark Panels as Ongoing ProcessesOngoing Processes

2003 – Executive Council Approves New 2003 – Executive Council Approves New Learning ObjectivesLearning Objectives

5

Recent ImprovementsRecent Improvements

Evolution of CBTEvolution of CBT Improvement to Pass Mark Panel ProcessImprovement to Pass Mark Panel Process Expanded Sample Answer SetsExpanded Sample Answer Sets Increased CommunicationIncreased Communication CAS Board White PaperCAS Board White Paper

6

Learning ObjectivesLearning ObjectivesThe way things wereThe way things were

– What topics should What topics should successful candidates successful candidates understandunderstand

– What readings should What readings should they know?they know?

The way things are nowThe way things are now

– What should What should successful candidates successful candidates be able to be able to DODO??

7

Learning ObjectivesLearning Objectives

The way things wereThe way things were– Individual topics and Individual topics and

readings were the readings were the basis for assigning basis for assigning the writing of exam the writing of exam questionsquestions

The way things are nowThe way things are now– Learning Objectives Learning Objectives

are the basis for are the basis for assigning the writing assigning the writing of exam questionsof exam questions

8

Learning ObjectivesLearning Objectives

The CAS Syllabus Committee has developed The CAS Syllabus Committee has developed Learning Objective Documents for CAS Learning Objective Documents for CAS Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9

There are more general Learning Objectives There are more general Learning Objectives for the joint Exams 1, 2, & 4 for the joint Exams 1, 2, & 4

9

Learning Objective DocumentsLearning Objective Documents

Five ElementsFive Elements

• Overview Statement for a Group of Overview Statement for a Group of Learning ObjectivesLearning Objectives

• Learning ObjectivesLearning Objectives

• Knowledge StatementsKnowledge Statements

• Syllabus ReadingsSyllabus Readings

• WeightsWeights

10


Overview StatementsOverview Statements

• Certain Syllabus sections can have Certain Syllabus sections can have multiple Learning Objectives (e.g., multiple Learning Objectives (e.g., ratemaking)ratemaking)

11


Learning ObjectivesLearning Objectives• What successful candidates should be able to doWhat successful candidates should be able to do• Learning Objectives should:Learning Objectives should:

Clearly state a main intentClearly state a main intentReflect a measurable outcomeReflect a measurable outcomeSupport an attainable behaviorSupport an attainable behaviorRelate to the learner’s needs or job functionRelate to the learner’s needs or job function

12


Knowledge StatementsKnowledge Statements

• Support Learning ObjectivesSupport Learning Objectives

• In order to accomplish the objective, In order to accomplish the objective, what does the candidate need to know?what does the candidate need to know?

13


ReadingsReadings

• An individual reading may be listed An individual reading may be listed under more than one learning objectiveunder more than one learning objective

• Readings listed under multiple objectives Readings listed under multiple objectives may facilitate exam questions that may facilitate exam questions that involve more synthesisinvolve more synthesis

14


Weights (by Learning Objective)Weights (by Learning Objective)

• Shown as rangesShown as ranges

• Guidelines - not intended to be absoluteGuidelines - not intended to be absolute

• Should end practice of candidates Should end practice of candidates calculating de facto weights by reading or calculating de facto weights by reading or topic from past examstopic from past exams

15

Learning Objectives and the SyllabusLearning Objectives and the Syllabus

• Learning Objective documents provide high Learning Objective documents provide high level guidancelevel guidance

– Review of current syllabus materialReview of current syllabus material

– Identification of topics requiring new Identification of topics requiring new syllabus materialsyllabus material

• Weights help Syllabus and Exam Weights help Syllabus and Exam Committees target specific objectivesCommittees target specific objectives

16

Future Changes to Learning Future Changes to Learning Objective DocumentsObjective Documents

• These are living documentsThese are living documents– Never perfectNever perfect– Subject to changeSubject to change

• Updates – When and how often?Updates – When and how often?– Not yet determinedNot yet determined– Annually per exam seems reasonableAnnually per exam seems reasonable– At least disruptive time for candidatesAt least disruptive time for candidates

17

Future Changes to Learning Future Changes to Learning Objective DocumentsObjective Documents

• CAS Executive Council (VP-Admissions) CAS Executive Council (VP-Admissions) will perform oversight and final approval of will perform oversight and final approval of any changes any changes

– Just as it does with changes to the Just as it does with changes to the SyllabusSyllabus

– Just as it has with the current Learning Just as it has with the current Learning Objective DocumentsObjective Documents

18

Learning Objective SummaryLearning Objective Summary

Transition to published Learning Objectives Transition to published Learning Objectives should help the CAS achieve:should help the CAS achieve:

– Better syllabus content and exam questionsBetter syllabus content and exam questions

– More transparent basic education processMore transparent basic education process

– Better model for evaluating future changes Better model for evaluating future changes to the syllabusto the syllabus

– Better model for evaluating future changes Better model for evaluating future changes to the desired education of casualty to the desired education of casualty actuariesactuaries

19

Writing Exam QuestionsWriting Exam Questions

A.A. One-Best-Answer QuestionsOne-Best-Answer Questions

B.B. Essay QuestionsEssay Questions

20

For Each Type:For Each Type:

What makes a good question? The questionsWhat makes a good question? The questions

Should measure students’ mastery of stated Should measure students’ mastery of stated Learning Objectives, and not merely recalling Learning Objectives, and not merely recalling factual informationfactual information

Should be clearly stated and not subject to different Should be clearly stated and not subject to different interpretationsinterpretations

Should be perceived by students to be fairShould be perceived by students to be fair Should be possible to answer by a typical student in Should be possible to answer by a typical student in

the allotted timethe allotted time

21

A: One-Best-Answer QuestionsA: One-Best-Answer Questions

QuestionsQuestions

Should focus on an important conceptShould focus on an important concept Should assess skills in practical applicationsShould assess skills in practical applications Should avoid flaws that provide special benefits to Should avoid flaws that provide special benefits to

“test wise” students “test wise” students Should not start with “Which of the following Should not start with “Which of the following

statements is correct” – heterogeneous optionsstatements is correct” – heterogeneous options

22

Distracters-Incorrect AnswersDistracters-Incorrect Answers

Distracters directly affect the difficulty of a Distracters directly affect the difficulty of a questionquestion

Should be plausible – selected by some Should be plausible – selected by some studentsstudents

Common misconceptions and faulty Common misconceptions and faulty reasoning provide good source of plausible reasoning provide good source of plausible distracters.distracters.

None should stand out as being obviously None should stand out as being obviously incorrectincorrect

23

Example – Bad DistractersExample – Bad Distracters

Who was the primary author of the Who was the primary author of the Declaration of Independence?Declaration of Independence?

A.A. Abraham LincolnAbraham Lincoln

B.B. Thomas JeffersonThomas Jefferson

C.C. Franklin RooseveltFranklin Roosevelt

D.D. King George IIKing George II

E.E. Catherine the GreatCatherine the Great

24

DistractersDistracters

Should be homogeneous in contentShould be homogeneous in content Plausible and attractive to the unprepared studentPlausible and attractive to the unprepared student Should not be totally wrong Should not be totally wrong Similar to the correct answer in construction and Similar to the correct answer in construction and

lengthlength Grammatically consistent and logically compatible Grammatically consistent and logically compatible

with the correct answerwith the correct answer ““None of the above” usually not a good distracter None of the above” usually not a good distracter

25

One-Best-Answer Questions One-Best-Answer Questions FormatFormat

All relevant information should be given in All relevant information should be given in the questionthe question

No additional information in the optionsNo additional information in the options Options should be relatively shortOptions should be relatively short

26

One-Best-Answer Questions One-Best-Answer Questions Summary Item FlawsSummary Item Flaws

““Test Wise” issues: grammatical, logical, Test Wise” issues: grammatical, logical, absolute terms, long correct answer, word absolute terms, long correct answer, word repeatsrepeats

Irrelevant Difficulty issues: inconsistent, Irrelevant Difficulty issues: inconsistent, long/complicated, vague, “None of the long/complicated, vague, “None of the above”, language not parallelabove”, language not parallel

27

One-Best-Answer Questions One-Best-Answer Questions Construction GuidelinesConstruction Guidelines

Item can be answered without looking at optionsItem can be answered without looking at options Include as much information as possible – but Include as much information as possible – but

avoid extra words - Options shortavoid extra words - Options short Avoid superfluous informationAvoid superfluous information Avoid “tricky” and overly complex itemsAvoid “tricky” and overly complex items Options grammatically consistentOptions grammatically consistent Avoid using absolute terms – always, neverAvoid using absolute terms – always, never Focus on important items – Do not waste time Focus on important items – Do not waste time

testing trivial facts testing trivial facts

28

B: Essay QuestionsB: Essay Questions

Basically two types of Essay Questions:Basically two types of Essay Questions:

Extended Response - Students free to Extended Response - Students free to determine content, format of response determine content, format of response

Restricted Response – Limit both content Restricted Response – Limit both content and form, this is the appropriate form when and form, this is the appropriate form when content is to be testedcontent is to be tested

29

Constructing Essay QuestionsConstructing Essay Questions

Allow adequate preparation time – Effective items Allow adequate preparation time – Effective items need deliberation, reflectionneed deliberation, reflection

Limit to Learning Objectives that can not be Limit to Learning Objectives that can not be measured by One-Best-Answer itemsmeasured by One-Best-Answer items

Design to test only one or just a few Learning Design to test only one or just a few Learning Objectives per itemObjectives per item

Select items that can be answered briefly-Several Select items that can be answered briefly-Several short essay items better than one long answeredshort essay items better than one long answered

Clearly indicate task – both content and processClearly indicate task – both content and process Explicitly state the time or number of pointsExplicitly state the time or number of points

30

Essay Questions -AvoidEssay Questions -Avoid

Open ended broad type of items Open ended broad type of items Vague poorly worded items subject to Vague poorly worded items subject to

multiple interpretationsmultiple interpretations Items asking simply for “Opinion” – Items asking simply for “Opinion” –

rather ask for evidence in support of a rather ask for evidence in support of a “position”“position”

Items where different students can give Items where different students can give answers to different questions. answers to different questions.

31

The Time FactorThe Time Factor

In both type of questions and especially In both type of questions and especially essay questions, consider the allotted timeessay questions, consider the allotted time

Time should not be a major factorTime should not be a major factor The rule of thumb is that if more than 10% The rule of thumb is that if more than 10%

of the students fail to complete the test in of the students fail to complete the test in the allotted time, Time has become a factor the allotted time, Time has become a factor in test scoresin test scores

32

Checklist for Effective Test ItemsChecklist for Effective Test Items

Item matches Leaning Objective?Item matches Leaning Objective? Item relates to what was actually covered in the Item relates to what was actually covered in the

Syllabus?Syllabus? Item relates to important concept rather than Item relates to important concept rather than

trivia?trivia? Item measures ability to make practical Item measures ability to make practical

application, rather than simply measuring recall?application, rather than simply measuring recall? Item free from vaguely defined terms, Item free from vaguely defined terms,

ambiguous wording, irrelevant information, ambiguous wording, irrelevant information, unintentional clues? unintentional clues?

33

ReferencesReferences

B.B. Zimmerman et. al., How to prepare B.B. Zimmerman et. al., How to prepare better tests: Guidelines for University better tests: Guidelines for University FacultyFaculty

National Board of Medical Examiners, National Board of Medical Examiners, Constructing Written Test Questions for Constructing Written Test Questions for Basic and Clinical Sciences Basic and Clinical Sciences

W. Cashin, Improving Essay testsW. Cashin, Improving Essay tests

34

Grading Exam QuestionsGrading Exam Questions

Assignments for grading will be distributed immediately after the exam as to which question(s) you are assigned to grade.

Questions are graded in pairs, just like writing exam questions.

A sample solution and a copy of the exam will be sent to the grading pair for review prior to getting the actual papers.

The actual papers come to the grading pair in the week or two following the examination.

35

Grading Exam QuestionsGrading Exam QuestionsGraders are encouraged to develop a grading key that

accounts for the different combinations and permutations of answers that can be provided.

Graders are encouraged to use this grading key for a random set of questions (between 20-40) to ensure consistency in application of the grading key.

The graders should meet after grading this set of random questions to reconcile any differences and change the grading key if necessary.

CONSISTENCY AND ACCURACY IS VITAL

36

Grading Exam QuestionsGrading Exam Questions

In the subsequent weeks, individual graders will grade every candidate’s response and log the points in a Grading Program.

Graders are encouraged to reconcile scores along the way so that this work does not monopolize the time spent at the on-site Grading Session.

The on-site Grading Session is now being conducted in Las Vegas and lasts for two days in June and December.

37

Grading Exam QuestionsGrading Exam QuestionsOn-Site Grading SessionOn-Site Grading Session

The on-site Grading Session allows the following:

Allows those on the committee to meet with one another.

Small allowance of time to reconcile all scores between graders to no more than a ¼ point difference (differs by Part Chair).

Establishment by the committee of an appropriate passing score.

Re-grading (no point discrepancy at all) of those candidate’s questions that are within a range of the passing score.

Discussion on how to make the exam even better for next year.

38

Grading Exam QuestionsGrading Exam QuestionsOn-Site Grading SessionOn-Site Grading Session

After the first day, a group activity occurs at night to allow those on the committee to interact with one another in a less formal manner.

The on-site Grading Session is concluded when the group has established a mutually agreed upon passing score by inspecting all the relevant statistics and grading/re-grading those candidates around the passing score to ensure that their score is correct.

All final scores by candidate, including grading keys and model candidate solutions are left with the Part Chair to create the exam report to be sent to the Exam Officers.

39

Setting the Pass MarkSetting the Pass Mark

Identify Purpose of the Pass MarkIdentify Purpose of the Pass Mark Convene Pass Mark PanelConvene Pass Mark Panel Analyze Exam StatisticsAnalyze Exam Statistics Prepare RecommendationPrepare Recommendation Proceed through Approval ProcessProceed through Approval Process

40

Purpose of the Pass MarkPurpose of the Pass Mark

Pass Minimally Qualified (or better) Pass Minimally Qualified (or better) CandidatesCandidates– Those who have demonstrated a Those who have demonstrated a

sufficient grasp of the syllabus materialsufficient grasp of the syllabus material Fail OthersFail Others There is no predetermined pass ratioThere is no predetermined pass ratio

41

Purpose of the Pass MarkPurpose of the Pass Mark

Failers Passers

Minimally Qualified Candidate

42

Pass Mark PanelPass Mark Panel

Panel includes:Panel includes:New Fellows (1-3 years)New Fellows (1-3 years)Fellows experienced in practice areaFellows experienced in practice areaOfficers of exam committeeOfficers of exam committee

Recommends a pass mark independent of Recommends a pass mark independent of the normal exam committee proceduresthe normal exam committee procedures

43


Defines Minimally Qualified CandidateDefines Minimally Qualified CandidateWhat he or she should will knowWhat he or she should will knowWhat he or she will not knowWhat he or she will not knowWhat he or she will be able to What he or she will be able to

demonstrate on the examdemonstrate on the exam Relates Criteria to Learning Objectives for Relates Criteria to Learning Objectives for

defining the minimally qualified candidate.defining the minimally qualified candidate.

44


Each panelist independently estimates how 100 Each panelist independently estimates how 100 minimally qualified candidates will score on each minimally qualified candidates will score on each question (and sub-part of each question).question (and sub-part of each question).

Scores are assembled and shared in a group Scores are assembled and shared in a group format.format.

Group discusses ratings and may change estimatesGroup discusses ratings and may change estimates Facilitator compiles ratings and shares results with Facilitator compiles ratings and shares results with

exam committee officersexam committee officers

45

Analyze Exam Statistics (back at the Analyze Exam Statistics (back at the Grading Session)Grading Session)

Collect Initial Scores for All CandidatesCollect Initial Scores for All Candidates Review/Discuss Key MeasuresReview/Discuss Key Measures

High, Low, MeanHigh, Low, Mean Percentiles, Percentile RelationshipsPercentiles, Percentile Relationships Pass Mark Panel RecommendationPass Mark Panel Recommendation Prior statistics from previous examsPrior statistics from previous exams CAS Board goal, “…CAS Board goal, “…that 40% or more of the candidates that 40% or more of the candidates

should get a score of 70% or more on any given exam; and should get a score of 70% or more on any given exam; and all candidates that get such a score should pass.”all candidates that get such a score should pass.”

Pick an initial pass mark and re-grade candidates within Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)certain range of pass mark (+/- 3 points, for example)

46

Prepare RecommendationPrepare Recommendation

Recollect scores if any have changed and review all Recollect scores if any have changed and review all relevant statistics again.relevant statistics again.

Review borderline candidates (tighter threshold now Review borderline candidates (tighter threshold now – no discrepancy between graders) and – no discrepancy between graders) and re-grade/review for consistency with grading key.re-grade/review for consistency with grading key.

Repeat process until only looking at the 5 exams Repeat process until only looking at the 5 exams above and the 5 exams below the recommended pass above and the 5 exams below the recommended pass mark. mark.

Justify Recommended Pass ScoreJustify Recommended Pass Score

47

Approval ProcessApproval Process

Part Chair Part Chair

General Officer (Spring / Fall)General Officer (Spring / Fall)

Exam Committee Chair (Arlie Proctor)Exam Committee Chair (Arlie Proctor)

VP-Admissions (Jim Christie) – VP-Admissions (Jim Christie) – The final The final decision on the pass mark is the responsibility of this decision on the pass mark is the responsibility of this position.position.

48

Appeal ProcessAppeal Process

In the event of a candidate appeal, a In the event of a candidate appeal, a

grader may be called upon to review the grader may be called upon to review the

appeal and reconcile the score with the appeal and reconcile the score with the

grading key.grading key.

Questions and CommentsQuestions and Comments

More info on the Web – www.casact.org/admissions/process/