CAS EXAMINATION PROCESS 2005 CAS Annual Meeting – Session C8 Baltimore CAS Exam Committee presents: Steve Armstrong, FCAS Daniel Roth, FCAS Manalur Sandilya,

CAS EXAMINATION CAS EXAMINATION PROCESSPROCESS

20052005 CAS Annual Meeting – Session C8CAS Annual Meeting – Session C8

BaltimoreBaltimore

CAS Exam Committee presents:CAS Exam Committee presents:

Steve Armstrong, FCAS Steve Armstrong, FCAS

Daniel Roth, FCAS Daniel Roth, FCAS

Manalur Sandilya, FCAS Manalur Sandilya, FCAS

Tom Struppeck, FCASTom Struppeck, FCAS

2

CAS Examination CAS Examination Process UpdateProcess Update

Fall 2000 – CAS Issues RFP for External Fall 2000 – CAS Issues RFP for External Review of Admissions ProcessesReview of Admissions Processes

The Chauncey Group (Subsidiary of ETS) The Chauncey Group (Subsidiary of ETS) SelectedSelected

Spring 2001 – Chauncey Group Conducts Spring 2001 – Chauncey Group Conducts Audit of CAS Admissions ProcessesAudit of CAS Admissions Processes

3

Audit FindingsAudit Findings

The CAS Does Many Things Well:The CAS Does Many Things Well:

• Good Communication with CandidatesGood Communication with Candidates

• Sound Procedures for Maintaining Sound Procedures for Maintaining Confidential InformationConfidential Information

• Exams are Administered with Exams are Administered with Appropriate Controls and Standardized Appropriate Controls and Standardized ProceduresProcedures

4

Audit FindingsAudit Findings

Several Areas for Improvement:Several Areas for Improvement:

• Need Better Link Between Learning Need Better Link Between Learning Objectives and Exams/ReadingsObjectives and Exams/Readings

• Learning Objectives and Exam Blueprints Learning Objectives and Exam Blueprints Should be PublishedShould be Published

• Need Better Training of Item WritersNeed Better Training of Item Writers

• Need to Consider Alternative Processes for Need to Consider Alternative Processes for Selecting Pass MarksSelecting Pass Marks

5

Major ObjectivesMajor Objectives

The Chauncey Group Engaged to Help CAS The Chauncey Group Engaged to Help CAS With Three Issues:With Three Issues:

• Write Better Learning Objectives and Write Better Learning Objectives and Establish Links to Readings/ExamsEstablish Links to Readings/Exams

• Develop a Process for Training Item WritersDevelop a Process for Training Item Writers

• Pilot an Alternative Process for Selecting Pilot an Alternative Process for Selecting Pass MarksPass Marks

6

Major MilestonesMajor Milestones 2001 – Chauncey Began Facilitating Meetings 2001 – Chauncey Began Facilitating Meetings

to Write Learning Objectivesto Write Learning Objectives 2001 – Pass Mark Panels Pilot2001 – Pass Mark Panels Pilot 2002 – Item Writer Training Pilot2002 – Item Writer Training Pilot 2003 – Executive Council Agrees to Fund 2003 – Executive Council Agrees to Fund

Item Writer Training and Pass Mark Panels as Item Writer Training and Pass Mark Panels as Ongoing ProcessesOngoing Processes

2003 – Executive Council Approves New 2003 – Executive Council Approves New Learning ObjectivesLearning Objectives

7

Learning ObjectivesLearning ObjectivesThe way things wereThe way things were

– What topics should What topics should successful successful candidates candidates understandunderstand

– What readings What readings should they know?should they know?

The way things are nowThe way things are now

– What should What should successful successful candidates be able candidates be able to to DODO??

8

Learning ObjectivesLearning ObjectivesThe way things wereThe way things were

– Individual topics Individual topics and readings were and readings were the basis for the basis for assigning the assigning the writing of exam writing of exam questionsquestions

The way things are nowThe way things are now

– Learning Objectives Learning Objectives are the basis for are the basis for assigning the assigning the writing of exam writing of exam questionsquestions

9

Learning ObjectivesLearning Objectives

The Syllabus Committee has developed The Syllabus Committee has developed Learning Objective Documents for CAS Learning Objective Documents for CAS

Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9 and Exams 3, 5, 6, 7-US, 7-Canada, 8 and 9 and also the VEE examsalso the VEE exams

There are simpler and more direct Learning There are simpler and more direct Learning Objectives for the Joint Exams 1, 2, & 4 Objectives for the Joint Exams 1, 2, & 4

10

Learning Objective DocumentsLearning Objective Documents

Five ElementsFive Elements

• Overview Statement for a Group of Overview Statement for a Group of Learning ObjectivesLearning Objectives

• Learning ObjectivesLearning Objectives

• Knowledge StatementsKnowledge Statements

• Syllabus ReadingsSyllabus Readings

• WeightsWeights

11


Overview StatementsOverview Statements

• Certain Syllabus Sections Can Have Certain Syllabus Sections Can Have Multiple Learning Objectives (e.g., Multiple Learning Objectives (e.g., Ratemaking)Ratemaking)

12


Learning ObjectivesLearning Objectives• What successful candidates should be able to What successful candidates should be able to

dodo• Learning Objectives Should:Learning Objectives Should:

Clearly state a main intentClearly state a main intentReflect a measurable outcomeReflect a measurable outcomeSupport an attainable behaviorSupport an attainable behaviorRelate to the learner’s needs or job functionRelate to the learner’s needs or job functionHave a definitive time frameHave a definitive time frame

13


Knowledge StatementsKnowledge Statements

• Support Learning ObjectivesSupport Learning Objectives

• In order to accomplish the objective, In order to accomplish the objective, what does the learner need to know?what does the learner need to know?

14


ReadingsReadings

• An individual reading may be listed An individual reading may be listed under more than one learning objectiveunder more than one learning objective

• Readings listed under multiple objectives Readings listed under multiple objectives may facilitate more may facilitate more synthesis/reasoning/cross-topic Exam synthesis/reasoning/cross-topic Exam questionsquestions

15


Weights (by Learning Objective)Weights (by Learning Objective)• Will be shown as rangesWill be shown as ranges• The ranges are guidelines and are not The ranges are guidelines and are not

intended to be absoluteintended to be absolute• Advantages of old-style “blueprints” Advantages of old-style “blueprints”

without disadvantageswithout disadvantages• Will (perhaps) end practice of candidates Will (perhaps) end practice of candidates

calculating de facto weights by reading or calculating de facto weights by reading or topic from past Examstopic from past Exams

16

Learning Objectives Learning Objectives and the Syllabusand the Syllabus

Learning Objective Documents Provide Learning Objective Documents Provide High Level GuidanceHigh Level Guidance

– Review of Current Syllabus MaterialReview of Current Syllabus Material

– Identification of Topics Requiring New Identification of Topics Requiring New Syllabus MaterialSyllabus Material

Weights help Syllabus Committee Target Weights help Syllabus Committee Target Specific ObjectivesSpecific Objectives

17

Future Changes to Learning Future Changes to Learning Objective DocumentsObjective Documents

These are Living DocumentsThese are Living Documents– Never PerfectNever Perfect– Subject to ChangeSubject to Change

Updates – When and How Often?Updates – When and How Often?– Not Yet DeterminedNot Yet Determined– Once a Year Per Exam Seems ReasonableOnce a Year Per Exam Seems Reasonable– At Least Disruptive Time for CandidatesAt Least Disruptive Time for Candidates

18

Future Changes to Learning Future Changes to Learning Objective DocumentsObjective Documents

CAS Executive Council (VP-Admissions) CAS Executive Council (VP-Admissions) Will Perform Oversight and Final Approval Will Perform Oversight and Final Approval of Any Changes of Any Changes

– Just as it does with changes to the Just as it does with changes to the SyllabusSyllabus

– Just as it has with the initial Learning Just as it has with the initial Learning Objective DocumentsObjective Documents

19

Learning Objective SummaryLearning Objective Summary

Transition to Published Learning Objectives Transition to Published Learning Objectives Should Help the CAS Achieve:Should Help the CAS Achieve:– Better Syllabus Content and Exam Better Syllabus Content and Exam

QuestionsQuestions– More Transparent Basic Education ProcessMore Transparent Basic Education Process– Better Model for Evaluating Future Changes Better Model for Evaluating Future Changes

to the Syllabusto the Syllabus– Better Model for Evaluating Future Changes Better Model for Evaluating Future Changes

to the Desired Education of Casualty to the Desired Education of Casualty ActuariesActuaries

20

Integrated Syllabus MaterialIntegrated Syllabus Material

Ratemaking for CatastrophesRatemaking for Catastrophes Multiple issuesMultiple issues Multiple papersMultiple papers Feedback from candidatesFeedback from candidates Need for Integrated Study Material Need for Integrated Study Material RFP processRFP process

21

Learning Objectives related to Learning Objectives related to Ratemaking for CatastrophesRatemaking for Catastrophes

Two Learning ObjectivesTwo Learning Objectives Operational IssuesOperational Issues Ratemaking IssuesRatemaking Issues 0 – 10 weight spread0 – 10 weight spread Two or more readings; Candidate feedbackTwo or more readings; Candidate feedback Exam Committee feedbackExam Committee feedback Can we create an integrated study noteCan we create an integrated study note

22

Learning Objectives to Integrated Learning Objectives to Integrated Study NotesStudy Notes

Outline begins with Learning ObjectivesOutline begins with Learning Objectives Expand the outline based on Knowledge Expand the outline based on Knowledge

StatementsStatements Add new Knowledge Statements where Add new Knowledge Statements where

appropriateappropriate Review the flow of ideasReview the flow of ideas Finalize the Study NotesFinalize the Study Notes

23

Writing Exam QuestionsWriting Exam QuestionsWhat makes a good exam question?What makes a good exam question?

1)1) Should be easy to grade.Should be easy to grade.2)2) Should be answerable in a reasonable amount of Should be answerable in a reasonable amount of

time.time.3)3) Should measure the student’s mastery of the Should measure the student’s mastery of the

material, ideally by doing something.material, ideally by doing something.

Not easy to achieve all of these.Not easy to achieve all of these.

24

Question 1Question 1

According to the errata, the “4.2” on page According to the errata, the “4.2” on page 93, line 15 should read “7.9”.93, line 15 should read “7.9”.

25


According to the errata, the “4.2” on page According to the errata, the “4.2” on page 93, line 15 should read “7.9”.93, line 15 should read “7.9”.

This satisfies two of the criteria.This satisfies two of the criteria.

26


Struppeck gives six example test questions, Struppeck gives six example test questions, list them.list them.

27


Struppeck gives six example test questions, Struppeck gives six example test questions, list them.list them.

This is better than Question 1, but it still isn’t This is better than Question 1, but it still isn’t giving the student a chance to show mastery giving the student a chance to show mastery of the material, only recall.of the material, only recall.

28


Struppeck gives six example test questions, Struppeck gives six example test questions, list them and describe them.list them and describe them.

29


Struppeck gives six example test questions, Struppeck gives six example test questions, list them and describe them.list them and describe them.

This is better than Question 2.This is better than Question 2.

30


Struppeck claims that Question 3 is better Struppeck claims that Question 3 is better than Question 2, explain why.than Question 2, explain why.

31


Struppeck claims that Question 3 is better Struppeck claims that Question 3 is better than Question 2, explain why.than Question 2, explain why.

This would actually be a pretty good question.This would actually be a pretty good question.

32


Write Question 7 and use Struppeck’s three Write Question 7 and use Struppeck’s three criteria to evaluate it.criteria to evaluate it.

33


Write Question 7 and use Struppeck’s three Write Question 7 and use Struppeck’s three criteria to evaluate it.criteria to evaluate it.

This might be a bit too open-ended to be This might be a bit too open-ended to be graded easily.graded easily.

34


Use Struppeck’s three criteria to evaluate Use Struppeck’s three criteria to evaluate Question 6.Question 6.

35



1)1) Easy to grade. OKEasy to grade. OK

36




2)2) Can be done quickly. OKCan be done quickly. OK

37




2)2) Can be done quickly. OKCan be done quickly. OK

3)3) Illustrates that we can use the three Illustrates that we can use the three criteria to do something. OKcriteria to do something. OK

38

Grading Exam QuestionsGrading Exam Questions

Assignments for grading will be distributed immediately after the exam as to which question(s) you are assigned to grade.

Questions are graded in pairs, just like writing exam questions.

A sample solution and a copy of the exam will be sent to the grading pair for review prior to getting the actual papers.

The actual papers come to the grading pair in the week or two following the examination.

39


Graders are encouraged to develop a grading key that accounts for the different combinations and permutations of answers that can be provided.

Graders are encouraged to use this grading key for a random set of questions (between 20-40) to ensure consistency in application of the grading key.

The graders should meet after grading this set of random questions to reconcile any differences and change the answer key if necessary.

40


In the subsequent weeks, individual graders will grade every candidate’s response and log the points in a Grading Program.

Graders are encouraged to reconcile scores along the way so that this work does not monopolize the time spent at the on-site Grading Session.

The on-site Grading Session is now being conducted in Las Vegas and lasts for two days.

41

Grading Exam QuestionsGrading Exam QuestionsOn-Site Grading SessionOn-Site Grading Session

The on-site Grading Session allows the following:

Allows those on the committee to meet with one another.

Small allowance of time to reconcile all scores between graders to no more than a ¼ point difference (differs by Part Chair).

Establishment by the committee of an appropriate passing score.

Re-grading of those candidate’s questions that are within a range of the passing score.

Discussion on how to make the exam even better for next year.

42

Grading Exam QuestionsGrading Exam QuestionsOn-Site Grading SessionOn-Site Grading Session

After the first day, a group activity occurs at night to allow those on the committee to interact with one another in a less formal manner.

The on-site Grading Session is concluded when the group has established a mutually agreed upon passing score by inspecting all the relevant statistics and grading/re-grading those candidates around the passing score to ensure that their score is correct.

All final scores by candidate, including grading keys and model solutions are left with the Part Chair to create the report to be sent to the Exam Officers.

43

What Have We Learned From What Have We Learned From The Chauncey Initiatives?The Chauncey Initiatives?

Questions should be focused on learning Questions should be focused on learning objectives, rather than individual papersobjectives, rather than individual papers

Triple True/False is not the only kind of Triple True/False is not the only kind of multiple choice questionmultiple choice question

Art of selecting good “wrong” multiple Art of selecting good “wrong” multiple choice answerschoice answers

44

What Changes Should The Candidates What Changes Should The Candidates See On The Exams?See On The Exams?

Better questionsBetter questions Questions with many possible full-credit Questions with many possible full-credit

answersanswers Less “according to” and “based on” Less “according to” and “based on”

questionsquestions Heavy “list” papers may become open-bookHeavy “list” papers may become open-book

45

Setting the Pass MarkSetting the Pass Mark

Identify Purpose of the Pass MarkIdentify Purpose of the Pass Mark Convene Pass Mark PanelConvene Pass Mark Panel Analyze Exam StatisticsAnalyze Exam Statistics Prepare RecommendationPrepare Recommendation Proceed through Approval ProcessProceed through Approval Process

46

Purpose of the Pass MarkPurpose of the Pass Mark

Establish Objective Pass/Fail CriterionEstablish Objective Pass/Fail Criterion Pass Minimally Qualified (or better) Pass Minimally Qualified (or better)

CandidatesCandidates Fail OthersFail Others Ensure Consistency between SittingsEnsure Consistency between Sittings

47

Purpose of the Pass MarkPurpose of the Pass Mark

Failures Passers

Minimally Qualified Candidate

48

Pass Mark PanelPass Mark Panel

Panel includes:Panel includes:New Fellows (1-3 years)New Fellows (1-3 years)Fellows experienced in practice areaFellows experienced in practice areaOfficers of exam committeeOfficers of exam committee

Recommends a pass mark independent of Recommends a pass mark independent of the normal exam committee proceduresthe normal exam committee procedures

49


Defines Minimally Qualified CandidateDefines Minimally Qualified CandidateWhat he or she will should knowWhat he or she will should knowWhat he or she will not knowWhat he or she will not knowWhat he or she will be able to What he or she will be able to

demonstrate on the examdemonstrate on the exam Relates Criteria to Learning Objectives for Relates Criteria to Learning Objectives for

defining the minimally qualified candidate.defining the minimally qualified candidate.

50


Each panelist independently estimates how 100 Each panelist independently estimates how 100 minimally qualified candidates will score on each minimally qualified candidates will score on each question (and sub-part of each question).question (and sub-part of each question).

Scores are assembled and shared in a group Scores are assembled and shared in a group format.format.

Group discusses ratings and may change estimatesGroup discusses ratings and may change estimates Facilitator compiles ratings and shares results with Facilitator compiles ratings and shares results with

exam committee officersexam committee officers

51

Analyze Exam Statistics (back at the Analyze Exam Statistics (back at the Grading Session in June)Grading Session in June)

Collect Initial Scores for All CandidatesCollect Initial Scores for All Candidates Review/Discuss Key MeasuresReview/Discuss Key Measures

High, Low, MeanHigh, Low, Mean Percentiles, Percentile RelationshipsPercentiles, Percentile Relationships Pass Mark Panel RecommendationPass Mark Panel Recommendation Prior statistics from previous examsPrior statistics from previous exams CAS Board goal CAS Board goal that 40% or more of the candidates that 40% or more of the candidates

should get a score of 70% or more on any given exam; should get a score of 70% or more on any given exam; and all candidates that get such a score should passand all candidates that get such a score should pass

Pick an initial pass mark and re-grade candidates within Pick an initial pass mark and re-grade candidates within certain range of pass mark (+/- 3 points, for example)certain range of pass mark (+/- 3 points, for example)

52

Prepare RecommendationPrepare Recommendation

Recollect scores if any have changed and Recollect scores if any have changed and review all relevant statistics again.review all relevant statistics again.

Review borderline candidates (tighter Review borderline candidates (tighter threshold now) and re-grade/review for threshold now) and re-grade/review for consistency with answer key.consistency with answer key.

Repeat process until only looking at the 5 Repeat process until only looking at the 5 exams above and the 5 exams below the exams above and the 5 exams below the recommended pass mark. recommended pass mark.

Justify Recommended Pass ScoreJustify Recommended Pass Score

53

Approval ProcessApproval Process

Part Chair Part Chair

General Officer (Spring / Fall)General Officer (Spring / Fall)

Exam Committee Chair (Arlie Proctor)Exam Committee Chair (Arlie Proctor)

VP-Admissions (Jim Christie)VP-Admissions (Jim Christie)