Assessing Educational Assessing Educational EffectivenessEffectiveness

Lisa M. Beardsley-Hardy, PhD, MPH, MBALisa M. Beardsley-Hardy, PhD, MPH, MBADirector of EducationDirector of Education

General Conference of Seventh-day AdventistsGeneral Conference of Seventh-day Adventists

Why Assessment MattersWhy Assessment MattersProvide Feedback Provide Feedback Direct efforts towards learning Direct efforts towards learning outcomesoutcomesStudentStudentTeacherTeacher

Demonstrate accountabilityDemonstrate accountabilityBasis for curriculum refinement Basis for curriculum refinement and institutional learningand institutional learning

Goal of EducationGoal of EducationHigh mean, small SDHigh mean, small SD

Junior medicine clerkship, First Junior medicine clerkship, First and Third-Year Internal Medicine and Third-Year Internal Medicine ResidentsResidents

Vital exam results for sophomore Vital exam results for sophomore medical studentsmedical students

Role of FeedbackRole of FeedbackEssential cognitive activity to Essential cognitive activity to ensure accuracy in acquisition ensure accuracy in acquisition of new knowledge and skill of new knowledge and skill (summarize, elaborate, monitor (summarize, elaborate, monitor for accuracy)for accuracy)

– Radiology: practice does not make Radiology: practice does not make perfect without feedback!perfect without feedback!

Direct Efforts to Learning Direct Efforts to Learning OutcomesOutcomes

StudentStudent

TeacherTeacherDetermine competencies required at Determine competencies required at next level and assess that next level and assess that knowledge and skillsknowledge and skills

““WeWe’’re getting better studentsre getting better students””

Demonstrate AccountabilityDemonstrate AccountabilityStudentsStudents

ParentsParents

Board of TrusteeBoard of Trustee

DonorsDonors

Accrediting AgenciesAccrediting Agencies

Basis for Curriculum Basis for Curriculum RefinementRefinement

Course (instructor)Course (instructor)Major (department faculty)Major (department faculty)General Education (interdisciplinary)General Education (interdisciplinary)Degree (faculty senate, graduate Degree (faculty senate, graduate council, academic affairs)council, academic affairs)Alverno College: videotaped interviewsAlverno College: videotaped interviewsCSU Monterey Bay: capstone projects CSU Monterey Bay: capstone projects http://csumb.edu/site/x13842.xmlhttp://csumb.edu/site/x13842.xml

University of Illinois: Objective University of Illinois: Objective Structured Clinical Exams and Senior CompsStructured Clinical Exams and Senior Comps

What to Measure?What to Measure?KnowledgeKnowledgeSkills (procedures, clinical Skills (procedures, clinical skills, complex behavioral skills, complex behavioral sequences, interpersonal, sequences, interpersonal, teamwork)teamwork)

Problem-solving skills (PBL)Problem-solving skills (PBL)AttitudesAttitudesValuesValuesSpiritual commitmentSpiritual commitment

KnowledgeKnowledgeMCQMCQ

Norm-referenced vs. criterion Norm-referenced vs. criterion referenced (Nedelsky or Angoff referenced (Nedelsky or Angoff method)method)

Essay, journal, thesisEssay, journal, thesis

SkillsSkillsObjective Structured Clinical Objective Structured Clinical Exam (OSCE)Exam (OSCE)

Multi-station, composed of subscalesMulti-station, composed of subscales

Assess interpersonal skills Assess interpersonal skills (subjective)(subjective)

Recording by standardized patientRecording by standardized patient

OSCEsOSCEsReduce test variance Reduce test variance (standardized patients/problems)(standardized patients/problems)

Reduce rater variance (eagles Reduce rater variance (eagles and doves)and doves)

Reduce environmental variance Reduce environmental variance (same time, testing conditions)(same time, testing conditions)

Reduce cheating (proctors, Reduce cheating (proctors, application of knowledge)application of knowledge)

INCREASE variance due to student INCREASE variance due to student

Problem-Solving SkillsProblem-Solving SkillsTriple Jump (assign problem and Triple Jump (assign problem and assess problem-solving strategy; assess problem-solving strategy; assess outcome after 24 hours)assess outcome after 24 hours)

Modified essay exam Modified essay exam (progressively provide (progressively provide additional information)additional information)

Peer assessment on teamworkPeer assessment on teamworkProblems requiring application Problems requiring application of knowledge and skillsof knowledge and skills

O = T + EO = T + E

Observed score is composed of Observed score is composed of true score and error (Classical true score and error (Classical test theory)test theory)

Basis for calculation of Basis for calculation of CronbachCronbach’’s alpha and reliability s alpha and reliability coefficients such as KR-20coefficients such as KR-20

Fair ExamsFair Exams

Do not use dishonest standards when Do not use dishonest standards when measuring length, weight or measuring length, weight or quantity. Use honest scales and quantity. Use honest scales and honest weights, honest weights, honest multiple honest multiple choice questionschoice questions and an honest and an honest comprehensive examcomprehensive exam. I am the LORD . I am the LORD your God, who brought you out of your God, who brought you out of Egypt. Egypt.

(Lev. 19:35-36, ephah and hin in NIV)(Lev. 19:35-36, ephah and hin in NIV)

Sources of ErrorSources of ErrorStudent (illness, mistakes)Student (illness, mistakes)Teacher (eagles vs. doves)Teacher (eagles vs. doves)Environment (noise, Environment (noise, distractions, defective distractions, defective equipment)equipment)

Testing instrument (ambiguity in Testing instrument (ambiguity in items, two distracters correct)items, two distracters correct)

CheatingCheating

Sources of Error: CheatingSources of Error: CheatingPrevention: integrity statement in Prevention: integrity statement in syllabus to include plagiarism, syllabus to include plagiarism, appropriate testing behavior, appropriate testing behavior, methods of prevention such as methods of prevention such as electronic monitoring of papers and electronic monitoring of papers and videotaping)videotaping)

Control testing environmentControl testing environmentConfirm suspicions in writing; legal Confirm suspicions in writing; legal counselcounsel

Deal with substantiated incidents as Deal with substantiated incidents as part of studentpart of student’’s educations education

Fair ExamsFair ExamsUse test blueprintUse test blueprintEstablish cut-off scoreEstablish cut-off scoreProvide opportunity to review Provide opportunity to review test results (after scoring but test results (after scoring but before final cut-off score)before final cut-off score)

Use item-analysis to modify Use item-analysis to modify cut-offcut-off

Apply SEM for high-stakes examsApply SEM for high-stakes exams

Test BlueprintTest BlueprintBased on learning objectivesBased on learning objectives

Determines test construction Determines test construction (items sampled and weight (items sampled and weight assigned to subscales)assigned to subscales)

Basis for retestingBasis for retesting

Test Blueprint Mirrors ObjectivesTest Blueprint Mirrors ObjectivesObjective: Upon Objective: Upon completion of this completion of this course (program) the course (program) the student will be able student will be able to: to:

Main Main testtest

Main Main testtest

Make-Make-up up testtest

Make-Make-up up testtest

1 define…1 define…

2 interpret….2 interpret….

3 perform a….3 perform a….

4 evaluate…4 evaluate…

5 determine…5 determine…

6 solve…6 solve…

7 create…7 create…

Mene, Mene, Tekel, ParsinMene, Mene, Tekel, Parsin

MeneMene : God has numbered the days of : God has numbered the days of your reign and brought it to an end. your reign and brought it to an end.

TekelTekel : You have been weighed on the : You have been weighed on the scales and found wanting. scales and found wanting.

PeresPeres : Your kingdom is divided and : Your kingdom is divided and given to the Medes and Persians.given to the Medes and Persians.““

(Dan 5:26-28 NIV)(Dan 5:26-28 NIV)

DiscussionDiscussionObservationsObservations

QuestionsQuestions

RecommendationsRecommendations

Next stepsNext steps