Embed Size (px)
Citation preview
|||© 2016 MCC | CMC
André De Champlain, PhDDirector, Psychometrics and
Assessment Services
Presentation at the MCC’s Annual General Meeting
Sept. 11, 2016
Setting the Stage for a More Continuous and Integrated Model of Assessment for Medical Licensure in Canada
Bridging the Transition
#MCCagm#AssembléeCMC
||© 2016 MCC | CMC
Rethinking the Nuts and Bolts of Assessment
• Reconceptualization of assessment over the past 2 decades• Over the past two decades, tremendous amount of thought & activity
aimed at proposing models of assessment & related processes that are:◦ More transparent & flexible◦ Better linked to learning activities◦ More informative from an educational standpoint
• Concurrently, effort has been devoted to improving processes necessary to support these re-envisioned assessments
• Revisitation of assessment’s epistemological core
||© 2016 MCC | CMC
Rethinking the Nuts and Bolts of Assessment• Reconceptualizition of assessment over the past 2
decades• Assessment paradigm shift
• Programmatic assessment (van der Vleuten et al., 2012)• Post-modern test theory (Mislevy, 1997)• Cognitively-based assessment of, for & as learning (CBAL
[Bennett, 2010])
• Use of technology to support this shifto Enhancement of test development practices (e.g. - automated
item generation [AIG])o For more frequent and flexible assessments
o Marking of open-ended responses & narrative texto To accommodate a broadening of measurement strategies
4||© 2016 MCC | CMC
Education and Assessment As Willing Partners
• Mechanisms through which learning occurs have shifted
• From traditional (paper-based) to electronic media◦ Tablet & mobile device-based learning is
ubiquitous (e.g., MedPage Today, QuantiaMD, etc.)
• Exponential growth of knowledge in medicine
5||© 2016 MCC | CMC
Education and Assessment As Willing Partners
• From a traditional view of education• Teacher-centered with high exam
scores as main goal
• To alternate models which stress learning, retention & integration of knowledge & skills using a host of assessment modalities (e.g., PBL, competency-based education, etc.)
6||© 2016 MCC | CMC
Education and Assessment As Willing Partners• Evolution of learning models & modalities not
completely mirrored by similar changes in educational assessment
• Educational assessment must evolve alongside learning models or risk fostering an antagonistic relationship
• Educational assessment must (Bennett, 2002):• Provide meaningful information• Satisfy multiple purposes• Use modern conceptions of competency as a design
basis• Design for positive impact & engagement• Use technology to achieve substantive goals
7||© 2016 MCC | CMC
Education and Assessment As Willing Partners• Evolution of learning models & modalities not
completely mirrored by similar changes in educational assessment
• Educational assessment must evolve alongside learning models or risk fostering an antagonistic relationship
• Educational assessment must (Bennett, 2002):• Provide meaningful information• Satisfy multiple purposes• Use modern conceptions of competency as a design
basis• Design for positive impact & engagement• Use technology to achieve substantive goals
||© 2016 MCC | CMC
Assessment Paradigm Shift
• Increasing dissatisfaction with established educational assessment models
• Unidirectional relationship between learning & assessment◦ Discrete, episodic hurdles to “overcome”◦ Decay/minimal transfer of knowledge and
skills
• Unlinked assessments
||© 2016 MCC | CMC
Assessment Paradigm Shift• Concerns
• Lack of overarching framework (program) to guide the design of the assessment tools along an educational/professional training continuum
◦ Plea for a macroscopic rather than microscopic view of assessment (de Rosnay, 1979)• Reductionist lens that is applied to what is a complex, adaptive system with interconnected
components & dynamic relationships• Missed opportunity to view learning & assessment in a rich, recursive relationship
◦ Both activities can dynamically inform each other◦ Critical to feed forwarding information or handover
||© 2016 MCC | CMC
Assessment Paradigm Shift
• Programmatic assessment• Calls for a deliberate, arranged set of longitudinal
assessment activities• Joint attestation of all data points for decision & remediation
purposes• Input of expert professional judgment is a cornerstone of this
model• Purposeful link between assessment & learning/remediation• Dynamic, recursive relationships between assessment & learning
points
||© 2016 MCC | CMC
Assessment Paradigm Shift
• Programmatic assessment• Application of a program evaluation framework to assessment
• Systematic collection of data to answer specific questions about a program
• Gaining popularity within several medical education settings• Competency-based workplace learning• Medical schools (e.g., Dalhousie, U of T, UBC, etc.)• Etc.
||© 2016 MCC | CMC
Wading Into the Programmatic Assessment Waters – The MCC Experience
Assessment Review Task Force (ARTF)• As the MCC approached it’s 100th anniversary (2012), a
task force of eminent Canadian medical educators was convened to undertake a reflective & strategic review of the MCC’s assessment purposes and objectives, their structure & their alignment with MCC’s major stakeholder requirements
• Report published in 2011 which contained six recommendations to fulfill including validating & updating the blueprint for MCC examinations, offering exams with greater flexibility, enhancing & standardizing IMG assessments & engaging in in-practice assessment
||© 2016 MCC | CMC
Tacit Sub-recommendation – The Need for a Macro-analysis
An intimated challenge outlined in the ARTF report pertains to the need to conduct a macro-analysis & review of the MCC Qualifying Examination (MCCQE)• Applying a systemic (macroscopic) lens to the MCCQE
as an integrated examination system • How are the components of the MCCQE interconnected &
how do they inform key markers along a physician’s educational & professional continuum?
• How can the MCCQE continue to evolve towards embodying an integrated, logically planned & sequenced system of assessments that mirrors the Canadian physician’s journey?
||© 2016 MCC | CMC
Wading Into the Programmatic Assessment Waters in Canada
Assessment Continuum for Canada (ACC)• Purpose
• To define the ‘life of a physician’ from the beginning of medical school to retirement in terms of assessments
• To propose a common, national framework of assessment using a programmatic approach
• ACC composition includes representation from the MCC, UGME/PGME, CPD, certification colleges, FMRAC and MRAs
• First outcomes include a definition of a CMG physician pathway with partners as well as broad agreement on key principles of assessment
• Next step – National summit (spring of 2017)
||© 2016 MCC | CMC
Practical Implications of a Pan-Canadian Program of Assessment for Medical Licensure
• Programmatic assessment is predicated on more frequent & flexible assessment via a combination of tools:
• Traditional exam formats• Lower-stakes, in-practice and other observations• Ancillary sources (e.g.: narratives)
• This shift impacts core assessment tasks including test development & scoring activities
• Assessments need to be developed, administered & scored more frequently
||© 2016 MCC | CMC
Practical Implications of a Pan-Canadian Program of Assessment for Medical Licensure
• Programmatic assessment rests on a concurrent development of a linked training and assessment continuum
• Both activities are equally important in informing each other• Assessment in and of itself is viewed as a critical learning activity
• Decisions are based not only on a sum of discrete elements but on a carefully developed composite of measures which reflects key learning outcomes in the trajectory of a physician’s medical education• What are the key data points that inform a decision in regard to entry into
supervised/independent/continuous practice and what learning outcomes do they reflect?
• How can key handoff (milestone) points be identified for all stakeholders that make sense within a licensing/certification context?
|||© 2016 MCC | CMC
PANEL Q&A
Q&AMedical Council of Canada 2016 Annual Meeting
#MCCagm#AssembléeCMC
