Upload
tranthuy
View
228
Download
2
Embed Size (px)
Citation preview
Clinical Simulation TestingClinical Simulation TestingA Webinar for Program Directors
March 2008National Board for Certification in Occupational Therapy, Inc.
Seminar FormatSeminar Format
Section 1Section 1Background and rationale for using clinical Background and rationale for using clinical simulation testing (CST) for occupational therapy simulation testing (CST) for occupational therapy credentialingcredentialing
Section 2Section 2Format and flow Format and flow Exam development processExam development process
Section 3Section 3Demonstration of a CST itemDemonstration of a CST item
Q & AQ & A
Section 1:Section 1:Background and RationaleBackground and Rationale
What is CST?What is CST?It is a format of assessment designed to It is a format of assessment designed to simulate the types of situations newly simulate the types of situations newly certified occupational therapy practitioners certified occupational therapy practitioners may encounter in practicemay encounter in practice
Background and RationaleBackground and Rationale
The items are designed to measure a The items are designed to measure a candidate’s knowledge and critical reasoning candidate’s knowledge and critical reasoning ability ability sequentiallysequentially across the continuum of across the continuum of care:care:
ScreeningScreeningFormulating conclusions regarding treatment Formulating conclusions regarding treatment needs and prioritiesneeds and prioritiesImplementing interventionsImplementing interventionsAssessing outcomesAssessing outcomes
Background and RationaleBackground and Rationale
How is CST different from Multiple Choice How is CST different from Multiple Choice Testing (MCT) formats?Testing (MCT) formats?
MCT measures breadth of knowledgeMCT measures breadth of knowledgejudgment is made in isolation and the items are judgment is made in isolation and the items are nonnon--sequential in naturesequential in nature
CST complements MCT formats by examining CST complements MCT formats by examining depth of knowledge sequentially where decisions depth of knowledge sequentially where decisions are tied to previous decisions and resultsare tied to previous decisions and results
Why Now?Why Now?
The move to CST came from the need to The move to CST came from the need to measure critical reasoning skills and was measure critical reasoning skills and was driven by:driven by:
NBCOT NBCOT practice analysis studiespractice analysis studiesmissionmission
Revised ACOTE standardsRevised ACOTE standardsCertification industry trendsCertification industry trends
NBCOTNBCOT®®
Practice Analysis StudiesPractice Analysis StudiesCritical foundation for developing a Critical foundation for developing a psychometrically sound and legally defensible psychometrically sound and legally defensible credentialing examinationcredentialing examinationAccreditation agencies require systematic reAccreditation agencies require systematic re--evaluation of content standards to ensure evaluation of content standards to ensure appropriateness of the examination:appropriateness of the examination:
content validcontent validappropriately evaluates what is required to function appropriately evaluates what is required to function as a competent practitioneras a competent practitioner
NBCOTNBCOT®®
Results from the latest NBCOT practice Results from the latest NBCOT practice analysis study:analysis study:
0101 Evaluate the client on an ongoing basis using 0101 Evaluate the client on an ongoing basis using appropriate tools, procedures, and protocols in order appropriate tools, procedures, and protocols in order to determine factors that impact participation in to determine factors that impact participation in occupationoccupation0303 Use critical reasoning to select and implement 0303 Use critical reasoning to select and implement interventions and approaches consistent with interventions and approaches consistent with psychosocial and cognitive abilities, and client needs psychosocial and cognitive abilities, and client needs in order to facilitate outcomes within areas of in order to facilitate outcomes within areas of occupationoccupation
(NBCOT OTR Practice Analysis Study, 2007)(NBCOT OTR Practice Analysis Study, 2007)
NBCOTNBCOT®® MissionMission
Mission “to serve the public interest:”Mission “to serve the public interest:”Assure key stakeholders that certificants have met Assure key stakeholders that certificants have met a minimum practice standard by differentiating a minimum practice standard by differentiating between competent and incompetent practitionersbetween competent and incompetent practitionersJust as in real life, through the sequential nature of Just as in real life, through the sequential nature of the CST, errors may be compounded on top of the CST, errors may be compounded on top of errors resulting in nonproductive assessment and errors resulting in nonproductive assessment and intervention proceduresintervention procedures
Revised ACOTE StandardsRevised ACOTE Standards
Transition to postTransition to post--baccalaureate degree and baccalaureate degree and revised ACOTE standards:revised ACOTE standards:
“The rapidly changing and dynamic nature of “The rapidly changing and dynamic nature of contemporary health and human services delivery contemporary health and human services delivery systems requires the occupational therapist to systems requires the occupational therapist to possess basic skills as a direct care provider, possess basic skills as a direct care provider, consultant, educator, manager, researcher, and consultant, educator, manager, researcher, and advocate for the profession and the consumer.”advocate for the profession and the consumer.”
Preamble to The Standards 2008Preamble to The Standards 2008
ACOTE StandardsACOTE Standards
B5.26 Organize, collect and analyze data in a B5.26 Organize, collect and analyze data in a systematic manner for evaluation of practice systematic manner for evaluation of practice outcomes. Report evaluation results and outcomes. Report evaluation results and modify practice as needed to improve modify practice as needed to improve outcomes.outcomes.
Certification Industry TrendsCertification Industry Trends
Consistent with certification examination Consistent with certification examination practices of other allied health and medical practices of other allied health and medical certification bodies who use defensible certification bodies who use defensible constructs to assess critical clinical constructs to assess critical clinical competenciescompetencies
i.e. Physician Assistants, Counseling i.e. Physician Assistants, Counseling
Historical BackgroundHistorical Background
Branching logic/simulation testing introduced by the Branching logic/simulation testing introduced by the National Board for Medical Examiners in the 1960s National Board for Medical Examiners in the 1960s as an objective, reliable evaluation of clinical as an objective, reliable evaluation of clinical competence in the medical professioncompetence in the medical profession
(Hubbard et al, 1965)(Hubbard et al, 1965)Seminal studies in the medical professions have Seminal studies in the medical professions have concluded there is a statistically significant concluded there is a statistically significant relationship between results obtained by exam relationship between results obtained by exam candidates and those obtained by experienced candidates and those obtained by experienced practitioners taking CST examinationspractitioners taking CST examinations
(Grace et al, 1977)(Grace et al, 1977)
Historical BackgroundHistorical Background
The 1980s saw the emergence of CST testing The 1980s saw the emergence of CST testing formats in the behavioral sciences formats in the behavioral sciences
(Berven & Scofield, 1980)(Berven & Scofield, 1980)
This led to CST being identified as a way to This led to CST being identified as a way to evaluate competence for high stakes evaluate competence for high stakes certification and licensure purposescertification and licensure purposes
(Forker & McDonald, 1996; (Forker & McDonald, 1996; Bersky, Krawczak & Kumar, 1998)Bersky, Krawczak & Kumar, 1998)
ReferencesReferences
Hubbard, J., Levit E., Schumacher, C., & Schnabel, T. (1965). AHubbard, J., Levit E., Schumacher, C., & Schnabel, T. (1965). An n objective evaluation of clinical competence. objective evaluation of clinical competence. The New England The New England Journal of Medicine.Journal of Medicine. 272 (25): 1321272 (25): 1321--1328.1328.
Grace, M., Hanson, J, Fincham, S., Skakun, E & Taylor, W (1977).Grace, M., Hanson, J, Fincham, S., Skakun, E & Taylor, W (1977). A A scoring technique for computerized patient management scoring technique for computerized patient management problems. problems. Medical EducationMedical Education, 11: 335, 11: 335--340.340.
Berven N. & Scofield, M. (1980). Evaluation of clinical problemBerven N. & Scofield, M. (1980). Evaluation of clinical problem--solving skills through standardized casesolving skills through standardized case--management management simulations. simulations. Journal of Counseling PsychologyJournal of Counseling Psychology, 27 (2):199, 27 (2):199--208.208.
Forker, J., & McDonald, M. (1996). Methodologic trends in the Forker, J., & McDonald, M. (1996). Methodologic trends in the healthcare professions: computer adaptive and computer healthcare professions: computer adaptive and computer simulation testing. simulation testing. Nurse EducationNurse Education. 21 (4): 13. 21 (4): 13--4.4.
Bersky A., Krawczak, J., & Kumar, T. (1998). Computerized cliniBersky A., Krawczak, J., & Kumar, T. (1998). Computerized clinical cal simulation testing. A new look for the NCLEXsimulation testing. A new look for the NCLEX--RN RN examination? examination? Nurse EducationNurse Education. 23(1): 20. 23(1): 20--5.5.
NBCOTNBCOT®®
CST Alpha TestingCST Alpha TestingCompleted last quarter of 2007Completed last quarter of 2007N = 50 N = 50
(Beta test N = 250)(Beta test N = 250)
OTR OTR certificantscertificants across geographic and practice across geographic and practice settingssettings0 0 –– 18 months since initial certification18 months since initial certificationEnd of test surveyEnd of test surveyPerformance statistics Performance statistics
NBCOTNBCOT®® CST Alpha TestingCST Alpha Testing
Preliminary Results:Preliminary Results:Over 80% indicated CST as an appropriate style Over 80% indicated CST as an appropriate style of testing for occupational therapy credentialingof testing for occupational therapy credentialingMajority said the format was a fair assessment of Majority said the format was a fair assessment of their occupational therapy knowledgetheir occupational therapy knowledge
Alpha ResultsAlpha Results
“Really replicates what I’m doing everyday “Really replicates what I’m doing everyday in my practice.”in my practice.”“MCT just tests snippets of knowledge, the “MCT just tests snippets of knowledge, the CST goes into more depth CST goes into more depth –– tested my tested my problem solving ability.”problem solving ability.”“… gave me more information to build up a “… gave me more information to build up a picture about the client.”picture about the client.”
Alpha ResultsAlpha Results
“I preferred having questions related to one “I preferred having questions related to one case rather than jumping around from case rather than jumping around from diagnosis to diagnosis like in the MCT diagnosis to diagnosis like in the MCT format.”format.”“If you’ve done 2 good level II fieldwork “If you’ve done 2 good level II fieldwork placements, you really should have no placements, you really should have no problem answering these CST questions.” problem answering these CST questions.”
Alpha ResultsAlpha Results
“Fieldwork is where you put all your “Fieldwork is where you put all your knowledge into practice by using your knowledge into practice by using your clinical judgment and common sense. clinical judgment and common sense. That’s exactly what you have to do when That’s exactly what you have to do when answering a CST item.”answering a CST item.”
Section 2:Section 2:Format and FlowFormat and Flow
2006Advisory Taskforce
BOD approves projectPresentation to program directors
2007CST Development Team
Alpha TestingPresentation & mailing to program directors
2008Beta testing
Program director webinarsStudy tools
Project Timeframe
Timeframe ContinuedTimeframe Continued
A Standard Setting study will be completed in A Standard Setting study will be completed in May:May:
Determine passing standard for the OTR Determine passing standard for the OTR certification examinationcertification examination
Beginning 2009, the OTR examination will be a Beginning 2009, the OTR examination will be a hybrid test consisting of:hybrid test consisting of:
MCT itemsMCT itemsCST itemsCST items
Exam will continue to be ADA compliantExam will continue to be ADA compliant
Exam Development Exam Development ProcessProcess
Practice Analysis StudyPractice Analysis Study
Completed systematically to ensure the Completed systematically to ensure the examination evaluates what is required examination evaluates what is required to function as a competent practitionerto function as a competent practitioner
Practice Analysis StudyPractice Analysis Study
Results from the practice analysis study are Results from the practice analysis study are used at all stages of examination development:used at all stages of examination development:
Creation of test blueprintCreation of test blueprintItem writingItem writingExamination constructionExamination construction
Latest Practice Analysis StudyLatest Practice Analysis Study
Completed in 2007Completed in 2007Initial development and validationInitial development and validationPilot studyPilot studyFull validation surveyFull validation surveyDevelopment of test blueprintsDevelopment of test blueprints
Implemented January 2009Implemented January 2009
2009 OTR Blueprint Specifications2009 OTR Blueprint Specifications
Domain 1Domain 1Gather information regarding factors that influence Gather information regarding factors that influence occupational performanceoccupational performance
Domain 2Domain 2Formulate conclusions regarding the client’s needs and Formulate conclusions regarding the client’s needs and priorities to develop a clientpriorities to develop a client--centered intervention plancentered intervention plan
Domain 3Domain 3Select and implement evidenceSelect and implement evidence--based interventions to support based interventions to support participation in areas of occupation throughout the continuum participation in areas of occupation throughout the continuum of careof care
Domain 4Domain 4Uphold professional standards and responsibilities to promote Uphold professional standards and responsibilities to promote quality in practicequality in practice
MCT Development ProcessMCT Development Process
Written to 2009 blueprint specificationsWritten to 2009 blueprint specificationsValidatedValidated
Subject matter experts (OTR practitioners and Subject matter experts (OTR practitioners and educators)educators)References References Performance statisticsPerformance statistics
Statistics collected to ensure psychometric Statistics collected to ensure psychometric integrity and validityintegrity and validityReviewed and approved by independent testing Reviewed and approved by independent testing agencyagencyComplies with accreditation standardsComplies with accreditation standards
Format of MCT ItemsFormat of MCT Items
Computer deliveredComputer deliveredTutorialTutorialQuestion stemQuestion stem4 response options4 response options
1 correct answer1 correct answer3 3 distractorsdistractors
Example of an MCT ItemExample of an MCT Item
An inpatient who hasAn inpatient who has Guillain BarreGuillain Barre syndrome is syndrome is referred to OT to maximize independence with ADL. referred to OT to maximize independence with ADL. The patient has proximal weakness in upper and lower The patient has proximal weakness in upper and lower extremities. Which method is extremities. Which method is BESTBEST for the OTR to for the OTR to use when assessing the patient’s initial functional use when assessing the patient’s initial functional status?status?
A.A. Ask the nursing staff about the patient’s abilitiesAsk the nursing staff about the patient’s abilitiesB.B. Have the patient complete a lifestyle inventoryHave the patient complete a lifestyle inventoryC.C. Interview family members during visiting hoursInterview family members during visiting hoursD.D. Observe the patient completing ADL tasksObserve the patient completing ADL tasks
Correct answer is…Correct answer is…
D. Observe the patient completing ADL D. Observe the patient completing ADL tasks.tasks.
OObserving the patient’s performance of ADL bserving the patient’s performance of ADL tasks will provide the most accurate and tasks will provide the most accurate and measurable information about the individual’s measurable information about the individual’s functional level.functional level.
Overview of CST Development ProcessOverview of CST Development Process
Written to 2009 OTR blueprint specificationsWritten to 2009 OTR blueprint specificationsValidatedValidated
Subject matter experts (OTR, practitioners and Subject matter experts (OTR, practitioners and educators)educators)References References Performance statisticsPerformance statistics
Statistics collected to ensure psychometric Statistics collected to ensure psychometric integrity and validityintegrity and validityReviewed and approved by independent testing Reviewed and approved by independent testing agencyagencyComplies with accreditation standardsComplies with accreditation standards
CST Item MakeupCST Item Makeup
TutorialTutorialAn opening sceneAn opening sceneSections with section headersSections with section headersA series of decisions/action statementsA series of decisions/action statements
Opening Scene
Section Heading
Format & FlowFormat & Flow
Candidates should select the decisions/actions Candidates should select the decisions/actions that they deem appropriate from the list of that they deem appropriate from the list of options providedoptions providedAs the decision/action statement is selected, As the decision/action statement is selected, a feedback box will appeara feedback box will appearFeedback contains information regarding the Feedback contains information regarding the consequence of selecting that decision and is consequence of selecting that decision and is used to make future selectionsused to make future selections
Feedback Boxes
Format & FlowFormat & FlowThe decisions/action list consists of:The decisions/action list consists of:
Positive responses Positive responses Optimal routeOptimal routePoints awarded if candidate selects these responsesPoints awarded if candidate selects these responses
Neutral responsesNeutral responsesMay be helpful, but not absolutely necessary in solving May be helpful, but not absolutely necessary in solving the problemthe problemPoints neither awarded nor deductedPoints neither awarded nor deducted
Negative responsesNegative responsesHinder or distract from solving the problemHinder or distract from solving the problemMay be harmful to the clientMay be harmful to the clientPoints deducted if candidate selects these optionsPoints deducted if candidate selects these options
Format & FlowFormat & Flow
Candidate will have the ability to scroll back Candidate will have the ability to scroll back through the CST item to view:through the CST item to view:
Opening sceneOpening sceneSection headersSection headersItems they have selectedItems they have selected
Just as in real life, once the decision/action has Just as in real life, once the decision/action has been checked, the candidate will not be able to been checked, the candidate will not be able to dede--select the responseselect the response
SummarySummary--Each CST Will Have:Each CST Will Have:
Opening Scene
Series of Sections
Each section has a section header
List of decisions/actions+ neutral -
As a decision is made, a feedback box willprovide additional information assisting the candidate
to progress through the item
Section 3Section 3
PrimerPrimerDemonstrationDemonstrationQ&AQ&A
PrimerPrimer
CST item is about an OTR working in a school CST item is about an OTR working in a school systemsystemTeacher refers student for OT due to concerns Teacher refers student for OT due to concerns with classroom performance and fine motor with classroom performance and fine motor skillsskillsStudent has an older sibling with autismStudent has an older sibling with autismCST progresses from screening through to CST progresses from screening through to selection of activities to meet IEP goalsselection of activities to meet IEP goals
Primer ContinuedPrimer Continued
Contains 4 sections:Contains 4 sections:Screening Screening
11 potential decisions/actions listed11 potential decisions/actions listed
Assessment Assessment 9 potential decisions/actions listed9 potential decisions/actions listed
Recommendations to IEP teamRecommendations to IEP team11 potential decisions/actions listed11 potential decisions/actions listed
Activities supporting IEP goalsActivities supporting IEP goals7 potential decisions/actions listed7 potential decisions/actions listed
DemonstrationDemonstration
NBCOT® Inc.12 S. Summit Ave, Suite 100
Gaithersburg, MD 21877-4150www.nbcot.org