Upload
examsoft
View
20
Download
0
Tags:
Embed Size (px)
Citation preview
2ExamSoft Assessment Conference | #ASSESS2015
Implementing & Launching Rubrics in a Clinical Setting
4ExamSoft Assessment Conference | #ASSESS2015
Learning Objectives
• Develop a plan to implement ExamSoft rubrics on a large scale at their institution and in a clinical setting.
• Discuss the benefits of using ExamSoft rubrics in both a simulated and live clinical setting.
• Determine methods of addressing issues during pilot testing of performance-based assessments.
• Understand the importance of pulling longitudinal reports using rubric results to help improve student outcomes.
5ExamSoft Assessment Conference | #ASSESS2015
Presentation Overview
• Introduction of Presenters• Background on College of Dental Medicine at Western
University of Health Sciences• Laying out the Logistics• Faculty Buy-in and Training• Pilot Testing with Lead Faculty• Addressing Issues during Pilot Test Period• Launching ExamSoft Rubrics in a Live-Patient Setting• Live Demo Simulation of Rubrics • Our Conclusions and Plans for Improvement• Interactive Q&A with Presenters
8ExamSoft Assessment Conference | #ASSESS2015
College of Dental Medicine
• Clinical Exams– On The Fly– Challenged Individually– Two Faculty Graders– 3rd & 4th year students– 10 Clinical Exams
9ExamSoft Assessment Conference | #ASSESS2015
Previous Process
• Staff Time• Resources (i.e. 2 Staff Members + Paper)• Temporarily Misplacing Rubrics• Turnaround Time for Releasing Scores• Calculation Errors
10ExamSoft Assessment Conference | #ASSESS2015
Previous Process
Student challenges
clinical exam
2 faculty grade on
paper rubrics
Faculty submit
rubrics to clinic staff member
Staff member
calculates scores
Staff member delivers
rubrics to Office of
Academic Affairs (OAA)
OAA staff scans & files
rubric, releases score to
student, and updates
grade sheet.
1 clinic session
1 clinic session Varies 3 hours Varies 8 hours
12ExamSoft Assessment Conference | #ASSESS2015
Simulated Clinical Exams
• Beta-tested ExamSoft rubric feature (Feb. through April 2014)– Mock WREB Endodontics – Complex Restoration
13ExamSoft Assessment Conference | #ASSESS2015
Before Implementation
• Establish long-term goals• Have a clear vision of what you need• Develop an implementation process
14ExamSoft Assessment Conference | #ASSESS2015
Establish Long-Term Goals
• Determine types of reports to generate• Create a timeline for implementation• Assign roles or tasks
16ExamSoft Assessment Conference | #ASSESS2015
Clear Vision
• Live Patient Clinical Exams• Support from Administration• Unique clinical exam logistics requires
customized ExamSoft set-up
17ExamSoft Assessment Conference | #ASSESS2015
Implementation Process
• Set deadlines• Assign roles to staff members
– Inputting rubrics– Assigning graders– Training faculty members– Pulling reports– Managing grade score release
18ExamSoft Assessment Conference | #ASSESS2015
Implementation Process
• ExamSoft set-up– Individual Courses for Students– Created Rubric & Rubric Templates– Assigned all 10 rubrics to each student
19ExamSoft Assessment Conference | #ASSESS2015
Implementation Process
• SharePoint form developed– Form submitted by student notifying staff of
CE date, CE rubric, and faculty graders– Submission of form initiates SharePoint
workflow – Workflow generates email sent to college
ticketing system– OAA staff member tracks CE tickets and
manages grader assignments in ExamSoft
20ExamSoft Assessment Conference | #ASSESS2015
Overview of Implementation Process
Set Deadlines
Assign roles to staff members
Set-Up ExamSoft
Develop Info Gathering
Form
21ExamSoft Assessment Conference | #ASSESS2015
Faculty Buy-In and Training
• Support from college administration• Explain advantages of ExamSoft rubrics
– Prevents misplacement of rubrics – Minimizes use of paper – Mobile device grading– Decreases likelihood of miscalculations due to
human error– Rubric creation is under 10 minutes– Immediate release of results to students
22ExamSoft Assessment Conference | #ASSESS2015
Faculty Buy-In and Training
• Small group faculty training session – 2 sessions held– 1 hour long– 12 faculty selected for pilot test– Staff provided cheat sheet for reference
• Staff developed step-by-step instruction manual
• Faculty encouraged to utilize personal devices• Staff offered technical support
24ExamSoft Assessment Conference | #ASSESS2015
Pilot Testing
• Pilot test period: December 2014 through February 2015
• Clinical faculty leaders were assigned as ExamSoft graders
• 4 of the 10 clinical exam’s were tested• Provided feedback and suggestions for
improvement
25ExamSoft Assessment Conference | #ASSESS2015
Pilot Testing
• Staff support during test period– Observed faculty graders for 2 clinical
sessions– Obtained feedback from faculty while on the
clinic floor – Made available via phone and e-mail for any
technical or operator-related issues
26ExamSoft Assessment Conference | #ASSESS2015
Pilot Testing
• Modification of SharePoint form– Version 1: Form fields for 2 graders– Version 2: Form fields for up to 6 graders
• Allowed for 2 graders per dimension
– Version 3: Finalized following pilot test period
27ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
• Major Issues:– Faculty login
• Graders had difficulty remembering passwords• Last minute changes to grader assignments
– Multiple faculty grading various dimensions within single rubric
– Lack of tracking for releasing scores
28ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
• Minor Issues:– Students requesting faculty graders not in
pilot test group– Incomplete assessments– High volume of reminder e-mails sent to
faculty
29ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
• Solutions:– Created 2 general faculty accounts per clinic
group– Updated SharePoint form for better tracking of
clinical exams– Changed deadline of clinical exam
assessments to the end of each week
30ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
Western University of Health
Sciences
College of Dental Medicine
31ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
Western University of Health
Sciences
College of Dental Medicine
32ExamSoft Assessment Conference | #ASSESS2015
Addressing Issues During Pilot Test Period
Western University of Health
Sciences
College of Dental Medicine
33ExamSoft Assessment Conference | #ASSESS2015
Launching ExamSoft in a Live-Patient Setting
• Clinical Dean issued a deadline to faculty for final submission of paper rubrics
• Faculty in pilot group trained remaining clinical faculty members
• Academic Affairs staff provided students with updated protocol for challenging clinical exams
34ExamSoft Assessment Conference | #ASSESS2015
Launching ExamSoft in a Live-Patient Setting
• Academic Managers set up auto login to ExamSoft grading portal on clinic iPads– Devices were secured through passcode
• Academic Managers facilitated ExamSoft set up for off-site clinics
35ExamSoft Assessment Conference | #ASSESS2015
New Clinical Exam Process
Student challenges
clinical exam/
Initiates workflow
OAA staff assigns faculty graders
2 faculty grade clinical
exam
OAA staff releases score to
student and updates
grade sheet.
1 clinic session
5 minutes
1 clinic session 2 hours
37ExamSoft Assessment Conference | #ASSESS2015
Conclusions
• Streamlines process of clinical exam grading– Smoother workflow– More efficient use of staff time– Minimizes likelihood of miscalculations due to
human error– Quicker turnaround for releasing scores
38ExamSoft Assessment Conference | #ASSESS2015
Conclusions
• Generate meaningful data– Demonstrate connections between
dimensions of a rubric and tagged categories– Translate data into robust reports for
university’s outcomes assessment committee and accreditation bodies
– Short-term & long-term reports for individual students and as a cohort
– Recommend improvements to the curriculum
39ExamSoft Assessment Conference | #ASSESS2015
Conclusions
• Generate meaningful data– Identify longitudinal trends such as
competencies in which students either excel in or need more focused attention
– Tag rubrics with “soft skills” such as professionalism, communication, and humanism
40ExamSoft Assessment Conference | #ASSESS2015
Plans for Improvements
• Brainstorm methods to identify critical errors in grading
• Refine rubric content • Continuous feedback from faculty, staff and
student• Offer development suggestions to ExamSoft
account managers