11

IMPROVEMENT OF RELIABILITY, VALIDITY, AND IMPROVEMENT OF RELIABILITY, VALIDITY, AND FEASIBILITY OF THE OBJECTIVE STRUCTURED FEASIBILITY OF THE OBJECTIVE STRUCTURED

CLINICAL EXAMINATION (OSCE) WHEN CLINICAL EXAMINATION (OSCE) WHEN ASSESSING THE PROBLEM SOLVING SKILLS OF ASSESSING THE PROBLEM SOLVING SKILLS OF

FINAL YEAR SURGICAL CLERKSHIPFINAL YEAR SURGICAL CLERKSHIP

Mohammed Y. Al-Naami*, Omar El FarouqMohammed Y. Al-Naami*, Omar El Farouq,, Gamal Khairy, Safdar Mofti, and Nawaz AnjumGamal Khairy, Safdar Mofti, and Nawaz Anjum

22

BIO-DATABIO-DATA**

Mohammed Y. Al NaamiMohammed Y. Al Naami MBBS (1986),FRCSC (1991), M Ed. (1992)MBBS (1986),FRCSC (1991), M Ed. (1992)

Associate ProfessorAssociate Professor Head, General Surgery DivisionHead, General Surgery Division

Director, Faculty Development ProgramDirector, Faculty Development ProgramCollege of Medicine, King Saud UniversityCollege of Medicine, King Saud University

Riyadh, Saudi ArabiaRiyadh, Saudi Arabia

33

INTRODUCTIONINTRODUCTION

• King Saud University founded in 1957.King Saud University founded in 1957.

• College of Medicine founded in 1967.College of Medicine founded in 1967.

• Affiliated King Abdulaziz University Hospital # of beds = 104 Affiliated King Abdulaziz University Hospital # of beds = 104

• Affiliated King Khalid University Hospital # of beds = 785.Affiliated King Khalid University Hospital # of beds = 785.

• Number of students admitted annually approximately 300.Number of students admitted annually approximately 300.

• Traditional 6 years Annual Curriculum.Traditional 6 years Annual Curriculum.

44

INTRODUCTIONINTRODUCTION

• Department of Surgery has 2 undergraduate courses: 1. Basic Surgery - during 3rd year - Clinical: 6 weeks Lectures: 4 weeks .

2. Clinical surgery - during 5th year - Clinics,OR’s, Bedside, Rounds &

Tutorials – 10 weeks

• Number of students averages 100 per course.

• Clinical examinations (1 long & 2 short cases X many years) were associated with rising dissatisfaction.

• First OSCE was introduced in May 2005 (r=0.68-0.79), validity at least better than clinical examinations, and feasibility was very low.

55

AIMAIM

To improve previous courses’ OSCEsTo improve previous courses’ OSCEs

by using more problem solving skills by using more problem solving skills stationsstations

66

METHODOLOGYMETHODOLOGY• 2 mornings OSCE involving 48 and 47 students in days

1 and 2 respectively.

• Each morning students were split into 2 halves examining them simultaneously through 5 minutes x 24 stations in 2 separate surgical wards (120 minutes / OSCE).

• 15 active and 9 rest OSCE stations - stability, internal consistency, and concurrent validity were determined using BMPD® statistical software.

77

Table 1: OSCE stations reliability (Day1) CompetencyCompetencyDescriptionDescriptionCronbach’s AlphaCronbach’s Alpha(Stability)(Stability)

SMC-Carmine’s ThetaSMC-Carmine’s Theta(Internal consistency)(Internal consistency)

Surgical Surgical SpecialtySpecialty

11Laryngoscopy and intubation Laryngoscopy and intubation (PS)(PS)

0.87400.87400.993360.99336AnesthesiaAnesthesia

22Intercostal chest tubing Intercostal chest tubing system(PS)system(PS)

0.87350.87350.985640.98564Thoracic Thoracic SurgerySurgery

33Incisional hernia (Phx)Incisional hernia (Phx)0.87120.87120.994480.99448General SurgeryGeneral Surgery

44Abdominal examination (Phx)Abdominal examination (Phx)0.87030.87030.996220.99622General SurgeryGeneral Surgery

55Subarachnoid hemorrhage (PS)Subarachnoid hemorrhage (PS)0.87420.87420.832450.83245NeurosurgeryNeurosurgery

66Lower intestinal obstruction (PS) Lower intestinal obstruction (PS) 0.89070.89070.573590.57359Pediatric Pediatric SurgerySurgery

77Small bowel obstruction (PS)Small bowel obstruction (PS)0.89480.89480.626880.62688General SurgeryGeneral Surgery

88Peripheral vascular occlusive Peripheral vascular occlusive disease (Hx-CS)disease (Hx-CS)

0.88940.88940.484650.48465Vascular Vascular SurgerySurgery

99Nephrostomy drain for pelvi-Nephrostomy drain for pelvi-ureteral ureteral obstruction (PS)obstruction (PS)

0.89010.89010.672350.67235Pediatric Pediatric UrologyUrology

1010Hematuria (Hx-CS)Hematuria (Hx-CS)0.89610.89610.541260.54126Adult UrologyAdult Urology

1111End-Colostomy (PS)End-Colostomy (PS)0.88180.88180.609790.60979General SurgeryGeneral Surgery

1212Parotid tumor picture (PS)Parotid tumor picture (PS)0.88610.88610.450370.45037General SurgeryGeneral Surgery

1313Triple lumen catheter (PS)Triple lumen catheter (PS)0.87890.87890.576380.57638General SurgeryGeneral Surgery

1414Gallstones Jar (PS)Gallstones Jar (PS)0.86790.86790.999720.99972General SurgeryGeneral Surgery

1515Burn picture (PS)Burn picture (PS)0.86780.86780.999710.99971Plastic SurgeryPlastic Surgery

Total Total AverageAverage

0.870.870.760.76SMC,Squared Multiple Correlations; PS, Problem Solving; Phx, Physical Examination; Hx-CS, History-Communication Skills.

RESULTS: Table 1: OSCE stations reliability (Day 1)

88

Results: Table 2: OSCE stations reliability (Day 2)

CompetencyCompetencyDescriptionDescriptionCronbach’s Alpha Cronbach’s Alpha (Stability)(Stability)

SMC-Carmine’s Theta SMC-Carmine’s Theta (Internal consistency)(Internal consistency)

Surgical SpecialtySurgical Specialty

11Spinal anesthesia picture (PS)Spinal anesthesia picture (PS)0.88180.88180.992870.99287AnesthesiaAnesthesia

22Pneumothorax chest x-ray (PS)Pneumothorax chest x-ray (PS)0.88180.88180.984920.98492Thoracic SurgeryThoracic Surgery

33Inguinal hernia (Phx)Inguinal hernia (Phx)0.87680.87680.994450.99445General SurgeryGeneral Surgery

44Abdominal examination (Phx)Abdominal examination (Phx)0.87710.87710.995920.99592General SurgeryGeneral Surgery

55MRI-multiple brain lesions (PS) MRI-multiple brain lesions (PS) 0.88050.88050.722310.72231NeurosurgeryNeurosurgery

66Doudenal atresia X-ray (PS)Doudenal atresia X-ray (PS)0.88170.88170.734070.73407Pediatric SurgeryPediatric Surgery

77Colon cancer-barium enema (PS)Colon cancer-barium enema (PS)0.90230.90230.474350.47435General SurgeryGeneral Surgery

88Peripheral occlusive vascular Peripheral occlusive vascular disease (Phx)disease (Phx)

0.88740.88740.567680.56768Vascular SurgeryVascular Surgery

99Nasogastric tube (PS)Nasogastric tube (PS)0.88480.88480.710590.71059General SurgeryGeneral Surgery

1010Hematuria (Hx-CS)Hematuria (Hx-CS)0.89320.89320.643240.64324Adult UrologyAdult Urology

1111Clift lip picture (PS)Clift lip picture (PS)0.88550.88550.580380.58038Plastic SurgeryPlastic Surgery

1212Lateral neck mass picture (PS)Lateral neck mass picture (PS)0.87640.87640.743200.74320General SurgeryGeneral Surgery

1313Lower urinary tract syndrome Lower urinary tract syndrome (Hx-CS)(Hx-CS)

0.88600.88600.700750.70075Adult UrologyAdult Urology

1414Foley’s catheter (PS)Foley’s catheter (PS)0.87050.87050.999670.99967General SurgeryGeneral Surgery

1515Appendix jar (PS)Appendix jar (PS)0.87040.87040.999790.99979General SurgeryGeneral Surgery

Total Total averageaverage

0.880.880.790.79

SMC,Squared Multiple Correlations; PS, Problem Solving; Phx, Physical Examination; Hx-CS, History-Communication Skills.

99

VALIDITYVALIDITY

Concurrent validity of the OSCE was determined by Pearson’s correlation with the written one best answer MCQs (P=0.65).

Content and face validities were considered good by majority of staff and students. Predictive and construct validities were not addressed in this study.

1010

FEASIBILITYFEASIBILITY

Initially our OSCEs were labor-intensive

Mostly related to inadequate management.

However; with repetitions and experience, subsequent OSCEs were more feasible,especially when assessing problem solving skills.

1111

DISCUSSIONDISCUSSION• Since its introduction in 1979 by Professors R.

Harden & F. Gleeson, OSCEs have developed into different forms depending on the desired competencies to be rested.

• Our OSCE addressed problem solving skills mainly • OSCE preparation including staff commitment,

blueprint, stations development, etc. is of paramount importance

1212

DISCUSSIONDISCUSSION

• The measure of reliability, validity, and feasibility are keys to successful examination.

• To achieve high reliabilities OSCE has to be longer than is often practical (D Newbel 2004).

• However, high reliabilities have been achieved by some authors with shorter OSCEs (Cohen R et al.1990; Stillman P et al.1991; Sloan D et al. 1993; and others)

• Validity is the hallmark of OSCEs despite arguments that encounters are short.( Stillman P et al.; Newbel D et al.)

• Feasibility is the most critiqued aspect of OSCEs as being costly and labor-intensive. (Barman A 2005)

1313

CONCLUSIONSCONCLUSIONS

• Clinical competence encompasses different aspects that no single examination can cover with high degrees of reliability and validity.

• Well designed OSCE can assess most of these aspects with variable degrees of reliability and validity.

• OSCE can be highly reliable and valid when testing problem solving skills.

• OSCE is more practical when used for testing problem solving skills.

1414

THANK THANK YOUYOU