Proposal for Assessment of Clinical Placement Andrew Kilgour

TO APPLY PAGE STYLESRIGHT CLICK YOUR PAGE>LAYOUT

SCHOOL OF DENTISTRY AND HEALTH SCIENCES

Proposal for Assessment of Clinical Placement

Andrew Kilgour



Factors involved in design

•Transition table•Alignment of University standards and professional accreditation body standards•REA/competency review•Educational literature•Input from the profession locally



Transition Table



Transition tableAt the end of Yr 1Preliminary/Initial

At the end of Yr 2 Novice/Beginner

At the end of Yr 3Advanced Beginner/ Functional

At the end of Yr 4 S1Competent/Consolidating

At the end of Yr 4 S2Proficient/Advanced

General description of attainment

Capable of interacting in the workplace in a professional manner and demonstrating clinical skills.

Capable of safe practice. Work is rule based with limited or no translation and interpretation of concepts, skills and procedures and limited adaptations to meet situational factors unless aided

Can function independently in a limited range of contexts, adapting concepts, skills and procedures to meet situational factors. Demonstrates an appreciation of own limitations and can set personal goals.

Exhibits independence in a breadth of contexts, adapting concepts, skills and procedures to meet situational factors. Can provide theoretical, defensible arguments for their own interpretations and adaptations. Can engage in productive critical reflection.

Exhibits a high level of independence adapting concepts, skills and procedures to meet situational factors. Can use principles to generate new understanding and can provide theoretical, defensible arguments for their own interpretations and adaptations. Can engage in productive critical reflection.

Practice Knowledge Understand basic facts and concepts

Understands a limited range of facts and analyse simple concepts

Understand a breadth of complex facts and analyse concepts

Understand and integrate a breadth complex concepts and principles

Understand and evaluate complex concepts and principles (limitations and strengths). Justify principles, protocols and hypotheses.

Degree ofindependence

Consistently requires instruction

Regularly requiresdirection orcorrection

Sometimesrequires directionor correction

Rarely requiresdirection orcorrection

Independent

Time efficiency Consistently requiresadditional tasktime for a limited range of simple tasks

Regularly requiresadditional tasktime

Sometimesrequiresadditional tasktime to considersurrounding issues

Rarely requiresadditional tasktime to considerpatient needs

Time efficient

Professionalreasoning

Limited to comprehending fundamental professional considerations

Regularly requiresassistance toidentify & solveproblems

Sometimesrequiresassistance toidentify & solveproblems

Rarely requiresassistance toidentify & solveproblems

Consistently identifies & solves problems without assistance

Task Ability Limited to performing simple tasks without consideration of anything else

Tends to focus onperforming tasks without consideration for surrounding issues

Tends to focuson performingtasks with someconsideration forsurrounding issues

Mostly integrates the complexity surrounding tasks

Consistently integrates the complexity of surrounding tasks

Interpersonal Capabilities

Show capacity for working as a team member. Adequate and respectful communication in a limited range of settings

Demonstrates effective contribution to a team. Awareness of professional behaviour such as code of conduct. Shows capacity for effective communication in a range of settings

Demonstrates commitment to professional values and effective communication in a range of settings

Takes on a professional identity and works as a member of the workplace. Demonstrates a culturally sensitive approach to all work interactions

Highly developed teamwork and communication ability in a breadth of settings. Show ability to lead and take initiative



•Original transition table excellent for progression from one year of program to the next•Expanded version used for progression from one placement block to the next•Based on original, but extra levels inserted to define expectations/achievement levels for each block



Yr 1 Yr 2A Yr 2B Yr 2C Yr 3A Yr 3B Yr 4 Sem 1 Yr 4 Sem 2

Practice Knowledge

Understands basic facts and concepts

Beginning to understand more complex facts and concepts

Developing skills to analyse basic facts and concepts

Understand a broadening range of facts and can successfully analyse simple facts and concepts

Understands the majority of complex facts and beginning to analyse more complex facts and concepts

Understands breadth of complex facts and can analyse increasingly complex facts and concepts

Understands and integrates a breadth of complex concepts and principles

Understands and evaluates complex concepts and principles (limitations and strengths). Can justify principles, protocols and hypotheses

Degree of Independence

Consistently requires instruction

Dependent on instruction, but beginning to apply instruction to practice

Some dependence on instruction. Increasing application of instruction to practice

Often requires direction or correction. Applies direction to practice

Sometimes requires direction or correction, and intuitively applies it to practice

Occasionally requires direction or correction

Rarely requires direction or correction

Independent

Time Efficiency

Consistently requires additional task time for a limited range of simple tasks

Consistently requires additional task time. Increasing number and complexity of tasks attempted

Often requires additional task time

Regularly requires additional task time

Sometimes requires additional task time. Uses additional time effectively

Occasionally requires additional task time to consider surrounding issues

Rarely requires additional task time to consider patient needs

Time efficient

Professional/Clinical Reasoning

Limited to comprehending fundamental professional considerations

Beginning to apply fundamental professional considerations to real situations with close supervision

Comprehends increasingly complex situations, but consistently requires assistance to handle them appropriately

Regularly requires assistance to identify and solve problems

Often requires assistance to identify and solve complex problems. Fundamental problems successfully solved

Sometimes requires assistance to identify and solve problems

Rarely requires assistance to identify and solve complex problems

Consistently identifies and solves most problems without assistance

Task Ability

Limited to performing simple tasks without consideration of other factors

Developing understanding of issues surrounding tasks

Sees issues surrounding tasks, but focuses on task achievement

Regularly focuses on performing tasks without consideration for surrounding issues

Sometimes takes surrounding issues into consideration

Focuses on performing tasks with increasing consideration for surrounding issues

Mostly integrates the complexity of surrounding tasks

Consistently integrates the complexity of surrounding tasks


Shows capacity for working as a team member. Adequate and respectful communication in a limited range of settings

Developing team integration. Adequate communication in an expanding range of settings

Good team integration within limits of experience. Regularly demonstrates good communication skills in a variety of settings

Demonstrates effective contribution to a team. Awareness of professional behaviour such as adherence to code of conduct. Shows capacity for effective communication in a range of settings

Regularly demonstrates adherence to professional code of conduct, and effective communication in a widening range of settings

Consistently demonstrates commitment to professional values and effective communication in a range of settings

Takes on a professional identity and works as a member of the workplace. Demonstrates a culturally sensitive approach to all work interactions

Highly developed teamwork and communication ability in a breadth of settings. Shows ability to lead and to take initiative



Using the expanded transition table to assess student performance

Assessment of a student’s clinical performance is OBSERVATIONAL ASSESSMENT

“Assessment of clinical performance during training ..... range(s) from informal observations of students in clinical settings to very formal (and sometimes complex) systems of data gathering from multiple raters about the performance of health professions students in actual clinical settings, with real patients...” Downing (2009)- p 6



Downing (2009) states that “...no more than seven quality rating categories should be used on observational assessment instruments.” (p 209)

Williams, Klamen & McGaghie (2003) agree:

“Five to seven quality ratings are optimal.... Additional quality rating categories add no performance information and just complicate the task for raters.”

Downing, S. M. (2009). Assessment in health professions education: Routledge.Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine, 15(4), 270-292. doi: 10.1207/S15328015TLM1504_11



Yr 1- Expected Standards

Practice KnowledgeUnderstands basic facts

and concepts

UnacceptableWell below expected standard

Below expected standardAt expected standard

Above expected standardWell above expected standard

ExceptionalComments

Degree of IndependenceConsistently requires

instruction




ExceptionalComments

Time Efficiency

Consistently requires additional task time for

a limited range of simple tasks




ExceptionalComments

Professional/Clinical Reasoning

Limited to comprehending

fundamental professional

considerations




ExceptionalComments

Task Ability

Limited to performing simple tasks without

consideration of other factors




ExceptionalComments


Shows capacity for working as a team

member. Adequate and respectful

communication in a limited range of

settings




ExceptionalComments



Standards based assessment of clinical performanceAssessment is the making of judgments about how students’ work meets appropriate standards (Boud & Dochy 2010)

Assessment of student achievements is judged against consistent national and international standards that are subject to continuing dialogue, review and justification within disciplinary and professional communities (Boud & Dochy 2010)

In addition to knowing about standards, students need to compare their actual levels of performance with these standards (Sadler 1989)

Teachers must be prepared to assess based on judgments about the quality of work, not based on wanting to reward effort or improvement (Sadler 2005)

Boud, D., & Dochy, F. (2010). Assessment 2020. Seven propositions for assessment reform in higher education. status: published. Sadler *, D. R. (2005). Interpretations of criteria based assessment and grading in higher education. ‐ Assessment & Evaluation in Higher Education, 30(2), 175-194. doi: 10.1080/0260293042000264262Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. doi: 10.1007/BF00117714



Application of standards-based assessment

The performance of the student is measured against the standard expected in each of the six categories: Practice knowledge; Degree of independence; Time efficiency; Professional/clinical reasoning; Task ability; Interpersonal capabilities



The standard expected of students in each of these categories varies from one placement block to the next, but the layout and format of the assessment form is the same for each block

Williams, Klamen and McGaghie (2003) emphasise the importance of keeping rating instruments short, stating:

“Adding more items only boosts rater time and makes the task harder.”

Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine, 15(4), 270-292. doi: 10.1207/S15328015TLM1504_11



Global assessment

Williams, Klamen and McGaghie (2003) also suggest that a global item should be included in the assessment form used. In the transition table, the global item is referred to as “general description of attainment”

These have also been expanded – see next slide

Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine, 15(4), 270-292. doi: 10.1207/S15328015TLM1504_11



Global items at each placement blockGeneral Description of Attainment at Each Level

Yr 1 Capable of interacting in the workplace in a professional manner. Demonstrates limited clinical skills under close supervision

Yr 2ADemonstrates beginner level professional patient interaction. Expanding range of clinical skills performed under close supervision

Yr 2BDemonstrates advancing professional patient interaction. Demonstrates limited independence in clinical skills, and rudimentary clinical decision making

Yr 2CCapable of safe practice under supervision. Work is rule based with limited or no translation and interpretation of concepts, skills and procedures, and limited adaptations to meet situational factors unless aided

Yr 3ARequires minimal supervision to demonstrate safe practice in basic procedures. Demonstrates functional clinical reasoning skills in simple situations. Aware of own limitations

Yr 3BCan function independently in a limited range of contexts, adapting concepts, skills and procedures to meet situational factors. Demonstrates an appreciation of own limitations and can set personal goals

Yr 4 Sem 1

Exhibits independence in a breadth of contexts, adapting concepts, skills and procedures to meet situational factors. Can provide theoretical, defensible arguments for their own interpretations and adaptations. Can engage in productive critical reflection

Yr 4 Sem 2

Exhibits a high level of independence, adapting concepts, skills and procedures to meet situational factors. Can use principles to generate new understanding and provide theoretical, defensible arguments for their own interpretations and adaptations. Can engage in productive critical reflection



How global items appear on the assessment form

Global Description of Expected Performance for Yr 1 Placement Block:Capable of interacting in the workplace in a professional manner. Demonstrates limited clinical skills under close supervision. Please write your comments regarding the student’s performance below:________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________Signed ______________________ Date _______________________



Alignment of University standards and professional accreditation body standards



Work done so far.........

A lot of work has already been done mapping the “Professional and Practice-Based Education Standards” to the “AIR Competency Based Assessment” and the “ANZSNM Competency Based Standards”

(See MRS_Clinical_Subject_Mapping_2Feb.docx)



In order to track a student’s progress in meeting these standards, a longitudinal documentation is required

How do we do this?



The e-portfolio...“In health professions education, a portfolio is a collection of evidence documenting progress, accomplishments and achievements over time...... portfolios provide a means to assess competencies such as self-directed learning, which are demonstrated over the course of months or years. Portfolios also comprise a vehicle for the longitudinal, multi-method, multi-source assessment of learner achievement.” Downing (2009) p 287

Downing, S. M. (2009). Assessment in health professions education: Routledge





How we use portfolios...

“...portfolios are best used as part of a comprehensive assessment system that can triangulate on learner competence.” (Melville, Rees, Brookfield, & Anderson, 2004)

The formative and summative functions of the portfolio should be separated, and learners should be allowed to select “best work” or “best evidence” to submit (pinsky & Fryer-Edwards, 2004)

Melville, C., Rees, M., Brookfield, D., & Anderson, J. (2004). Portfolios for assessment of paediatric specialist registrars. Medical Education, 38(10), 1117-1125.

Pinsky, L. E. M. D., & Fryer-edwards, K. (2004). Diving for PERLS. Journal of General Internal Medicine, 19(5), 582-587.



CSU PortfolioCommunications and Interactions (CI)1. Demonstrate ethical, respectful, supportive and culturally competent communication and interaction consistent with professional codes of practice. 2. Demonstrate proficient and professional communication, through a variety of delivery media/modes to specialist and non specialist audiences. 3. Demonstrate teamwork abilities, leadership, collegiality, conflict management and professional conventions at the level of an emerging professional.

Professional Judgement (PJ)1. Demonstrate critical and creative decision making and problem solving that is context relevant. 2. Make work-related decisions that are aligned with professional values, standards and ethics and address legal requirements. 3. Demonstrate accountability by being able to report and articulate the basis for professional decisions and actions.

Profession Competence and Work readiness (WR)1. Demonstrate the discipline-specific technical capabilities of a beginning practitioner or professional. 2. Integrate discipline, practical and social knowledge and skills in contemporary professional practice. 3. Demonstrate an understanding of legal and ethical requirements and the boundaries in which to work. 4. Recognise and respond appropriately to unsafe practice. 5. Demonstrate an ability to plan and manage workloads.

Professionalism and Citizenship (PC)



Professionalism and Citizenship (PC)

1. Demonstrate commitment, and an ability to undertake life-long learning through reflection, self-evaluation and self-improvement2. Exhibit qualities and behaviours consistent with professional values informed by social justice, global citizenship, Indigenous and cultural competencies and inclusion principles. 3. Explain how practice is informed by knowledge of continuous quality improvement, sustainability and global trends in practice.

Information Literacy (IL)1. Demonstrate an ability to critique new information and determine its relevance to a given situation. 2. Demonstrate efficacy in the use of information and communication technologies as part of: a) learning b) professional practice.



REA/Competency Review



Existing REA....

•Either a student is competent in an examination or they are not- little merit in giving them a score for different facets of an examination•Competency should be a high standard. It is a mastery process, and if a student doesn’t achieve it the first time, they are free to try again•Old forms use outdated terminology (ie “film type”)





Where Do REAs Fit Into Assessment Scheme?



Alternate Model

•The model can be adapted to include a “gateway” into Year 4•This is considered essential to ensure that our Yr 4 students are at the standard expected in the workforce





The proposed assessment schedule meets the criteria..........

•Triangulation: e-portfolio is run parallel to clinical supervisor assessment and individual examination category assessment (REAs)•Formative and summative functions separated: reflective journal entries formative, supervisor assessments summative. Entered in separate “assets” in e-portfolio•Tied together in capstone assessment



Educational Literature



Holistic Knowledge Integration“By concentrating on discrete pieces of knowledge and skill, there is a distinct danger of only dealing with fairly superficial aspects of professional practice, whilst ignoring the holistic way in which such knowledge and skill is integrated and coordinated in actual professional practice.” (Hager & Gonczi, 1991)

We need to look at the big picture of student performance, rather than assessing each little aspect of performance in detail

Hager, P., & Gonczi, A. (1991). Competency based standards: a boon for continuing professional education? ‐ Studies in Continuing Education, 13(1), 24-40. doi: 10.1080/0158037910130103



The Reason for Standards-based Assessment.......

Students deserve to have work graded strictly according to quality, without comparison to other students, and without their academic history being considered. Students deserve to know the bases on which judgments about their work are made (Sadler 2009)

Sadler, D. R. (2009). Grade integrity and the representation of academic achievement. Studies in Higher Education, 34(7), 807-826. doi: 10.1080/03075070802706553



Judgment of Professional Competence

Professional competence involves being able to do high-level tasks consistently well. When it comes to grading, we need to arrive at judgments which are or would be consistent with peers even when there are no independent sources of data to substantiate those professional judgments (Sadler 2011)

Sadler, D. R. (2011). Academic freedom, achievement standards and professional identity. Quality in Higher Education, 17(1), 85-100. doi: 10.1080/13538322.2011.554639



Assessment Should Facilitate Students Monitoring Their Own Learning

Even when teachers provide students with valid and reliable judgments about the quality of their work, improvements do not necessarily follow. Assessments should develop the ability of students to exercise executive control over their own productive activities, and eventually to become independent and fully self-monitoring (Sadler 1989)

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. doi: 10.1007/BF00117714



The Role of Reflective Writing in Assessment

“The process of learning is a process of students devising learning strategies to solve the challenges their motives have defined for them” (Wilson & Fowler 2005)

Reflective writing facilitates students devising their own learning strategies

Wilson *, K., & Fowler, J. (2005). Assessing the impact of learning environments on students' approaches to learning: comparing conventional and action learning designs. Assessment & Evaluation in Higher Education, 30(1), 87-101. doi: 10.1080/0260293042003251770



Integration of Formal and Informal Learning

Work-related programs need to be flexible and responsive to the circumstances of the learner and of the work setting. They must not focus on a narrow competency-based approach, rather be more open to the learning derived from the student experience. The design of the assessments must allow both formal and informal learning to be integrated in ways which will demonstrate their graduate qualities (Clements & Cord, 2011)

Clements, M. D., & Cord, B. A. (2011). Assessment guiding learning: developing graduate qualities in an experiential learning programme. Assessment & Evaluation in Higher Education, 38(1), 114-124. doi: 10.1080/02602938.2011.609314



Learning as a ProcessTraditional education has focused on content as the object of learning. When we view learning as a process, we focus on learning as the creation of circumstances that favour acquisition of ideas by the mind. Many capabilities and abilities that are necessary for competence are developed by the view of learning as a process. These include the capacity to analyse a situation, the ability to plan ahead, and the ability to gather pertinent information. These processes make learning processes more likely to be effective (Hager 2004)

Hager, P. (2004). The competence affair, or why vocational education and training urgently needs a new understanding of learning. Journal of Vocational Education & Training, 56(3), 409-433. doi: 10.1080/13636820400200262



Input From The Profession Locally



Why Invite Input From the Profession?

“Stakeholder involvement in the design of assessment programs not only promotes input of creative ideas, but also ensures a certain fitness for practice. It can give stakeholders a sense of ownership of the program, thereby gaining their support, without which goals can remain elusive” (Dijkstra, Vleuten & Schuwirth, 2010)

Dijkstra, J., Vleuten, C. P. M., & Schuwirth, L. W. T. (2010). A new framework for designing programmes of assessment. Advances in Health Sciences Education, 15(3), 379-393. doi: 10.1007/s10459-009-9205-z



How Information Was Gathered....

Survey, whose link was published by the AIR in 2012:

https://www.surveymonkey.com/s/VBH6YHD

103 responses nationwide



Who Responded?







What They Said....







Comments on What Should Be Assessed•I think an importance should be placed on CPD at a earlier stage in Radiography it may impact behaviours at a later stage•With the influx of international students, perhaps clinical communication skills appropriate or inappropriate•Attitude and professionalism•Ability to accept feedback and improve•Social skills are vitally important and too many students have absolutely none! should be emphasised more•Ability to work as part of a team. Lack of teamwork can impact on clinical competency and patient outcomes•Communication - with patients AND staff. Extremely important



What we are proposing..........



CSU will be introducing the new assessment form for 2B placement, commencing 24 June

We have already started the reflective journals, using pebblepad, and are getting some excellent results

We propose that Sydney begin the trial of the new form with the 2.2 placement, commencing in July



Along with the new forms, we will also send out the existing ones during the trial period, to give information on reliability of the new version

This is proven methodology in trialling a new assessment protocol

Results will be tabulated and a comparison made



Questions??



ReferencesBoud, D., & Dochy, F. (2010). Assessment 2020. Seven propositions for assessment reform in higher education. status: published. Clements, M. D., & Cord, B. A. (2011). Assessment guiding learning: developing graduate qualities in an experiential learning programme. Assessment & Evaluation in Higher Education, 38(1), 114-124. doi: 10.1080/02602938.2011.609314Dijkstra, J., Vleuten, C. P. M., & Schuwirth, L. W. T. (2010). A new framework for designing programmes of assessment. Advances in Health Sciences Education, 15(3), 379-393. doi: 10.1007/s10459-009-9205-zDowning, S. M. (2009). Assessment in health professions education: Routledge.Hager, P. (2004). The competence affair, or why vocational education and training urgently needs a new understanding of learning. Journal of Vocational Education & Training, 56(3), 409-433. doi: 10.1080/13636820400200262Hager, P., & Gonczi, A. (1991). Competency based standards: a boon for continuing professional education? ‐ Studies in Continuing Education, 13(1), 24-40. doi: 10.1080/0158037910130103Melville, C., Rees, M., Brookfield, D., & Anderson, J. (2004). Portfolios for assessment of paediatric specialist registrars. Medical Education, 38(10), 1117-1125. Pinsky, L. E. M. D., & Fryer-edwards, K. (2004). Diving for PERLS. Journal of General Internal Medicine, 19(5), 582-587. Sadler *, D. R. (2005). Interpretations of criteria based assessment and grading in higher education. ‐ Assessment & Evaluation in Higher Education, 30(2), 175-194. doi: 10.1080/0260293042000264262Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18(2), 119-144. doi: 10.1007/BF00117714Sadler, D. R. (2009). Grade integrity and the representation of academic achievement. Studies in Higher Education, 34(7), 807-826. doi: 10.1080/03075070802706553Sadler, D. R. (2011). Academic freedom, achievement standards and professional identity. Quality in Higher Education, 17(1), 85-100. doi: 10.1080/13538322.2011.554639Williams, R. G., Klamen, D. A., & McGaghie, W. C. (2003). SPECIAL ARTICLE: Cognitive, Social and Environmental Sources of Bias in Clinical Performance Ratings. Teaching and Learning in Medicine, 15(4), 270-292. doi: 10.1207/S15328015TLM1504_11Wilson *, K., & Fowler, J. (2005). Assessing the impact of learning environments on students' approaches to learning: comparing conventional and action learning designs. Assessment & Evaluation in Higher Education, 30(1), 87-101. doi: 10.1080/0260293042003251770

Documents

Proposal for Assessment of Clinical Placement Andrew Kilgour