Deliverable 2.1: Tool for matrix mapping of work- place ...€¦ · Deliverable 2.1: Tool for matrix mapping of work-place-based curricula with Entrustable Professional Activities

This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619349.

Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education Deliverable 2.1: Tool for matrix mapping of work-place-based curricula with Entrustable Professional Activities and competencies

Delivery month Annex I: M10 Actual delivery month M10 Lead participant: UMCU Work package: 2 Nature: O Dissemination level: PU Version: 1.0 Project coordinator Dr. Marieke van der Schaaf Utrecht University Faculty of Social and Behavioral Sciences

Department of Education PO Box 80.140

3508TC Utrecht The Netherlands

Telephone: +31 (0)30 253 4944 Email: [email protected]

Workplace-based e-Assessment Technology for Competency-based Higher Multi-professional Education

Deliverable D2.1

2

1. Executive Summary ............................................................................................ 3 2. Introduction ......................................................................................................... 4

2.1 Background .......................................................................................................... 4 2.2 Scope of the deliverable ......................................................................................... 4

3. Content ................................................................................................................. 4 4. Conclusion ............................................................................................................. 4 3. 5. References ....................................................................................................... 5 4. 6. Tables and Figures .......................................................................................... 5 7. History of the document ........................................................................................ 5

7.1 Document history .................................................................................................... 5 7.2 Internal review history ............................................................................................. 5

APPENDIX 1: TOOL FOR MATRIX MAPPING FOCUSED ON HEALTH CARE TRAINING .................................................................................................................... 7

Guidelines for Competency-based Workplace Curriculum Development based on Entrustable Professional Activities - a matrix mapping approach ................................. 7

How can progression of the resident in this EPA be assessed? ...................................... 70 Which criteria must be met by the resident to fulfil the EPA satisfactory (= at a level of indirect supervision) ................................................................................................... 70 How can progression of the resident in this EPA be assessed? ...................................... 72 Which criteria must be met by the resident to fulfil the EPA satisfactory (= at a level of indirect supervision) ................................................................................................... 72 How can progression of the resident in this EPA be assessed? ...................................... 73 How can progression of the resident in this EPA be assessed? ...................................... 75 Which criteria must be met by the resident to fulfil the EPA satisfactory? .................... 75 How can progression of the resident in this EPA be assessed? ...................................... 76 How can progression of the resident in this EPA be assessed? ...................................... 77 - are committed to students and their learning; ............................................................................................. 83 - know the subjects they teach and how to teach those subjects to students; .................................... 83 - are responsible for managing and monitoring student learning; ........................................................ 83 - think systematically about their practice and learn from experience; .............................................. 83 - are members of learning communities (NBPTS, 1987). ............................................................................ 83


Deliverable D2.1

3

1. Executive Summary This Tool for Matrix Mapping of Workplace-Based Curricula with Entrustable Professional Activities (EPAs) and Competencies is a large document that consists of the following parts, all written to be foundational for the WATCHME project. 1. An introductory text explaining the relationship of all parts and the scope of the de-‐

liverable. 2. An extensive tool description, worked out as Guidelines for workplace curriculum

development for the health care professions, based on matrix mapping, supplement-‐ed with three appendices: (I) proposed EPAs for undergraduate medical education (i.e. the course to become a medical doctor); (II) proposed EPAs for one medical spe-‐cialty training course, in particular Anesthesiology training; (III) proposed EPAs for veterinary education.

3. An extensive background description, worked out as New Perspectives on Formative Evaluation of Student Teachers’ Teaching Competence, supplemented with an ap-‐pendix with proposed EPAs for teacher training, described in detail.

These deliverable will serve (a) to inform subsequent work packages in WATCHME and will be disseminated for a wider audience in the three content domains at stake (medi-‐cine, veterinary medicine and teacher training). This deliverable was written to support educators in building a competency-‐based workplace curriculum providing an up-‐to-‐date overview of the literature on EPAs, sup-‐plemented with suggestions for practical issues regarding curriculum construction, as-‐sessment and educational technology around EPAs-‐curricula. It sets out with concepts and definitions. Next, it provides guidance with the identification, elaboration and vali-‐dation of EPAs, while explaining common misunderstandings about EPAs. Next, the ma-‐trix-‐mapping approach of combining EPAs with competencies is discussed, and related to existing concepts such as milestones. A chapter is devoted to entrustment decision-‐making as an inextricable part of working with EPAs. In using EPAs, assessment in the workplace is translated to entrustment decision-‐making for designated levels of permit-‐ted autonomy, ranging from full supervision present during the act to the permission to provide supervision. Finally, a chapter is devoted to the use of technology of mobile de-‐vices and electronic portfolio to support feedback to trainee and entrustment decision-‐making by program directors or clinical teams about trainee progress. This part con-‐cludes with many examples of EPAs from human and veterinary health care that are planned to implemented in programs in Utrecht, Berlin, Budapest and San Francisco WATCHME partner institutions. This approach has been adapted to serve Teacher Education. The terminology of EPAs has interchangeable been used with “Core Practices”, a concept that was recently intro-‐duced in the teacher education literature and that corresponds to a certain extent with the EPA concept. This concludes with a list of teacher education EPAs. To keep this de-‐liverable manageable and readable, the content is kept short, referring to long appen-‐dices at the end of the deliverable. The next section may be read as an extension of this executive summary and refers to content that is attached as appendices.


Deliverable D2.1

4

2. Introduction

2.1 Background Matrix mapping basically is establishing the relationship between professional practice and its component activities, and the competencies that individuals must acquire in training to enable them to execute these activities. This is a fundamentally different way to look at professional education than is current mainstream thinking. The use of en-‐trustable professional activities, a concept developed in medical education in the past decade, has many implications for feedback and assessment in the workplace, that have been described in this deliverable and can be used in professional domains outside med-‐icine.

2.2 Scope of the deliverable This deliverable is of relevance for workplace curriculum development in the domains and institutions that contributed (undergraduate and postgraduate medical education, veterinary education and teacher training at Utrecht University in The Netherlands, Charité University in Germany, Tartu University in Estonia, Szent Istvan University in Budapest), but also more broadly for these domains at other universities and countries, and possibly for other professional domains. As the foundation is laid for technology-‐based feedback and entrustment decision-‐making as an approach to assessment, it will stimulate learning analytics development that is yet in its infancy in these domains of higher education.

3. Content The Deliverable 2.1 has an extensive content description that is divided in two main parts: 1. part 1 relating to health care education (undergraduate and anesthesiology (post-‐

graduate)medical education, veterinary education). 2. part 2 relating to teacher education. Both parts are subdivided into a detailed description of the approach of matrix mapping of EPA-‐based competency curricula that can serve as guideline for practice, and an ap-‐pendix that lists EPAs that have been developed to be applied empirically within the WATCHME for the development of learning analytics in further work packages. For details of the content we refer to the overall appendix with this deliverable.

4. Conclusion Work package 2 has successfully delivered their foundational work for the WATCHME project, to be supplemented with a literature review and empirical data on markers in the workplace to inform feedback and entrustment decision-‐making (deliverables WP2-‐2.2 and 2.3)


Deliverable D2.1

5

3. 5. References About 180 literature references used in both parts of this deliverable are listed in he respective parts of the appendix

4. 6. Tables and Figures The deliverable contains 12 tables and 10 figures, excluding the EPAs, to be found in the appendix.

7. History of the document

7.1 Document history Version Author(s) [see doc-‐

ument parts for com-‐plete lists of authors]

Date Changes

V#1 Part1 Main doc Ten Cate July 2014 First section V#2 Part1 Main doc Ten Cate November 2014 Complete document V#3 Part1 Main doc Ten Cate et al December 2014 Many changes V#1 EPAs UMCU* Reyman et al September 2014 Many previous ver-

sions V#2 EPAs UMCU* Welink et at December 2014 Many previous ver-

sions V#1 EPAs UCSF Chen et al September 2014 Many previous ver-

sions V#1 EPAs Charité* Peters et al December 2014 Many previous ver-

sions V#1 EPAs Vet.educ Van Duijn et al December 2014 Many previous ver-

sions V#1 EPAs Anesthe-siology training

Wisman-Zwarter et al December 2014 Delphi-based previ-ous version

V#1 Part2 Main doc Krull et al December 2014 Several previous version

V#2 EPAs Teacher Training

Slof et al December 2014 Extensive prevision Excel version

*separately developed but combined in this document

7.2 Internal review history Internal Reviewer Date Comments Serban Ovidiu 12-12-2014 No substantial comments Marieke van der Schaaf 27-12-2014 and 29-12- No substantial comments


Deliverable D2.1

6

2014


Deliverable D2.1

7

APPENDIX 1: TOOL FOR MATRIX MAPPING FOCUSED ON HEALTH CARE TRAINING

Guidelines for Competency-based Workplace Curriculum Development based on Entrustable Professional Activities - a matrix mapping ap-proach AUTHORS: Olle ten Cate (1), Carrie Chen (2), Reinier Hoff (1), Harm Peters (3), Harold Bok (4), Marieke van der Schaaf (5) (1) University Medical Center Utrecht (2) University of California San Francisco (3) Charité University, Berlin (4) Utrecht University (Department of Veterinary Medicine) (5) Utrecht University (Department of Education)

Table of contents

1. Introduction

2. The ground work: concepts and definitions 2.1 Workplace and workplace curriculum 2.2 Competency-‐based education 2.3 Entrustable professional activities 2.4 The EPAs-‐competencies matrix and the significance of matrix mapping 2.5 Learning to bear responsibility 2.6 Assessment, entrustment, and supervision 2.7 Portfolio as a tool to organize and support competency-‐based development

3. Translating the professional work into EPAs

3.1 Identifying EPAs 3.2 Some common misunderstandings 3.3 Elaborating EPAs 3.4 Validating EPAs

4. Building and maintaining an individualized workplace curriculum with EPAs

4.1 A general framework for the workplace curriculum 4.2 Supervision levels related to entrustment decisions 4.3 Task-‐based instructional strategy 4.4 Connecting EPAs and competencies with milestones and supervision levels 4.5 Core, specific and elective EPAs

5. Monitoring and evaluating trainees using entrustment decisions for EPAs

5.1 Factors determining entrustment decisions 5.2 Trainee features that allow supervisors to entrust them with a critical task 5.3 Arriving at formative and summative entrustment decisions 5.4 Instruments to assess trainees based on EPAs 5.5 Proposed general reporting format for observed performance


Deliverable D2.1

8

5.6 Self-‐entrustment 5.7 Expiry and reconfirmation of summative entrustment decisions

6. Technology to support feedback and entrustment decision-‐making

6.1 Formative feedback and entrustment decisions 6.2 Summative entrustment decisions 6.3 E-‐portfolio and learning analytic

7. Discussion References Appendices


Deliverable D2.1

9

1. Introduction The purpose of this guide is to provide a practical framework for workplace curriculum devel-‐opment drawing from the existing literature on competency-‐based education in the health pro-‐fessions that refers to entrustable professional activities (EPAs). Central in our thinking of workplace curriculum development is matrix mapping.(1,2). A matrix mapping approach signifies the two-‐dimensional nature of working with competencies on one hand and activities on the other hand. Competencies are descriptors of the qualities of individual trainees and professionals, while activities describe the work that is being done in the workplace. Essential in our approach is that competencies are mapped to work. The authors’ background is from medical and veterinary education, but the approach would hold just as well for other professions in healthcare and other professional domains. Education within the health professions is characterized by a substantial portion of training in a real work setting. Much of the work in health care can be defined in tasks or responsibilities that must be entrusted to individuals. These entrustable professional activities as we shall call them, usually require different qualities of the practitioner. Each EPA may draw from several domains of competence, such as content expertise, skills in collaboration, communication, management et cetera. Conversely, each domain of competence is relevant to many different activities (i.e. EPAs) in health care. Combining domains of competence and EPAs in a matrix reveals which compe-‐tencies a trainee must have before being trusted to execute an EPA (3,4). This matrix serves the specification of assessment and feedback procedures for individual development and grounds entrustment decisions for critical tasks in health care. Four key questions within this approach are (i) What is the work to be done? This question leads to the identification and description of EPAs. These units of work can be very small (measuring and reporting blood pressure) or very big (managing a clinical ward), but al-‐ways have a professional nature, which excludes ‘taking a break’, ‘cleaning your desk’ and ‘listen-‐ing to your colleague’s experience at yesterday’s football match’. For practical purposes it is nec-‐essary to apply EPAs as significant units related to requirements or expectations at the end of a designated training period, i.e. when transiting to a new phase. Why this is important will be explained below. (ii) Which requirements must trainees meet before we trust them to do the work? For each of the EPAs, required competencies must be determined. These competencies are the qualities of the trainee that serve as necessary conditions to enable summative entrustment decisions. If these conditions are not met, trainees should not be entrusted with that critical task. For educational purposes it is useful to specify the experience, knowledge, skill and attitude re-‐quirements to guide trainees in their preparation for entrustment decisions. (iii) How do we train trainees to meet these requirements? If EPAs are the focus of training, the conditions set for entrustment decisions should guide the training activities. For each of these professional activities trainees must understand what it


Deliverable D2.1

10

takes to actually execute an EPA, or the components constituting it. Specific knowledge and skills can usually be taught and trained. In the workplace, development of proficiency requires in-‐formed self-‐assessment, guided by feedback from the environment, and a motivation to keep improving. (iv) How do we assess trainees’ readiness to pass the threshold of entrustment? When using EPAs, the assessment of trainees becomes meaningful for trainees and supervisors when we translate assessment to entrustment decisions related to a specified level of supervi-‐sion. Instead of using grades that are often unreliable in the workplace (5), the simple questions are Do I need to assist this trainee? Can I leave the ward to come back later? Can I trust the in-‐formation in the electronic patient record to be adequate and sufficient when I see it tomorrow? Such questions represent assessment framed as supervision language. These questions are addressed in this Guide. This Guide is meant to support the building of a workplace curriculum with tools for learning and training, feedback, assessment and entrust-‐ment decision-‐making. 2. The ground work: concepts and definitions 2.1 Workplace and workplace curriculum The workplace is the context in which much, maybe most, of the learning occurs for professions with a vocational nature. Although implicitly assumed for millennia (6), workplace learning has only recently been acknowledged for its huge contribution to the attainment of competencies that professionals need to possess(7). The establishment of schools and universities in the past centuries, and the development of educational theories and principles for classroom learning may have overshadowed the significance of learning in the workplace (7), with its many unwrit-‐ten, and informal requirements that guide the learning. As students are employed with tasks in the workplace, at the discretion of supervisors, they are expected to gradually learn the tricks of the trade. Efforts to analyze what happens in workplace learning have guided improvements to its quality and effectiveness as a learning environment (8,9). In the 1980s, educational scientists introduced experiential learning (10) and reinvented the term (cognitive) apprenticeship (11–13) to stress the importance of authentic activities and social interaction to learning. This is in contrast with the implicit assumption that schools and universities provide learners with the majority of their universally applicable knowledge and skills. Situated learning and legitimate peripheral participation within a community of practice were concepts introduced by Lave & Wenger (14) with the same purpose. In health care education, clinical teachers have always complained about the lack of connection with what students have been taught in the classroom and what they can apply once they are placed in real work settings with patients. Integration of learning of theoretical concepts in the classroom with learning in situ has therefore been stressed to optimize medical curricula (15–17). In particular, vertical curriculum integration (18,19) implies an enhanced connection between basic sciences and clinical practice. Published objectives for medical training in the past decades have consistently proposed basic science


Deliverable D2.1

11

knowledge to be instrumental to a higher clinical purpose (20–22), and not to be an objective of medical training in itself. A workplace curriculum can be defined as an organized set of experiences in a real-‐world setting that foster the acquisition of competence or competencies that are necessary to act as a profes-‐sional. Features that set a workplace curriculum apart from just engaging in work are, according to Billett, (i) a trajectory of participation from low to high accountability, (ii) access to knowledge that would not be learned by discovery alone, (iii) direct guidance from more experi-‐enced others and experts, and (iv) indirect guidance provided by the physical and social envi-‐ronment (7). It is against this background that curriculum building with EPAs takes place. 2.2 Competency-based education Whereas knowledge and to some extent also attitude and skills can be viewed as the result of academic study separate from the workplace, competency-‐based education is defined by the outcome of education as a tangible capability to perform in the workplace. Competency-‐based education and the assessment of it require a workplace environment, even though preparation for it can happen before entering the workplace. The acquisition of competencies, which inte-‐grate knowledge, skills and attitudes for the sake of working in practice (23–25), must be con-‐firmed in a workplace environment. Competency-‐based professional education can be defined as education for a profession that is targeted at a fixed level of proficiency in one or more professional competencies (24,26). Com-‐petency-‐based medical education is “an approach to preparing physicians for practice that is fundamentally oriented to graduate outcome abilities and organized around competencies de-‐rived from an analysis of societal and patient needs. It de-‐emphasizes time-‐based training and promises a greater accountability, flexibility, and learner-‐centeredness” (27). Key elements of competency-‐based education are a focus on outcome and independence of time (28). As work-‐place experience for trainees is usually organized in a rotational system with fixed periods of time, competency-‐based education may pose logistical challenges, as it requires some flexibility in time. Fundamental is that trainees are only being certified for competencies that they have been shown to possess, or, phrased differently, for which they have passed a threshold that al-‐lows for unsupervised practice (2). Competency-‐based training with EPAs is basically a mastery learning approach to education. Mastery learning leads to certification only if trainees meet all requirements, no matter how much time they need to get there. This curriculum approach has proven its effectiveness (29,30). 2.3 Entrustable professional activities Competency-‐based education in the workplace is about learning to execute the professional ac-‐tivities that must be done. EPAs can best be considered tasks or responsibilities that faculty members entrust to a trainee to execute with limited supervision, once he or she has obtained adequate competence.(1,31) EPAs are executable within a given time frame; their execution is observable and measurable, and suitable for focused entrustment decisions. EPAs are units of work (e.g., the anesthetic care of an uncomplicated patient, or the chairing of a multidisciplinary meeting), while competencies describe people’s abilities (e.g., knowledge, professional attitude, communication skill). Competencies are often felt to be theoretical constructs (32–36) and EPAs


Deliverable D2.1

12

were introduced to ground competencies in day-‐to-‐day practice (1,37). As EPAs are the pro-‐posed focus of assessment, they lead to integrated, holistic evaluation of learners, that include quantifiable assessments, but also the more tacit but important impressions of the trust-‐readiness of a trainee concerning a critical activity. EPAs are now being introduced in many postgraduate medical education programs and fellow-‐ships, including obstetrics/gynaecology (38), pediatrics (39), internal medicine (40), family medicine (41), psychiatry (42) and pulmonary and critical care (43). 2.4 The EPAs-competencies matrix and the significance of matrix mapping Units of work and abilities of persons can be viewed as two dimensions of a grid. Mapping EPAs to competencies is basically answering the question: Which competencies must an individual have before a critical activity can be entrusted to this person to complete unsupervised? In most cases an educational program has an existing list of competencies, outcome-‐oriented objectives, or knowledge, skills and attitudes that define the desired qualities of graduates. A well-‐known framework in the medical domain is CanMEDS, that defines these qualities in seven roles or competency-‐domains (44), including, among others, content expertise, communication and collaboration ability, and a professional attitude. These desired person-‐descriptors are sep-‐arate from the activities (EPAs) that must be done, such as “admitting a patient to the hospital with an initial workup” or “doing a caesarian section in a cow”. The matrix combines EPAs with competencies as depicted in Figure 1. Likewise, in veterinary medicine, an internationally rec-‐ognized competency framework (VetPro) defines the qualities for the veterinary professional (45,46) that can build such a matrix. Figure 1 The basic form of an EPAs-Competencies Matrix EPA1 EPA2 EPA3 EPA4 EPA5 EPA6

Competency 1 ! ! ! !

Competency 2 ! ! !

Competency 3 ! ! ! ! Competency 4 ! !

Competency 5 ! ! ! ! ! Competency 6 !

Competency 7 ! ! ! The figure signifies the notion that competencies almost invariably map to multiple EPAs and that the trustworthy execution of any EPA usually requires multiple competencies. In the exam-‐ple shown, some EPAs are broad and complex, requiring competencies in various domains (EPA3), while others may be more limited or focused (EPA5). Conversely, some competencies


Deliverable D2.1

13

may be so general that they are important requisites for many activities (Competency 5), while others are rather specialized or specific, and only needed for few EPAs (Competency 6). The literature shows several examples of such matrices (3,47–49). The significance of the matrix lies in the guidance it provides for both learners and teachers. Trainees know the expectations for earning trust to complete a specific EPA; supervisors know what to evaluate before making an entrustment decision. The assessment of a trainee’s readi-‐ness for unsupervised practice of an EPA can be designed using the matrix. For each dot, the most appropriate sources of data to inform entrustment decisions and feedback to the trainee should be determined. Matrix mapping of an educational program requires the analysis of the profession for which stu-‐dents are being trained. Ideally, a comprehensive list of EPAs is identified that constitute the complete core of the profession, expressed in the activities that professionals carry out. As a next step, these activities are mapped against an existing framework of competencies. Each compe-‐tency is weighed against each EPA and valued as to their significance for entrustment decisions, asking is this (specified) competency an important requirement for any trainee, before he or she can be trusted to execute this EPA without supervision? We will expand on this in chapter 3. 2.5 Learning to bear responsibility One of the major challenges for medical educators is to facilitate and stimulate trainees to take responsibility for patient care. This has become a more significant obligation in the past decades. Pressure on patient safety in health care has resulted in increased and stricter supervision in several countries, while decreasing trainee responsibility. Work time restrictions for residents, the need to provide care as quickly and efficiently as possible, and the introduction of managed care, only reimbursable if provided by licensed specialists, has put attending physicians in more dominant roles and decreased trainee responsibility (50,51). This trend is completely justifiable from a patient perspective but not from an education perspective, and ultimately counter pro-‐ductive for safe patient care.(52) Trainees must practice bearing responsibility, in a safe envi-‐ronment. Full supervision and avoidance of full responsibility for patients until the end of resi-‐dency jeopardizes safe patient care directly after certification. With graduated responsibilities, medical trainees can practice with some autonomy while still having the opportunity to debrief and correct things with a supervisor. Graduates from residency programs bear full responsibility on the first day of a new job deprived of any form of supervision. If they have never learned to bear responsibility, they place themselves and their patients in potential danger. An EPA-‐based competency curriculum aims to establish this gradual increase of responsibility and responsive autonomy in a safe and justifiable way (2). 2.6 Assessment, entrustment and supervision EPA-‐based assessment is framed as entrustment to carry out critical activities under a designat-‐ed level of supervision. In other words, a trainee is primarily evaluation to determine how much supervision he or she needs for a specified EPA. This leads to the distinction of five levels of su-‐pervision: (1) no permission act, (2) permission to act with direct, pro-‐active supervision pre-‐sent in the room, (3) permission to act with indirect supervision, outside the room but quickly available of needed, (4) permission to act under distant supervision not directly available or (5)


Deliverable D2.1

14

permission to provide supervision to junior trainees (3,37). This supervision concept will be explained in detail in section 4.2 2.7 Portfolio as a tool to organize and support competency-based development The portfolio, in an original sense, is a showcase for students of art, crafts, architecture and oth-‐er visual domains, that enable the exposition of personal accomplishments at job applications and for potential customers. Portfolios in health professions education are personal containers, usually trainee-‐owned, of information to document progress and to stimulate reflection. Since the turn of the century, portfolios gradually have become accepted as a useful tool in health pro-‐fessions education with a two-‐fold purpose to better capture information about individual pro-‐gress of trainees and to stimulate students to reflect on their learning progress. (26) They are rapidly adopted on electronic platforms, allowing access to different parts of the content by var-‐ious target groups as workplace learning is individualized and its assessment dependent on mul-‐tiple information sources, a portfolio to document progress is a better tool for evaluation than traditional standardized test administrations for groups of students. One possible use in compe-‐tency-‐based workplace training is to turn summative entrustment decisions for EPAs (meaning that “the trainee is now permitted to do EPA X with only distant supervision”(3)) into digital badges in an electronic portfolio (53). These badges could be accessible to supervisors, or, in the medical domain, nurses, other health care worker colleagues, and patients. This requires a care-‐ful and authorized validation of competence by an authorizing body (e.g., an examination com-‐mittee at institutional level, or some organization at a national or international level). It also means that this part of the portfolio must be validly maintained. 3. Translating the professional work into EPAs Many schools or programs that consider curriculum development with EPAs already have an existing framework or blueprint of educational objectives, established at local, regional or na-‐tional levels. They may have existing competencies, milestones, problem lists, knowledge-‐skills-‐attitudes lists or other. These can be further enhanced by the use of EPAs. An EPA-‐based compe-‐tency curriculum reframes those objectives in the context of the workplace. Entrustable profes-‐sional activities are basically not an educational concept but simply structured descriptions of professional work. Of course, the purpose of defining EPAs is to serve education. Curriculum building with EPAs begins with an elaboration of what professionals do in practice. This elabora-‐tion is something professionals can best do themselves. It is job analysis with an educational purpose in mind. It resembles what Jonassen et al. have called “job task analysis”, leading to an overview of tasks sometimes categorized with labels such as frequency (very rarely to multiple times per day), importance (not to very) and difficulty to master (easy to difficult) (54), and framed in a language that supervisors would use related to a real workflow including entrust-‐ment decisions for trainees at the right time. For the use of EPAs in health care education, ‘im-‐portance’ would also include how critical it is that the task is done safely. Labeling EPAs this way is not always necessary, but it may help trainees focus on the most important experiences. The steps to be taken are Identifying EPAs – Elaborating EPAs – Validating EPAs.


Deliverable D2.1

15

3.1 Identifying EPAs EPAs are units of professional practice that can be regarded as discrete tasks or responsibilities that are only entrusted to a professional to be executed unsupervised if the individual is suffi-‐ciently competent. EPAs reflect what graduates of a program or course must be able to do more or less by themselves. Identifying EPAs is usually an iterative back-‐and-‐forth process among professionals. One useful method to start is to have a small group of professionals with similar background analyze a week of work in the profession, starting Monday morning and ending Sunday evening, at a typi-‐cal location, such as a health care subspecialty ward, and identify units of work that can serve as an EPA. To establish EPAs adequate for education, an important question to ask is what graduates of the program are expected to do when starting a new phase in the trajectory, such as a residency after MD graduation, a fellowship after residency or unsupervised practice after a residency or fellowship. Several authors have recently provided such lists of activities. (55,56) Activities can be small or comprehensive. Clustering small activities into EPAs is an important topic of debate and sometimes of confusion. It relates to the question of how broad or granular EPAs should be, and consequently how many EPAs need to be distinguished. There is no straight answer to the ‘right’ breadth of EPAs and consequently to the number of EPAs to be distinguished. If the question is “what is the scope of responsibility that is covered when an EPA is entrusted to a trainee for indirect supervision?”, then clearly big differences can arise depending on the level of trainee in question. The first EPA that may be entrusted to a med-‐ical student could be ‘measuring blood pressure’. If we consider this unit of professional prac-‐tice, an activity that one can trust a trainee to complete unchecked, then it is a true EPA. But clearly, this tiny responsibility can and must be part of a full standard physical examination that is a more logical unit for more advanced medical trainees. The full standard physical examina-‐tion, in turn, can be included in a broader EPA of a standard outpatient consultation that also includes the history. In technical terminology, smaller EPAs are nested within larger EPAs (Fig-‐ure 2).

Figure 2 Nesting EPAs


Deliverable D2.1

16

This leads to the conclusion that the breadth of an EPA is related to the end-‐of-‐training program requirement, or the entrance requirement for the next phase of training. Examples are the “EPAs for entering residency” (57) or the geriatric “end-‐of-‐training EPAs” (58). This does not mean that the EPAs are only mastered at the end of that training period. Indeed, the key to competency-‐based training is that EPAs may be mastered and awarded with decrease in supervision as soon as the trainee demonstrates the required competence. Another important consideration relates to the granularity of EPAs. EPAs are meant to be units of practice that can be awarded a STAR (statement of awarded responsibility, see section 5), i.e. a formal acknowledgement of the ability and right to practice at a certain level of supervision. As this signifies a significant step towards joining a professional community, it does not make sense to distinguish hundreds of small EPAs, loosing their significance. Warm et al. have recently pro-‐posed to name such small units ‘observable practice activities’ to be clustered in larger EPAs. (59). Even for early EPAs such clustering makes sense. One proposed early EPA for a first clerk-‐ship at University Medical Center, the Netherlands is ‘Routine check-up of a stable adult patient’. This EPA includes measuring vital functions heart rate, breathing, temperature, blood pressure, saturation—by hand and with devices, explaining all actions to the patient and documenting and reporting results to the members of the health care team. Each of these is an activity, but logical-‐ly they constitute together one EPA that allows for formal permission to do all included activities with only indirect supervision. A medical student, entrusted with this EPA at a level of indirect supervision, is trusted to do any of these or all of these without a supervisor present in the room. The breadth or size of EPAs is directly linked to the number of them. In an educational program, entrustment decisions for EPAs are meant to be significant moments that constitute increasing trust and increasing responsibility in trainees aligned with a generally supported need for pro-‐gressive independence or autonomy (51,52,60–63). To maintain their significance, such formal entrustment decisions cannot be taken every day or every week. An EPAs-‐based workplace cur-‐riculum should map out a route for individual trainees with summative entrustment decisions at significant moments (3) that lead to acknowledged privileges in patient care. This approach leads to the general recommendation that trainees have no more than 10, but preferably less moments per year of certification for EPAs. The quickly growing literature that describes EPAs for educational programs show the following numbers of EPAs (Table 1), mostly aligned with this recommendation. Table 1. Numbers of EPAs proposed, related to program length Source Program Length

(years) Number of EPAs

Mulder et al 2010 (49) Physician assistant education 2.5 5-8 Boyce et al 2011(42) Psychiatry residency, 1st year 1 4 Jones et al 2011 (47) Pediatric residency 3 17 Hauer et al 2013 (64) General Internal Medicine residency 3 30 Chang et al 2013 (65) Internal Medicine (Patient-Centered Med. Home program) unspec. 25 Shaughnessy et al 2013 (41) Family Medicine residency 3 76 O’Keefe 2013 (66) Developmental-Behavioral Pediatrics residency unspec 14


Deliverable D2.1

17

Flynn et al 2014 (57) Undergraduate medical education (+ 2.5 year clinical) 2.5 13 Fessler et al 2014 (43) Pulmonary care residency 1-2 18 Fessler et al 2014 (67) Critical Care medicine residency 1-2 13 Rose et al 2014 (48) Gastro-intestinal fellowship 3 13 Caverzagie et al 2014 (40) Internal Medicine residency 3 16 Chen et al 2015 (submitted) Undergraduate medical education pre-clerkship training 2 5

3.2 Some common misunderstandings Excellent examples of EPAs may be found in the literature mentioned in Table 1 (see also the appendix). While most of the EPAs that have been proposed comply with its definition as given earlier, i.e. observable, measurable, having a designated time frame, and being suitable for en-‐trustment decision, some published EPAs may pose problems when assessment and entrust-‐ment are operationalized (68). Several common misunderstandings are useful to elaborate. They are illustrated with examples drawn from the published literature and from conversations with educators designing EPAs for their programs. a. EPAs that are not discrete tasks and unsuitable for focused entrustment decisions Examples Practice personal habits of lifelong learning; Demonstrate professional behavior; Iden-‐tify system failures and contribute to a culture of safety and improvement; Minimize unneces-‐sary care including tests; Minimize unfamiliar terms during patient encounters; Enhance patient safety; Improve the quality of health care. There is no dispute that the ability to do these things is essential. It is however difficult to envi-‐sion a moment at which trainees are entrusted to carry these out with only indirect supervision, before which moment they should not permitted to do this. They are activities, and they are im-‐portant, but they do not fully meet the EPA definition. Rather, they are ongoing habits that should be present as trainees mature to be professionals. They should be addressed in educa-‐tion, but are conditions for entrustment of various different EPAs, rather than EPAs, units of work, in themselves. b. EPAs that are inseparable from other EPAs Examples Manage the sad patient; Recognize child abuse. Somewhat similar is this caveat. These may be important skills, but as sad patients may have various diseases and may be sad because of medical conditions, managing sad patients cannot easily be viewed as a stand-‐alone EPA. Rather one would hope that most medical graduates would be able to cope with sad patients across various EPAs. Recognizing signs of abuse when examining a child is important but not a stand-‐alone EPA. c. EPA titles that include reference to proficiency level Examples Skillfully facilitate a family meeting; Safely and efficiently perform common critical care procedures; Evaluate and manage an acute or new patient complaint as a 21st century phy-‐sician. Typical for EPAs is that as work descriptors, they contrast, with person descriptors such as competencies, skill and knowledge. It is not useful to refer to skill or proficiency in the title of an


Deliverable D2.1

18

EPA. To avoid this, think first of the job that must be done, and then about a person best equipped with skills to do it. Adjectives as to how well the job must be done are not necessary. “Skillfully” would hold for any EPA, just as carefully, adequately, safely; they all connote the per-‐son; the job however is neutral. It is helpful to keep titles short and include specification and limitations in an added elaboration, unless the short title would generate misunderstanding. “Share information about the patient’s care, including diagnosis and management plan, with a patient in no significant physical or emotional distress” was proposed as a pre-clerkship EPA (Chen et al, submitted). That is long, but probably necessary to avoid misunderstanding at first sight. d. EPAs that are too broad Example Care for acute or new patients. This EPA is too broad, as it includes almost all of health care. A clear description that includes limitations, specified for trainees in a general program, would yield a very long description that would make the EPA impossible to evaluate for indirect supervision. ‘Care’ may not have a speci-‐fied beginning and end and can lead to different interpretations and confusion (1). ‘Care for psy-‐chiatric patients’ could include a full profession, as if one would consider acting as a psychiatrist one EPA. The purpose of EPAs is the distinction of units of practice that allows for separate en-‐trustment decisions and gradually increasing responsibilities. e. EPAs that are discrete tasks, but not suitable for entrustment decisions Examples Designing a personal development plan; Elaborating and presenting a critical apprais-‐al topic for colleagues in the department. These tasks may be important for education and for the quality of work, but it is not logical to restrict the task to be done with direct or indirect supervision only. Nor is it logical to ‘advance’ trainees to a higher level of autonomy for these tasks. EPAs can differ in the degree they meet components of the EPA definition. To illustrate this, Ta-‐ble 2 shows seven different EPAs that have critical features related to the question “can we trust someone to do this”. The first EPA (performing a lumbar puncture) is highly critical, even with direct (level 2) supervision, because only one person can do this and, if not done well, it can re-‐sult in permanent neurological disability. The last EPA is low risk, not at all irreversible and has basically no consequences for safe health care. EPAs 1 to 5 could be called true EPAs, 6 and 7 may rather not be called EPAs.

Table 2 Sample of suggested EPAs ranked by typicality of being an EPA

Features

Activity

High Risk Irreversible in its con-

sequences

Inherently dependent on

one actor

Key to safe health care on the spot

Is this typi-cally an EPA?

1. Performing a lumbar punc- ++ +++ +++ +++ yes


Deliverable D2.1

19

ture

2. Conducting a laparoscopic cholecystectomy

+++ ++ ++ +++ yes

3. Prescribing critical medica-tion

+ + - ++ yes

4. Breaking bad news to a patient

+/- +/- - - +/- yes

5. Designing and presenting a new therapy protocol

- - - - + yes

6. Conducting a literature review

- - - - - no

7. Designing a personal development plan

- - - - + - - no

We recommend ‘conducting a literature review’ not to consider an EPA, as one cannot envision an entrustment decision before which a trainee is not permitted to do this unsupervised. Design-‐ing one’s personal development plan is actually not a task that is part of the necessary tasks that must be carried out by the profession. One way to think of EPAs is to imagine a list of tasks that must be done and then to think who would be best equipped to do these tasks. They may, in some form, feature on personnel advertisements when new staff is hired. No one would hire personnel to design their own personal development plan. Are all activities of professionals EPAs, and do all EPAs together cover the profession? As stated above, mapping a workplace curriculum in EPAs aims at covering all professional ac-‐tivities that a program envisions their graduates should have mastered to be ready for practice or for continued education in an advanced course, such as a fellowship. If this is true and feasi-‐ble, the question arises do professionals ever do things that are not EPAs? The answer is yes, very often, and this is a source of confusion that requires clarification. First, not all activities are professional activities in the strict sense of EPAs. Professional activities are those that non-‐professionals are not usually trained, equipped or permitted to do(1). Some things physicians do may only be indirectly related to health care execution (such as personal development activities) and could not be envisioned as permitted only with close supervision until entrustment. Other activities are not stand-‐alone EPAs, such as ‘cost effective utilization of resources’ or ‘applying methods to maximize adequate patient experience’. Recalling to mind the matrix-‐mapping ap-‐proach described earlier (Figure 1), these ‘activities’ are component approaches to EPAs, includ-‐ed in competencies such as professionalism, management, or systems-‐based practice. They are important to verify when trusting trainees to execute genuine EPAs, such as “evaluating and managing low-‐acuity, low complexity stable patients in the ER”, but they are not EPAs in the true sense. A profession, e.g. a medical specialty, can be defined broadly or more narrow. The core EPAs of a profession can constitute the …


Deliverable D2.1

20

EPAs constitute standards for expected professional practice for all graduates. Core, and in some jurisdictions, specialty-‐specific EPAs designate minimum requirements. This sounds minimalis-‐tic, but it only means that standards for every graduate are by definition minimum standards. Many graduates, and hopefully most, may distinguish themselves on top of this, by their excel-‐lence above expectation, and by unexpected personal skills and qualities they bring to the work-‐place. We cannot capture these extras in EPAs as they are often unique qualities that we truly value but cannot ‘measure’ in any objective sense. 3.3 Elaborating EPAs For educational purposes it is not sufficient to identify EPAs as a simple list of tasks by titles. The reason is that most formulations of tasks are multi-‐interpretable. To enable an entrustment de-‐cision (“the trainee may now do this with only indirect supervision”), there must be specifica-‐tions. To illustrate this, if the EPA is “Gather a history and perform a physical examination”, the entrustment decision must include specifications and limitations. A near-‐death multi-‐trauma patient after a major accident arriving at an ER must be examined by a trauma team; a medical student would not be trusted to gather the history and perform the physical examination with this patient. Clearly, this EPA has multiple variants. For the medical student, high-‐risk, high-‐complexity patients requiring urgent care must be excluded. In addition, the matrix-‐approach to using EPAs requires a specification of which competencies, sub-‐competencies, knowledge, skill and behaviors should be present before the trainee may be trusted to perform unsupervised or with only indirect supervision. Also, a plan for assessment is needed to guide trainees in their preparation for entrustment decisions. The recommended full description of an EPA therefore includes the rubrics as mentioned in Ta-‐ble 3, evolved from earlier versions of this format (3,49). Some of these rubrics refer to assess-‐ment, supervision levels and entrustment decision-‐making explained later in this Guide. Table 3 Components of a fully described EPA 1. Title of the EPA

An EPA title should be concise and informative, i.e. readily understood. As it only reflects work, it should not be state as a learning objective or skill, merely as an activity. Try to limit to 10 words or less.

2. Specification and limitations

This specification should clearly list what is included in the activity and what is not included, given the level of the intended trainees. It should also include the context and targeted transition (e.g., entering residen-‐cy, fellowship, autonomous practice)

3. Most relevant do-‐mains of competence

This section relates the EPA to the competency framework used. Those competencies, sub-‐competencies or domains of competence of the framework that are most applicable may be mentioned.

4. Required experience, knowledge, skills, at-‐titude and behavior

Trainees should be aware what knowledge, skills and attitudes are ex-‐pected before they can be trusted to carry out the EPA; this will help them to prepare for entrustment. It may also be helpful to understand which workplace experiences are considered necessary before en-‐


Deliverable D2.1

21

trustment (type of rotation, type of patients, number of procedures).

5. Assessment informa-‐tion sources to assess progress and ground a summative en-‐trustment decision

Teachers should be aware what sources of information should be used to determine progress. That can be observed behavior or skill at the bedside or at morning reports meetings, a skills test, information from colleagues, nursing and patients, a double-‐checked procedure, a case-‐based discussion and other sources. For trainees as well as teachers it is important to state how many times an EPA or its constituent parts must have been observed to enable taking a summative entrustment decision, and to state who takes this decision. It is highly recommended that multiple staff members sign off such decisions. Supervisors should feel personal responsibility of these important decisions.

6. Entrustment for which level of super-‐vision is to be reached at which stage of training?

The consequence of an entrustment decision is stated as the permission to act under a designated level of supervision (e.g. indirect supervision, or distant supervision) not generally permitted before that time

Next, it is necessary to state at which transition of training trainees must ultimately master the EPA at that level. Graduation should re-‐quire that all core EPAs of the program be mastered.

When building an individual workplace curriculum it is useful to esti-‐mate when this trainee is expected to receive the entrustment decision, based on prior training and expected rotations and experiences.

7. Expiry date Optional but recommended is stating expiry dates. Entrustment should drop if no maintenance of competence for this EPA happens, e.g. over a period one up to five years, depending on the EPA. Revalidation may be a marginal or a more substantive check.

Most of the rows of this description can be more or less generalized and applicable to EPAs in multiple settings (departments, hospitals, clinics). Some may be affected by local jurisdictions. As appendices to this guide examples of EPAs are provided for undergraduate and postgraduate medical education and for veterinary medicine. The text table shows one early undergraduate EPA as designed at UMC Utrecht.

Table 4 Example of an early EPA in undergraduate medical education 1. Title of the EPA Routine check-up of the stable adult patient

2. Specification and limitations

This EPA includes no more and no less than 1. Measuring vital parameters heart rate, respiratory rate, tempera-‐

ture, blood pressure, saturation 2. Explaining all actions to the patient 3. Reporting results to the health care team including interpretation,

orally and/or written Context ambulatory and inpatient setting Targeted transition first fulltime clinical clerkship to next clerkship


Deliverable D2.1

22

Limitations only with circulatory stable patients of 18 year and older

3. Most relevant do-‐mains of competence

X Medical Expert " Health Advocate X Communicator " Scholar X Collaborator " Professional " Manager

4. Required experience, knowledge, skills, at-‐titude and behavior

Knowledge -‐ basic knowledge of anatomy including relevant arteries -‐ normal values of vital parameters Skill -‐ skill in using necessary devices to measure vital parameters -‐ recognition of stable and unstable patients Attitude and behavior -‐ professional communication with the patient -‐ proactive alertness in case of adverse events -‐ willingness to ask for help if needed Experience -‐ all measurements done at least 5 times

5. Assessment informa-‐tion sources to assess progress and ground a summative en-‐trustment decision

-‐ Observation satisfactory observation of all measurements at least fully twice by experienced health care professionals (nurse, physi-‐cian or other)

-‐ Case-‐based discussions one CBD with an qualified health care pro-‐fessional

6. Entrustment for which level of super-‐vision is to be reached at which stage of training?

Indirect supervision (level 3) ultimately before the transition to the second full time clinical clerkship

7. Expiry date One year without practice after summative entrustment decision

3.4 Validating EPAs EPAs should be as relevant and correct as possible, and supported by those who work with it. With validation we therefore primarily aim at content validation of EPAs (Is an EPA truly part of work, does it comply with the EPA definition and is it fit for its purpose?). Validation of EPAs aims to align them as closely as possible with common requirements for graduates from the program or similar programs and should lead to wide and well-‐founded recognition of entrusted EPAs. Validating a set of EPAs also aims at covering the full professional domain. Content valida-‐tion can be done by comparing EPAs with existing documents such as curricular blueprints and publications in the literature, or with the opinion of experts, or by combining these. Soliciting expert opinions serves not only the purpose of improving the quality of the set of EPAs, but also informs and involves faculty who may be working with these EPAs in the future. Evidence for content validity of EPAs can be gathered with several techniques as elaborated below. A study


Deliverable D2.1

23

by Chen et al shows one excellent example various approaches can be used (Chen et al, submit-‐ted). Expert meetings. Chang et al gathered a wide range of internal medicine experts (program di-‐rectors, clinicians, educators, researchers) at a national two-‐day meeting preceded by multiple email and telephone preparatory conversations reviewing the literature. At the summit three sessions were held with different compositions of delegates to refine EPAs, resulting in a con-‐sensus list (65), somewhat similar to a procedure conducted by Fessler et al (43). Chen et al used the opportunity at local, national and international education conferences to conduct struc-‐tured group discussions to refine pre-‐clerkship EPAs (Chen et al, in preparation). Leipzig et al describe how two national meetings of geriatricians, a year apart, were used to validate EPAs in geriatrics, proposed by a working group. (58) Surveys and interviews among experts. Boyce et al surveyed 470 fellows of the Australia and New Zealand College of Psychiatry with 30 proposed EPAs for psychiatry training, asking which should safely be entrusted to unsupervised residents at the end of the first year, leading to four priorities, subsequently developed into EPAs.(42) Spenkelink et al interviewed urologists with the question what of your work would be suitable to trust well-‐trained physician assistants to take over as EPAs (70). Delphi procedure. In a Delphi procedure experts are approached individually to answer a sur-‐vey, are then fed back its aggregated results to refine their original responses and repeat this in a third round if necessary. Hauer et al applied this technique among 22 educators and 12 resi-‐dents at three hospitals (64) and Shaughnessy among 21 experts for family medicine residency (41). Delphi procedures are being used regionally to establish validity of EPAs for end of under-‐graduate medical education at Charité University medical school in Berlin (Peters et al, in prepa-‐ration), for residency training in anesthesiology in the Netherlands (Wisman-‐Zwarter et al in preparation), and for undergraduate veterinary training in the Netherlands (Duijn et al in prepa-‐ration). Nominal group technique. Touchie and colleagues used a nominal group technique to identify EPAs that residents in their first year in multiple specialties should accomplish doing indepen-‐dently. She asked 8 subject matter experts during a brainstorm session to each list as many EPAs as possible that could meet this condition, yielding 25. A consensus discussion in the group led to 10 EPAs that subsequently were ratified by a national panel of 9 medical educators (71). Q-sort. One way to validate and prioritize EPAs for an educational program is with the Q-‐sort technique. This method was applied by Englander after publication of the Core EPAs for Enter-‐ing Residency (57). The Q-‐sort method assumes that many variables naturally show a normal distribution. An interested group at an international medical education meeting was asked to sort and prioritize the 13 EPAs on cards according their importance and need of attention in undergraduate medical education. In a picture of a normal distribution curve, 13 slots were made, and the EPAs were placed in the slots according to their respective positions and the sub-‐


Deliverable D2.1

24

sequent discussion focused on high and low scoring EPAs, while the bulk could be placed near the center of the curve. Table 5 Strategies described in the literature to validate EPAs among experts Strategy Explanation References to examples Expert mee-tings, national or international

Meetings of experts during conferences or gathered for this purpose are used to build consensus about EPAs

Chang et al 2012 (65); Fessler et al 2014 ; Chen (in pre-paration); Hauer et al 2013 (72) Caverzagie et al (40)

Surveys Asking an expert populations to score the validity of EPAs for a designated purpose

Boyce et al 2011 (42)

Delphi proce-dure (73)

Carefully selected experts are surveyed with a list of EPAs to score their validity on a scale; aggregated results are pre-sented to the subjects to refine their original score. If needed, a third round is conducted.

Fessler et al 2014(43) ; Hauer et al 2012 (64); Wisman-Zwarter et al (in preparation); Duijn et al (in preparation); Peters et al (in preparation)

Nominal group technique (73)

Establish a listing of potential EPAs among an expert group until no new EPAs can be thought of. Then refine the list by grouping and prioritizing to finalize with a best consensus list.

Touchie et al 2014 (71)

Interviews Program directors can be interviewed asking “what activities would you expect incoming residents be able to do without direct supervision” or hospital department heads about which EPAs newly hired specialists should be able to do autono-mously.

Westerveld et al 2004 (74); Spenkelink et al (70)

Q sort Prioritizing EPAs on cards using slots that together resemble a normal curve

Englander et al (AMEE meeting 2014)

When validating EPAs through surveys or Delphi procedures, it is of great importance that re-‐spondents are aware of the EPA definition. Suggested but faulty EPAs such as “Minimize unnec-‐essary diagnostic tests” may be endorsed as very important by many respondents, or added to a list, if additions are asked for. However, it should not be identified as an EPA as it does not meet EPA requirements (1) and it is not a task that can move from being directly supervised to indi-‐rectly supervised. Validating EPAs often also serves buy-‐in by an important target group. Once adopted, faulty EPAs are more difficult to correct. This makes a combination of survey and face-‐to-‐face validation procedures that allow for explanation useful.

4. Building and maintaining an individualized workplace curriculum with EPAs

A workplace curriculum in health professions education is an important part of a full curriculum. While much of a curriculum is preferably standardized and uniform for all students, workplace curricula are far less standardized and hence different for each student. In a competency-‐based curriculum model individual adaptations in curricula are necessary, guided by workplace and


Deliverable D2.1

25

practice experiences. While standards can and must be adopted, the pathways for individual trainees must be flexible (16), and depend on both personal qualities and the context of the workplaces that characterize their particular training. Building a workplace curriculum using a set of validated EPAs can be viewed as a task-based instructional strategy, described by Merill (75), following his five research-‐based ‘first principles of instruction’ (1) Task centered – learning is promoted when learners acquire concepts and principles in the context of real world tasks, (2) Activation – learning is promoted when learners activate relevant prior knowledge, (3) Demonstration – learning is promoted when learners observe a demonstration of skills to be learned, (4) Application – learning is promoted when learners apply their newly acquired knowledge and skill, and (5) Integration – learning is pro-‐moted when learners integrate their new skills into their everyday life. A curriculum focused on EPAs with the prospect of acquiring the permission to execute these with no or indirect supervi-‐sion follows just that strategy. 4.1 A general framework for the workplace curriculum Workplace curriculum building begins with mapping the expected moments of entrustment de-‐cisions for EPAs against the years of training, in a way that can be adapted for individual trainees if necessary at any time. In Figure 3, five EPAs of a program show how trainees are expected to increase (in darker shades) in the competence until they have reached a moment at which they are expected to be trusted to perform this activity unsupervised, or in some cases, such as un-‐dergraduate medical education, to do this with only indirect supervision. The stars represent the Statements of Awarded Responsibility, following formal and summative entrustment decisions (3,37). Entrustment for EPAs A, C and E is expected to be reached later than for EPAs B and D. When a new trainee starts, a tailored version of the framework may be made, depending on the structure of the workplace curriculum, and the length, nature and sequencing of rotations and also on previously acquired competencies of the trainee. An individualized framework should be agreed upon with the trainee and can be viewed as a learning contract with committed efforts on both sides practice effort for the trainee, supervision and coaching from teachers. Figure 3 A general workplace curriculum framework PGY 1 PGY 2 PGY 3 PGY 4

EPA A EPA B EPA C EPA D EPA E 4.2 Supervision levels related to entrustment decisions Entrustment decisions require a specification of what exactly is decided. Trust relates to the acceptance that the trustee is permitted to act in the circumstance that risks are not excluded. Trainees may be trusted and licensed to drive a car unsupervised when adequate driving skill


Deliverable D2.1

26

and relevant knowledge has been demonstrated. Their competence has reached a threshold that permits them to do this. The risk of accidents is now considered low and manageable. For trainees in the health care domain, a more subtle transition between full supervision and unsupervised practice aligns better with heath care practice. The five levels of decreasing su-‐pervision, most used when applying EPAs, are as follows (2,3,37). Figure 4 states the trainee’s permissions, related to supervision levels. Figure 4 General framework of permissions, related to supervision levels Level 1 Be merely present and observe Level 2 Act with direct, pro-‐active supervision, i.e. with a supervisor physically present in the

room Level 3 Act with indirect, re-‐active supervision, i.e. readily available on request Level 4 Act without supervision readily available, but with post-‐hoc report or distant su-‐pervision Level 5 Provide supervision to junior trainees Trainees and their environment should know at any moment at which level they formally can act for any given EPA. This does preclude that, anticipating a next level, supervisors can and should grant ad-hoc permissions, to allow trainees to start acting with less supervision, for educational purposes. In educational terminology, the ‘zone of proximal development’ (76) is used to pre-‐pare student for a summative entrustment decision that permits them to act under the specified level of supervision without explicit instruction (see chapter 5 for a more detailed explanation of ad-‐hoc or formative versus summative entrustment decisions). This supervision framework aligns with the standards of the US Accreditation Council for Grad-‐uate Medical Education (ACGME). Level 2 equates with ACGME’s ‘direct supervision’, Level 3 with ‘indirect supervision’ and Level 4 with ‘Oversight’ (77). These standards are important as they are increasingly used to accredit hospitals for quality and safety, specifying to what extent workers are permitted to act according to their documented skills. If applied to the general framework, an individual curriculum can be built, showing not only the moment at which the major level 4 decision is to be taken but also the other levels of supervision (Figure 5) (4). This example may be the results of an agreement with the student, based, for in-‐stance, on documented prior experience with EPA D, to enable an early higher-‐level start. Such agreement can and should of course not lead to the right to work unsupervised from a pre-‐designated moment, if competence has not yet been demonstrated, but it gives direction to ex-‐pectations for trainees and supervisors. Flexibility to adapt moments of entrustment decisions is needed to realize true competency-‐based education. Figure 5 An individualized workplace curriculum framework with expected super-

vision levels Individual workplace curriculum

PGY 1 PGY 2 PGY 3 PGY 4

EPA A 1 2 2 3 4 4 5


Deliverable D2.1

27

EPA B 1 2 3 3 4 4 4 5 EPA C 1 1 2 3 3 4 4 EPA D 2 2 2 3 4 5 5 5 EPA E 1 1 2 3 3 4 4 4 This figure only shows a very schematic version of the time frame. In this example, EPA E could be practiced at level 2 from the beginning of the first semester of program year 2. But gradually, say after a few weeks, EPA E could be practiced ad hoc at level 3, with frequent close observa-‐tion, to make sure that by the end of that semester a formal and summative entrustment deci-‐sion can be taken that allows for working at level 3 from the beginning of second semester on. At start of a new a clinical rotation there may be a verification of the level that the trainee’s portfo-‐lio or digital badge indicates he or she has be certified for. Next, a supervisor may allow the trainee to take more responsibility on ad-‐hoc basis to enable monitoring whether he or she can be ready to be entrusted with a higher level of autonomy and advise a program director or the trainee to opt for more autonomy at a next progress interview. The framework in Figure 5 is widely used and appears adequate for postgraduate medical train-‐ing. For undergraduate training, Chen et al have recently recommended a more granular frame-‐work of supervision levels. (78) depicted in Table 6.

Table 6 General framework of permissions, elaborated for undergraduate medical education

Standard entrustment and su-pervision framework

Granular entrustment and supervision framework proposed for undergraduate medical education (79)

1. Not allowed to practice EPA 1. Not allowed to practice EPA

1a. Not allowed to be present because of insuffi-‐cient background for safe care

1b. Allowed to be present and observe

2. Allowed to practice EPA on-‐ly under proactive, full su-‐pervision

2. Allowed to practice EPA only under proactive, full supervision

2a. As co-‐activity with supervisor

2b. Alone, but with a supervisor in room ready to step in if needed

3. Allowed to practice EPA on-‐ly under reactive/on-‐demand supervision

3. Allowed to practice EPA only under reactive/on-‐demand supervision

3a. With supervisor immediately available, all find-‐ings being double checked

3b. With supervisor immediately available, key findings being double checked

3c. With supervisor distantly available (e.g. by phone), findings being reviewed


Deliverable D2.1

28

4. Allowed to practice EPA un-‐supervised

4. Allowed to practice EPA unsupervised

5. Allowed to supervise others in the practice of this EPA

5. Allowed to supervise others in the practice of this EPA

4.3 Task-based instructional strategy To prepare trainees for professional tasks, EPAs may lead to mini-‐curricula, derived from their description (see table XX). While the professional context may not be altered for educational purposes, experiences of trainees can be influenced by selecting and sequencing of activities (Chen et al, submitted). Complex EPAs may require Merill’s Demonstration, Application and In-‐tegration phases (75), or may require preceding practice in a simulated environment or self-‐directed study effort (80) shortly before entering the workplace. In the workplace, regular coaching, role modeling, instruction for specific EPAs, and practice opportunities with frequent, specific feedback are conducive to learning. The most important strategy is regular, ongoing contact with a preceptor for coaching and the provision of feedback. In medical courses this need for continuity is increasingly being stressed and often operationalized through longitudinal integrated clerkships (81–84). Indeed, time is needed to build the trust that is necessary for entrustment decisions (85). 4.4 Connecting EPAs and competencies with milestones and supervision levels. Milestones are behavioral descriptions on a scale that indicates a developmental trajectory. In 2013 and 2014 the Accreditation Council for Graduate Medical Education (ACGME) of the United States mandated that all residency programs in the United States develop Milestones, defined as a set of 5 descriptors for each sub-‐competency that a residency had developed under the six broad ACGME competency-‐domains (patient care, medical knowledge, interpersonal & commu-‐nication skill, professionalism, systems-‐based practice and practice-‐based learning & improve-‐ment). The question arises how milestones and (sub-‐) competencies connect with EPAs and su-‐pervision levels. This can best be illustrated with a figure. Figure 6 shows an EPA at the left side (65). Among the six ACGME competency domains two are identified as being particularly rele-‐vant for this EPA and given two asterisks. In practice each domain would show several sub-‐competencies, but for reasons of clarity they are left out. The milestones next to the competency-‐domains show shades of grey. In reality these cells are descriptions of trainee behavior devel-‐opment toward competence and proficiency. The arrows show how trainees must align with multiple behavior descriptions to be allowed to conduct this EPA under direct, or indirect su-‐pervision or with no supervision. Readers interested to see such behavior description are re-‐ferred to two supplements of the Journal of Graduate Medical Education (June 2013 and June 2014) for postgraduate programs and to Englander et al 2014 (57) for undergraduate medical education. The latter distinguishes only two levels (pre-‐entrustable learners and entrustable learners). The descriptions can be a great help for educators to develop an understanding how trainees impress at various stages of development.


Deliverable D2.1

29

Figure 6 Connecting EPAs and competencies with milestones and supervision levels.

4.5 Core, specific and elective EPAs Various authors have proposed to distinguish different type of EPAs within one program. (57,79) Core EPAs should be those mastered at a specified level (“unsupervised practice” for residents; “Indirect supervision” for medical students) by all trainees in the program. In other words, the is no possibility of graduating and finishing the program if any of these is not mas-‐tered at the required level, compliant with the fundamental philosophy of competency-‐based medical education (24,86). Core EPAs, such as those proposed for undergraduate medical educa-‐tion in the USA (57) by virtue of its name imply that non-‐core EPAs also should exist. In residen-‐cy training, non-‐core EPAs may pertain to focus areas of interest. As an example, the proposed EPA-‐based new national curriculum in Radiology & Nuclear Medicine in the Netherlands expects every graduate to choose one or two focus areas, to supplement the core EPAs in radiology. (87) As there are EPAs for eight focus areas (e.g., cardio-‐thoracic radiology, pediatric radiology, inter-‐vention radiology) these are sub-‐specialty specific EPAs. They allow for flexibility of competen-‐cy-‐based training, as some residents will end training being certified for two focus areas and others with only one focus area. Chen et al have proposed elective EPAs next to core and specialty-‐specific EPAs in undergradu-‐ate medical education. Students may, if time permits given their superb progression, attempt to become certified for one or more additional EPAs of their choice (79). 5. Monitoring and assessing trainees using entrustment decisions for EPAs


Deliverable D2.1

30

The final step in consolidating an EPA-‐based competency curriculum is making sure that the major “milestones” for the trainees for such curriculum, i.e. the decisions to entrust them with meaningful tasks for the professional community, are well founded, serve as landmarks to guide trainees in their learning activities and are the focus of feedback and monitoring by preceptors. Making entrustment decisions may be viewed as a specific approach to assessment. Instead of using neutral value statements such as numbers or labels on a scale (1-‐10, A-‐E, fail to outstand-‐ing) the focus with EPAs is on a statement about the level of required supervision. By doing this, educational objectives are linked to health care and patient safety objectives (88). Translated to every-‐day wordings, the questions for the staff are Can I leave the room? Do I need to return to check? Can the trainee finish without me? Can the trainee manage the admission of a patient without proactive assistance? Can the trainee now do this procedure, manage the case, work the apparatus, chair the meeting, hand over the patient et cetera without support? This is a different way of thinking than the mental transformation clinical teachers usually must do to turn obser-‐vations into scales related to competencies. That has become common for clinician educators but has proven troublesome (5,89–91). It is likely that quality (reliability and validity) of as-‐sessment increases when professionals can phrase their opinion in meaningful propositions such as the questions above (92,93). When encounters and evaluations are framed in such ques-‐tions, summative assessment can more easily arise from a summation of observations. Entrustment decisions may be distinguished in two forms (i) ad-‐hoc or formative entrustment decisions that happen every day, are taken by individuals and pertain to immediate permission for the trainee to act, and (ii) summative entrustment decisions that are grounded in more or less systematic observation, leading to lasting permission to act under a specified level of super-‐vision, comparable with the driver’s license that formalizes a permission to driving unsuper-‐vised from moment X on (94). They have previously been labeled as ‘ad-‐hoc’ versus ‘structural’ entrustment decisions, which refers to the same distinction (3). Formative entrustment deci-‐sions may sound as an oxymoron, as formative evaluation by definition does not lead to deci-‐sions. However, formative entrustment is without long-‐term consequences. It has a temporal nature for educational purposes; the permission is only granted at this moment, in this context for, this patient and by this supervisor to stimulate development and evaluate the eligibility for summative decisions. Conversely, a summative entrustment decision is a general statement that must be documented, awards a higher level of responsibility for future actions and should be recognizable for third parties. Both are important in EPA-‐based curricula. The ad-‐hoc decision experience of the particular supervisor may be documented in the trainee’s portfolio (was this a justified decision? If not, why not?). Summative decisions may be informed by multiple forma-‐tive decisions supplemented information gathered through other channels (multi-‐source feed-‐back, knowledge assessment, skills assessment). Summative entrustment decisions should be multi-‐source decisions and are more or less based on the summation of bits of information. 5.1 Factors determining entrustment decisions In the past years many authors have investigated the factors that influence the many entrust-‐ment decisions that supervisors make ad hoc regarding trainees in the clinical setting (60,95–97). There is consensus that four groups of factors come into play trainee features, supervisor features, the nature of the task and the circumstances (37), supplemented with trainee-‐


Deliverable D2.1

31

supervisor relationship (85) and patient or parental preference in pediatrics (98). Each of these groups has several variables that affect the outcome of the decision. Ad-‐hoc entrustment decisions are clearly based on many variables, and it is not useful to try to arrive at reliability. Systematization can be attempted to some extent by training clinicians to use a similar frame of reference, but rater differences may never be solved this way, and perhaps should not (99). Rather, ad-‐hoc entrustment decisions can be viewed as bound by a given con-‐text and by the nature of the task. That would sound as “I trust you to do this procedure with this patient, this afternoon, knowing that my colleague John is around who is familiar with the pa-‐tient and with the procedure. If you do well, I might ask you to do it tomorrow too, when John is not available. But let’s first evaluate this evening, and I’ll probe you with case-‐based what-if questions before deciding that you can be left alone”. That sounds like a complex entrustment decision, but it reflects the reality of de workplace and may in fact only be a quick thought that leads to sufficient swift trust for this case. For EPA-‐based curricula we propose to call these formative entrustment decisions, as they add to professional formation. The validity of summa-‐tive entrustment decisions may be supported by a series of such observations and try-‐outs. For the purpose of this Guide, we will not expand on the factors ‘supervisor’, ‘context’ and ‘EPA’, but focus on the features of the trainee. 5.2 Trainee features that allow supervisors to entrust them with a critical task Ten Cate et al reviewed the literature about entrustment decision-‐making and came up with a list of critical trainee features (94). Trusting a medical trainee does in some way resemble trust-‐ing a medical colleague. Choudry et al recently suggested that physicians weigh expertise, inter-‐action style with the patient and collegial interaction style when choosing a consultant for their patient (eg., Does this physician have enough expertise with the problem for which the patient requires consultation? Does the physician communicate well with patients and their families? Does the physician have genuine concern for the well-‐being of his/her patients? How well will the specialist communicate with the referring physician? Will this physician provide good conti-‐nuity of care and follow-‐up communication?) (100). They too align with the trainee features found in the literature. The 10 most important factors identified from the literature are the following (85,95–97,94,101,102). The first four were found by Kennedy et al (101), aligning well with the notions of trust-‐expert and philosopher O’Neill (103) and may be considered foundational. These quali-‐ties in a trainee enable supervisors to accept a vulnerability that comes along with trusting a trainee with a new critical activity.

Table 7 Qualities in trainees that enable trust

Foundational qualities, primarily based on Kennedy et al (101)

Competence and clini-cal reasoning

This pertains to the cognitive and physical skills needed to execute the EPA. Specific competencies may map to a competency frame-‐work, and, more generally, include knowledge, skills, and attitudes.

Conscientiousness and Conscientiousness and reliability reflect a thoroughness and con-‐


Deliverable D2.1

32

reliability

sistency in actions, e.g., when trainees do what they say they will do and show a thoroughness that is predictable across occasions.

Truthfulness or hones-ty

Truthfulness and honesty imply that trainees, if asked, tell what they observed, what they did, and why. It includes admitting what they should have done and did not.

Discernment of limita-tions and inclination to ask for help if truly needed

Crucial is a discernment of one’s own limitations and knowing when to refrain from procedures and ask for help. This ‘knowing’ is the cognitive component; willingness or propensity to ask for help is an attitudinal component that is just as important but that may not always align with the cognitive component. An adequate balance between proactive behavior and asking help when really needed is important.

Supplementary qualities summarized from the literature (85,95,97,102) Empathy, openness and receptiveness to-ward patients

Actively listening to patients and reacting verbally and nonverbally to the things the trainee hears in a way that encourages the sharing of information by the patients and that confirms involvement with the patient.

Skill in collegial and interprofessional communication and collaboration

Adequate communication about patients exemplifies a mastery of the situation necessary both for general supervision at levels 3 and 4 (“indirect supervision” and “unsupervised”) and for specific situa-‐tions such as patient handovers.

Self-confidence and feeling safe to act

Being self-‐confident and feeling safe to act are important to enable action, but overconfidence can be dangerous. An adequate balance is necessary.

Habits of ongoing self-evaluation, reflection, and development

A habit of self-‐evaluation, reflection and development are estab-‐lished qualities of well-‐functioning professionals. Seeking feedback to improve is part of that habit.

Sense of responsibility A trainee who is readily trusted is one who makes sure patients are cared for when he or she is gone, who picks up perceived lapses of care caused by others and accordingly initiates action, or who acts upon urgent needs of care when others are not available.

Adequately dealing with mistakes of self and others

As patient safety comes to the forefront of thinking about quality in health care, acknowledging errors and mistakes of oneself and oth-‐ers has become a crucial habit to acquire.

This clustering of factors is merely based on existing medical education literature. Other do-‐mains, such as organizational and occupation psychology (104) have yielded still other factors.


Deliverable D2.1

33

This is an important area of further empirical research that should inform us more in detail on the dominant factors that lead to trust in trainees. 5.3 Arriving at formative and summative entrustment decisions Each of the features of Table 7 and more may, consciously or unconsciously, weigh into the deci-‐sion to trust a trainee with care for patients at a particular moment. As these formative entrust-‐ment decisions are usually taken without much time to carefully deliberate, there is often little opportunity to take all, or even the most essential factors into account. Ad hoc entrustment is often based on gut feelings and limited information. This does not necessarily make such forma-‐tive decisions inaccurate. Human decision-‐making has been the object of numerous studies in psychology, economics and other domains. One conclusion is that humans often cannot take all relevant factors for rational decisions into account. In many cases, the capacity of human cogni-‐tion is too limited to weigh all relevant factors in making rational decisions (105). However, hu-‐mans can use rules-‐of-‐thumb or heuristics, defined by Gigerenzer as strategies that ignore part of the information, with the general goal of making decision more quickly, frugally and/or accurately than more complex methods, with remarkable success (106). Heuristics (“I saw she did this well before, so let her do it again” or “if my colleague Peter trusts her, I think I can too” or “the way he presents this case over the phone makes me really worried – I better come over right away”) do not necessarily lead to wrong decisions. The ‘more-is-less’ proposition in heuristics theory postu-‐lates that comprehensiveness in weighing all potential or available factors relevant to a decision does not always serve its validity, and that indeed ignoring information can increase validity (107). It is known from the medical education literature that global ratings about trainees are often more reliable than composite scores based on more complete checklist ratings of relevant ob-‐servations (108). If there is sufficient time, feelings of worry may be checked against the list above before the decision is made. The conclusion is that not everything that grounds an entrustment decision can be captured in numbers, scales or even words. We sometimes ‘feel’ we can trust a trainee or not. For formative decisions, this presumptive trust, based on prior credentials of the trainee, combined with initial trust derived from a short observation, may be sufficient to make formative entrustment deci-‐sions. For summative entrustment decision-‐making, grounded trust is necessary, based on a more or less systematic collection of information (94). An EPA-‐based competency curriculum requires the gradual building of acknowledged medical competence. In postgraduate medical training, the collective decisions of entrustment for unsupervised practice (level 4) of an individual, pref-‐erably documented in an electronic portfolio, constitute the foundation of the license to practice. These decisions are the agreed upon formal moments of certification. In undergraduate medical education the more logical level to reach is 3 (‘indirect supervision’). Summative entrustment decisions must be as valid and predictive as possible and must be taken by multiple professionals. As was illustrated above however, the number of potential variables that could affect entrustment decisions is large. Summative entrustment decisions, leading to permission to act unsupervised from a specified moment on, should be grounded in more sys-‐tematic exploration and weighing of these qualities if the trainee.


Deliverable D2.1

34

Program directors, teams or clinical competence committees regularly feel they lack information about individual trainees and would wish to be provided with sufficient, well-‐structured infor-‐mation that can efficiently but validly be evaluated to arrive at decisions. (109) Various sources of information in the workplace can inform entrustment decisions. Table 7 lists suggested sources of information that may inform such group decisions. The sources were pre-‐dominantly derived and reworked from ten Cate et al (submitted) and related to the ten factors listed above. This listing of information sources is preliminary and requires a more systematic investigation in the near future.


Deliverable D2.1

35

Table 8 Suggested sources of information to support factors weighing in entrust-ment decisions

Factors that affect entrustment decisions Potential information sources

Com

pete

nce

and

clin

ical

reas

onin

g

Con

scie

ntio

usne

ss a

nd re

liabi

lity

Trut

hful

ness

and

hon

esty

Dis

cern

men

t of l

imita

tions

and

incl

inat

ion

to a

sk fo

r hel

p

Empa

thy,

ope

nnes

s an

d re

cept

iven

ess

tow

ard

patie

nts

Col

legi

al a

nd in

terp

rofe

ssio

nal c

omm

unic

atio

n an

d co

llabo

ratio

n

Self-

conf

iden

ce a

nd fe

elin

g of

saf

ety

Hab

its o

f ong

oing

sel

f-eva

luat

ion,

refle

ctio

n, a

nd d

evel

opm

ent

Sens

e of

resp

onsi

bilit

y

Know

ing

how

to d

eal w

ith m

ista

kes

of o

nese

lf an

d ot

hers

• knowledge exams and skills exams X

• direct observations by supervisors, related to specific EPAs X X

• narrative observation-based feedback from patients and peers (e.g., MSF) X X X X X X X X

• audit of practice, incl. patient handovers and electronic medical record X X

• observing trainee teaching techniques (including one-minute preceptor) X X

• prior credentials and reputation reported by trusted colleagues X X

• sampled checks on accuracy of information reported X X

• patient presentations with cross-checks at morning rounds and handoffs X X

• review of events during night shifts X X X

• post hoc case-based discussions, including “what if” scenarios X X X

• guided self-reflection exercises and self-report (e.g. in a portfolio) X X X X

• significant event audit, root cause analysis and gap analysis X X X X

• multi-source feedback on interprofessional skills X

• self-initiated clinical or research projects X X

• signs of preparedness, initiative, and follow-through despite sacrifices X

• assigning a deliberate patient safety task that can be evaluated X

The list of potential information sources for inform summative entrustment decisions is based on collective expert knowledge gathered through discussion (94). While a useful overview of our current knowledge, there is a need for more systematic literature investigation one the suggest-‐


Deliverable D2.1

36

ed relationship with trustworthiness factors and empirical evidence to support to validity to these sources for the intended use and a need for translation into practical tools and instrument to operationalize these. It should be recognized that entrustment decisions resemble a summative assessment of the trainee, but one with direct consequences for the scope of responsibility of the trainee. It is a formal statement of trust. Trust can be viewed as a calculated risk that future adverse event are manageable (94,110). The analogy of the driver can illustrate this. A driver’s license can be viewed as a statement of trust that no accidents will happen. A license received in a rural town permits one to drive in a busy city in a different country with different traffic rules. In other words, the trustor makes the inference that the driver will do no harm in circumstances far from what has been observed during lessons and in a driving test. This also holds for entrustment decisions in health care education. The challenge is to collect sufficient information to make this inference reasonably grounded. 5.4 Instruments to assess trainees based on EPAs Collecting valid information to evaluate trainees on their readiness to advance to a next level of responsibility or autonomy requires the systematic use of instruments and methods. While there has recently appeared abundant literature presenting new instrument to evaluate trainees in workplace environments ((111) Wisman-‐Zwarter et al, in preparation) with many creative acronyms, they can be categorized within a limited number of approaches (Table 9) . Practice related instruments focus on trainee behavior or achievement during practice, while practice unrelated instruments focus on evaluation knowledge or skill not directly related to behavior or achievement in the workplace, although immediately relevant for it. The literature often distin-‐guishes between formative and summative instruments. We believe this distinction is less use-‐ful, as similar instruments can be used for summative and formative, i.e. leading to significant progress decisions or not. We believe that multiple formative evaluations can and should inform summative entrustment decisions, while each of these separately should clearly have a more formative nature. See our discussion in the previous section. In Table 9, we have summarized the instrument categories in knowledge, skills, attitude and products, acknowledging that instruments may or may now combine these (e.g. sampled prac-‐tice observations) and acknowledging that within these instruments competencies and compe-‐tency domains that have been deemed relevant for an EPA should be represented, according to the matrix mapping approach we have proposed.


Deliverable D2.1

37

Table 9 Instruments to be used in EPA-based assessment Practice unrelated instruments Practice related instruments

Knowledge • Written or electronic knowledge test-‐

ing

• Short practice observation

• Case-‐based discussion

Skill • Simulation testing • Short practice observation

Attitude • Simulation testing • Long practice observation

Products • Product evaluation • Product evaluation

Written or electronic knowledge testing Knowledge testing does not need further explanation Simulation testing Skills testing in a simulated and standardized environment can be applied in an objective struc-‐tured clinical examination (OSCE), with low or high fidelity simulation equipment, or with standardized patients. While most of attitude and professional behavior is best shown in prac-‐tice, some aspects can be evaluated in testing with standardized patients. Case-based discussion A case-‐based discussion (CBD) is a short oral discussion with the trainee on knowledge and clin-‐ical reasoning (5-‐20 minutes) after a clinical encounter, prompted with two types of questions (1) How was your reasoning during the encounter and (2) what would you have done differently if this patient had shown X, Y or Z? This second question is particularly relevant for entrustment decisions, as it captures situations that are less common. If the encounter is just prior to the CBD, its may coincide with the feedback of a practice observation. A CBD about patient record data that a trained has entered may be called chart-‐stimulated recall. Short practice observation A practice observation is usually a short observation (usually 5 to 15 minutes) of work in prac-‐tice (a patient consultation with history or physical examination, execution of a procedure, a case presentation, a lesson for allied health trainees), with feedback debriefing afterwards, meant to be conducted multiple times (112,113). One or more observers rate the observed indi-‐vidual on a scale related to supervision level (i.e. readiness for direct, indirect or distant supervi-‐sion) and provide feedback with specific ‘tops’ and ‘tips’ related to related to the competency domains relevant for this EPA. EPAs should have EPA-‐specific practice observation forms, pref-‐erably on mobile devices. The most well known example is the Mini Clinical Evaluation Exercise (Mini-‐CEX, (114)) but there are many examples of short observations of clinical encounters, pro-‐cedures and activities in practice (111). Practice observations are samples of work, preferably not solicited or planned, can be rated via live presence of an observer or via video recording (e.g.


Deliverable D2.1

38

in a consultation room), observed real-‐time elsewhere or rated post hoc. Short practice observa-‐tion forms may tailored to specific EPAs. Long practice observation Long practice observation pertains to evaluation of observed behavior over a period of a week to a month or longer, focused on intrinsic competency domains rather than medical expertise. Key to long practice observation is that it is not done in retrospect (as in many end-‐of rotation as-‐sessments) but prospectively. Observers are asked in advance to observe over a specified period to time which allows them to be alert when observing the trainee. This can be as short as an on-‐call weekend service, but is usually longer. A strong example is multi-‐source feedback, also named 360 degree evaluation. Multi-‐source feedback (MSF) information, collected among col-‐leagues (staff, peers and junior trainees), among other health professionals such as nursing, and among patients. MSF is particularly useful to evaluate attitudinal components of professional behavior, communication, collaboration and aspects of trustworthiness. Patients in MSF usually evaluate directly after an encounter, which in fact is a short observation, but some patient with multiple encounters with the same doctor may add to long practice observations. Product evaluation Products that may be evaluated are discharge summaries and letters, medication prescriptions and other entries into the electronic health record, slide presentations, case-‐reports. Products may be used to evaluate patient-‐related outcomes of training. Practice related products pertain to actual patients or happenings, practice unrelated products follow either from assignments for the purpose of assessment, or are generalized products such as clinical protocols, critically ap-‐praised topics extractions from the literature. 5.5 Proposed general reporting format for observed performance Next to knowledge and skills assessment (written, oral and simulation-‐based) much of this in-‐formation is to be collected through observation of work. As ultimately, summative decisions must be made about a permitted level of supervision, such phrasing (“can we leave this trainee to work unsupervised?”) is generally also usable for all individual encounters (115). Warm has used this EPA supervision levels approach for the collection of large amounts of data on individ-‐ual residents in internal medicine (59,116) to report on milestones toward the attainment of competence. The simple question that any observer may be asked, next to specifications about an EPA, is “Based on my observation today, I suggest for this EPA this trainee may be ready after the next upcoming review to (1) only observe, (2) act under direct supervision, (3) act under indirect supervision, (4) act with post-‐hoc report only, (5) supervise juniors. The response for each of these levels may be framed as, for instance, No, Hesitate, Yes. This should be an answerable question if used formatively. For formative entrustment decisions that are evaluated and re-‐ported this way, supervisors need to understand that their recommendation is not a formalized decision, but a suggested advice to the responsible program director, examiner or competency-‐committee to be weighed together with many other similar evaluations and additional different sources of information. Depending on the EPA, other sources could include knowledge and skill


Deliverable D2.1

39

test results, oral case-‐based discussions and other strategies (see Table 7). The report could in some cases also be provided by nursing or other non-‐physician co-‐workers. 5.6 Self-entrustment Summative entrustment decisions are significant and usually meant to bring the trainee to a higher level of responsibility and a step toward full certification for the profession. The program director or a team makes these decisions based on information such as described above. In that process one element has not been touched upon. That is the feeling of readiness by the trainee himself or herself. From reported studies we know that in general, medical trainees estimate their abilities and readiness to act unsupervised somewhat higher than what clinicians project as reasonable for that particular stage of training (95,117). In other cases trainees may feel inse-‐cure about tasks they are asked to execute. A summative entrustment decision for an EPA should generally not be made when the trainee feels significant insecurity to act unsupervised (or with only indirect supervision in undergraduate medical education). One way to realize this is to have the trainee opt for such a decision only when he or she feels ready for it. Some trainees may tend to postpone such requests, but adequate mentoring and feedback should help trainees choose strategies to progress at an adequate speed. 5.7 Expiry and reconfirmation of summative entrustment decisions Summative entrustment decisions, sometimes called statements of awarded responsibility (STARs) for an EPA at level 4 should be regarded similar to certification or licensing to practice for that particular unit of professional practice. The summative entrustment decisions for the EPAs of the profession together should establish a general certification to practice. The portfolio of STARS (trusted EPAs) should thus define the physician’s qualification. Two important limita-‐tions of this reasoning are important to note. One is the context-‐dependence of competence. Medical competence is predominantly general or canonic, in the sense that applicability should extend across different circumstances and condi-‐tions, but to some extent competence depends on the context (2,118,119). For that reason, trainees moving from one rotation or hospital to another may be briefly observed to reconfirm the validity of the entrustment decision for an EPA, depending on the risk level of the EPA. The other limitation is the fact that skills generally decrease when not practiced. The ability to execute an EPA is likely to reduce when practice does not occur, similar to the reduction in knowledge that is not applied (120). Entrustment decisions should therefore have an expiry date that invalidates the decision if no or too little practice has occurred. It is important that entrust-‐ment decisions are not considered as the conclusion of a training period, but as the beginning of a practice period. Expiry dates for EPAs after graduation are also suitable for recertification and maintenance of competence procedures. If certification for an EPA after graduation, as default, would expire after five years of inactivity and lead to a stricter level of supervision, the physician may choose to revalidate or restrict the scope of practice to a limited number of EPAs. This way, maintenance of competence regulations can be based on EPAs. Recertification of specific EPAs may become more meaningful than current procedures that focus on full recertification of a spe-‐cialty license. Expiry dates and reconfirmation appear to be a logical consequence of using EPAs. Of note however is that dates should relate to the nature of the EPA and the experience built


Deliverable D2.1

40

after the first entrustment decision, and may be much less than 5 years if no practice has taken place directly after the entrustment decision. 6. Technology to support feedback and entrustment decision-making In busy clinical environments, both trainees and supervisors may be supported by electronic means to optimize information about trainee progress. For trainees this feedback information should serve to inform next actions and next behavior (rehearse knowledge and skill, actively select next experiences) to proceed to readiness for a next entrustment decision about an EPA. For supervisors, the multitude of potential information bits about a trainee must be collected and aggregated to support entrustment decisions and inform supervisors in the work place. This is an ambitious enterprise that should be supported by electronic means. 6.1. Formative feedback and entrustment decisions In an era of rapid technological developments monitoring of trainees in workplaces to support their progress should use technology. In hospitals there is a continuous pressure on patient care that precludes many clinicians to spend much time on documentation of encounters with learn-‐ers. With the ubiquitous presence of mobile devices such as smartphones and tablets, every trainee and clinical educator can use these for the benefit of education and learning. Electronic portfolios are becoming common in clinical training (121–123) documentation of EPA-‐based progress monitoring should use both. Figure 7 illustrates how shot practice observations with feedback can be efficiently handled.

Figure 7 Possible sequence of events in reporting on formative entrustment

0. In a program that provides all trainees with a e-‐portfolio, for any given EPA, an electronic evaluation form is constructed, with scoring rubrics related to pertinent qualities and com-‐petencies for this EPA, derived from the EPAs-‐competencies matrix

1. A trainee or a supervisor requests or initiates a sampled practice observation procedure related to an EPA, usually planned on short notice

2. The observing supervisor sits with the trainee, during 5 to 15 minutes, at a patient encoun-‐ter, a clinical procedure, a case presentation or other event, related to this EPA

3. During the observation the supervisor uses a hand-‐held mobile device and logs in, online or via an app, to retrieve a observation form, tailored to the EPA and tailored to the trainee (both must be pre-‐programmed)

4. The supervisor fills out three related forms one global impression, formulated as “Based on my observation today, I suggest for this EPA this trainee may be ready after the next upcom-‐ing review to (1) only observe, (2) act under direct supervision, (3) act under indirect su-‐


Deliverable D2.1

41

pervision, (4) act with post-‐hoc report only, (5) supervise juniors. Each of these shows a scaling in three levels No, Hesitate,

5. Yes. Next, the supervisor is asked to state which relevant CanMEDS domain could be strengthened. Finally feedback is provided related to a specific set of knowledge, skills or at-‐titude related to this EPA that prompts for feedback. Clicking each of these opens voice re-‐cording, to record specific feedback tops and tips in a short verbal debriefing with the train-‐ee.

6. All forms and recordings are sent to the trainee’s e-‐portfolio. The trainee translates the rec-‐orded feedback in written sentences to be stored with the related EPA, competency domain, and date.

7. The supervisor approves of the student’s representation of the feedback

8. On an aggregate level, a program director or clinical competency committee or staff may decide on summative entrustment based on many or at least multiple observations, sup-‐plemented with other information.

The mobile technology should enable efficient feedback and support of entrustment decision-‐making. Figure 8 shows an impression of how the procedure could look on a mobile device. Figure 8 Representation of a potential EPA evaluation on a mobile device


Deliverable D2.1

42

A global evaluation shown in the first frame of this figure is expanded, based on the EPAs-‐Competencies matrix as elaborated early in this guide (see Table 3). That is, the suggested read-‐iness for a supervision level can be backed with information about the competencies that have been identified as critical for a particular EPA in the next frame. Depending on the preferences of the observer, feedback can be provided either in writing or orally. The required dialogue of a short practice observation can be recorded to maximize efficiency. A similar procedure can be applied for case-‐based discussions and case presentations, while the forms and frames used may be somewhat different. 6.2 Summative entrustment decisions Grounded entrustment decisions leading to progressive independence must be based on as many observations as reasonably feasible in an authentic context. This approach has been called a programmatic approach to assessment (124), in contrast with assessment on single moments of examination. Summative entrustment decisions are clearly summative, but the route to these decisions is less summative. All observations that inform program directors and competency committees to assist grounding summative entrustment decisions also serve as formative feed-‐back to trainees, to inform them about their progress. It concords with what Stiggins and Schuwirth & Van der Vleuten have called ‘assessment for learning’ (125,126). Warm has shown the feasibility of reporting large amounts of data collected in the clinical workplace using this data about internal medicine resident progression on ‘observable practice activities’ (small units nesting into EPAs) and milestones to establish personal learning curves serving both feedback and entrustment decisions making (59,116). 6.3. E-portfolio and learning analytics Collecting information by electronic means requires its storage in a personal electronic portfolio of the trainee. The portfolio repository should serve to inform trainees with aggregated, up-‐to-‐date information about their progress and inform program directors with specific information to support summative entrustment decisions. Data collected about individuals may be related to aggregated data across populations of current or past trainees. Clearly this involves large amounts of data. Analyzing big data for educational purposes has been called learning analytics. Learning analytics is the measurement, collection, analysis and reporting of data about trainees and their contexts, for the purpose of understanding and optimizing learning and the utilizing of environments in which it occurs (Society for Learning Analytics Research – www.solaresearch.org). Greller & Drachsler have identified five dimensions of learning analytics that may be operationalized for EPA-‐based competency curricula (127).

Table 10 Learning analytics and EPAs

Dimensions of learning analytics (cf. Greller & Drachsler 2011)

Values as suggested for EPA-based competency curricula

Stakeholders • Trainees (students/residents) • Program directors / supervisors / examiners / clinical competency committees


Deliverable D2.1

43

Purpose • Feedback to trainees related to EPAs • Support for summative entrustment decisions for EPAs

Data • All relevant individual data on observed performance, supplemented with data on tests • Aggregated data across (sets of) individuals

Instruments • Mobile devices, learners supplied information, multi-source feedback information • Future options patient provided health care outcome data, electronic medical record

data Output • Visualized graphical representations of progress of individuals compared with individual

objectives, development plan and past progress and with relevant groups External limitations • Storage of data and access to data must be limited according to ethical rules Internal limitations • Both trainees and teaching staff must be trained to understand and interpret data that

are provided in the visualized output

The following e-‐portfolio functions should be provided 1. Easy input by observers through mobile devices 2. Easy input by learners through mobile devices or computers 3. Easy input by educational administration about formal progress results (tests, scheduling of

rotations, assigned mentor) 4. Clear visualization of tailored output for distinct groups learners, program directors, men-‐

tors, and external groups that require information about certified EPAs such as hospital staff with adequate access permission conditions

5. Upload facility for various documents Graphical representations Figures 9 and 10 show two images what part of the portfolio of an imaginary medical student Peter Berk may look like if accessed by anyone who has permission to view his current creden-‐tials for acting in health care, and a specification of core EPA 1. Peter Berk’s portfolio may have a parts that are publically accessible, parts that are accessible by designated individuals, such as a program director or mentor, and parts that may only be acces-‐sible for Peter himself. The images of figures 9 and 10 may be accessible by anyone who needs to be informed about Peter’s credentials and permissions, such as clinical and nursing staff of a new rotation. It resembles digital badging as proposed by Mehta et al (53).


Deliverable D2.1

44

Figure 9 Representation of overview of EPAs of imaginary medical student Peter Berk

Figure 10 Representation of the history and current status of core EPA 1 of Peter Berk


Deliverable D2.1

45

6. Discussion This paper was written as a multi-‐purpose guide for competency-‐based curriculum development with EPAs. The guide should assist educators interested in building such curriculum and should serve developers of electronic solutions to support workplace based feedback and entrustment decision-‐making. It has pulled together the literature and expanded the knowledge about cur-‐riculum development using entrustable professional activities. Acknowledging that the EPA con-‐cept is less than a decade old – its first publication appeared in December 2005 (1) and that it is only now beginning to be used as a framework for workplace curriculum development, we ex-‐pect that many aspects will continue to be clarified, added or refined over de coming years, based on research and examples from practice. Entrustable Professional Activities are being welcomed by many programs, to reshape compe-‐tency-‐based postgraduate medical education and increasingly also undergraduate medical edu-‐cation and some other programs (see Table 1). The potential of EPAs is however broader. Defin-‐ing professional competence in term of EPAs opens the possibility to cross traditional bounda-‐ries. Boundaries between phases in the medical education continuum may be crossed when medical students have the opportunity to start practicing EPAs that are usually provided to jun-‐ior residents. Continuum projects such as that being explored in pediatrics (128) as well as ‘ded-‐icated transitional year’ experiments between undergraduate and postgraduate education in the Netherlands show serious attempts to do this. Boundaries between postgraduate training and


Deliverable D2.1

46

continuing professional development may also be crossed using EPAs (129). The personal com-‐petence of a physician may start to be defined by a dynamic portfolio of EPAs that may either be strengthened or expire, and be supplemented with newer ones. Boundaries between specialties may be crossed with EPAs pertaining to activities that are located on such boundaries (think of surgeons or rheumatologists interpreting radiological images or colonoscopies, family physi-‐cians conducting small surgical procedures, pathologists taking biopsies). Finally, even bounda-‐ries between professions may be crossed for very specific EPAs, such as between physicians and physician assistants (49). EPAs-‐based portfolios may grow into dynamic repositories that truly represent the actual competencies physicians have maintained as their current certified EPAs at any moment, leading from competency-‐based medical education to competency-‐based medical practice. The guide has limitations. As recommendations to curriculum development with EPAs can yet hardly or not be based on experience with fully developed EPA-‐based workplace curricula, many were derived from various literature sources and deliberations among expert educators. In the coming years research will be necessary to build evidence to further ground the proposed ap-‐proaches. For instance, tools to collect information to support entrustment decisions with tech-‐nology and learning analytics may be expected to substantially facilitate the richness of feedback and the quality of such decisions, but also the more conceptual and theory-‐based understanding of entrustment decision-‐making will serve from further research. Another necessary domain of progress will be faculty development and the valuing the efforts to supervise trainees. As ade-‐quate supervision is key in EPA-‐based curricula, the effort to coach, provide feedback and con-‐tribute to entrustment decisions should be valued and rewarded. One possible way to do this is monitor the efforts of clinical faculty related to electronically provided feedback and reward this with continuing professional development points. The guide is one milestone in a broader competency-‐based education project, both at the curric-‐ulum level and in information technology that will undoubtedly show further progress in the coming years. Entrustable professional activities are gradually becoming part of the language in competency-‐based medical education development in several countries, and we hope the many groups that are active in these developments will benefit from the thoughts shared in this guide. Acknowledgements The authors wish to express their gratitude to the following persons for commenting on previ-‐ous version of this paper. Christy Boscardin, Sjoukje van den Broek, Anouk van der Gijp, Gersten Jonker, Mira Mandoki, Hanneke Mulder, Sophie Querido and Nienke Wisman-‐Zwarter. This publication is part of a multi-‐institutional, multi-‐country, and multi-‐professional project (WATCHME) which has received funding from the European Union’s Seventh Framework Pro-‐gramme for research, technological development and demonstration, under grant agreement 619349.

References


Deliverable D2.1

47

1. Ten Cate O. Entrustability of professional activities and competency-‐based training. Med Educ. 2005 Dec;39(12):1176–7.

2. Ten Cate O, Snell L, Carraccio C. Medical competence: the interplay between individual ability and the health care environment. Med Teach. 2010 Jan;32(8):669–75.

3. Ten Cate O. Nuts and Bolts of Entrustable professional activities. J Grad Med Educ. 2013 Sep 21;5(1):157–8.

4. Ten Cate O. AM Last Page�: What Entrustable Professional Activities Add to a Competency-‐Based Curriculum. Acad Med. 2014;89(4):691.

5. Albanese M. Challenges in using rater judgements in medical education. J Eval Clin Pract. 2000 Aug;6(3):305–19.

6. Billett S. Mimetic learning at work: Learning in the circumstances of practice. 1st ed. Dordrecht: Springer; 2014.

7. Billett S. Learning in the Workplace. 1st ed. Crows Nest, NSW, Australia: Allen & Unwin; 2001.

8. Teunissen PW. Unravelling learning by doing. Doctoral Dissertation. Amsterdam: VU University; 2009.

9. Dornan T, Tan N, Boshuizen H, Gick R, Isba R, Mann K, et al. How and what do medical students learn in clerkships? Experience based learning (ExBL). Adv Health Sci Educ Theory Pract. 2014 Mar 18;

10. Kolb DA. Experiential learning. Experience as the source of learning and development. Englewood Cliffs, New Jersey: Prentice-‐Hall, Inc.; 1984.

11. Collins A, Brown J, Newman S. Cognitive apprenticeship: teaching the craft of reading, writing and mathematics. In: Resnick L, editor. Knowing, Learning and Instruction Essays in the honor of Robert Glaser. Hillsdale, NJ: Lawrence Erlbaum Associates; 1989. p. 453–94.

12. Collins A. Cognitive apprenticeship. In: Sawyer R, editor. The Cambridge Handbook of the Learning Sciences. 1st ed. Cambringde: Cambridge University Press; 2006. p. 47–60.

13. Brown JS, Collins A, Duguid P. Situated Cognition and the Culture of Learning. Educ Res. 1989;18:32–42.

14. Lave J, Wenger E. Situated Learning. Legitimate Peripheral Participation. Edinburgh: Cambridge University Press; 1991.

15. Harden RM, Sowden S, Dunn WR. Educational strategies in curriculum development: the SPICES model. Med Teach. 1984;18:284–97.

16. Cooke M, Irby D, O’Brien BC. Educating Physicians -‐ A Call for Reform of Medical School and Residency. San Francisco: Jossey-‐Bass; 2010.

17. Koens F, Mann K V, Custers EJFM, Ten Cate OTJ. Analysing the concept of context in medical education. Med Educ. 2005 Dec;39(12):1243–9.

18. Dahle LO, Brynhildsen J, Behrbohm Fallsberg M, Rundquist I, Hammar M. Pros and cons of vertical integration between clinical medicine and basic science within a problem-‐based undergraduate medical curriculum: examples and experiences from Linköping, Sweden. Med Teach. 2002 May;24(3):280–5.

19. Koens F. Vertical Integration in Medical Education. Utrecht, the Netherlands: Doctoral Dissertation, Utrecht University; 2005.

20. General Medical Council, editor. Tomorrow ’ s Doctors -‐ Outcomes and standards for undergraduate medical education [Internet]. London: General Medical Council; 2009. Available from: http://www.gmc-‐uk.org/TomorrowsDoctors_2009.pdf_39260971.pdf

21. Anderson B. Learning objectives for medical student education-‐-‐guidelines for medical schools: report I of the Medical School Objectives Project. Acad Med. 1999 Jan;74(1):13–8.


Deliverable D2.1

48

22. Van Herwaarden CLA, Laan RFJM, Leunissen RRM. The 2009 Framework for Undergraduate Medical Education in the Netherlands [Internet]. Utrecht; 2009. Available from: http://www.vsnu.nl/Media-‐item/Raamplan-‐Artsopleiding-‐2009.htm

23. Epstein RM, Hundert EM. Defining and Assessing Professional Competence. JAMA J Am Med Assoc. 2002;287(2):226–35.

24. Ten Cate O. Competency-‐Based Medical Education. In: Cockerham WC, Dingwall R, Quah S, editors. The Wiley-‐Blackwell Encyclopedia of Health, Illness, Behavior, and Society. Hoboken, NJ: John Wiley & Sons; 2014. p. 1329–35.

25. Albanese MA, Mejicano G, Mullan P, Kokotailo P, Gruppen L. Defining characteristics of educational competencies. Med Educ. 2008 Mar;42(3):248–55.

26. Bok HGJ. Competency-‐based Veterinary Education: an integrative approach to learning and assessment in the clinical workplace. PhD thesis. Utrecht: Utrecht University; 2014.

27. Frank JR, Mungroo R, Ahmad Y, Wang M, De Rossi S, Horsley T, et al. Toward a definition of competency-‐based education in medicine: a systematic review of published definitions. Med Teach. 2010 Jan;32(8):631–7.

28. Orgill BD, Simpson D. Toward a glossary of competency-‐based medical education terms. J Grad Med Educ. 2014 Jun;6(2):203–6.

29. Kulik C-‐LC, Kulik JA, Bangert-‐Drowns RL. Effectiveness of mastery learning programs: a meta-‐analysis. Rev Educ Res. 1990;60(2):265–99.

30. McGaghie WC, Issenberg SB, Barsuk JH, Wayne DB. A critical review of simulation-‐based mastery learning with translational outcomes. Med Educ. 2014 Apr;48(4):375–85.

31. Ten Cate O. Last Page: What Entrustable Professional Activities Add to a Competency-‐Based Curriculum. Acad Med. 2014;89(4):691.

32. Grant J. The Incapacitating Effects of Competence: A Critique. Adv Health Sci Educ Theory Pract. 1999 Jan;4(3):271–7.

33. Talbot M. Monkey see, monkey do: a critique of the competency model in graduate medical education. Med Educ. 2004 Jun;38(6):587–92.

34. Brooks MA. Medical education and the tyranny of competency. Perspect Biol Med. 2009 Jan;52(1):90–102.

35. Malone K, Supri S. A critical time for medical education: the perils of competence-‐based reform of the curriculum. Adv Health Sci Educ Theory Pract. 2010 Sep 14;

36. Glass JM. Competency based training is a framework for incompetence. Br Med J. 2014;348:g2909.

37. Ten Cate O, Scheele F. Competency-‐Based Postgraduate Training: Can We Bridge the Gap between Theory and Clinical Practice�? Acad Med. 2007;82(6):542–7.

38. Scheele F, Caccia N, Van Luijk S, Van Loon K, De Rooyen C. BOEG-‐Better Education for Obstetrics and Gynaecology. A National Competency-‐Based Curriculum for Obstetrics & Gynaecology. Utrecht, the Netherlands; 2013 p. 1–61.

39. Gilhooly J, Schumacher DJ, West DC, Jones MD. The Promise and Challenge of Entrustable Professional Activities. Pediatrics. 2014 May 1;133(Supplement):S78–9.

40. Caverzagie KJ, Cooney TG, Hemmer P a, Berkowitz L. The Development of Entrustable Professional Activities for Internal Medicine Residency Training: A Report From the Education Redesign Committee of the Alliance for Academic Internal Medicine. Acad Med. 2014 Nov 18;XX(X):1–6.

41. Shaughnessy AF, Sparks J, Cohen-‐osher M, Goodell KH, Sawin GL, Gravel J. Entrustable Professional Activities in Family Medicine. J Grad Med Educ. 2013;5(1):112–8.

42. Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ. 2011 Jan;11:96.

43. Fessler HE, Addrizzo-‐Harris D, Beck JM, Buckley JD, Pastores SM, Piquette CA, et al. Entrustable professional activities and curricular milestones for fellowship training in


Deliverable D2.1

49

pulmonary and critical care medicine: report of a multisociety working group. Chest. 2014 Sep 1;146(3):813–34.

44. Frank JR. The CanMEDS 2005 physician competency framework: Better standards, better physicians, better care. Ottawa: Royal College of Physicians and Surgeons in Canada; 2005.

45. Bok HGJ, Jaarsma DADC, Teunissen PW, van der Vleuten CPM, van Beukelen P. Development and validation of a competency framework for veterinarians. J Vet Med Educ. 2011 Jan;38(3):262–9.

46. Bok HGJ, Teunissen PW, Boerboom TBB, Rhind S, Baillie S, Tegzes J, et al. International survey of veterinarians to assess the importance of competencies in professional practice and education. J Am Vet Med Assoc. 2014;245(8):906–13.

47. Jones MD, Rosenberg A, Gilhooly JT, Carraccio CL. Perspective: Competencies, outcomes, and controversy-‐-‐linking professional activities to competencies to improve resident education and practice. Acad Med. 2011 Feb;86(2):161–5.

48. Rose S, Fix OK, Shah BJ, Jones TN, Szyjkowski RD, Bosworth BP, et al. Entrustable professional activities for gastroenterology fellowship training. Gastrointest Endosc. 2014 Jul;80(1):16–27.

49. Mulder H, Ten Cate O, Daalder R, Berkvens J. Building a competency-‐based workplace curriculum around entrustable professional activities: The case of physician assistant training. Med Teach. 2010 Jan;32(10):e453–9.

50. Teman NR, Gauger PG, Mullan PB, Tarpley JL, Minter RM. Entrustment of general surgery residents in the operating room: factors contributing to provision of resident autonomy. J Am Coll Surg. American College of Surgeons; 2014 Oct;219(4):778–87.

51. Kennedy TJT, Regehr G, Baker GR, Lingard LA. Progressive independence in clinical training: a tradition worth defending? Acad Med. 2005 Oct;80(10 Suppl):S106–11.

52. Halpern SD, Detsky AS. Graded Autonomy in Medical Education — Managing Things That Go Bump in the Night. N Engl J Med. 2014;370(12):1086–9.

53. Mehta NB, Hull AL, Young JB, Stoller JK. Just imagine: new paradigms for medical education. Acad Med. 2013 Oct;88(10):1418–23.

54. Jonassen D, Tessmer M, Hannum W. Task analysis methods for instructional design. 1st ed. Mahwah, New Jersy: Lawrence Erlbaum Associates; 1999.

55. Dijkstra IS, Pols J, Remmelts P, Bakker B, Mooij JJ, Borleffs JCC, et al. What are we preparing them for? Development of an inventory of tasks for medical, surgical and supportive specialties. Med Teach. 2013 Apr;35(4):e1068–77.

56. Raymond MR, Mee J, King A, Haist SA, Winward ML. What new residents do during their initial months of training. Acad Med. 2011 Oct;86(10 Suppl):S59–62.

57. Englander R, Flynn T, Call S, Carraccio C, Cleary L, Fulton T, et al. Core Entrustable Professional Activities for Entering Residency Curriculum Developers ’ Guide [Internet]. Washington DC; 2014 p. 1–114. Available from: www.aamc.org

58. Leipzig RM, Sauvigné K, Granville LJ, Harper GM, Kirk LM, Levine S a, et al. What Is a Geriatrician? American Geriatrics Society and Association of Directors of Geriatric Academic Programs End-‐of-‐Training Entrustable Professional Activities for Geriatric Medicine. J Am Geriatr Soc. 2014 May;62(5):924–9.

59. Warm EJ, Mathis BR, Held JD, Pai S, Tolentino J, Ashbrook L, et al. Entrustment and mapping of observable practice activities for resident assessment. J Gen Intern Med. 2014 Aug;29(8):1177–82.

60. Dijksterhuis MGK, Voorhuis M, Teunissen PW, Schuwirth LWT, ten Cate OTJ, Braat DDM, et al. Assessment of competence and progressive independence in postgraduate clinical training. Med Educ. 2009 Dec;43(12):1156–65.


Deliverable D2.1

50

61. Kashner TM, Byrne JM, Henley SS, Golden RM, Aron DC, Cannon GW, et al. Measuring progressive independence with the resident supervision index: theoretical approach. J Grad Med Educ. 2010 Mar;2(1):8–16.

62. Ten Cate O, Snell L, Mann K, Vermunt J. Orienting teaching toward the learning process. Acad Med. 2004 Mar;79(3):219–28.

63. Ten Cate TJ, Kusurkar RA, Williams GC. How self-‐determination theory can assist our understanding of the teaching and learning processes in medical education. AMEE guide No. 59. Med Teach. 2011 Jan;33(12):961–73.

64. Karen E. Hauer KE, Jeffrey Kohlwes J, Patricia Cornett P, Hollander H, ten Cate O, Ranji SR, et al. Identifying entrustable professional activities in internal medicine training. J Grad Med Educ. 2013;5(4):54–9.

65. Chang A, Bowen JL, Buranosky RA, Frankel RM, Ghosh N, Rosenblum MJ, et al. Transforming primary care training-‐-‐patient-‐centered medical home entrustable professional activities for internal medicine residents. J Gen Intern Med. 2013 Jun;28(6):801–9.

66. O’Keeffe M. Clinical competence in developmental-‐behavioural paediatrics: Raising the bar. J Paediatr Child Health. 2013 May 28;

67. Fessler HE, Addrizzo-‐Harris D, Beck JM, Buckley JD, Pastores SM, Piquette C a, et al. Entrustable professional activities and curricular milestones for fellowship training in pulmonary and critical care medicine: executive summary from the multi-‐society working group. Crit Care Med. 2014 Oct;42(10):2290–1.

68. Ten Cate O. Trusting Graduates to Enter Residency: What Does It Take? J Grad Med Educ. 2014;6(March):7–10.

69. AERA/APA/NCME. Standards for Educational and Psychological Testing. Plake B, Wise L, and others, editors. Washington DC: American Educational Research Association; 2014.

70. Spenkelink-‐Schut G, ten Cate TJ, Kort HSM. Toepassing van het concept EPA als verbinding tussen professionele activiteiten en CanMEDS competentie-‐gebieden: pilotstudie Physician Assistant Urologie (Applying EPAs to link CanMEDS competencies for the physician assistant workforce in urology). Tijdschr voor Med Ond (Dutch J Med Educ ). 2008 Oct;27(5):230–8.

71. Touchie C, De Champlain A, Pugh D, Downing S, Bordage G. Supervising incoming first-‐year residents: faculty expectations versus residents’ experiences. Med Educ. 2014 Sep;48(9):921–9.

72. Hauer KE, Soni K, Cornett P, Kohlwes J, Hollander H, Ranji SR, et al. Developing Entrustable Professional Activities as the Basis for Assessment of Competence in an Internal Medicine Residency: A Feasibility Study. J Gen Intern Med. 2013 Apr 18;(Im).

73. Jones J, Hunter D. Consensus methods for medical and health services research. Br Med J. 1995;311(August):376–80.

74. Westerveld H, Bos-‐Onderstal L, ter Braak E, Hendrix H, Lagaaij M, Voorn T, et al. Aansluiting van de studie geneeskunde op vervolg-‐opleidingen: een interviewstudie onder arts-‐assistenten en opleiders van medische specialismen (Linking medical school and residency: an interview study among residents and residency program directors). Tijdschr voor Med Ond (Dutch J Med Educ ). 2004;23(1):15–22.

75. Merrill DM. A Task-‐Centered Instructional Strategy. J Res Technol Educ. 2007 Sep;40(1):5–22.

76. Vygotsky LS. Mind in society. The development of higher psychological processes. Cambridge, MA: Harvard University Press; 1978.

77. Whalen T, Wendel G. Chaper 6 New Supervision Standards: discussion and justification. The ACGME 2011 Duty Hour Standards -‐ Enhancing Quality of Care, Supervision, and Resident Professional Development [Internet]. Chicago, IL: Accreditation Council for


Deliverable D2.1

51

Graduate Medical Education; 2011. p. 39–45. Available from: http://www.acgme.org/acgmeweb/Portals/0/PDFs/jgme-‐11-‐00-‐39-‐45[1].pdf

78. Chen H, Van den Broek W, ten Cate OTJ. Entrustable professional activities in undergraduate medical education. Acad Med. :in – press.

79. Chen HC, van den Broek WES, ten Cate O. The Case for Use of Entrustable Professional Activities in Undergraduate Medical Education. Acad Med [Internet]. 2014 Dec 2 [cited 2014 Dec 4];XX(X):1–6. Available from: http://www.ncbi.nlm.nih.gov/pubmed/25470310

80. Cohen ER, Barsuk JH, Moazed F, Caprio T, Didwania A, McGaghie WC, et al. Making July safer: simulation-‐based mastery learning during intern boot camp. Acad Med. 2013 Feb;88(2):233–9.

81. Hirsh D, Gaufberg E, Ogur B, Cohen P, Krupat E, Cox M, et al. Educational outcomes of the Harvard Medical School-‐Cambridge integrated clerkship: a way forward for medical education. Acad Med. 2012 May;87(5):643–50.

82. Hirsh D, Walters L, Poncelet AN. Better learning, better doctors, better delivery system: Possibilities from a case study of longitudinal integrated clerkships. Med Teach. 2012 Jan;34(7):548–54.

83. Hirsh D, Ogur B, Thibault GE, Cox M. “Continuity” as an organizing principle for clinical education reform. N Engl J Med. 2007 Feb 22;356(8):858–66.

84. Hirsh DA, Holmboe ES, Ten Cate O. Time to Trust: Longitudinal Integrated Clerkships and Entrustable Professional Activities. Acad Med. 2013 Dec 19;89(2):201–4.

85. Hauer KE, Ten Cate O, Boscardin C, Irby DM, Iobst W, O’Sullivan PS. Understanding trust as an essential element of trainee supervision and learning in the workplace. Adv Health Sci Educ Theory Pract. 2014 Aug;19(3):435–56.

86. Carraccio C, Wolfsthal SD, Englander R, Ferentz K, Martin C. Shifting paradigms: from Flexner to competencies. Acad Med. 2002 May;77(5):361–7.

87. Corona-‐Committee. Draft CORONA Curriculum for the Merged Radiology and Nuclear Medicine Residency Programs [in Dutch]. Utrecht, the Netherlands; 2014 p. 1–59.

88. Kogan JR, Conforti LN, Iobst WF, Holmboe ES. Reconceptualizing Variable Rater Assessments as Both an Educational and Clinical Care Problem. Acad Med. 2014 Mar 24;89(5):1–7.

89. Lurie SJ, Mooney CJ, Lyness JM. Commentary: pitfalls in assessment of competency-‐based educational objectives. Acad Med. 2011 Apr;86(4):412–4.

90. Lurie SJ, Mooney CJ, Lyness JM. Measurement of the general competencies of the accreditation council for graduate medical education: a systematic review. Acad Med. 2009 Mar;84(3):301–9.

91. Govaerts MJB, van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM. Broadening perspectives on clinical performance assessment: rethinking the nature of in-‐training assessment. Adv Health Sci Educ Theory Pract. 2007 May;12(2):239–60.

92. Weller JM, Misur M, Nicolson S, Morris J, Ure S, Crossley J, et al. Can I leave the theatre? A key to more reliable workplace-‐based assessment. Br J Anaesth. 2014 Mar 17;1–9.

93. Ten Cate O. Trust, competence, and the supervisor’s role in postgraduate training. BMJ. 2006 Oct 7;333(7571):748–51.

94. Ten Cate O, Hart D, Ankel F, Busari J, Englander R, Glasgow N, et al. Entrustment decision-‐making in clinical training. submitted. 2015;-‐(-‐):-‐.

95. Sterkenburg A, Barach P, Kalkman C, Gielen M, ten Cate O. When do supervising physicians decide to entrust residents with unsupervised tasks? Acad Med. 2010 Sep;85(9):1408–17.


Deliverable D2.1

52

96. Choo KJ, Arora VM, Barach P, Johnson JK, Farnan JM. How do supervising physicians decide to entrust residents with unsupervised tasks? A qualitative analysis. J Hosp Med. 2014 Jan 20;9(3):169–75.

97. Wijnen-‐Meijer M, van der Schaaf M, Nillesen K, Harendza S, Ten Cate O. Essential facets of competence that enable trust in medical graduates: a ranking study among physician educators in two countries. Perspect Med Educ. 2013 Nov;2(5-‐6):290–7.

98. Tiyyagura G, Balmer D, Chaudoin L, Kessler D, Khanna K, Srivastava G, et al. The Greater Good: How Supervising Physicians Make Entrustment Decisions in the Pediatric Emergency Department. Acad Pediatr. Elsevier Ltd; 2014 Nov;14(6):597–602.

99. Gingerich A, Kogan J, Yeates P, Govaerts M, Holmboe E. Seeing the “black box” differently: assessor cognition from three research perspectives. Med Educ. 2014 Nov;48(11):1055–68.

100. Choudhry NK, Liao JM, Detsky AS. Selecting a Specialist -‐ Adding Evidence to the Clinical Practice of Making Referrals. JAMA J Am Med Assoc. 2014;312(18):1861–2.

101. Kennedy TJT, Regehr G, Baker GR, Lingard L. Point-‐of-‐care assessment of medical trainee competence for independent clinical work. Acad Med. 2008 Oct;83(10 Suppl):S89–92.

102. Wijnen-‐Meijer M, Van der Schaaf M, Nillesen K, Harendza S, ten Cate O. Essential facets of competence that enable trust in graduates: A Delphi study among physician educators in the Netherlands. J Grad Med Educ. :in press.

103. O’Neill O. A Question of Trust. Cambridge UK: Cambridge University Press; 2002. 104. Mayer RC, Davis JH, Schoorman FD. An integrative model of organizational trust. Acad

Manag Rev. 1995;20(3):709–34. 105. Young JQ, Van Merrienboer J, Durning S, Ten Cate O. Cognitive Load Theory: Implications

for medical education: AMEE Guide No. 86. Med Teach. 2014 Mar 4;(86):1–14. 106. Gigerenzer G, Gaissmaier W. Heuristic decision making. Annu Rev Psychol. 2011

Jan;62:451–82. 107. Goldstein DG, Gigerenzer G. Models of ecological rationality: The recognition heuristic.

Psychol Rev. 2002;109(1):75–90. 108. Regehr G, Macray H, Reznick R, Szalay D. Comparing the psychometric properties of

checklists and global ratings scales for assessment performance on an OSCE-‐format Examination. Acad Med. 1998;73:993–7.

109. Hauer K, Chesluk B, Iobst W, Holmboe E, Baron R, Boscardin C, et al. Resident Competence Review in Graduate Medical Education: a Qualitative Study. Acad Med. -‐(-‐):-‐.

110. Cianciolo AT, Kegg JA. Behavioral Specification of the Entrustment Process. J Grad Med Educ. 2013;5(1):10–2.

111. Kogan JR, Holmboe ES, Hauer KE. Tools for Direct Observation and Assessment A Systematic Review. JAMA J Am Med Assoc. 2009;302(12):1316–26.

112. Norcini J, Burch V. Workplace-‐based assessment as an educational tool: AMEE Guide No. 31. Med Teach. 2007 Nov;29(9):855–71.

113. Ten Cate T., Fluit CRMG. Guidelined for Short Practice Observations [Richtlijn korte praktijkbeoordeling] [Dutch]. Netherlands J Med Educ [Internet]. 2011 Feb 27;29(5):s105–34. Available from: http://www.springerlink.com/index/10.1007/s12507-‐010-‐0101-‐6

114. Norcini JJ, Blank LL, Duffy FD, Fortna GS. The Mini-‐CEX�: A Method for Assessing Clinical Skills. Ann Intern Med. 2003;138:476–81.

115. Weller JM, Misur M, Nicolson S, Morris J, Ure S, Crossley J, et al. Can I leave the theatre�? A key to more reliable workplace-‐based assessment. Br J Anaesth. 2014;(EarlyOnline):1–9.

116. Warm E, Mathis B, Fichtenbaum C, Mueller C, Pain S, Tolentino J, et al. A novel assessment system created by mapping milestones to entrustble professional activities. University of Cincinnati; 2013. p. Poster.


Deliverable D2.1

53

117. Wijnen-‐Meijer M, Kilminster S, Van Der Schaaf M, Ten Cate O. The impact of various transitions in the medical education continuum on perceived readiness of trainees to be entrusted with professional tasks. Med Teach. 2012 Jan;34(11):929–35.

118. Ten Cate O, Billett S. Competency-‐based medical education: origins, perspectives and potentialities. Med Educ. 2014 Mar;48(3):325–32.

119. Cianciolo AT. Deciding “what to teach” health professionals: a human-‐centred systems engineering perspective. Med Educ. 2014 Dec 20;48(12):1150–6.

120. Custers EJFM, Ten Cate OTJ. Very long-‐term retention of basic science knowledge in doctors after graduation. Med Educ. 2011 Apr;45(4):422–30.

121. Dannefer EF, Henson LC. The portfolio approach to competency-‐based assessment at the Cleveland Clinic Lerner College of Medicine. Acad Med. 2007 May;82(5):493–502.

122. Dannefer EF, Bierer SB, Gladding SP. Evidence within a portfolio-‐based assessment program: what do medical students select to document their performance? Med Teach. 2012 Jan;34(3):215–20.

123. Van Tartwijk J, Driessen EW. Portfolios for assessment and learning: AMEE Guide no. 45. Med Teach. 2009 Sep;31(9):790–801.

124. Van der Vleuten CPM, Schuwirth LWT. Assessing professional competence: from methods to programmes. Med Educ. 2005 Mar;39(3):309–17.

125. Stiggins R. Assessment Crisis: The Absence Of Assessment FOR Learning. Phi Delta Kappan. 2002;83(10):758–67.

126. Schuwirth LWT, van der Vleuten CPM. General overview of the theories used in assessment: AMEE Guide No. 57. Med Teach. 2011 Jan;33(10):783–97.

127. Greller W, Drachsler H. Translating Learning into Numbers�: A Generic Framework for Learning Analytics. Educ Technol Soc. 2012;15(3):42–57.

128. Powell DE, Carraccio C, Aschenbrener CA. Pediatrics redesign project: a pilot implementing competency-‐based education across the continuum. Acad Med. 2011 Nov;86(11):e13.

129. Knape JT., ten Cate TJ. Still too old, too smart and too expensive [Nog altijd te oud, te knap, te duur][in Dutch]. Med Contact (Bussum). 2010;65(13):582–5.


Deliverable D2.1

54

APPENDIX I: PROPOSED EPAs for undergraduate medical education 1. UMC UTRECHT AND CHARITE BERLIN Main contributors: Marta Reyman, Sanne van den Munckhof, Lisanne Welink, Olle ten Cate, Harm Peters, Ylva Hotzhausen, Asja Maaz At this stage (December 2014) not all EPAs to be introduced in the undergraduate curricula in the coming years have been agreed upon. Based on a Delphi study in Berlin, literature review and a series of expert meetings in Utrecht, three EPAs were selected to serve for experimental investigation in the WATCHME project. Title of the EPA EPA 1 - Routine check-up of the stable adult patient

Specification and limita-‐tions

-‐ This EPA includes no more and no less thanMeasuring vital param-‐eters: heart rate, respiratory rate, temperature, blood pressure, saturationExplaining all actions to the patient

-‐ Reporting results to the health care team including interpretation, orally and/or written

Context: ambulatory and inpatient setting Targeted transition: first fulltime clinical clerkship to next clerkship Limitations: only with circulatory stable patients of 18 year and older

Most relevant domains of competence

X Medical Expert " Health Advocate X Communicator " Scholar X Collaborator " Professional " Manager

Required experience, knowledge, skills, atti-‐tude and behavior

Knowledge: -‐ basic knowledge of anatomy including relevant arteries -‐ normal values of vital parameters Skill: -‐ skill in using necessary devices to measure vital parameters -‐ recognition of stable and unstable patients Attitude and behavior: -‐ professional communication with the patient -‐ proactive alertness in case of adverse events -‐ willingness to ask for help if needed Experience: -‐ all measurements done at least 5 times

Assessment: informa-‐tion sources to assess progress and ground a structural entrustment decision

-‐ Observation: satisfactory observation of all measurements at least fully twice by experienced health care professionals (nurse, physi-‐cian or other)

-‐ Case-‐based discussions: one CBD with an qualified health care pro-‐fessional

Entrustment for which level of supervision is to be reached at which

Indirect supervision (level 3) ultimately before the transition to the second full time clinical clerkship


Deliverable D2.1

55

stage of training? Expiry date One year without practice after structural entrustment decisions

Title of the EPA EPA 2 - Common clinical procedures


This EPA includes no more and no less than: 1. Explanation of the procedure to the patient 2. Capillary and venous blood withdrawal 3. Arterial blood withdrawal 4. Inserting a peripheral catheter 5. Taking a blood culture 6. Taking a smear (mouth, nose, ears, sores, pus, cervical, anal, urogen-‐

ital and cervical) 7. Intracutaneous, subcutaneous and intramuscular injections 8. Giving infusions 9. Placing a nasogastric tube 10. Surgical suturing after administering local anesthesia 11. Administering a simple bandage 12. Taking an ECG Context: ambulatory and inpatient setting (not in ICU) Target transition: medical school to residency Limitations. Certification does not include the permission to apply in: -‐ Patients not capable of giving consent -‐ Unstable or vitally endangered patients -‐ Patients younger that 18 -‐ Pregnant women


X Medical Expert X Communicator X Collaborator " Manager X Professional " Scholar " Health Advocate

Required experience, knowledge, skills, atti-‐tude and behavior

Knowledge: -‐ Anatomy and function of the human body related to the medical

procedure, legal basis of performing medical procedures on pa-‐tients, hygiene and infection prevention, indication, goals and po-‐tential risks for the medical procedure

Skills: -‐ Preparation of all materials and equipment needed, technical exe-‐

cution of the skills listed above, post-‐procedure handling and pro-‐cessing of patients material gathered, writing inquiries.

Attitude and behavior: -‐ Putting patient at ease; being aware of own and patient safety; cor-‐

rectly identifying the patient Experience:


Deliverable D2.1

56

-‐ Minimum of 10 executions of each medical procedure, except naso-‐gastric tube placement (minimum 5)


-‐ Satisfactory observation of each procedure at least fully three times by qualified health care professionals (nurse, physician or other) with practice observation report

-‐ Case-‐based discussions: each procedure must at least once be in-‐clude in a CBD with a qualified health care professional

Entrustment for which level of supervision is to be reached at which stage of training?

-‐ Indirect supervision (level 3) reached ultimately at the graduation from medical school

-‐ Parts of the full list of procedures may be entrusted early in medical school as a smaller, nested EPA (e.g., “Venous and capillary blood withdrawal and infusion” to be reached at level 3 ultimately at the transition to the second fulltime clinical clerkship).

Expiry date One year without practice after structural entrustment decisions

Title of the EPA EPA 3 - General history and physical examination


This EPA includes no more and no less than: 1. Taking a complete medical history of present illness 2. Performing a full physical examination 3. Focused oral report to a supervisor or a health care team 4. Full report in an electronic health record Context: ambulatory and inpatient setting Transition: medical school to residency Limitations: This EPA does not cover specialty-‐specific physical examination Certification does not include the permission to apply in: • Patients not capable of giving consent • Unstable or vitally endangered patients • All patient younger that 18 and pregnant women.


X Medical Expert X Communicator " Collaborator " Manager X Professional " Scholar " Health Advocate

Required experience, knowledge, skills, atti-‐tude, and behavior

Experience: -‐ The student must a have passed a structured clinical examination in

a simulated environment. -‐ Before entrustment, the student must have experience with exam-‐

ining at least 20 patients


Deliverable D2.1

57

Knowledge and skill: -‐ This EPA requires ability to (1) actively search for typical signs and

symptoms for the diseases and health disorders, to construct a dif-‐ferential diagnosis during investigation, (2) put together prelimi-‐nary findings and documents related to the patient’s history, cur-‐rent medication and, if necessary, consultation by physicians and conferring with family members, (4) structure a summary as de-‐fined by major and minor complaints, previous diagnoses, current and previous medication. It requires skills in techniques for history taking.

-‐ Knowledge of anatomy and most common illness scripts.


-‐ Minimum 8 satisfactory observations of history and physical exam-‐ination in primary, medical and surgical contexts with practice ob-‐servation reports

-‐ Case-‐base discussions: at least 3 CBDs, including one with a prima-‐ry, a medical and a surgical specialist, with structured report

Entrustment for which level of supervision is to be reached at which stage of training?

-‐ Post-‐hoc report (level 4) expected in the final year and required ultimately at graduation for medical school

-‐ Indirect supervision (level 3) expected at transition to the final year of medical school

Expiry date One year without practice after structural entrustment decisions


Deliverable D2.1

58

4. UNIVERSITY OF CALIFORNIA, SAN FRANCISCO

Based on a Delphi procedure medical educators and a series of expert meetings (Chen et al, submitted), the following EPAs were identified as important for pre-‐clerkship medical educa-‐tion. All EPAs aim at a Level 3 supervision (‘indirect supervision’) at graduation. Main contributors: H. Carrie Chen, Margaret McNamara, Arianne Teherani, Olle ten Cate, Patricia O’Sullivan. EPA title

EPA 1 -‐ Gather information from a medically stable patient with a common chief complaint

Detailed descrip-tion

Use the chief complaint to gather a history and perform a complete or fo-‐cused physical exam appropriate to the context, within a reasonable timeframe (i.e. considering setting, complexity) in the following circum-‐stances: • The patient has a common chief complaint (e.g. earache, headache, cough,

shortness of breath, abdominal pain, vomiting/diarrhea, back pain, dysu-‐ria, fever, or rash)

• The patient may have underlying medical problems (e.g. chronic condi-‐tions such as hypertension, COPD/asthma, or diabetes)

• The physical exam does not include the genitourinary, rectal, or female breast exam

• The patient is medically stable and is not in significant physical or emo-‐tional distress as determined by a supervising clinician

• The setting can be in the outpatient clinic, emergency department, or inpatient ward (but not intensive care units)

• The patient is mostly cooperative (e.g. non-‐combative, adult or child greater than 7 years of age), relatively cognitively intact (e.g. non-‐sedated, not delirious or demented or psychotic) • History could be obtained from a cooperative family member of pa-‐

tient (e.g. parent of a child) • Physical exam where the patient is able to ambulate/transfer

him/herself • Interactions are conducted in a language in which both parties are fluent

or through a qualified interpreter. Specific knowledge, skills, and attitudes needed to execute the EPA well

The following abilities are required to perform this EPA: • Knowledge of differential diagnoses for common chief complaints, the

types of information to elicit in the history and types of findings to assess for in the physical exam based on the signs and symptoms of common ill-‐nesses (Gained from the EC courses and FPC)

• Knowledge of the structure and parts of a history, including what is en-‐tailed in a medication reconciliation (FPC)

• Skills in rapport-‐building, clear communications with patients, history-‐taking, and medication reconciliation (FPC)

• Knowledge of how to perform all parts of the basic complete physical examination (FPC)

• Skills in physical examination techniques and performing these maneu-‐vers while ensuring patient comfort and modesty (FPC)

• Knowledge of and adherence to HIPAA guidelines (FPC) • Demonstration of compassion, respect for and sensitivity to patients’


Deliverable D2.1

59

backgrounds/cultures, identities, perspectives, communication prefer-‐ences, and needs (FPC)

• Demonstration of dependability, accountability, and integrity in interac-‐tions with patients (FPC)

Link to FPC precep-‐torship objectives and course compe-‐tencies

Associated preceptorship objectives: • Conduct a complete medical history on a patient (real or standardized) in

the ambulatory, ED, or inpatient setting • Conduct a focused history appropriate to the presenting complaint(s) • Perform a complete or appropriately focused basic physical examination

on a cooperative patient • Extract relevant information from the medical record including the PMH,

medications, allergies, Social History, Family History, and problem list • Complete an accurate Medication Reconciliation (e.g. using health coach-‐

ing techniques or other tools as appropriate) • Abide by HIPPA guidelines in all patient-‐related encounters and commu-‐

nications Associated FPC course competencies: • Gather, synthesize and organize patient information into a standard med-‐

ical history • Perform a complete physical examination while attending to patient

comfort and modesty • Establish rapport and demonstrate respectful behaviors that address

patients’ needs and preferences • Demonstrate sensitivity and responsiveness to patients’ diversity and

identity, and advocate for patients and communities • Demonstrate respect, compassion, accountability, dependability, integri-‐

ty, and collaboration with patients and families • Describe approaches to healthcare maintenance and management of

acute and chronic illness for patients across the age spectrum Link to UCSF Grad-uation competen-cies and milestones most applicable to this EPA

Patient care • Gather complete and focused histories in an organized fashion, appropri-‐

ate to the clinical situation and specific population • Conduct relevant, complete, and focused physical examinations Interpersonal and communication skills • Establish collaborative and constructive relationships with patients and

families • Communicate effectively with patients and families of diverse back-‐

ground and cultures • Elicit and address patients’ concerns, needs and preferences and incor-‐

porate them into management plans Professionalism • Form doctor-‐patient relationships demonstrating sensitivity and respon-‐

siveness to culture, race/ethnicity, age, socioeconomic status, gender, sexual orientation, spirituality, disabilities, and other aspects of diversity and identity, and advocate for care for the underserved

• Demonstrate respect, compassion, accountability, dependability, and integrity when interacting with peers, inter-‐professional healthcare pro-‐viders, patients, and families

• Show accountability and reliability in interactions with patients, families,


Deliverable D2.1

60

and other health professionals • Practice ethically and with integrity, including maintaining patient confi-‐

dentiality, obtaining appropriate informed consent, and responding to medical errors

Link to EPAs from professional or-ganizations

UME (AAMC core EPAs for entering residents) • Gather a history and perform a physical examination GME • Manage care of patients with acute common diseases across multiple

care settings (medicine) • Manage patients with acute, common, single system diagnoses in an am-‐

bulatory, emergency or inpatient setting (pediatrics) Information sources to gauge progress

List of evaluation and assessment sources

Basis for formal entrustment deci-‐sion

Description of number of times ability needs to be demonstrated in what circumstances

Implications of en-‐trustment for the student

• Students entering clerkships are expected to have reached en-‐trustment level 3a (be trusted to perform the EPA with reactive supervision, where the supervising clinician is not with the student but is nearby and available, and will recheck the student’s findings)

• Student will be allowed to independently (without in-‐room super-‐vision) gather information from history and physical examination of medically stable patients with common chief complaints to sup-‐port his/her role as a primary care provider in the outpatient clin-‐ic, emergency department, or inpatient ward.

EPA title

EPA 2 -‐ Integrate information gathered about the patient to construct a rea-‐soned and prioritized differential diagnosis as well as a preliminary plan for common chief complaints.


Integrate information from the history and physical exam in the following circumstances and with the following characteristics: • The patient has a common chief complaint (e.g. earache, headache, cough,

shortness of breath, abdominal pain, vomiting/diarrhea, back pain, dysu-‐ria, fever, or rash).

• The patient has up to three significant, stable medical problems (e.g. con-‐trolled hypertension, asthma, or diabetes).

• The differential diagnosis and plan: • Are based on the patient’s history of present illness • Incorporate factors from the patient’s past medical, social, and family

histories, and the patient’s medical record (e.g., considers a patient’s travel history in the differential diagnosis of fever.)

• Incorporate foundational science knowledge (e.g. pathophysiology or molecular mechanisms of disease)

• The differential diagnosis includes more than one possible diagnosis, and is prioritized and supported by clinical reasoning.

• The plan includes suggestions for next steps as appropriate (e.g. common-‐ly ordered diagnostic tests/imaging and/or initial treatment, medications, or interventions).


Deliverable D2.1

61

Specific knowledge, skills, and attitudes needed to execute the EPA well

The following abilities are required to perform this EPA: • Knowledge of the differential diagnoses for common chief complaints, the

typical characteristics of the disease entities that cause them, and man-‐agement of the more common disease entities (Gained from EC courses and FPC).

• Knowledge of and skill in using a systematic approach to generating a list of possibilities for the differential diagnoses

• Skill in integration and application of information obtained in the history and physical to determine likelihood of diagnoses under consideration


Associated preceptorship objectives: • Complete a new patient write-‐up or interval visit SOAP note, with particu-‐

lar focus on the problem list, and review it with your preceptor. Associated FPC course competencies: • Apply clinical reasoning to information gathering and reporting, including

medical documentation, and oral presentations • Demonstrate integration and application of fundamental sciences to clini-‐

cal encounters • Use clinical reasoning to generate differential diagnoses and problem lists

Link to UCSF Graduation com-petencies and milestones most applicable to this EPA

Patient Care • Present encounters including reporting of information and development

of an assessment and plan efficiently and accurately • Document encounters including reporting of information and develop-‐

ment of an assessment and plan efficiently and accurately Medical Knowledge • Establish and maintain knowledge necessary for the preventive care, di-‐

agnosis, treatment, and management of medical problems • Demonstrate curiosity, objectivity, and the use of scientific reasoning in

acquisition of knowledge, and in applying it to patient care • Select, justify and interpret diagnostic clinical tests and imaging • Diagnose and explain clinical problems • Select and apply basic preventive, curative, and/or palliative therapeutic

strategies for the management of clinical conditions Interpersonal and Communication Skills • Present patient information efficiently in an organized, accurate, and logi-‐

cal fashion appropriate for the clinical situation, including assessment and plan


UME (AAMC core EPAs for entering residents) • Develop a prioritized differential diagnosis and select a working diagnosis

following a patient encounter • Recommend and interpret common diagnostic and screening tests GME • Manage care of patients with acute common diseases across multiple care

settings (medicine) • Manage care of patients with acute complex diseases across multiple care

settings (medicine) • Manage care of patients with chronic diseases across multiple care set-‐

tings (medicine) • Provide perioperative assessment and care (medicine) • Manage patients with acute, common, single system diagnoses in an ambu-‐


Deliverable D2.1

62

latory, emergency, or inpatient setting (pediatrics) • Manage patients with acute complex multi-‐system disease in an ambulato-‐

ry, emergency, or inpatient setting (pediatrics) • Recognize, provide initial management and refer patients presenting with

surgical problems (pediatrics) • Assess and manage patients with common behavior/mental health prob-‐

lems (pediatrics) Information sources to gauge progress



Description of number of times ability needs to be demonstrated in what cir-‐cumstances

Implications of entrustment for the student

• Students entering clerkships are expected to have reached entrustment level 3a (be trusted to perform the EPA with reactive supervision, where the supervising clinician is not with the student but is nearby and availa-‐ble, and will verify the student’s reasoning).

• Student will be allowed to independently develop an initial assessment and plan after a patient encounter (to be verified by the supervising clini-‐cian) to support his/her role in direct patient care.

EPA title

EPA 3 -‐ Communicate information relevant to patient’s care with other mem-‐bers of the health care team


The following conditions and limitations apply: • Findings following a patient encounter (e.g. patient interview, physical

exam, chart review, test results, etc) are organized and prioritized and then communicated via: • Oral case presentation using an accepted standard format • Written documentation using an accepted standard format (e.g. EHR or

other) • The setting can be in the outpatient clinic, emergency department, or inpa-‐

tient ward (but not intensive care units) • Encounters may include also include interactions outside the clinical set-‐

ting (e.g. home visit, telephone call, email correspondence, etc.) • Findings are presented and discussed with the supervising clinician before

sharing with other members of the health care team (e.g. nursing staff, con-‐sulting service, etc.)


The following abilities to perform this EPA: • Knowledge of the structure and components of an oral case presentation

(FPC) • Knowledge of the structure and components of an encounter note, includ-‐

ing that of the H&P and SOAP note formats (FPC) • Knowledge of and adherence to HIPAA guidelines (FPC) • Skills in clearly and accurately presenting patient information to team

members in a structured and organized fashion • Skills in clearly and accurately documenting patient encounters in standard

accepted formats such as the H&P and SOAP notes. • Demonstration of dependability, accountability and integrity in interac-‐

tions with other health care providers


Deliverable D2.1

63


Associated preceptorship objectives: • Clearly and succinctly communicate relevant history and physical findings

through oral presentation • Clearly and succinctly communicate relevant history and physical findings

through written documentation • Abide by HIPPA guidelines in all patient-‐related encounters and communi-‐

cations Associated FPC course competencies: • Apply clinical reasoning to information gathering and reporting, including

medical documentation, and oral presentations • Demonstrate integration and application of fundamental sciences to clini-‐

cal encounters • Use clinical reasoning to generate differential diagnoses and problem lists • Demonstrate respect, compassion, accountability, dependability, integrity,

and collaboration with patients, families and the health care team Link to ACGME/UCSF Graduation com-petencies and milestones most applicable to this EPA

Patient care • Present encounters including reporting of information and development of

an assessment and plan efficiently and accurately • Document encounters including reporting of information and development

of an assessment and plan efficiently and accurately Interpersonal and communication skills • Present patient information efficiently in an organized, accurate, logical

fashion appropriate for the clinical situation, including the assessment and plan

• Communicate oral and written clinical information that accurately and efficiently summarizes patient data

• Communicate effectively and respectfully with all members of the inter-‐professional team involved in a patient’s care

Professionalism • Demonstrate respect, compassion, accountability, dependability, and integ-‐

rity when interacting with peers, inter-‐professional healthcare providers, patients, and families

• Show accountability and reliability in interactions with patients, families, and other health professionals


UME (AAMC core EPAs for entering residents) • Provide an oral presentation/summary of a patient encounter • Provide documentation of a clinical encounter in written or electronic for-‐

mat • Give or receive a patient handover to transition care responsibility to an-‐

other health care provider or team • Participate as a contributing and integrated member of an inter-‐

professional team GME • Lead and work within inter-‐professional health care teams (medicine and

pediatrics) • Manage transitions of care (medicine) • Facilitate handovers to another healthcare provider either within or across

systems (pediatrics) Information List of evaluation and assessment sources


Deliverable D2.1

64

sources to gauge progress Basis for formal entrustment deci-‐sion



• Students entering clerkships are expected to have reached entrustment level 3a (be trusted to perform the EPA with reactive supervision, where the supervising clinician is not with the student but is nearby and availa-‐ble)

• Student will be allowed to independently (without in-‐room supervision) share information about a patient’s care with the supervising clinician or with other members of the health care team after discussion with the su-‐pervising clinician.

EPA title

EPA 4 -‐ Share information about the patient’s care, including diagnosis and management plan, with a patient in no significant physical or emotional dis-‐tress.


The following conditions and limitations apply: • The information to be shared is straightforward and has been vetted by the

supervising clinician. • Information can include diagnosis, management plan, next steps, pa-‐

tient education, anticipatory guidance, or health coaching. • The discussion is anticipated not to surprise or provoke undue anxiety

in the patient. (e.g.counseling patients on eating habits, medications, or hgb A1C but not providing cancer diagnosis).

• The patient is medically stable and has a common acute or chronic diagno-‐sis that is not immediately life threatening, critical, or emergent.

• The patient is generally cooperative (non-‐combative, older child or adult), relatively cognitively intact (non-‐sedated, not delirious or demented or psychotic). • Communication may be with a family member of the patient (e.g. par-‐

ent of a child) • Communication is in language in which the provider and patient are both

fluent or through a qualified interpreter. • Sharing information includes checking the patient’s understanding of the

information conveyed and seeking assistance from a supervising clinician if there is notable patient surprise or anxiety.

Specific knowledge, skills, and atti-tudes needed to execute the EPA well

To perform this EPA, the pre-‐clerkship student will require the following abili-‐ties: • Knowledge of typical clinical course and management for common chief

complaints, and patient education for common health concerns (Gained from EC courses and FPC).

• Skills in rapport-‐building, clear communications with patients, patient edu-‐cation, and health coaching, utilizing techniques such as ask-‐teach-‐ask, closing the loop/teach back, and looking ahead/action plan (FPC).

• Skills in tailoring patient education information to patients’ health literacy and developmental levels (FPC).

• Demonstration of compassion, respect for and sensitivity to patients’ back-‐grounds/cultures, identities, perspectives, communication preferences,


Deliverable D2.1

65

and needs (FPC). • Demonstration of dependability, accountability, and integrity in interac-‐

tions with patients (FPC). Link to preceptor-‐ship objectives and course com-‐petencies

Associated preceptorship objectives: • Demonstrate appropriate verbal communication with the patient during

physical examination to inform the patient about next steps • Provide anticipatory guidance about a health related behavior (e.g. healthy

diets, adequate physical activity, harm reduction in substance use) • Provide health coaching for common conditions (e.g. obesity, diabetes, hy-‐

percholesterolemia, hypertension) Associated FPC course competencies: • Establish rapport and demonstrate respectful behaviors that address pa-‐

tients’ needs and preferences • Communicate with patients and families in a manner appropriate to di-‐

verse populations (including those across the age and developmental spec-‐trum), including discussions of sensitive and serious topics

• Demonstrate sensitivity and responsiveness to patients’ diversity and iden-‐tity, and advocate for patients and communities

• Demonstrate respect, compassion, accountability, dependability, integrity, and collaboration with patients, families and the health care team


Interpersonal and communication skills • Establish collaborative and constructive relationships with patients and

families • Communicate effectively with patients and families of diverse background

and cultures • Effectively and empathetically discuss serious, sensitive, and difficult top-‐

ics; share information and negotiate treatment plans with patients and their families

• Communicate effectively with diverse patients and ensure patient under-‐standing

Professionalism • Form doctor-‐patient relationships demonstrating sensitivity and respon-‐

siveness to culture, race/ethnicity, age, socioeconomic status, gender, sex-‐ual orientation, spirituality, disabilities, and other aspects of diversity and identity, and advocate for care for the underserved

• Demonstrate respect, compassion, accountability, dependability, and integ-‐rity when interacting with peers, inter-‐professional healthcare providers, patients, and families



UME (AAMC core EPAs for entering residents) • Enter and discuss patient orders/prescriptions • Obtain informed consent for tests and/or procedures that the day 1 intern

is expected to perform or order without supervision GME • Manage care of patients with acute common diseases across multiple care

settings (medicine) • Provide age-‐appropriate screening and preventative care (medicine) • Facilitate family meetings (medicine)


Deliverable D2.1

66

• Facilitate the learning of patients, families, and members of the interdisci-‐plinary teams (medicine)

• Manage patients with acute, common, single system diagnoses in an ambu-‐latory, emergency or inpatient setting (pediatrics)

• Provide recommended pediatric health screening (pediatrics) Information sources to gauge progress





• Students entering clerkships are expected to have reached entrustment level 3a (be trusted to perform the EPA with reactive supervision, where the supervising clinician is not with the student but is nearby and available, and has preapproved the information to be shared)

• Student will be allowed to independently (without in-‐room supervision) share information with patients about diagnoses, next steps, and health education to support his/her primary role in direct patient care.

EPA title

EPA 5 -‐ Provide the health care team with resources to improve an individual patient’s care or collective patient care.


Information that has been researched and appraised may only be shared with permission of the supervising clinician. Resources include: • Information from the medical literature such as practice guidelines and

possible treatment options from clinical reviews and studies weighted by quality and relevance of evidence

• Patient education materials from the electronic medical record system or other vetted, evidence-‐based sources (e.g. Up To Date Patient Handouts, clinic-‐specific information, instructions on how to take medications).

• Local, community-‐based resources for support of patients and/or patients’ families (e.g. non-‐profit organizations, support groups, food bank, hotline numbers).

• National organizations for information, support, and advocacy for patients and/or patients’ families (e.g. American Heart Association, American Can-‐cer Society, Cystic Fibrosis Foundation).

• Expert opinion related to a given problem from other members of the health care team (e.g. medical or nurse specialist, pharmacist, nutritionist, social worker, etc)


The following abilities are required to perform this EPA: • Knowledge of potential information resources (Gained from the EC cours-‐

es and FPC) • Skill in the appraisal of information sources and content (EC and FPC) • Skill in the use of information technology to access electronic and online

information (EC and FPC) • Skill in summarizing and applying information to individual patients (EC

and FPC) • Demonstrate dependability, accountability, and integrity in interactions

with patients (FPC)


Deliverable D2.1

67

Link to preceptor-‐ship objectives and course competen-‐cies

Associated preceptorship objectives:

• Use of available practice-‐based and web-‐based information to provide local, community, or national resources to patients for the management of acute or chronic concerns

Associated FPC course competencies:

• Demonstrate sensitivity and responsiveness to patients’ diversity and identity, and advocate for patients and communities

• Demonstrate respect, compassion, accountability, dependability, integrity, and collaboration with patients, families and the health care team

• Demonstrate knowledge of different health care teams and systems and ways in which they can be improved


Patient care • Demonstrate confidence and efficacy with the primary provider role in the

acute and ambulatory settings and the provision of longitudinal care; Man-‐age and prioritize patient care tasks for a group of patients; Anticipate pa-‐tients’ needs, conduct discharge planning, and create individualized dis-‐ease management and/or prevention plans including patient self-‐management and behavior change

Problem based learning and improvement • Use information technology to access online medical information, manage

information, and assimilate evidence from scientific studies in patient care • Identify clinical questions as they emerge in patient care activities and

identify and apply evidence relevant to answering those questions; ap-‐praise and assimilate the scientific evidence from the literature and apply it to clinical decision making for individual patients

• Apply evidence-‐based medicine to improve the care of individual patients and populations

Professionalism • Form doctor-‐patient relationships demonstrating sensitivity and respon-‐

siveness to culture, race/ethnicity, age, socioeconomic status, gender, sex-‐ual orientation, spirituality, disabilities, and other aspects of diversity and identity, and advocate for care for the underserved

• Demonstrate respect, compassion, accountability, dependability, and in-‐tegrity when interacting with peers, inter-‐professional healthcare provid-‐ers, patients and families


Systems-‐based practice • Participate effectively as a member of the healthcare team with physicians

and inter-‐professional healthcare providers Link to EPAs from professional or-ganizations

UME (AAMC core EPAs for entering residents) • Form clinical questions and retrieve evidence to advance patient care • Participate as contributing and integrated member of an inter-‐professional

team GME • Improve the quality of health care at both the individual and systems level

(medicine)


Deliverable D2.1

68

• Advocate for individual patients (medicine) • Manage information from a variety of sources for both learning and appli-‐

cation to patient care (pediatrics) Information sources to gauge progress




Implications of en-‐trustment for the student

• Students entering clerkships are expected to have reached entrustment level 3a (be trusted to perform the EPA with reactive supervision, where the supervising clinician is not with the learner but is nearby and availa-‐ble).

• Student will be allowed to independently research and appraise infor-‐mation from a variety of sources to share with the health care team with permission from the supervising clinician.


Deliverable D2.1

69

APPENDIX II: PROPOSED EPAs for postgraduate medical education – Anesthesiology Based upon a Delphi procedure among academic and non-‐academic anesthesiologists, 45 EPAs were identified as important for residency education. Of these three were selected for empirical investigation in the WATCHME project. All EPAs aim at a Level 4 supervision (‘unsupervised’) at graduation. Main contributors: Nienke Wisman-‐Zwarter, Reinier Hoff. Title EPA EPA 1 -‐ Resuscitation of the multiple trauma patient in the Emergency

Room Detailed description (including limitation of the EPA to specific patient groups and/or context)

Resuscitation of trauma patients, of all age groups, in the Emergency Room. Active participation in the trauma team. Assessment and control of vital parameters. Pain management in trauma patients.

Which specific knowledge, skills and attitudes are neces-‐sary to perform the EPA at an adequate level?

1. Trauma mechanisms & pathophysiology 2. Organization of trauma care 3. Collaboration in the trauma team 4. Trauma diagnoses & treatment 5. Primary & secondary survey 6. Trauma airway management 7. Emergency IV1 & IO2 access 8. Emergency thoracostomy 9. Hemorrhage / massive transfusion 10. Emergency Room administrative procedures

Which CanMeds com-‐petency fields are mostly addressed in this EPA?

Ø Medical expert Ø Communicator Ø Collaborator Ο Manager Ο Scholar Ο Health advocate Ø Professional


Deliverable D2.1

70

How can progression of the resident in this EPA be assessed?

1. Trauma Mini-‐CEX3 2. OSATS4 on trauma airway management 3. OSATS on emergency IV & IO access 4. OSATS on emergency thoracostomy 5. Multi-‐source feedback 6. Trauma case-‐based discussions 7. Participation in trauma Simulator Sessions

Which criteria must be met by the resident to fulfil the EPA satisfac-‐tory (= at a level of indirect supervision)

1. Completion of two-‐month rotation Anesthesia in the Emergency Room 2. ATLS5 certification 3. APLS6 certification 4. Positive result (pass) on (at least) five trauma Mini-‐CEX, on different

days, by different assessors in different contexts 5. Positive result (pass) on (at least) two case-‐based discussions on

trauma, on different days, by different assessors in different contexts 6. Positive result (pass) on OSATS on trauma airway management, emer-‐

gency IV & IO access and emergency thoracostomy 7. Positive result (pass) on two trauma Simulator Sessions, in different

contexts 8. Positive result (pass) on the evaluation of the EPA by (at least) 3 dif-‐

ferent assessors


Deliverable D2.1

71

Title EPA EPA 2-‐ Peripartum pain management

Detailed description (including limitation of the EPA to specific patient groups and/or context)

Consenting the patient and determine the indicated treatment. Executing the indicated treatment, and providing aftercare re-‐garding aspects of pain relief during labour

Which specific knowledge, skills and attitudes are nec-‐essary to perform the EPA at an adequate level?

1. Knowledge of physiologic changes during pregnancy and labour

2. Knowledge of pharmacological considerations during preg-‐nancy and labour

3. Knowledge of current (options in) pharmacological labour pain treatment

4. Informing and consenting the patient 5. Performing epidural analgesia during labour 6. Performing opioid-‐based analgesia during labour 7. Monitoring vital parameters 8. Recognition and treatment of labour analgesia complica-‐

tions 9. Collaboration with the obstetric team 10. Labour suite administrative procedures

Which CanMeds competency fields are mostly addressed in this EPA?

Ø Medical expert Ø Communicator Ø Collaborator Ο Manager Ο Scholar Ο Health advocate Ø Professional


Deliverable D2.1

72

How can progression of the resident in this EPA be as-‐sessed?

1. Labour suite Mini-‐CEX1 2. OSATS2 epidural 3. Multi-‐source feedback 4. Labour analgesia case-‐based discussions 5. Participation in labour suite Simulator Sessions

Which criteria must be met by the resident to fulfil the EPA satisfactory (= at a level of indirect supervision)

1. Positive result (pass) on (at least) three peripartum pain management Mini-‐CEX, on different days, by different as-‐sessors in different contexts (with at least one epidural mini-‐CEX and one non-‐epidural mini-‐CEX = Mini Clinical Evaluation Exercise)

2. Positive result (pass) on (at least) two case-‐based discus-‐sions on peripartum pain management, on different days, by different assessors in different contexts

3. Positive result (pass) on OSATS on epidural = Objective Structured Assessment of Technical Skills

4. Positive result (pass) on the evaluation of the EPA by (at least) 3 different assessors

Title EPA

EPA 3 -‐ Preoperative assessment


Preoperative screening and designing a tailored anesthetic management plan at the preoperative outpatient clinic and on the hospital ward

Which specific knowledge, skills and attitudes are nec-‐essary to perform the EPA at an adequate level?

1. Knowledge of surgical procedures 2. Knowledge of comorbidities 3. Knowledge of appropriate anesthetic procedures 4. Taking a focused history and performing physical examina-‐

tion 5. Perioperative risk assessment 6. Ordering and interpreting relevant diagnostic tests 7. Collaborating with specialists in other medical fields 8. Collaborating with colleagues at the preoperative outpatient

clinic 9. Communicating lifestyle and general health issues 10. Informing and consenting the patient 11. Demonstrates discernment of own limitations 12. Time management

Which CanMeds competency fields are mostly addressed in this EPA?

Ø Medical expert Ø Communicator Ø Collaborator Ο Manager Ο Scholar


Deliverable D2.1

73

Ο Health advocate Ø Professional

How can progression of the resident in this EPA be as-‐sessed?

1. Mini-‐CEX1 preoperative outpatient clinic 2. Video assessment and feedback 3. Multi-‐source feedback 4. Preoperative case-‐based discussions 5. Participation in preoperative Simulator Sessions

Which criteria must be met by the resident to fulfil the EPA satisfactory (= at a level of indirect supervision)

1. Completion of one-‐month rotation at the preoperative out-‐patient clinic

2. Positive result (pass) on (at least) five preoperative assess-‐ment Mini-‐CEX, on different days, by different assessors in different patient groups

3. Positive result (pass) on (at least) two case-‐based discus-‐sions on preoperative assessment, on different days, by dif-‐ferent assessors

4. Positive result (pass) on the evaluation of the EPA by (at least) 3 different assessors


Deliverable D2.1

74

APPENDIX III: PROPOSED EPAs for veterinary education Based upon a Delphi procedure among 47 academic and non-‐academic veterinarians, 35 EPAs were identified as important for veterinarian education. Of these two were selected for empiri-‐cal investigation in the WATCHME project. All EPAs aim at a Level 4 supervision (‘unsuper-‐vised’) at graduation Title EPA 1 -‐ History taking, general impression and general examination


History taking: iatrotrophic problem, global functioning (individual animal and herd), living conditions (food, housing) and history (previ-‐ous treatment, etc.) General impression (individual and torque): behaviour, posture and pace, nutrition status and clinical disorders General examination: respiration rate, pulse rate, temperature, skin, hair / feathering, claws and horns, mucous membranes and lymph nodes

-‐ Poultry: plumage, state of oviposit and crop -‐ Cattle: rumen movements, rumen displacement, udder and rec-‐

tal examination (EPA Rectal examination), -‐ including assessing dropping -‐ Sheep and goat: rumen movements, steel band and udder -‐ Pig: No specific extras

Estimate the relevance of the data, rank the data and use it as a start point for further research Problem definition and ddx Data documentation


1. History taking 2. Give a general impression 3. Do the general examination 4. Come to a problem definition and ddx 5. Data documentation 6. Collaboration with the farmer Kuiper en Van Nieuwstad, 2008;

-‐ chapter 3, Disease history -‐ chapter 4, General impression -‐ chapter 5, General examination

Which veterinary competency fields are mostly addressed in this EPA?

• Veterinary expertise • Communication • Collaboration • Entrepeneurship • Health and Welfare • Scholarship • Personal Development


Deliverable D2.1

75


1. Mini-‐CEX1 a. Veterinary expertise b. Communication c. Collaboration

2. Multi-‐source feedback Which criteria must be met by the resi-‐dent to fulfil the EPA satisfactory?

Block 1: Focus on the Animal -‐ At least 3x Mini-‐CEX in the clinic (1x ruminants; 1x pigs; 1x poultry.

Each competency must be evaluated at least 3 times). -‐ At least 3x Mini-‐CEX by fellow student (each competency must be

evaluated at least 3 times). Block 2: -‐ Minimum 2x Mini-‐CEX Tolakker lecturer (each competency must be

evaluated at least 2 times). -‐ At least 2x Mini-‐CEX by fellow student (each competency must be

evaluated at least 2 times) Block 3: -‐ For each farm, at least 1x Mini-‐CEX by lecturer and 1x Mini-‐CEX by

fellow student (each competency must be evaluated by lecturer and fellow students at least 1 times). These Mini-‐CEXs are filled in dur-‐ing the final meeting (this requires coordination between the su-‐pervisor and the lecturer who evaluates the student during the presentation).

-‐ At least 1x MSF for the farm analysis (farmer, veterinarian, supervi-‐sor and at least 1x fellow student).

Block 4: UFAP (University Farm Animals Practice) -‐ At least 1x Min-‐CEX during the 5-‐week ULP, filled in over several

case studies during the period (each competency must be evaluated at least 1 time).

-‐ Skills (see EPASS). -‐ 1x MSF for the farm analysis (farmer, veterinarian, supervisor and

at least 1x fellow student). Block 6: External courses -‐ At least 2X Mini CEX by a lecturer on location (supervising veteri-‐

narian. Each competency must be evaluated at least 2 times). -‐ Minimum 1x MSF (livestock owners, veterinarians, assistants, col-‐

league). Block 7: Senior clinical clerkship -‐ At least 1x Mini-‐CEX by fellow student (each competency must be

evaluated at least 1 time). -‐ At least 1x Mini CEX (supervisor, at least 2x senior clinical place-‐

ment students and 2x basic clinical placement student).


Deliverable D2.1

76

Title EPA 2 -‐ Pain relief


• Recognition of pain • Knowledge of the relevant pathophysiology and pharmacology • Determine and remove the cause of the pain and • Treatment and aftercare • Communicate and explain the benefits of analgesia to farmers • Instruction for working with painkillers • Prevention of pain


Knowledge of the pathofysiologie of pain -‐ Hellebrekers, L.J. (2001). Pathophysiology of pain

Murrell, J.C. & Hellebrekers, L.J. (2006). Post-‐operative care and pain managment. Chapter 12. The cuttong edge; basic operation skills for the veterinary surgeon


• Veterinary expertise • Communication • Collaboration • Entrepeneurship • Health and Welfare • Scholarship • Personal Development


1. Mini-‐CEX1 a. Veterinary expertise b. Communication c. Health and Welfare Mini-‐CEX d. Scholarship

2. Multi-‐source feedback Which criteria must be met by the resi-‐dent to fulfil the EPA successfully

Block 1: Focus on the Animal -‐ At least 3x Mini-‐CEX in the clinic (1x ruminants; 1x pigs; 1x poultry.

Each competency must be evaluated at least 3 times). -‐ At least 3x Mini-‐CEX by fellow student (each competency must be

evaluated at least 3 times). Block 4: UFAP (University Farm Animals Practice) -‐ At least 1x Min-‐CEX during the 5-‐week ULP, filled in over several

case studies during the period (each competency must be evaluated at least 1 time).

-‐ Skills (see EPASS). -‐ 1x MSF for the farm analysis (farmer, veterinarian, supervisor and

at least 1x fellow student). Block 6: External courses -‐ At least 2X Mini CEX by a lecturer on location (supervising veteri-‐

narian. Each competency must be evaluated at least 2 times). -‐ Minimum 1x MSF (livestock owners, veterinarians, assistants, col-‐

league). Block 7: Senior clinical clerkship -‐ At least 1x Mini-‐CEX by fellow student (each competency must be

evaluated at least 1 time).


Deliverable D2.1

77

At least 1x Mini CEX (supervisor, at least 2x senior clinical placement students and 2x basic clinical placement student).

Title EPA 3 -‐ Dealing with a respiration problem


• History taking, general impression and general examination • Recognition of the respiration problem and evaluating the prognosis • Respiration examination; respiratory movements, noise and cough,

nasal cavities and thorax, lung percussion, larynx and trachea • Problem definition and ddx • Additional research • Diagnosis and treatment • Knowledge of the relevant pathophysiology, zoonosis, notifiable

diseases and pharmacology (vaccinations and formulary) Which specific knowledge, skills and attitudes are neces-‐sary to perform the EPA at an adequate level?

Knowledge of the respiration system and respiration examination Kuiper en Van Nieuwstad, 2008; chapter 6 The respiration system


Veterinary expertise Communication Collaboration Entrepeneurship Health and Welfare Scholarship Personal Development


1. Mini-‐CEX1 a. Veterinary expertise b. Communication c. Collaboration d. Health and Welfare e. Scholarship

2. Multi-‐source feedback Which criteria must be met by the resi-‐dent to fulfil the EPA satisfactorily?

Block 1: Focus on the Animal -‐ At least 3x Mini-‐CEX in the clinic (1x ruminants; 1x pigs; 1x poul-‐

try. Each competency must be evaluated at least 3 times). -‐ At least 3x Mini-‐CEX by fellow student (each competency must be

evaluated at least 3 times). Block 4: UFAP (University Farm Animals Practice)

-‐ At least 1x Min-‐CEX during the 5-‐week ULP, filled in over several case studies during the period (each competency must be evalu-‐ated at least 1 time).

-‐ Skills (see EPASS). -‐ 1x MSF for the farm analysis (farmer, veterinarian, supervisor

and at least 1x fellow student).


Deliverable D2.1

78

Block 6: External courses -‐ At least 2X Mini CEX by a lecturer on location (supervising veter-‐

inarian. Each competency must be evaluated at least 2 times). -‐ Minimum 1x MSF (livestock owners, veterinarians, assistants,

colleague). Block 7: Senior clinical clerkship

-‐ At least 1x Mini-‐CEX by fellow student (each competency must be evaluated at least 1 time).

At least 1x Mini CEX (supervisor, at least 2x senior clinical placement students and 2x basic clinical placement student)


Deliverable D2.1

79

APPENDIX 2: TOOL FOR MATRIX MAPPING FOCUSED ON TEACHER TRAINING Title: New perspectives on formative evaluation of student teachers’ teaching

competence

Authors: Edgar Krull [1] , Äli Leijen [1], Bert Slof [2], Marieke van der Schaaf [2]

[1] University of Tartu, Estonia

[2] Utrecht University, the Netherlands

Abstract The aim of this study is to analyze possibilities for the performance-‐based formative evaluation of student teachers’ teaching skills in their field. The targeted analysis is a part of the interna-‐tional project “Workplace-‐based e-‐Assessment Technology for Competency-‐based Higher Multi-‐professional Education” WATCHME. As there is no overall consensus in what is meant by quality or effective teaching the presented analysis starts with a brief survey of approaches to define characteristics of effective teaching. It comes to a conclusion that national standards and compe-‐tence requirements represent the lowest level of specification in which consensus of all stake-‐holders can be achieved. Three widely known teacher standards frameworks are introduced as coordinating ideas for defining specific indicators of teaching competencies. The following anal-‐ysis of available specifications of teaching competencies like those of the Model Core Teaching Standards and Learning Progressions for Teachers introduced above (Council of Chief State School Officers , 2013) reveals that using them imposes a need for documenting too many atom-‐ic indicator behaviors of student teachers in quantified format. Therefore, focusing on essential teaching competencies based on a rather holistic approach is suggested by applying the emerg-‐ing concept of core practices. Also, using portfolios for documenting data on teaching skills is taken into consideration. Finally, the performance-‐based approach in the format of several spec-‐ifying matrixes (up to description of performance levels of the indicator behaviors) for as-‐sessing student teachers teaching skills that was developed on the basis of SBL competence re-‐quirements for Dutch teachers is described and issues of adapting to Estonian context are dis-‐cussed. Keywords: teacher education, teaching practice, teaching competences, professional develop-‐ment, competency-‐based instruction, performance indicators, core practices 1. Introduction The aim of this literature review-‐based study is to prepare the ground for optimizing student teachers’ professional learning in school practice by providing them with adaptive formative evaluation of their progress. Studying possibilities for advancing performance-‐based assessment of student teachers teaching skills is part of the “Workplace-‐based e-‐Assessment


Deliverable D2.1

80

Technology for Competency-‐based Higher Multi-‐professional Education” WATCHME project. This international project aims at improving workplace-‐based feedback and assessment and professional development by means of learning analytics. Learning analytics (LA) is a computer-‐based approach for documenting assessment of learning results, for providing specified feedback on the achievement of expected objectives and for providing suggestions for corrective and further learning activities (e.g. Clow, 2013; Dringus, 2012; Ferguson, 2012; Greller and Drachsler, 2012). This paper provides an overview of what student teachers should know and be able to do to be qualified as ‘good teachers’. Validity and reliability issues and suggestions for advancing the assessment of student teachers teaching skills in Estonia and the Netherlands (part of the WatchMe project) are discussed. 2. Competence requirements for the beginning and experienced teachers Our study on approaches to defining good or quality teaching1 and its assessment is based on learning relevant research reviews and studies on these topics. Academic Search Complete, E-‐journals, ERIC, PsycARTICLES, PsycINFO, and Teacher Reference Center as databases were re-‐peatedly searched for locating publications on research in the field for the last 20 years. The main keywords, used for searching in combination with restricting term “teacher education”, were teaching competences, competence requirements, professional standards, teaching stand-‐ards performance-‐based assessment, professional development, competency-‐based instruction, teaching practice, performance indicators meta-‐analysis, core practices, learning analytics. Also, the contents of academic journals like Teaching and Teacher Education, Journal of Teacher Edu-‐cation, European Journal of Education, Educational Leadership were learned for finding works not covered by used keywords. 2.1. Theoretical underpinnings of defining characteristics of effective teaching The attempts of defining knowledge and skills necessary for teaching profession date almost hundred years ago when researchers started to compile lists of teaching skills on the basis of teacher inquiries what makes up a good teaching. This resulted in producing of long and varying lists of attributes of good teaching that had of little use for practice of teacher education (Good, 1996). Since the 1960s more specific models for describing teacher education approaches or defining attributes of good teaching appeared. For example, recently Cochran-‐Smith (2014) in her keynote speech at the EERA conference in Porto listed four consecutively appearing re-‐search questions for teacher education aimed at discovering the secrets of good teaching: (1950–1960) what are attributes of good teachers; (1960–1980) what teaching processes lead to effective teaching; (1980–1990) what should teachers know, be able to do; and (since 2001) the nature (teachers are born) versus nurture (teachers are educated) issues reappeared. Corre-‐spondingly, in the first case the main research question is How can inborn teachers be recruited?; in the last case How can teacher candidates learn to teach? 1 Also, terms like teaching competence or competency, professional competence, basic teach-ing competence, teaching skills, teaching knowledge, effective teaching are used over the paper as approximate synonyms.


Deliverable D2.1

81

Another list of research traditions for discovering effective teaching (with approximate domination periods) is provided by Good: (1960s) focus on teacher personality; (mid-‐1960s–1970) search for teacher-‐proof curriculum; investigation of teaching in naturalistic settings (1968–1990s); relating teacher behavior to student learning (1960–1990s); examination how teachers utilized the classroom time (1960s–1990s); centering on teacher cognition (1970s–1990s); focusing on student mediation of teaching (1980s–1990s); and (1990s) studies on teaching for understanding (Good, 1996). The presented lists of research questions for teacher education as well as those of teach-‐er effectiveness research traditions illuminate the complexity of teaching as of a professional activity and point to the fact that construction of all-‐comprehensive models of good or effective teaching is a very controversial and challenging task. With some concession it can be argued that a common trend for the research of teacher education as well as for teacher effectiveness re-‐search is moving away from attempts of discovering specific and simplistic indicators of good teaching (like specific teacher behavior or classroom time utilization) to indicators embracing teacher professionalism in all its aspects (like teacher learning as personal professional growth or teaching for understanding). As Good points out in his survey of research traditions of effec-‐tive teaching, each research tradition has yielded relevant ideas for evaluating teaching and he suggests to rely on a paradigm of inclusion rather than of exclusion (Good, 1996), meaning that advantage should be taken of all positive sides of former research. This variability in conceiving teacher education as well as in teacher effectiveness re-‐search traditions illuminate the complexity of teaching as of a professional activity and point to the fact that construction of all-‐comprehensive models of good or effective teaching is a very controversial and challenging task. With some concession it can be argued that a common trend for the research of teacher education as well as for teacher effectiveness research is moving away from attempts of discovering specific and simplistic indicators of good teaching, like it was practiced in competency based teacher education (e.g. Andrews & Barnes, 1990; Bowles, 1973) to indicators embracing teacher professionalism in all its aspects like teacher learning as per-‐sonal professional growth or teaching for understanding (Good, 1996). The issue of creating models of good teaching leads to questions of more general character: to what extent is good teaching identifiable, does it have a permanent character in the sense that it manifests itself when working with different students and in different contexts of teaching, meaning that there exist certain general professional skills of teaching or are these skills rather context dependent? The first position (that there exists a universal competence of teaching) was, for example, sup-‐ported by Medley (1985) and Stodolsky (1985). Instead, Shulman (1992) pointed out that teach-‐ing quality depends on the subject to be taught as well as on the situation in which the teaching takes place. Yet, even Medley (1982) found that correlations between qualities of teaching activi-‐ties of a teacher in different contexts are less than 0.3. Consequently, it is rather difficult to con-‐struct a model of teaching activities with indicators that would enable comparing teachers and even assessing teaching effectiveness of one teacher in different school conditions. This task becomes even more complicated if we take into consideration that different stakeholders in educational systems and even teacher educators might have differing expecta-‐tions for teacher professional competences. For example, several scholars (Joyce, 1975; Doyle, 1990; Zeichner, 1983) have proposed five major profiles of ideal teachers: good employee, jun-‐


Deliverable D2.1

82

ior professor, fully functioning professor, innovator, and reflective practitioner. Furthermore, teaching at different school levels and of different subjects might impose variations in needed teaching competences and correspondingly lead to a need of specifying competence require-‐ments for teacher certification. 2.2. National standards and competence requirements for teachers Despite of these theoretical concerns warning against using simplistic or biased models of teach-‐ing competences and consequently competence requirements, there have been standards along with indicators in use for assessing teaching competences of beginning and even experienced teachers in many countries. As stated by Arends, ” …effective teaching is dependent upon a knowledge base and clear definitions of what constitutes effective teaching” (2006, p. 17). Most-‐ly, the standards and competencies describing “effective teaching” have been defined in general terms that leave a lot of flexibility for taking into account specific contextual features. A need for “…formulation of clear professional standards and also criteria to assess them” is also emphasized by Zgaga as a conclusion made on the basis of a recent survey of teacher education in 12 European countries (2011, p. 31). However, despite of these critical notes regarding quality, the most frequent practice of defining good teaching in many countries is introduction of teacher competence requirements or standards (e.g., Bourgonje and Tromp, 2011; Roth, 1996). Three of them are introduced in the following sections and validity and reliability issues regarding the assessment student teachers competence requirements will be discussed. 2.2.1. Dutch teacher competence requirements In Europe an elaborated system of teacher competence requirements has been developed in the Netherlands. Public discussions over its suitability lasted almost for six years until it was ap-‐proved by the Dutch Parliament in 2006 (Bourgonje & Tromp, 2011). The Dutch teacher compe-‐tence requirements are defined two-‐dimensionally as combinations of teacher roles and situa-‐tions in which they implement their roles. The competence requirements introduce four roles (interpersonal, pedagogical, organizational, and organizational roles) and four types of contexts (with students, colleague, working environment, and him/herself). The sets of teacher roles and types of situations form a 4 by 4 matrix. According to SBL (the Association for the Professional Quality of Teachers), seven partial competences are enough to cover all essential aspects of a teacher competence (Bourgonje & Tromp, 2011). These seven competences are defined as fol-‐lows:

˗ Interpersonal competence in creating a pleasant, safe and effective classroom environment;

˗ Pedagogical competence to support children’s personal development by helping them to become independent and responsible;

˗ Subject knowledge and methodological competence that demonstrates substantial knowledge of their subject and appropriate teaching methods (including pedagogical content knowledge);

˗ Organizational competence in organizing curricula that support


Deliverable D2.1

83

student learning; ˗ Competence to collaborate with colleagues and thus contribute to a

well-‐functioning school organization; ˗ Competence to collaborate with those in the school environment who

also play a role in students’ well-‐being and development (i.e. students’ parents or guardians, colleagues at educational and youth welfare institutions);

˗ Competence to reflect and to develop as professionals over the long term (Snoek, 2011)

The seven competences are specified by introducing competence requirements and indicators (Bourgonje & Tromp, 2011). The Dutch model has been tried out with teachers in primary, sec-‐ondary and vocational education (SBL, 2004). However, as the described requirements have not been tested with scientific rigor and corrected for their validity and reliability, it is not surpris-‐ing that the issue of subjectivity of evaluation raises, i.e. different evaluators assess differently. Therefore, these teacher competence requirements can serve rather as coordinating guidelines or ideas for teacher education programs but not directly as a tool for assessing teaching. 2.2.2. NBPTS standards Probably the most thoroughly investigated conception of teacher standards and related proce-‐dures for teacher certification of highly accomplished teachers belong to the National Board for Professional Teacher Standards (NBPTS) in the United States of America. All specific standards and procedures of NBPTS are based on five core propositions about good teaching. Effective teachers …

- are committed to students and their learning; - know the subjects they teach and how to teach those subjects to students; - are responsible for managing and monitoring student learning; - think systematically about their practice and learn from experience; - are members of learning communities (NBPTS, 1987).

NBPTS certification is performance-‐based. Candidates, applying for certification present portfolios containing four videos of teaching performance in different contexts along with de-‐tailed explanations of observable teaching activities. However, for understanding the instru-‐ments providing validity and reliability of NBPTS certification procedures, it has to be taken into account that the competence requirements of these standards are specified by school levels and subjects taught (25 certification areas; see NBPTS, 2014) and provided with detailed assessment guidelines for experts. A comprehensive study by Bond and his colleagues (2000) aimed at the investigation of construct validity of the NBPTS teacher certification procedures by comparing the instructional practices and outcomes of teachers who have been certified by the agency with those of teachers who have applied for certification but were not certified, yielded a detailed list of teacher attrib-‐


Deliverable D2.1

84

utes in which certified teachers typically outperformed not certified teachers. The list of expert teachers’ attributes was developed on the basis of meta-‐analysis of big number of studies and its validity was confirmed by relatively high inter-‐rater agreement on the relevance of the listed attributes and by demonstrating superiority of certified teachers in comparison with uncertified teachers in the case of 11 attributes from 13 (Bond, et al., 2000). These 11 attributes were: use of knowledge, having deep representations, problem solving, improvisation, challenge of objec-‐tives, creating classroom climate, multidimensional perception, monitoring learning and provid-‐ing feedback, respect for students, passion for teaching and learning, motivation and self-‐efficacy (Bond, et al., 2000, p. 104). Another study aimed at validating the NBPTS certification standards was carried out by Vandevoort and his colleagues (Vandevoort, et al., 2004). The study revealed that students with higher learning achievement were taught by certified teachers. In this study academic perfor-‐mance of students in the elementary classrooms of 35 certified teachers and their non-‐certified peers, in 14 Arizona school districts were compared. Four years of results from the Stanford Achievement Tests in reading, mathematics and language arts, in grades three through six, were analyzed. In the 48 comparisons (four grades, four years of data, three measures of academic performance), using gain scores adjusted for students’ entering ability, the students in the clas-‐ses of National Board Certified Teachers surpassed students in the classrooms of non-‐Board certified teachers in almost three quarters of the comparisons (Vandevoort, et al., 2004). 2.2.3. INTASC standards Alongside of investigating the validity of competence requirements for practicing teachers there have been a few attempts to put under scrutiny competence requirements for the beginning teachers. It is obviously wrong to consider competence requirements for experienced teachers and for student teachers to be achieved in their teaching practice as the same things. In the case of student teacher school practice the main focus should be on formative evaluation of the be-‐ginning teachers’ progress in learning to teach rather than on the final professional competences to be achieved (Andrews & Barnes, 1990). Therefore the performance-‐based objectives to be achieved at teaching practice cannot be directly derived from standards for expert teaching. Nevertheless the awareness of the main competences of the expert teachers might be helpful for coordinating and guiding learning to teach in the school practice conditions as well. From this point of view, the work done by the New Teacher Assessment and Support Consortium (INTASC), USA is of interest. The model of core standards (10 INTASC principles) for beginning teachers grew out of the five propositions about effective teaching adopted by the NBPTS (Arends, 2006). Recently a revision of these standards was launched for updating. The beta draft version of these standards publicly available is aimed at beginning as well as practicing teachers (Council of Chief State School Officers, 2013). All ten standards define expected teaching competences in terms of per-‐formances, essential knowledge, critical dispositions, and of descriptions for progression. How-‐ever, the most interesting feature of the introduced draft standards is descriptions of progres-‐sion for the listed standards that outline three levels of professionalism in teaching. The main categories of these standards are …

- learner development;


Deliverable D2.1

85

- learning differences; - learning environments; - content knowledge; - application of content; - assessment; - planning for instruction; - instructional strategies; - professional learning and ethical practice, and - leadership and collaboration (Council of Chief State School Officers, 2013, pp. 16–47).

2.3. Difficulties in defining (inter-) national standards for effective teaching Whereas in European countries little empirical evidence has been reported on effects of the use of teaching standards, in the USA rigorous evaluations of certification systems based on those standards emerge. However, the results of those studies are not sufficient to draw firm conclu-‐sions about their effectiveness (Cantrell, Fullerton, Kane, & Staiger, 2008; Hakel, Koenig, & El-‐liott, 2008; Ingvarson, 2009). What is considered as good teaching in one country is not necessarily good teaching in another country. Furthermore, expectations for teachers’ roles might be different even in one country. This might explain why there is not much international validation studies of teacher competence requirements or teaching standards specified up to measurable teaching perfor-‐mances. The diversity in emphasizing relevant aspects of good teaching in the introduced stand-‐ards and scarcity of reliable research on teacher competence requirements or certification pro-‐cedures based on these requirements or standards points, as already mentioned above, to the complexity of teaching as of a professional activity and to the difficulties of modelling a good teaching in a reliable way. For example, if the quality of learning foreign languages and progress in it at basic level can be quite reliably and satisfactory described as a gradual acquisition of vo-‐cabulary and phrases then the quality teaching cannot be reduced to single performances with-‐out assessing the quality that these performances produce integrally. The alternation in ap-‐proaches to defining attributes of good teaching as exemplified by periodization of the related research by Cochran-‐Smith (2014) and Good (1996) only confirm that all-‐comprehensive cap-‐turing what is a quality teaching is a very challenging task.

Due to the lack of valid and reliable models of good teaching with clearly identified vari-‐ables allowing rigorous discrimination between poor and good teaching it is clear that prospects for optimizing student teachers’ study paths is limited as descriptions of expected teaching com-‐petence integrally as well as identification of partial competences underlying this competence tend to be subjected to different interpretations and are context dependent. Furthermore, con-‐sidering that educator’s models and representations of good teaching and its component skills are never ideally correct and unchangeable, the adequate assessment of teaching should always involve an interpretative component (Tigelaar & Van Tartwijk, 2010; Van der Schaaf, Stokking & Verloop, 2008b).

Teacher educators have always tried to imagine and describe what they mean by good teaching and which teaching skills have to be developed for learning to teach. Typically this has


Deliverable D2.1

86

been taken lead of following national competence requirements or teacher standards as of a coordinating framework for specifying tasks or activities to perform for student teachers in their field-‐practice expectedly leading to anticipated teaching competences. However, this does not mean that necessary performance indicators are directly derivable from standards. 3. Assessing student teachers’ professional competence: A new perspective Irrespective of theoretical problems with ensuring rigorous validity in modelling of good teach-‐ing and identifying component teaching skills, the teacher educators have always tried to imag-‐ine and describe what they mean by a good teaching and which teaching skills have to be devel-‐oped for learning to teach. Depending on traditions of teacher education these teaching activities and component skills to be mastered and criteria of assessment to be used would be more or less specific. As a rule, these specifications are validated on the basis of negotiations and agree-‐ments among teacher educators involved (e.g. Van der Schaaf & Stokking, 2011). 3.1. Introducing performance-based indicators for the basic teacher competences A good example of specifying teaching competences that are defined at general level into specific indicator activities is CCSSO (Council of Chief State School Offices) Model Core Teaching Stand-ards and Learning Progressions for Teachers introduced above (Council of Chief State School Of-‐ficers , 2013). Its ten standards of teaching competences are specified by listing expected per-formances in teaching, essential knowledge, and critical dispositions – all defined in terms of be-‐haviors characterizing the achievement of these sub-‐competences. For example, the Standard # 6: Assessment is specified by 9 statements for performances in assessment, 7 for essential knowledge, and 6 for critical dispositions (Council of Chief State School Officers, 2013). The first statements for these three domains are: (1) the teacher balances the use of formative and sum-‐mative assessment as appropriate to support, verify, and document learning; (2) the teacher understands the differences between formative and summative applications of assessment and knows how and when to use each; and (3) the teacher is committed to engaging learners actively in assessment processes and to developing each learner’s capacity to review and communicate about their own progress and learning. However, the most valuable feature from the point of view of assessing student teachers’ progress in acquiring teacher competences is definitions of progression indicators for standards. For example, the standard # 6 is provided with three aspects or dimensions of progress in the assessment competences: (1) the teacher uses, designs or adapts multiple methods of assess-‐ment to document, monitor, and support learner progress appropriate for learning goals and objectives; (2) the teacher uses assessment to engage learners in their own growth; and (3) the teacher implements assessments in an ethical manner and minimizes bias to enable learners to display the full extent of their learning. In its turn, every aspect of progress for standards is described on three, gradually refin-‐ing levels of proficiency. For the dimension two (the teacher uses assessment to engage learners in their own growth) the nature of these levels can be apprehended from the first sentences of the corresponding descriptions: (1) the teacher engages each learner in examining samples of quality work … (2) and the teacher engages learners in generating criteria for quality work … (3)


Deliverable D2.1

87

and the teacher engages learners in giving peers feedback on performance using criteria gener-‐ated collaboratively. Totally, the whole model offers descriptions of 19 progress dimensions with 54 indicator performances suitable for documenting in quantified format the beginning teachers’ progression in acquiring required teaching competences. However, observing simulta-‐neously such large numbers of teacher attributes and teaching skills without losing an integral sight of a student teacher progression in acquiring teaching competence is rather complicated. One solution to this problem would be the emerging concept of core practices (Gross-‐man, et al., 2009). This approach is different from prior attempts to orient teacher education around practice. In her comparison of this innovative concept with former practice-‐centered teacher education Forzani points out that “… it is playing out in both the decisions teacher edu-‐cators are making about what novices should learn – and in particular in how they decompose practice into learnable parts—and in the pedagogical approaches used in professional training” (2014, p. 360). She explains that unlike former practice-‐based teacher education where re-‐searchers tried to decompose teaching into specific performances … core or high-‐leverage prac-‐tices …have used carefully developed criteria to identify a smaller number of items, recognizing that the short duration of teacher education must be used strategically” (Forzani, 2014, p. 363). It means that teaching tasks for teaching practice should be carefully selected in a way that im-‐plementation of these tasks supports the best way linking theory with practice and competence in teaching. It should be noticed that the core practices approach differs radically from competency-‐based teacher education (CBTE) that was based on the Stanford taxonomy of teaching tasks that contained hundreds of teaching tasks organized into 19 categories (Baral, Snow, & Allen, 1968; Bush, 1968). The mentioned approach was based on a belief that learning to teach consisted in mastering specific teaching activities by training them not leaving much place for reflection or experimentation in Dewey’s sense (Dewey, 1965/1904). The concept of core practices as applied to teacher education has a high potential for increasing the coherence between general competency requirements to teaching profession and selection of practice-‐based teaching tasks that implementation is expected to lead to meeting these requirements. Therefore, paying more attention to applying ideas of core practices in addi-‐tion to taking lead of national competence requirements for defining and selecting teaching tasks for student teachers’ school practice is very justified. This would increase the effectiveness of students’ learning to teach and also the validity of performance-‐based assessment of relevant teaching competences by introducing more authentic performance indicators representing teaching in its integrity. 3.2. Portfolios as tools for assessing teaching competence Besides defining performance-‐based criteria for the assessment of student teachers’ pro-‐gress in learning to teach, the identification of sources allowing to collect necessary evidences or data for the assessment procedures is not an easier task, either. Typically different documents providing information on teaching competences are drawn together into student teacher portfo-‐lios. For example, Van der Schaaf, Stokking & Verloop (2008b) examined the validity and relia-‐bility of portfolios as a tool for assessing teaching skills. The WATCHME assessment matrix for applying LA methodology foresees collecting evidence materials in an electronic portfolio called


Deliverable D2.1

88

EPASS (2014). These evidence materials involve lesson plans, filled in student placement evalua-‐tion forms by the student teacher, lesson observation forms from supervisors, lesson observa-‐tion forms from other student teachers, samples of used and developed tests, QTI (Questionnaire on Teacher Interaction) information from two students, and reflection reports. Though many researchers point out that portfolios have been introduced as tools to col-‐lect evidence about the development on various competencies, the implementation of portfolios has met with mixed success, particularly if they were not tailored to show what really happened in the workplace (Driessen, Van Tartwijk, Van der Vleuten, & Wass, 2007; Van der Schaaf, Stokking, & Verloop, 2008a). Arends observes: “To be valid, the work included in the portfolio must cover the range of standards and the scoring must be reliable” (2006, p. 105) and he pro-‐vides two examples of how the University of Nôtre Dame and Connecticut have developed pro-‐cedures to make portfolio a valid and reliable summative assessment of teacher candidates and beginning teachers’ performance. A main problem with using portfolios as tools for assessing teaching competences is that their composing is time consuming and quality of reflection teach-‐ing activities is often unsatisfactory (Tigelaar & Van Tartwijk, 2010). Another example of using teacher portfolios is The Performance Assessment for Califor-‐nia Teachers (PACT) system that uses multiple measures approach based on two types of portfo-‐lio-‐based assessment strategies (Pecheone & Chung, 2006). The assessment procedures are de-‐fined by the California Teacher Performance Expectations (Commission on Teacher Credential-‐ing, 2013). The formative assessment of prospective teachers is implemented through embed-‐ded signature assessments that occur throughout teacher education program. A summative as-‐sessment of teaching competences takes place during teaching practice (Pecheone & Chung, 2006). The purpose of the embedded signature assessment (ESA) serves for providing formative feedback to the student teachers and teacher educators. Many generic guiding questions were formulated for creating developmental performance assessment portfolios that meet psycho-‐metric rigor. As explained by Larsen and Calfee:

The ESAs are campus-‐specific assignments chosen from standard criteria that track a teacher candidate’s growth over time. The ESA label signifies that the assessments (1) already were part of preparation courses (embedded) and (2) provided significant snap-‐shots of teacher candidate competency over time (signature). ESAs include case studies, lesson plans, observations, classroom management plans, and other assignments or ac-‐tivities that fulfill certain selection criteria (2005, p. 151).

Also, these authors point out that “…for universities using an e-‐portfolio system, ESAs push the boundaries one step further. An e-‐portfolio is a computer-‐based collection of artifacts showcasing a teacher candidate’s growth over time in key teaching competencies” (Larsen & Calfee, 2005, pp. 155–156) The PACT summative assessment of teaching events (TE) use multiple sources of data (teacher plans, teacher artifacts, student work samples, video clips of teaching, and personal reflections and commentaries) that are organized on four categories of teaching: planning, in-‐struction, assessment, and reflection (PIAR). To complete the TE, candidates plan and teach a


Deliverable D2.1

89

learning segment (i.e., an instructional unit or part of a unit), videotape and analyze their in-‐struction, collect student work and analyze student learning, and reflect on their practice. The TEs are designed to measure and promote candidates’ abilities to integrate their knowledge of content, students, and instructional context in making instructional decisions and to stimulate teacher reflection on practice (Pecheone & Chung, 2006). TE scoring is based on task-‐based model that follows the design of the portfolio in se-‐quential manner of tasks. The candidates’ performance is rated for each task on the basis of guiding questions (GQ) (on a 4-‐point continuum). As a result, a detailed score profile is generat-‐ed that provides information at the GQ level and at the PIAR task level and can be used for de-‐veloping individual induction plans (Pecheone & Chung, 2006). 3.3. Defining performance-based indicators for assessing teacher competence in the WatchMe project This performance-‐based approach for assessing student teachers’ teaching skills is advocated by the international team of WATCHME (2014). It is developed on the basis of Dutch teacher com-‐petence requirements (SBL, 2004) by defining related indicator performances for teacher educa-‐tion needs. To this end the original list of seven competences was shorten to five professional roles (actually extended by one additional role) and those in their turn were specified through descriptions of tasks or activities (also, in terms of entrustable professional activities – EPA2s) that the student teacher is expected and entrusted to perform without supervision. The whole assessment tool consists of 8 frames (tables) from which the essence of five that are relevant for defining performance indicators is exemplified shortly in the Table 1. A more detailed analysis of indicator performances by of performance levels reveals that not only issues of the construct validity of the competence requirements as a model of good teaching in regard of indicators performances arise in the Dutch school context (for what this

Table 1. WATCHME draft performance-based teacher evaluation matrix

Fra–me

Title Content

1 Matrix mapping tool

1) Lists 5 types of professional roles, provides descriptions of profes-‐sional activities related to these roles, and lists 11tasks to be per-‐formed as evidences of being able to implement these roles. The professional roles are: (1) designer, supervisor, and evaluator of learning activities; (2) manager of the work environment; (3) Peda-‐gogue; (4) member of professional community; and (5) Manager of own professional development.

2 Curriculum map-‐ Defines requirements for study phases Curriculum_ internship 1 and

2 An EPA is ‘a critical part of professional work that can be identified as a unit to be entrusted to a trainee once suffi-‐cient competence has been reached’. Competence is thus translated and made manageable in terms of the tasks or activities that can be safely entrusted to someone who has shown the required ability (Mulder, Ten Cate, Daalder, et al., 2010; Ten Cate, 2013). One of many innovations of the WATCHME project is introduction of minimal quality criteria for professional activities. The identification and definittion of these criteria for teaching are subject of another study.


Deliverable D2.1

90

ping tool Curriculum_internship 2. From student completing phase one imple-‐mentation of roles of the pedagogue and member of the professional community is not expected.

3 Performance indi-‐cators

States for tasks to perform indicator activities (2–4 for each task) that evidence coping with tasks related to implementing professional roles

4 Performance levels Defines four performance levels for the tasks evidencing meeting of the roles’ requirements [Level 1 (starting); Level 2 (sufficient); Level 3 (good); Level 4 (excellent)]

5 Performance indi-‐cators by perfor-‐mance levels

Combines performance indicators of tasks with performance level indicators creating a matrix 34 x 4 = 136 combinations altogether.

model was initially developed) but also the question of appropriateness of these performance indicators for assessing teaching skill in school context of another country. For example, in a country like Estonia national curricula for general education have a much more prescriptive nature than in the Netherlands (see e.g. Education, Audiovisual and Culture Executive Agency P9 Eurydice, 2012). Consequently, Dutch student teachers are expected to be ready for more inde-‐pendent professional work than their Estonian colleagues. This becomes even more evident when analyzing the indicator performances expected from Dutch student teachers by complet-‐ing their teaching practice at schools (see Table 2). This means that relatively specific criteria for performance-‐based assessment of teaching developed in the conditions of one country for being effective in the context of another country with different education traditions should be validated for this context. One possible way for adapting the scoring rubric to specific school context is to use Delphi method for deciding whether a specific performance indicator should be accepted, revised or removed. The construct validation of content standards for teaching students research skills used by Van der Schaaf and Stokking (2011) can serve as a prototype approach here.

Table 2. Defining performance indicators by 4 performance levels for task 1 (Formulates

a vision of the subject content and the subject didactics) in the WATCHME draft teacher

evaluation matrix.

Role/task Performance indicators Performance

Level 1

Performance

Level 2

Performance

Level 3

Performance

Level 4

Role: Designer,

1. The teacher does/does not formu-late (self-formulated) learning goals in con-nection with specific content (subject con-tent/didactic compe-

The teacher takes over the learning goals of others and the course book and occasionally stops to think

The teacher takes over the learning goals of others and the course book and often checks to see whether the

The teacher formulates his/her own learning goals which partially match the spe-cific subject

The teacher formulates his/her own learning goals which match those of the subject content.


Deliverable D2.1

91

supervisor and evaluator of learning activities (subject didacti-cian and supervi-sor of the learn-ing process)

Task 1: Sets learning goals for the whole curricu-lum and specific lessons.

tences).

about the cohe-sion between the set of learn-ing goals and the specific subject content.

set learning goals match those of the specific subject content.

content.

2. The teacher does/does not make use of SMART (specif-ic, measurable, ac-ceptable, realistic and time related) formulated learning goals (subject content/didactic compe-tences).

The teacher does not check if the set learning goals are SMART formu-lated.

The teacher regularly checks if the set learn-ing goals are SMART formu-lated.

The teacher formulates his/her own learning goals which partially meet SMART guidelines.

The teacher formulates his/her own learning goals which meet SMART guide-lines.

3. The teacher does/does not take into consideration the start-ing situation of students when formulating learn-ing goals (subject content/didactic compe-tences).

The teacher incidentally stops to think about the con-sistency be-tween the set of learning goals and the starting situation of the students

The teacher regularly checks if the learning goals match the starting situation of the students

The teacher formulates his/her learning goals which partially match with the starting situation of the students

The teacher formulates his/her learning goals which match with the measured start-ing situation of students

Also, the issue of reliability of documenting (measurement) cannot be ignored, as the identifica-‐tion of performance levels would heavily depend on the context and personal interpretation of definitions. For example, the performance criterion “the teacher incidentally stops to think about the consistency between the set of learning goals and the starting situation of the stu-‐dents” for identifying the performance level might be subject of very different interpretations. The fact that student teachers’ performance-‐based assessment is typically based on information collected from different sources (lesson plans, lesson observations and recordings, conferences etc.) additionally emphasizes a need for clear and unambiguous definition of performance level indicators. 5. Discussion and concluding remarks Describing quality teaching and identifying its components that can be used as indicators of quality or progress towards quality is a difficult task. One common way of creating models for quality teaching for is to define professional standards or competence requirements. However, there is a lot of variety in standards for quality teaching and these differ from one country to another. Only few of them provide valid and reliable procedures for specific assessment of teaching. In general, teaching standards are formulated on the basis of agreements between ex-‐perts and used as coordinating ideas for defining specific requirements for good teaching. The validity of specific performance indicators might depend on the context (on the subject taught,


Deliverable D2.1

92

traditions of a particular country or on a specific teacher education program). Learning Analytics (LA) procedures can be helpful for validating such indicators by providing processed data for external interpretation by experts. For example, a graphical presentation of progress profiles and a typology of progress might be beneficial for more general interpretations and validating of the underlying model. Yet, as Greller and Drachsler (2012) stated, there is always an on-‐going challenge to formulate indicators from the available datasets that bear relevance for the evalua-‐tion of the learning process (i.e. learning to teach and modelling of it in our case) but do not rep-‐resent the real teaching competence in its integrity, i.e. embracing all relevant professional skills. Another issue besides modelling teaching competence is collecting entry data on indica-‐tor performances of teaching by means of an e-‐portfolio for applying LA. Main problems are that composing portfolios for assessing teaching competences as well as assessing procedures are time consuming, and the quality of reflection on teaching activities is often unsatisfactory. Quite promising procedures for portfolio-‐based formative and summative assessing of teaching have been elaborated by the PACT (Larsen & Calfee, 2005) involving embedded signature assessment (ESA) and assessment of teaching events (TE). For ESA an e-‐portfolio encloses a computer-‐based collection of artifacts that show a teacher candidate’s growth over time in key teaching compe-‐tencies. Probably, the most challenging problem when using portfolio-‐based assessments as in-‐put for LA, is having reliable assessment scales of defined indicator performances enabling con-‐version of interpretative ratings into quantitative data. Here the strong and weak sides of three options of scaling should be taken into consideration: (1) introduction of relatively independent-‐ly stated indicators of performance levels; (2) cumulatively defined indicators of performance levels (like in the case of CCSSO Model Core Teaching Standards); (3) Likert-‐type scales for as-‐sessing quality of indicator performances. This literature study raised issues in validating models of good teaching and their com-‐ponents in the context of e-‐portfolio assessment and LA. Most issues derived from an atomistic perspective on teaching and teacher assessment and can be solved in an integral way by using a more holistic concept, such as core practices.

References Andrews, T.E. & Barnes, S. (1990). Assessment of teaching. In Robert Houston (Ed.). Handbook of

research on teacher education, (569 − 598).New York Macmillan Publishing Company. Arends, R. I. (2006). Performance assessment in perspective: History, Opportunities, and chal-‐

lenges. In Sharon Castle and Beverly D. Shaklee (eds.) Assessing Teacher Performance: Per-formance-Based Assessment in Teacher Education,(pp. 3–22). Rowman & Littlefield Educa-‐tion.

Baral, D. P., Snow, R. E., & Allen, E. W. (1968). A taxonomy of teaching behaviors: progress report

(Research and Development Memorandum No. 36, Stanford, CA: Stanford University School of Education. ERIC Document Reproduction Service No. ED049157).


Deliverable D2.1

93

Bond, L., Smith, T., Baker, W.K., & Hattie, J.A. (2000). The Certification System of the National Board for Professional Teaching Standards: A Construct and Consequential Validity Study, Center for Educational Research and Evaluation. The University of North Carolina at Greens-‐boro, Greensboro, NC.

Bourgonje, P. and Tromp, R. (2011). Quality Educators: An International Study of Teacher Competences and Standards. Education International / Oxfam Novib. Retrieved from

http://download.ei-‐ie.org/Docs/WebDepot/Quality%20Educators.pdf Bowles, F. D. (1973). Competency-‐based teacher education? The Houston story. Educational

Leadership, 30 (6), 510–12 . Bush, R. N. (1968, April). Second annual report (Annual Report No. 2,. Stanford, CA: Stanford Cen-‐

ter for Research and Development in Teaching. ERIC Document Reproduction Service No. ED024642)

Cantrell, S., Fullerton, J., Kane, T.J., & Staiger, D.O. (2008). National Board Certification and teach-

er effectiveness: Evidence from a random assignment experiment (NBER Working Paper 14608). Cambridge: National Bureau of Economic Research.

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, (18) 6, p 683–

695. Cochran-‐Smith, M. (2014). A U.S. perspective on teacher preparation research: past, present and

future. A paper presented at the EERA Conference “The past, present and future of educa-‐tional research in Europe”, 1–5 September in Porto, Portugal.

Commission on Teacher Credentialing. (2013). California Teaching Performance Expectations.

Retrieved from http://www.ctc.ca.gov/educator-‐prep/standards/adopted-‐TPEs-‐2013.pdf Council of Chief State School Officers (CCSSO). (2013, April). Interstate Teacher Assessment and

Support Consortium InTASC Model Core Teaching Standards and Learning Progressions for Teachers 1.0: A Resource for Ongoing Teacher Development. Washington, DC: Author.

Dewey, J. (1965/1904). The relation of theory to practice in education. In C. A. McMurry (Ed.),

The Third Yearbook of the National Society for the Scientific Study of Education. Part I, ( pp. 9–30). Chicago: The University of Chicago Press. Retrieved from https://archive.org/details/r00elationoftheorynatirich

Doyle, W. (1990). Themes in teacher education research. In W. Robert Houston ed. Handbook of

research on teacher education, (pp. 3 –24). New York: Macmillan Pub. Co. Driessen, E., Van Tartwijk, J., Van der Vleuten, C., & Wass, V. (2007). Portfolios in medical

education: why do they meet with mixed success? A systematic review. Medical education, 41(12), 1224–33.

Dringus, L. P. (2012). Learning Analytics Considered Harmful. Journal of Asynchronous Learning

Networks, 16(3), 87–100.


Deliverable D2.1

94

Education, Audiovisual and Culture Executive Agency P9 Eurydice. (2012). Key data on education in Europe 2012. Brussels: Education, Audiovisual and Culture Executive Agency

Retrieved from http://eacea.ec.europa.eu/education/eurydice/index_en.php) EPASS. (2014). About EPASS. Retrieved from http://www.epass.eu/en/over-‐epass/ Ferguson, Rebecca. The State of Learning Analytics in 2012: A Review and Future Challenges.

Technical Report. Knowledge Media Institute: The Open University, UK, 2012. http://kmi.open.ac.uk/publications/pdf/kmi-‐12-‐01.pdf

Forzani, F.M. (2014). Understanding “Core Practices” and “Practice Based” Teacher Education

Learning From the Past. Journal of Teacher Education, 65(4), 357–368. Good, T.L. (1996). Teaching effects and teacher education. In John Sikula (Ed.). Handbook of re-

search on teacher education. Second edition, (pp. 617–665). New York: Simon & Schuster Macmillan.

Greller, W.; Drachsler, H. (2012) . Translating Learning into Numbers: A Generic Framework for

Learning Analytics. Educational Technology & Society, 15(3), 42–57. Forzani, F. M. (2014). Understanding “Core Practices” and “Practice-‐Based” Teacher Education: Learning From the Past. Journal of Teacher Education, 65(4), 357–36. Grossman, P., Karen Hammerness, K, & McDonald, M. (2009). Redefining teaching, re-‐imagining

teacher education. Teachers and Teaching: Theory and Practice, 15(2), 273–289 Hakel, M., Koenig, J. & Elliott, S.(2008). Assessing Accomplished Teaching: Advanced-‐Level Certification Programs. Washington, D.C.: National Academies Press. Ingvarson, L.C. (2009). Developing and rewarding excellent teachers: The Scottish Chartered

Teacher Scheme. Professional Development in Education, 35(3), 451–468. Joyce, B. (1975). Conceptions of man and their implications for teacher education. In K. Ryan

(Ed.).Teacher education (74th yearbook of the National Society for the Study of Education, Part II, (111 − 145). Chicago: University of Chicago Press..

Larsen, L. R.; Calfee, R. C. (2005). Assessing Teacher Candidate Growth over Time: Embedded Signature Assessments. Clearing House, 78 (4), 151–157.

Medley, D. (1985). Issues and problems in the validation of teaching and teacher professional be-

haviors. Paper presented at the Annual Meeting of the American Educational Research Asso-‐ciation, Chicago, IL, March 31– April 4.

Mulder, H., Ten Cate, O., Daalder, R., & Berkvens, J. (2010). Building a competency-‐based

workplace curriculum around entrustable professional activities: The case of physician assistant training. Medical Teacher, 32(10), e453–9.

NBPTS. (1987).The Five Core Propositions. http://www.nbpts.org/five-‐core-‐

propositions#sthash.rdAKZJ2w.dpuf NBPTS. (2014). Certificate Areas. Retrieved from http://www.nbpts.org/certificate-‐areas


Deliverable D2.1

95

NCATE. (2008). Professional Standards for the Accreditation of Teacher Preparation Institutions. Retrieved from

http://www.ncate.org/Portals/0/documents/Standards/NCATE%20Standards%202008.pdf

Pecheone, R. L.; Chung, R. R. (2006). Evidence in Teacher Education: The Performance Assess-‐

ment for California Teachers (PACT). Journal of Teacher Education, 57 (1), 22–36. Pecheone, R. L.; Chung, R. R. (2007). Technical Report of the Performance Assessment for Califor-

nia Teachers (PACT): Summary of Validity and Reliability Studies for the 2003-04 Pilot Year. California: PACT Consortium. Retrieved from: http://www.pacttpa.org/_files/Publications_and_Presentations/PACT_Technical_Report_March07.pdf

Roth, R. A. (1996). Standards for certification, licensure, and accreditation. . In John Sikula (Ed.).

Handbook of research on teacher education. Second edition, (pp. 242–278). New York: Simon & Schuster Macmillan.

SBL (2004). Competence requirements. Teachers. Retrieved from http://www.bekwaamheidsdossier.nl/cms/bijlagen/SBL_Introduction.pdf

Shulman, L. (1992). Research on teaching: A historical and personal perspective. In F. Oser, A.

Dick, & J. Patry (Eds.). Effective and responsible teaching,( pp. 14 –29). San Francisco: Jossey-‐Bass.

Snoek, M. (2011). Teacher education in the Netherlands. Balancing between autonomous institu-‐

tions and a steering government. In Zuljan, M. V. and Vogrinc J. (eds.). ( European Dimen-sions of Teacher Education – Similarities and Differences (pp. 53–83). Ljubljana: Faculty of Education, University of Ljubljana, Slovenia and The National School of Leadership in Edu-‐cation. Retrieved from http://www.pef.uninlj.si/fileadmin/Datoteke/Zalozba/pdf/Zuljan_Vogrinc_Teacher_education.pdf

Stodolsky, S. (1985). Teacher evaluation: The limits of looking. Educational Research, 13(9),

11−18. Ten Cate, O. (2013) Nuts and Bolts of Entrustable professional activities. Journal of Graduate

Medical Education, 5(1), 157–8. The Council of Chief State School Officers (2013). Interstate Teacher Assessment and Support

Consortium InTASC Model Core Teaching Standards and Learning Progressions for Teachers 1.0: A Resource for Ongoing Teacher Development. Washington, DC: Author. http://www.ccsso.org/Documents/2013/2013_INTASC_Learning_Progressions_for_Teachers.pdf

Tigelaar, D. E. H. and van Tartwijk, J. (2010). Ther evaluation of prospective teachers in teacher

education. In Penelop Peterson, Eva Baker and McGaw (Eds.-‐in-‐chief) International Encyclopedia of Education,Third ed., ( pp. 511–517). Elsvier.

Van der Schaaf, M., Stokking K., & Verloop, N. (2008a). Teacher beliefs and teacher behaviour in

portfolio assessment. Teaching and Teacher Education, 24 (7), 1691–1704.


Deliverable D2.1

96

Van der Schaaf, M., Stokking, K., & Verloop, N. (2008b). Developing and validating a design for

teacher portfolio assessment. Assessment & Evaluation in Higher Education, 33 (3), 245–262. Van der Schaaf, M. F. and Stokking, K. M. (2011). Construct Validation of Content Standards for

Teaching. Scandinavian Journal of Educational Research, 55(3), 273–289. Vandevoort, Amrein-‐Beardsley, and Berliner (2004). National Board Certified Teachers and

Their Students’ Achievement. Education Policy Analysis Archives, 12( 46), 1–117. WATCHME: Workplace-‐based e-‐Assessment Technology for Competency-‐based Higher Multi-‐

professional Education (2014). Retrieved from http://cordis.europa.eu/project/rcn/189095_en.html

Zeichner, K.M. (1983). Alternative paradigms of teacher education. Journal of Teacher education,

34(3), 3 – 9. Zgaga, P. (2011). Why we need to learn one from another and work together? In Zuljan, M. V.

and Vogrinc J. (eds.). ( European Dimensions of Teacher Education – Similarities and Differ-ences (pp. 11–31). Ljubljana: Faculty of Education, University of Ljubljana, Slovenia and The National School of Leadership in Education. Retrieved from http://www.pef.uninlj.si/fileadmin/Datoteke/Zalozba/pdf/Zuljan_Vogrinc_Teacher_education.pdf


Deliverable D2.1

97

Appendix: EPAs for Teacher Education Main contributors: Bert Slof, Äli Leijen, Edgar Krull, Marieke van der Schaaf, & Jan van Tartwijk A) Introduction Teacher education takes place at teacher institutes and at schools for primary education (Esto-‐nia) and secondary education (The Netherlands) where students have a placement (i.e., school practicum). The supervisor (from the teacher institute) and the mentor (from the placement school) are both involved in the assessment and evaluation of the student teacher.

In Estonia, three types of school practicums take place. Firstly, students visit schools in groups of 2-‐3 students in every two weeks throughout three semesters. Every group has a men-‐tor teacher under whose supervision they carry out the practicum. The main focus of this school practicum is to associate studies of the pedagogical subjects with teacher’s activities in an educa-‐tional institution. During this practicum students carry out different observational assignments and regular mentor discussions are held. The final part of this practicum also includes different assignments related to class teacher’s work. Secondly, students visit schools in groups of 2-‐3 students throughout a semester to associate studies of subject, domain-‐ and subject-‐didactics with teacher’s activities in an educational institution. During this practicum, students are in a role of assistant teachers supporting teachers in different activities (e.g. giving feedback, carry-‐ing out parts of the lesson, guiding pupils individually). Thirdly, the main school practicum takes place in the second year of teacher education. This is a 10-‐week period where student teachers actually teach at school and carry out other activities related to teacher’s work. During this practicum formative (feedback is provided) and summative (proficiency level is judged) evalua-‐tion is planned to take place for 11 defined entrusted professional activities (EPAs) in the future. This is the common framework of teacher education in the University of Tartu. Primary school teachers also have some additional school practicums related to the level of education (first years of primary school separately from the later years).

In The Netherlands the assessment and evaluation of the one-‐year programme is divided in two phases. In the first phase, student teachers, in triads, visit their placement school and ob-‐serve experienced qualified teachers’ performance and, in addition, give lessons themselves. In the second phase the student teachers visit the placement school by their self and have to carry out the professional activities with guidance from the mentor and the supervisor from the teacher institute. There is, however, a difference what will be assessed and evaluated in each phase. In the first phase a formative (feedback is provided) and summative (proficiency level is judged) evaluation will take place for eight of the 11 defined entrusted professional activities (EPAs). The other three EPAs will only be formatively evaluated. In the second phase a formative and summative evaluation will take place for all EPAs.

The WATCHME project focuses on the school practicum period of teacher education and, there-‐fore, does not specifically address the acquisition of knowledge about: (1) the subject matter and (2) interpersonal communication and classroom management and (3) development of the pupil as a person and possible problems associated with it. This is assessed and evaluated in other parts of the teacher education programme. Since student teachers’ performance of the EPAs is affected by this kind of knowledge, it is referred to as prerequisite knowledge. Deliverable 2.1 concentrates on what should be assessed and evaluated during student teachers’ school practi-‐


Deliverable D2.1

98

cum. It concerns the content and not how the EPAs should be assessed and evaluated. In this appendix, the five professional roles (student) teachers have to fulfil and the associated EPAs will be described.

To determine what should be assessed and evaluated during school practicum in the teacher education programme a Delphi procedure was carried out at institutes for teacher edu-‐cation in Estonia and The Netherlands. In Step one a first list of professional roles and entrusted activities were developed and presented to the four teacher educators and the head of the teacher educator in the Netherlands. In addition, two Estonian teacher educators were involved in developing the list with roles and activities. Three discussion and revision rounds were held to develop a final list with professional roles and entrusted activities student teachers should master during their internship placement. In Step two, a rubric containing four proficiency level descriptions (i.e., beginning, sufficient, good and excellent) for each EPA was composed. Again, three discussion and revision rounds were held to develop the proficiency levels for each EPA and their underlying performance criteria. Below the results of both rounds are presented and suggestions information sources and assessors for the assessment and evaluation procedure are provided. B) Teachers’ professional roles Student teachers have to fulfil five different professional roles and within each role one or more EPAs have to be carried out. In both countries the roles and EPAs were recognized as important, but especially the first two roles and the associated seven EPAs were seen as the most crucial ones since they directly relate to the teaching responsibilities in the classroom.


Deliverable D2.1

99

Role 1. Designer, supervisor and evaluator of learning activities

Description A teacher is able to plan learning activities based on self estab-‐lished goals and the starting situation of the students. The teacher is able to implement different types media, such as ICT, and target different types of methods flexibly. The teacher is able to clarify to the students the how and why of a lesson. Is able to handle differ-‐ent types of students. Additionally, the teacher is capable of ade-‐quately testing students’ progress.

EPAs • Sets learning goals for the whole curriculum and specific les-‐sons.

• Designs learning activities (incl. materials and media) for the set learning goals.

• Plans the execution and supervision of learning activities. • Supervises the execution of learning activities. • Tests to which extend the set learning goals have been met.

Prerequisite knowledge The teacher expresses readily available content knowledge that goes beyond the course material. He/she expresses these choices mainly from their own subject and educational conception and insight how students acquire subject knowledge. The teacher oversees the annual curriculum.

Formative assessment School practicum 1 and 2 Summative assessment School practicum 1 and 2 Role 2. Manager of the work environment

Description A teacher seeks personal contact with students. He/she has a clear overview of the communication process in the group and is can manage to steer it in different directions in order to maintain an orderly and pleasant work environment.

EPAs • Engages in interpersonal relationships with (groups of) stu-‐dents.

• Directs the communication processes in the group. Prerequisite knowledge The teacher has a personal vision on how to communicate effec-‐

tively and to steer and maintain and orderly and pleasant work environment. This vision is based on basic theories on interper-‐sonal relations and classroom management.

Formative assessment School practicum 1 and 2 Summative assessment School practicum 1 and 2

Role 3. Pedagogue

Description A teacher processes a good sense of identity, self-‐knowledge, and feeling of responsibility to develop a pedagogical vision. The teacher is able to supervise the independence and social-‐emotional and moral development (personal growth) of the stu-‐dents.


Deliverable D2.1

100

C) Teachers’ entrusted professional activities For assessment and evaluation purposes at least two performance criteria were formulated for each EPA. Please be aware that the order in which the criteria are described does not provide an indication for the importance or difficulty of a specific criterion. The criteria were used to assess and evaluate at which proficiency level (i.e., beginning, sufficient, good and excellent) a student teacher is able to carry out a specific EPA. Based on the proficiency levels formative (i.e., sugges-‐tions for improvement) and summative feedback (i.e., pass or fail decision) can be provided by the mentor and the supervisor. Student teachers pass the school practicums when they, at least, master all EPAs at the sufficient level.

EPAs • Supervises the development of the student as a person. Prerequisite knowledge The teacher has a pedagogical vision based on her/his own

norms and values and relevant basic theories related to the de-‐velopment of the student as a person. She is able to demonstrate this based on him-‐/herself as example in the class. He/she is able to work out supervision forms for individual students and the group as a whole and knows how to communicate this to stake-‐holders such as colleagues and parents.

Formative assessment School practicum 1 and 2 Summative assessment School practicum 2

Role 4. Member of the professional community

Description A teacher has a view on aspects of school and educational policies that affect her or his personal performance. The teacher carries out tasks that go beyond the teaching tasks, and is prepared to use her or his influence to improve school and educational policies. This is done constructively in collaboration with colleagues and others.

EPAs • Carries out tasks that go beyond the lesson, class and subject. • Collaborates with colleagues and, if necessary, parents and

other stakeholders. Prerequisite knowledge -‐ Formative assessment School practicum 1 and 2 Summative assessment School practicum 2


Deliverable D2.1

101

EPA 1. Sets learning goals for the whole curriculum and specific lessons Assessment and evaluation criteria

The teacher does/does not formulate (self formulated) learning goals in connection with specific subject content The teacher does/does not make use of SMART (specific, measurable, ac-‐ceptable, realistic and time related) formulated learning goals. The teacher does/does not take into consideration the starting situation of students when formulating learning goals.

Proficiency levels The teacher takes over the learning goals or course material from others. He/she incidentally considers the starting situation of the students and the connection with specific subject content. The teacher does not check if the learning goals are SMART formulated. (starting) The teacher regularly checks if the learning goals of others or the course material connect to specific subject content and the starting situation of the students. The teacher checks if the set learning goals are SMART formulat-‐ed. (sufficient) The teacher formulates his/her own learning goals, which usually connect to the specific subject content and the starting situation of the students. These learning goals are partially SMART formulated. (good) The teacher formulates his/her own coherent learning goals, which con-‐nect to the specific subject content and the investigated starting situation of the students. The learning goals are SMART formulated. (Excellent)

Suggested infor-‐mation sources

Lesson plans/series of lessons and student placement evaluation form.

Suggested assessors Mentor and supervisor.


Deliverable D2.1

102

EPA 2. Designs learning activities (incl. materials and media) for the set learning goals Assessment and evaluation criteria

The teacher, if desired, does/does not make use of self-‐developed learning activities (incl. methods and materials). The teacher does/does not relate the learning activities to the set learning goals. The teacher does/does not create possibilities to involve students actively in the lesson. The teacher does/does not make use of possibilities to differentiate be-‐tween students.

Proficiency levels The teacher makes use of learning activities (incl. material and media), which are developed by the publisher or others. He/she incidentally stops to think about how it connects with the learning goals. The teacher mainly uses instruction oriented working methods, which are targeted to the class as a whole. The teacher supports students with the help of standard exer-‐cises. (starting) The teacher is able to adapt learning activities (incl. material and media), which are developed by the publisher or others. He/she usually stops to think about how the learning goals and the learning activities connect. The teacher alternately makes use of direct instruction and activating working methods. He/she searches for solutions for the differences in student levels in the class. (sufficient) The teacher develops his or her own learning materials (incl. material and media) and usually relates these to the desired learning results. He/she uses self-‐developed and existing learning activities and is able to bring var-‐iation in the lesson. The teacher takes into consideration the differences of between students at the class level. He/she chooses targeted brush-‐up ac-‐tivities and enrichment activities at the class level. (good) The teacher knows when new learning activities (incl. material and media) are desired, shapes these accordingly and considerately, and relates the learning activities clearly to the desired learning results. He/she makes use of the rich diversity of activating working methods. The teacher lets indi-‐vidual students work at their own level as much as possible and chooses appropriate brush-‐up activities and enrichment exercises. (Excellent)





Deliverable D2.1

103

EPA 3. Plans the execution and supervision of learning activities Assessment and evaluation criteria

The teacher does/does not make use of structured lesson plans. The teacher does/does not take into consideration (to be expected) events. The teacher does/does not take on a variety of learning activities and ma-‐terials. The teacher does/does not make use of the possibility to differentiate be-‐tween students.

Proficiency levels The teacher has problems to plan a realistic and clearly structured lesson. He/she often does not have alternative plans available in order to antici-‐pate (to be expected) situations. The teacher only includes a couple of in-‐struction oriented working methods, which are meant for the group as a whole, in the lesson plan. (starting) The teacher mainly uses lesson plans, which have a clearly distinguishable introduction, core and closing. She/he takes into consideration (to be ex-‐pected) situations. The teacher incorporates instruction oriented and acti-‐vating working methods, which are meant for the group as a whole in the lesson plan. (sufficient) The teacher plans the construction of the lesson realistically. The learning activities are marked by a clear construction, with alternated activating working methods and take into consideration the heterogeneity of students at class level. The teacher partially anticipates (to be expected) situations. (good) The teacher plans the content and the construct of the lesson realistically. The learning activities are marked by a variety of activating working meth-‐ods. The teacher takes into consideration the features of individual stu-‐dents. He/she, if necessary, has an alternative plan available to play into (to be expected) situation. (Excellent)





Deliverable D2.1

104

EPA 4. Supervises the execution of learning activities Assessment and evaluation criteria

The teacher does/does not carry out effectively and with flexibility the planned learning activities. The teacher does/does not implement a variety of learning activities and learning materials which do/do not take into consideration the level of the students. The teacher does/does not stimulate the students to choose their own learning activities. The teacher explains to the students the learning goals and lesson phases

Proficiency levels The teacher still has difficulty to execute on time and effectively the planned learning activities (methods). He/she has little attention to the relationship between the learning goals and learning activities. He/she is often lead by events which take place during the lesson. The teacher mainly offers instruction oriented working methods, which are targeted to the class as a whole. (starting) The teacher usually executes the planned learning activities effectively and on time. The teacher explicates the relationship between the learning goals and the learning activities at the beginning of the lesson. The teacher adapts instruction oriented and activating working methods and alternates between them. The teacher seeks for solutions in order to take into consid-‐eration the differences between students in the class. (sufficient) The teacher effectively executes the planned learning activities in the class. Gives clear instructions and explains the phases of the lesson. He/she is capable of adapting activities adequately if the situation requires it. The teacher applies instruction oriented and activating working methods and is able to bring variation to them. The teacher stimulates students to choose their own learning activities. She/he explains the relationship between the learning goals and learning activities at the beginning and at the end of the lesson. (good) The teacher effectively executes the planned learning activities in the class. Gives clear instructions and explains the phases of the lesson. He/she has to his/her disposal a large set of adequate working methods that can be used depending on the situation and often driven by students. Additionally, he/she stimulates students to choose their own learning activities. The teacher explains the relationship between the learning goals and learning activities throughout the entire lesson. (excellent)


Lesson observation, video material and student placement evaluation form.

Suggested assessors Mentor, supervisor and pupils.


Deliverable D2.1

105

EPA 5. Tests to which extend the set learning goals have been met. Assessment and evaluation criteria

The teacher, if desired, does/does not make use of self-‐developed tests (incl. correction sheet). The teacher does/does not interpret (e.g. the contemplated results The teacher does/does not apply different tests (incl. correction sheets) as an instrument to diagnose and evaluate the learning of students and the self-‐learning.

Proficiency levels The teacher makes use of tests and correction models, which are offered by publishers and others. She/he makes use of evaluation forms, which only serve as an evaluation instrument. The teacher has little attention for the interpretation of the acquired test results. (starting) The teacher is aware of shortcomings of existing tests (incl. the correction models) and adapts them. The teacher interprets acquired test results and corrects, if needed, the grade accordingly. He/she occasionally includes other evaluation forms for the assessment. The teacher does not use the evaluation forms as a diagnostic instrument for the advancement of the learning of the students and self-‐learning. (sufficient) The teacher realizes when existing tests are inadequate and designs for that purpose a new test (incl. correction model). He/she interprets the ac-‐quired test results and offers suggestions for improvement. The teacher uses different evaluation forms as assessment instrument and, where rele-‐vant, as diagnosis instrument to promote the learning of students and self-‐learning. (excellent)


Tests (incl. correction sheets and analysis) and student placement evalua-‐tion form.



Deliverable D2.1

106

EPA 6. Engages in interpersonal relationships with (groups of) students Assessment and evaluation criteria

The teacher does/does not show interest in her/his own students. The teacher is/is not open to student initiatives. The teacher does/does not have an overview of the social relationships within a group and is/is not able to respond accordingly.

Proficiency levels The teacher shows little interest in what moves and motivates students. He/she provides little room for the students to come with own initiatives. The teacher has little regard for the social relationships in a group and has difficulty to respond to this. (starting) The teacher regularly shows interest in what moves and motivates stu-‐dents. The teacher regularly offers students the possibility to come with own initiatives. The teacher sees how different (groups of) students associ-‐ate with each other and is able to respond to this. (sufficient) The teacher knows the relevant background information of most students and knows, at a class level, what moves and motives students. Students can come with their own initiatives and the teacher makes use of this now and then. The teacher knows which place most students have in the social structure of the group and is able to respond to this. (good) The teacher is able to build good relationships with students due to having a permanent interest in the students, as individuals and as a group. He/she stimulates students to come with own initiatives and decides, in collabora-‐tion with the students, to make use of these. The teacher has a good insight into the social relationships in the group and is able to respond to this. (ex-cellent)


Interpersonal relationship questionnaire, notes, and student placement evaluation form.



Deliverable D2.1

107

EPA 7. Directs the communication processes in the group Assessment and evaluation criteria

The teacher does/does not communicate effectively related to different ways of leading, supervising, and confronting students. The teacher does/does not create a safe environment for the students in which they communicate and listen well. The teacher is/is not able to appropriately and with flexibility set behav-‐ioral norms and to reward and correct the behavior of students.

Proficiency levels The teacher still has difficulties to communicate effectively with students and is insufficiently aware what is going on in the class. Behavioral norms are not always clear. He/she only limitedly corrects and rewards and re-‐sponds too much or too little to reactions from the class. The teacher has, as a result, difficulties to create a safe environment in the class where stu-‐dents listen and communicate well towards each other. (starting) The teacher communicates well at a content level. He/she is conscious of the communication at a relational level. The teacher reward and corrects the students according to a fixed set of recognizable patterns. Based on personal behavioral norms, he/she creates a safe environment for the class where students can communicate and listen well towards each other. (suf-ficient) The teacher communicates effectively at a content and relational level. He/she can create a nice working atmosphere in different groups and lead-‐ing and supervising and rewarding and correcting different types of behav-‐ior maintain these. The teacher involves students when determining the norms of behavior and creates a safe environment with them where stu-‐dents can communicate and listen well towards each other. (good) The teacher and students communicate effectively at a content and rela-‐tional level. He/she can uphold a good working environment in the class by leading and supervising and reward and correct behavior with different intensity and with flexibility. The teacher makes the teachers responsible for the establishment of behavioral norms and creates a safe environment where students can communicate and listen well towards each other. (ex-cellent)


Interpersonal relationship questionnaire, notes, and student placement evaluation form.



Deliverable D2.1

108

EPA 8. Supervises the development of the student as a person Assessment and evaluation criteria

The teacher is/is not able to detect and pass on problems in the develop-‐ment of the student as person and the accompanied behavioral problems The teacher is/is not able to adequately supervise the development of the student as person and the possible complications (e.g. fear of failure, bully-‐ing). The teacher does/does not stimulate the students to think critically about their own perceptions and behavior.

Proficiency levels The teacher does not yet have a good overview of the development of the students as person and possible problems associated with it. He/she does not yet stimulate students to think critically about their own behavior and personal views. The teacher is not yet able to pick up everyday problems in the classroom. (starting) The teacher recognizes the development of the student as a person at the group level. She/he stimulates students to think about their own behavior and personal views. The teacher is, to a certain extent, able to supervise the daily problems during the development of the person of the student in and after class time. (sufficient) The teacher recognizes the development of the student as a person in indi-‐vidual students and the group as a whole. She/he regularly initiates group discussions where students are asked to critically reflect on their own be-‐havior and personal views. The teacher is able to supervise the personal development of individual students during and after class time from the perspective of their own developmental characteristics. Also in case of the-‐se being daily problems. (good) The teacher has a good insight in the development of the student as a per-‐son in individual students and the group as a whole. He/she stimulates students to discuss their own behavior and personal views. The teacher is, considering diversity and multiculturalism; able to supervise students with their personal development, also when they need specialized care. (excel-lent)


Lesson observation, video material, notes, and student placement evalua-‐tion form.



Deliverable D2.1

109

EPA 9. Carries out tasks that go beyond the lesson, class and subject Assessment and evaluation criteria

The teacher is/is not aware of the priorities of the school and educational policies. The teacher does/does not carry out a variety of additional tasks (e.g. test development, mentoring, policy making) The teacher does/does not show initiatives to carry out new tasks.

Proficiency levels The teacher is primarily focused on the primary teaching task, but is pre-‐sent at outside class and school activities. (starting) The teacher has an overview of the tasks in the school and carries out sev-‐eral tasks, which fit his/her own competences and the needs of the school. He/she contributes, when requested, to the execution and the vision and/or profiling of the school. (sufficient) The teacher carries out and is responsible for several tasks. He/she has, if necessary, trained to become a professional. The teacher represents the school professionally and contributes to the execution of the vision and/or profiling of the school. (good) The teacher takes initiatives or is a policy-‐maker for subject, team, and cross-‐curricular activities including (inter) national cooperation within and outside his/her school. (excellent)


Assignment subject didactics 2, notes, and student placement evaluation form



Deliverable D2.1

110

EPA 10. Collaborates with colleagues and, if necessary, parents and other stakeholders Assessment and evaluation criteria

The teacher is/is not able to work together with colleagues, and, if neces-‐sary, parents and other stakeholders. The teacher does/does not show initiatives to work together with others outside the direct school context to improve the education.

Proficiency levels The teacher primarily works together with others on the execution of her primary teaching task. He/she contributes to the subject cluster and the team. (starting) The teacher is conscious of his/her own qualities and what this means in cooperation with others. He/she has his/her own tasks within the subject group, the team and the school and coordinates the execution of these tasks with others. (sufficient) The teacher is constructively and actively involved with different forms of discussions at school and consciously applies her/his personal qualities. She/he professionally supervises students and collaborates, if necessary, with parents and other stakeholders. (good) The teacher systematically participates in and initiates different forms of discussions within and outside his/her own school. He/she is able to utilize his/her own quality as well as those of others in the cooperation process. (excellent)


Assignment subject didactics 2 and student placement evaluation form

Suggested assessors


Deliverable D2.1

111

EPA 11. Takes initiatives to improve his/her personal professional activities Assessment and evaluation criteria

The teacher is/is not able to make a well-‐grounded strength/weaknesses analysis of his or her own professional activities. The teacher does/does not attempt to carry out follow up activities target-‐ed to the improvement of his or her own professional activities. The teacher is/is not able to extent his/her own learning strategies.

Proficiency levels The teacher can describe his/her actions and incidentally is open for feed-‐back from the work placement supervisor, teacher educators and other teachers (in training). He/she is able to distinguish several cause and ef-‐fects, which might influence her/his actions. The teacher incidentally seeks applicable alternatives. He/she is not always conscious of his/her own learning activities. (starting) The teacher describes her/his own actions and accordingly regularly and actively seeks feedback from the work placement supervisor, the teacher educators, other teachers (in training) and students. She/he is conscious of behavioral patterns and the effects of these. The teacher regularly seeks, based on the given feedback, alternatives. He/she is conscious of his/her own learning activities, but does not take any steps to systematically ex-‐pand on these. (sufficient) The teacher actively and systematically asks for feedback based on his/her own learning goals and processes this feedback systematically in follow-‐up activities. He/she also involves the perspectives of the organization (school and team) and the students in his/her analysis. The teacher is based on his/her awareness of his/her behavioral patterns able to consciously make and steer appropriate steps. He/she is able to provide arguments for and expand partially his/her learning activities. (good) The teacher is able to choose independent of the situation different forms of reflection and is able to organize for him-‐/herself 360° feedback. He/she can, based on his/her awareness of his/her professional actions, make con-‐scious steps and is able to justify these from sources of knowledge. The teacher involves in his/her analysis different societal and theoretical per-‐spectives and shows an interaction between experience and practice theo-‐ry. He/she is conscious of his/her own learning strategies and behavioral patterns, is able to justify these from sources of knowledge, and is able to expand on these systematically. (excellent)


Reflection report and student placement evaluation form
