Embed Size (px)
Citation preview
Full Terms & Conditions of access and use can be found athttp://www.tandfonline.com/action/journalInformation?journalCode=imte20
Medical Teacher
ISSN: 0142-159X (Print) 1466-187X (Online) Journal homepage: http://www.tandfonline.com/loi/imte20
Benchmarking for research-related competencies– a curricular mapping approach at medicalfaculties in Germany
Maria Lammerding-Koeppel, Olaf Fritze, Marianne Giesler, Elisabeth Narciss,Sandra Steffens, Annette Wosnik & Jan Griewatz
To cite this article: Maria Lammerding-Koeppel, Olaf Fritze, Marianne Giesler, Elisabeth Narciss,Sandra Steffens, Annette Wosnik & Jan Griewatz (2018) Benchmarking for research-relatedcompetencies – a curricular mapping approach at medical faculties in Germany, Medical Teacher,40:2, 164-173, DOI: 10.1080/0142159X.2017.1395403
To link to this article: https://doi.org/10.1080/0142159X.2017.1395403
Published online: 16 Nov 2017.
Submit your article to this journal
Article views: 201
View related articles
View Crossmark data
Benchmarking for research-related competencies – a curricular mappingapproach at medical faculties in Germany
Maria Lammerding-Koeppela� , Olaf Fritzea� , Marianne Gieslerb, Elisabeth Narcissc, Sandra Steffensd ,Annette Wosnike and Jan Griewatza
aCompetence Centre for University Teaching in Medicine Baden-Wuerttemberg, Faculty of Medicine, University of Tuebingen, Tuebingen,Germany; bCompetence Centre for Evaluation in Medicine Baden-Wuerttemberg, Medical Faculty, University of Freiburg, Freiburg,Germany; cCompetence Centre of Final Year, Medical Faculty Mannheim, University of Heidelberg, Mannheim, Germany; dDean’s Officefor Medical Education, Hannover Medical School, Hannover, Germany; eDean’s Office of Student Affairs, Faculty of Medicine, Universityof Tuebingen, Tuebingen, Germany
ABSTRACTObjectives: Internationally, scientific and research-related competencies need to be sufficiently targeted as core outcomesin many undergraduate medical curricula. Since 2015, standards have been recommended for Germany in the NationalCompetency-based Learning Objective Catalogue in Medicine (NKLM). The aim of this study is to develop a multi-centermapping approach for curricular benchmarking against national standards and against other medical faculties.Method: A total of 277 faculty members from four German medical faculties have mapped the local curriculum against thescientific and research-related NKLM objectives, using consented procedures, metrics, and tools. The amount of mappingcitations of each objective is used as indicator for its weighting in the local curriculum. Achieved competency levels afterfive-year education are compared.Results: All four programs fulfill the NKLM standards, with each emphasizing different sub-competencies explicitly in writing(Scholar: 17–41% of all courses; Medical Scientific Skills: 14–37% of all courses). Faculties show major or full agreement inobjective weighting: Scholar 44%, scientific skills 79%. The given NKLM competency level is met or even outperformed in78–100% of the courses.Conclusions: The multi-center mapping approach provides an informative dataset allowing curricular diagnosis by externalbenchmarking and guidance for optimization of local curricula.
Introduction
Scientific and research skills and related attributes (attitude,ethos and way of thinking) (Laidlaw et al. 2012) are crucialfor medical doctors in daily clinical practice and the basisfor adequate medical decision-making and patient safety(Frenk et al. 2010; Stigler et al. 2010). Medical programshave to ensure a minimum competency level that can beacquired by each graduate, irrespective of potential careerplans (Laidlaw et al. 2012; van Schravendijk et al. 2013;German Council of Science and Humanities (WR) 2014).However, it has been argued repeatedly that research-related competencies have not been sufficiently targeted inmedical education (van Schravendijk et al. 2013; WR 2014;Nature Editorial 2015). They need to be defined more expli-citly as measurable core outcomes of medical curricula(Laidlaw et al. 2012; Nature Editorial 2015).
Internationally, this issue has led to an increasing trendtoward formal integration of scientific and research-relatedtraining into undergraduate medical and other healthcareprofessions education (Spencer et al. 2008; Griffin andHindocha 2011; Nikkar-Esfahani et al. 2012; Mabvuure 2012;Lawson McLean et al. 2013; Macleod et al. 2014). In orderto give better orientation by standards, competency-basedframeworks have been worked out in many countries fol-lowing the widely accepted CanMEDs model (e.g. Frank2005; Scottish Deans’ Medical Education Group 2008;
Working Group of the Swiss Medical Faculties 2008; Fischeret al. 2015). Among other core competencies, these guide-lines define competencies and milestones regarding med-ical scientific and research-related skills including theScholar role of medical doctor.
Despite these efforts, evidence is given that medical fac-ulties currently display various degrees of integratingresearch and scientific skills in the compulsory undergradu-ate medical curriculum (Hautz et al. 2016). In particular,
Practice points� Multi-center mapping against the NKLM reveals
great variety in weighting of research-relevantcompetencies.
� The mapping approach provides insights in cur-ricular status quo, starting discussions for targeteddevelopment.
� It enables benchmarking against other programs.� It is applicable for any competency comparison to
a given reference.� Multi-center datasets offer evidence-based guid-
ance for revision of a competency-basedframework.
CONTACT J. Griewatz [email protected] Competence Centre for University Teaching in Medicine, Baden-Wuerttemberg, Eberhard-Karls University, Elfriede-Aulhorn-Str. 10, Tuebingen D-72076, Germany�These author equally contributed to this work.� 2017 Informa UK Limited, trading as Taylor & Francis Group
MEDICAL TEACHER, 2018VOL. 40, NO. 2, 164–173https://doi.org/10.1080/0142159X.2017.1395403
views on what should be a minimum exposure for all med-ical students appear to differ (van Schravendijk et al. 2013).During the last decades, numerous faculties around theworld reformed their undergraduate medical programs.They used a wide variety of different approaches, adjustedtheir traditional medical curricula or implementedreformed/model educational programs (Frenk et al. 2010).Given the increasing mobility of medical students and prac-titioners, there is a growing need for harmonization ofmedical education and qualifications (Hautz et al. 2016).
Under these conditions, it is relevant to collect dataabout the current link between the medical curriculum andgiven standards. Benchmarking methods are often used asa common practice and sensible exercise in business enter-prises to establish baselines, gain (external) reference data,define best practices and identify improvement opportuni-ties (Camp 1989). Besides, these efforts can stimulate data-driven curricular communication inside the local institutionand/or with external partners. Benchmarking appears to bean appropriate approach in medical educational context, aswell.
The German situation shows important parallels to theabove international experiences; a national competency-based framework, including scientific and research-relatedstandards, has been worked out recently (Fischer and Fabry2014; Fischer et al. 2015; HRK 2016). Four German medicalfaculties with either a more traditional concept or with areformed curriculum, decided to face the problem above ina cooperative mapping project. For this they implementeda benchmarking approach. The National Competency-basedLearning Objective Catalogue in Medicine (NKLM) serves asan adequate referential framework (NKLM 2015). First, thedegree of matching shall be measured between the localeducational program and the newly given national stand-ards. Second, the quality of curricular performance shall becompared with other undergraduate medical faculties inorder to detect strengths, weaknesses and chances forimprovement and to determine one’s position in the fieldof others.
The superordinate research question in the present studyis whether a cross-university benchmarking approach in med-ical context will be helpful in quality management and cur-riculum improvement? The detailed research questions are:
1. Measuring the quality of curricular performance: Towhat extent do the local curricula fulfill the require-ments of the science and research-related NKLM-stand-ards after five years of studies (coverage of objectives,competency level)?
2. Comparison with other medical faculties: Do theundergraduate medical curricula currently display moresimilarities or more differences in respect to promotingscientific and research-related NKLM standards (specificlocal gaps and/or redundancies)? Do traditional orreformed medical faculties present these competenciesin training more intensely?
Methods
Sample
From June 2016 to January 2017 four German medical fac-ulties participated in the NKLM mapping project for
external benchmarking, led by the Competence Centre ofUniversity Teaching in Medicine Baden-Wuerttemberg(CCMD). All courses after five years of compulsory curriculawere evaluated. This study includes only data with regardto scientific and research-related competencies (NKLMchapter 6 resp. 14a; NKLM 2015).
Procedures
To gain meaningful data for benchmarking, consented rulesfor data collection, including procedures, metrics, and toolsas well as training and exchange were agreed upon by thefour faculties. The mapping progress from all partners waslogged continuously and monitored by supervisinglocal coordinator and central CCMD administrator.Recommendations for effective implementation of curricu-lum mapping, previously described in detail (Lammerding-Koeppel et al. 2017a, 2017b), were followed closely. Theprocedures are described briefly below.
Data collectionFor sake of content validity the mapping was conducted by277 faculty members from each discipline (47–80 individu-als per university), often preceptors with content-relatedexpertise of courses or medical teachers with educationalbackground, coordinating and/or supervising courses of thedepartment. Plausibility controls were carried out by dean’soffice, and consistency checks were done by the centralCCMD administrator.
Mappers’ trainingTo facilitate and harmonize the mapping process and dataquality, curriculum mappers and local supervisors wereinstructed by experts with educational, medical, and tech-nical background from the CCMD. Database manual andshort hands-on mapping instructions allowed easy practicalconsultation at any time. For fast troubleshooting, mapperswere supported via mail, telephone or by face-to-face map-ping assistance on request. Intramural and/or cross-univer-sity meetings like kickoff meetings, hands-on workshops,and interim meetings were organized, fostering exchange.
Mapping software and design
For data collection, the web-based MERlin-database wasused as common mapping tool, ensuring standardized dataformats and consistent data management (Fritze et al.2014). The database-driven application (PHP, MySQL) allowseach medical school entering its curricular data in a propri-etary data space. The database is structured in analogy tothe NKLM-hierarchy: competencies, sub-competencies, andlearning objectives.
Table 1 visualizes the basic scheme and categoriesincluding the varying granularity, when mapping a specificcourse in the MERlin-database (Note: only those mappingcategories relevant in this study are recorded in this sche-matic table). Options can be selected from pre-definedmenus. Mapping data are recorded mainly on sub-compe-tency level. The highest achieved level is categorized foreach course. Transparency in teaching is recorded: “explicit”means written in a study-guide, module manual or other
MEDICAL TEACHER 165
material; “implicit” means implied, but not documented inwriting. In order to analyze the extent of sub-competencycompleteness, the underlying objectives are ticked off incase they are taught (“objective covered”). The coverage ofobjectives is calculated automatically from number ofticked objectives (e.g. in Table 1).
For our specific study questions only explicitly taughtsub-competencies and underlying objectives are included;for characterization of curricula implicit data, as well, areinvolved (Table 2).
Terminology, data processing, and visualization
For the identification of specific (sub-) competencies andunderlying objectives in figures and tables, the NKLM chap-ter identification numbers are used; the textual content isshortened. Long versions of (sub-) competencies and objec-tives are listed in Table 3.
The amount of mapped courses in a faculty is specifiedas total number of curricular courses. Mapping citationsrefer to any objective taught in a course and ticked off bythe mapper. Teaching an objective on one or more occa-sions in a course corresponds to one citation. Since thenumber of curricular courses differs in the four faculties(Table 2), faculty’s mapping citations are calculated in % forcomparison of curricula. The percentage of citationsexpresses the curricular weighting of an objective: themore courses present an objective resp. superordinate sub-competency, the higher it is emphasized in the curriculum.
For benchmarking of a local curriculum against theNKLM, the percentage of citations (weighting) is displayedfor every objective (Figure 1(a,b)). The resulting datapoints of a faculty are connected by a line to facilitate datatracking and to visualize the characteristics of local
weighting profiles. To facilitate orientation and benchmark-ing against the other curricula (“own position in the groupof others”), the median of the weightings is calculated inevery objective category over the four faculties. For roughlypointing out high or low weightings, a “general median” iscalculated for each NKLM chapter. Serving as a chapter-related reference line, it includes all weightings from all fac-ulties (Figure 1).
In order to analyze the sub-competencies for differencesin mapping citations across the reformed and traditionalcurricula, the Mann–Whitney U test for independent sam-ples is used. The Kruskal–Wallis H test is conducted to com-pare sub-competencies across the four curricula regardingpotential differences in the amount of citations. Statisticaldata analyses are conducted using Excel (version 2010) andSPSS version 23 (SPSS, Chicago, IL).
For analyzing the quality of competency level achieve-ment, the highest levels achieved after five years of studiesare listed in Table 4, as reported by each faculty. They arecompared to the given NKLM levels as reference. Status ofquality is coded by gray-shaded table fields, thus allowingdiagnosis at a glance.
Results
General description of curricular status quo
The four German medical faculties participating in thisstudy are characterized by educational, structural, andorganizational differences. Table 2 gives an overview abouttheir general curricular status quo. Regarding the educa-tional concepts, two of them represent a more traditionaldisciplinary approach with some reformed elements inte-grated (Trad-1, Trad-2), whereas the other two follow areformed, modular and more interdisciplinary curriculum(Ref-1, Ref-2). All four faculties show a huge range of man-datory courses of various lengths. Overall, 85–100% of thecompulsory curricula were mapped after 5 years of study.Three of the four programs meet the objectives of Scholarand Medical Scientific Skills to a similar extent of explicitand implicit transparency. The only exceptions are detectedfirst in the Scholar chapter where Trad-2 shows a remark-ably low rate of explicit courses and a high rate of implicitcourses simultaneously; and second vice versa in the scien-tific skills chapter.
Diagnosis of curricular profiles
Each of the four curricula widely meets the NKLM objec-tives from Scholar and Medical Scientific Skills as well,
Table 1. Mapping citation pattern showing organizational hierarchy andmapping categories from this study. Competency level: 1¼ knowledge/understanding/basic skills; 2¼ applied knowledge and skills in training;3¼ competency in practice (3a¼ supervised, 3b¼ independent). For furtherinformation refer to the text.
Course identification and characteristics
Mapping data
NKLM Level Explicit Implicit Objective covered
Competency 1Sub-competency 1.1 2 � – 100%Objective 1.1.1 �Objective 1.1.2 �Objective 1.1.3 �
Sub-competency 1.2 3a – � 33%Objective 1.2.1 –Objective 1.2.2 �Objective 1.2.3 –
Table 2. Curricular status quo including all courses addressing scientific and research-related objectives as defined in the NKLM: Scholar (NKLM Chapter 6)and Medical Scientific Skills (NKLM Chapter 14a).
Scholar Medical Scientific Skills
Explicit Implicit Explicit Implicit
Total numbers ofcurricular courses
Courses Citations Courses Citations Courses Citations Courses Citations
Medical faculty n n % n n % n n % n n % n
Ref-1 173 65 37.6 227 58 33.5 181 63 36.4 289 6 3.5 13Ref-2 99 40 40.4 185 51 51.5 312 27 27.3 166 22 22.2 49Trad-1 201 82 40.8 368 99 49.2 378 42 20.9 156 48 23.9 114Trad-2 154 27 17.5 89 67 43.5 182 21 13.6 110 28 18.2 116
Ref: reformed curriculum; Trad: traditional curriculum
166 M. LAMMERDING-KOEPPEL ET AL.
Table 3. Full-text wording and weighting of research-related NKLM objectives (unofficial translation), including Scholar (NKLM Chapter 6) and MedicalScientific Skills (NKLM Chapter 14a). (Sub-)Competencies are displayed in shades of gray. As a preliminary estimate, weighting of faculties is differentiated in(a) tendency (high weighting, above the general median¼ì ; low weighting, below the general median¼î ; medium weighting, on/near the generalmedian¼è and (b) agreement between faculties (low agreement with high variance¼� ; split agreement with pairwise positioning¼ �; major agreementwith one clear outlier¼� ; and full agreement between all faculties¼�).
Chap. ID Competency level (degree of detail) Tendency Agreement
6.1. As lifelong learners, graduates maintain and improve their professional behavior bycontinuous learning.
6.1.1 They master the principles of learning in terms of identifying, reflecting on andmeeting their own learning needs as well as applying learning outcomes. Theyare able to …
6.1.1.1 Take responsibility for their continuing training to maintain and to further developmedical competencies by adequately assessing their stage of development andtaking appropriate measures if necessary.
ì �
6.1.1.2 Excerpt, save and manage gained information and its sources. ì �6.1.1.3 Constantly keep themselves informed about developments and changes in medicine
as well as relevant framework conditions and evaluate them.ì �
6.1.2 They design and document an adequate learning process based on a concretelearning question. They are able to …
6.1.2.1 Formulate a specific (learning-) question to assess themselves and others. ì �6.1.2.2 Search for, select and interpret relevant literature and other sources of information
with the help of appropriate research systems and effective search strategies.ì �
6.1.2.3 Adequately document their own learning process. è �6.2 Graduates critically evaluate scientific information and its sources applying this infor-
mation to their own professional behavior in a suitable manner.6.2.1 They use the principles of a critical evaluation of scientific sources. They are able
to …6.2.1.1 Name and recognize quality criteria of scientific work as well as disturbing
influences.ì �
6.2.1.2 Use methodological knowledge in planning and evaluating scientific studies. ì �6.2.1.3 Critically question possibilities and limits of gaining medical knowledge. ì �6.2.2 They collect and critically evaluate especially primary but also secondary evidence
concerning a medical problem. They are able to …6.2.2.1 Gather the scientific information necessary to answer medical questions. ì �6.2.2.2 Critically question the available information and assess it concerning its evidence for
their own research.ì �
6.2.3 They integrate conclusions based on critical evaluations into their daily medicalpractice. They are able to …
6.2.3.1 Critically question their own professional conduct in terms of content and methods. ì �6.2.3.2 Make evidence-based clinical decisions and critically evaluate data, at the same time
correlating it with the available evidence.ì
6.2.3.3 Present scientific data and findings in a commonly understandable manner. è �6.3 Graduates serve as teachers for different target groups (e.g. patients, students,
others).6.3.1 They recognize that the principles of learning are relevant to medical and other
health-care professions education and apply the principles in these fields. Theyare able to …
6.3.1.1 Apply important methods of teaching and learning used in adult education as wellas examination and evaluation techniques.
î �
6.3.1.2 Detect the learning needs of students, peers or members of other health-careprofessions.
î �
6.3.1.3 Choose and apply the best-fitting methods and contents for students, peers andmembers of other health-care professions.
î �
6.3.2 They recognize the relevance of learning principles in the education of patients andtheir family members and apply them. They are able to …
6.3.2.1 Recognize the learning needs of patients and their family members. è �6.3.2.2 Apply and evaluate appropriate procedures and contents for the education of
patients and their family members.î �
6.3.2.3 Communicate contents to patients or their family members (e.g. in the form of aconversation, presentation or in another appropriate manner).
ì �
6.3.3 They reflect and assess teaching-learning situations. They are able to …6.3.3.1 Evaluate and reflect on the progress of a teaching-learning situation. ì �6.3.3.2 Recognize their own limitations and skills in a teaching-learning situation. ì �6.3.3.3 Evaluate their own education for quality improvement. ì �6.3.3.4 Willingly give feedback to learners in concrete situations. ì �6.4 Graduates contribute as innovators to the creation, dissemination, application and
translation of new findings and practices.6.4.1 They internalize the principles of scientific research. They are able to …6.4.1.1 Reflect on the historical development of experiments, animal and human testing as
well as the epistemology of medical research.î �
6.4.1.2 Realize the ambiguity of current medical knowledge based on their knowledge ofthe main features of scientific theory.
î �
6.4.2 They are familiar with the ethical and legal principles of scientific research. They areable to …
6.4.2.1 Comply with the ethical and legal norms of good scientific practice. î �6.4.2.2 Identify the difference between descriptive, empirical, experimental, mathematical
and hermeneutic-interpretive methods of knowledge creation.î �
6.4.3 They contribute to the creation of new knowledge. They are able …6.4.3.1 Develop and formulate a research question and generate a testable hypothesis. î �6.4.3.2 Systematically gain information on the state of research. î �6.4.3.3 Plan and carry out a research project. î �6.4.3.4 Disseminate their own findings in an appropriate manner. î �
(continued)
MEDICAL TEACHER 167
Table 3. Continued
Chap. ID Competency level (degree of detail) Tendency Agreement
14a.1 As critical users of the principles and methods of evidence-based medicine, gradu-ates explain and apply these principles as part of the treatment of individualpatients.
14a.1.1 They collect and critically assess especially primary but also secondary evidence con-cerning a medical problem. They are able to …
14a.1.1.1 Translate problems into precise scientific questions, which are researchable in pro-fessional and literature databases.
ì �
14a.1.1.2 Search the literature for the best available evidence in relation to these problems. ì �14a.1.1.3 Critically evaluate the evidence found referring to these problems in terms of their
relevance and validity.ì �
14a.1.2 They use collected and critically evaluated primary and secondary evidence for deci-sion-making in their daily medical practice. They are able to …
14a.1.2.1 Present the collected and evaluated evidence to patients in an understandable man-ner and integrate it into the treatment process.
î �
14a.1.2.2 Explain and evaluate their own scientific and medical approaches to theseproblems.
î �
14a.1.2.3 Apply the methods of medical-clinical decision-making. ì �14a.1.2.4 Outline the validity criteria and explain the clinical benefits of guidelines and apply
them in daily practice.ì �
14a.2 Graduates contribute to the creation of new knowledge.14a.2.1 They develop a research question, formulate it and generate scientific hypotheses.
They are able to…14a.2.1.1 Explain different approaches when generating a hypothesis. è �14a.2.1.2 Explain different forms of hypotheses. è �14a.2.1.3 Transform a problem into a precise, researchable question. è �14a.2.1.4 Investigate, critically appraise and synthesize the current state of knowledge in rela-
tion to a question.î �
14a.2.1.5 Develop questions and deriving from them testable hypotheses considering the cur-rent state of knowledge.
î �
14a.2.2 They plan and conduct a scientific study. They are able to …14a.2.2.1 Explain and apply the ethical and legal framework of scientific work. è �14a.2.2.2 Explain and apply the guidelines of good scientific as well as good clinical practice. è �14a.2.2.3 Explain the framework conditions for different types of studies. î �14a.2.2.4 Perceive their own specialization/limitations and request additional expert opinions
if necessary.î �
14a.2.2.5 Name possible research objects, select and justify it scientifically. è �14a.2.2.6 Specify possible research methods (basic medical research, clinical research, epi-
demiology, evidence-based medicine, natural sciences relevant to human medi-cine as well as methods used in the humanities and social sciences), select andjustify them scientifically.
è �
14a.2.2.7 Operationalise research variables, select and justify the operationalizationscientifically.
ì �
14a.2.2.8 Explain and apply sampling techniques, derive and justify them scientifically. è �14a.2.2.9 Explain and apply experimental designs and justify the selection scientifically. è �14a.2.2.10 Explain and apply bias control techniques and justify them scientifically. ì �14a.2.2.11 Justify the necessity and name the preconditions of a case number estimate. î �14a.2.2.12 Take and document a measurement. è �14a.2.2.13 Apply the fundamentals of project management to their study. î �14a.2.3 They apply statistical methods of hypothesis testing appropriately. They are able
to …14a.2.3.1 Derive statistical hypotheses from research hypotheses. è �14a.2.3.2 Use appropriate procedures to test statistical hypotheses, derive and justify them
scientifically. They are able to recognize their own need for statistical advice andprofessionally prepare a consultation by a biometrician.
î �14a.2.3.3 Explain decision errors in statistical hypothesis tests, derive, and justify scientifically
as well as apply methods to minimize those errors.è �
14a.3 Graduates contribute to the dissemination, application and translation of new find-ings and practices.
14a.3.1 They present and discuss the results of a scientific study. They are able to …14a.3.1.1 Use different methods to present results in a textual, graphical and tabular form. ì �14a.3.1.2 Interpret and present the result of a statistical hypothesis test. ì �14a.3.1.3 Critically discuss the significance of a scientific study in terms of methodological
aspects.ì �
14a.3.1.4 Critically discuss the results of a study within the context of existing findings. è �14a.3.1.5 Present the knowledge derived from a study and discuss it critically regarding future
research needs.ì �
14a.3.2 They make findings of a scientific study available to medical practice. They are ableto …
14a.3.2.1 Reflect on the conflicts between the roles of practitioner and researcher. î �14a.3.2.2 Use terms and concepts from different disciplines relevant to medicine in a differen-
tiated manner and in line with scientific standards.î �
14a.3.2.3 Prepare scientific findings for a specialist audience according to the rules of scien-tific publications.
î �
14a.3.2.4 Give an oral presentation of scientific findings to a specialist audience. î �14a.3.2.5 Present scientific findings in a form understandable to laypeople. î �
168 M. LAMMERDING-KOEPPEL ET AL.
albeit at varying degrees (Figure 1(a,b)). For example, thecompetencies (C-) 6.1 (lifelong learning) and C-6.2 (evaluat-ing and applying scientific information) as well as partiallyC-6.3 (teaching different groups) obviously tend to rank inthe upper range of weightings, though having a wide scat-ter of values. In contrast, other competency such as C-6.4(contributing to innovations) clearly rank in the lower oreven lowest range of weightings in at least three of thefour programs.
In both chapters, every curriculum appears to locallyfocus on certain sub-competencies (SC-) or objectives (O-).Outliers lying either far beyond or far below the majoritygroup, reveal potential over- or underperformance in com-parison to the majority: e.g. Trad-2 significantly emphasizesSC-14a.2.1 (deriving a research question and generating sci-entific hypotheses) and a few objectives from SC-14a.2.2(planning and conducting a scientific study). However,among others, Trad-2 clearly shows low values in SC-14a.1.2 (Using evidence for decision-making). The overallanalysis reveals in both chapters that each faculty priori-tizes certain topics, sometimes single objectives; each
shows selective gaps. Thus each faculty makes its ownmarks resulting in a local profile.
Taken as a preliminary estimate, tendencies and consis-tencies of weightings are summarized in Table 3 at object-ive level. Comparing group valuation in both chapters, thefaculties reveal a great variety in tendency of curricularweighting and in consistency of faculties’ agreement: InScholar, the faculties show high weightings in 53% ofobjectives, but mostly only low or split agreement. In con-trast, in Medical Scientific Skills, the faculties tend to lowweighting in 37% and to medium weighting in 34% ofobjectives, but mostly with major or full agreementbetween the faculties. All in all, agreement is clearly higherin Medical Scientific Skills (79%) than in Scholar (44%).
Comparison between traditional and reformedprograms
The Scholar sub-competency SC-6.2.2 (assessing evidence)shows significantly more mapping citations in the reformed
Figure 1. Curricular profiles of research-related objectives based on faculties weightings. Competencies are specified in abbreviated form; underlying objectivesare identified by NKLM code numbers: e.g. 6.1.1.1 (For full-text wording see Table 3). Ref: reformed curriculum; Trad: traditional curriculum; Mdn: median; gen.Mdn: general median of a chapter.
MEDICAL TEACHER 169
group (p¼ 0.034). In the Medical Scientific Skills chapter,the traditional group shows significantly more mappingcitations than the reformed group for the SC-14a.2.1(p¼ 0.009). Testing differences in the amount of mappingcitations across the four faculties, moderate differences aredetected in SC-6.2.1 (p¼ 0.037), SC-6.3.1 (p¼ 0.013) and SC-6.3.3 (p¼ 0.029) (Kruskal–Wallis H test).
Diagnosis of competency level
To analyze status and outcomes of scientific and research-related promotion of students, it is relevant to evaluate thecompetency levels at a defined milestone in comparison tothe given NKLM-standards. This study refers to the givenmaximum level of competency that must have beenreached at the milestone of final practical year. The gray-shaded results (Table 4) clearly show that the facultiesmeet most of the NKLM recommendations exactly (lightgray fields), often recommendations were even outper-formed (dark gray). Fields of underperformance (white) caneasily be identified (e.g. Trad-2). One faculty (Ref-1) fulfillsthe requested competency levels of all sub-competencies.
Discussion
Our descriptive study examines in four German facultiesthe current curricular depiction of scientific and research-related competencies as defined by the NKLM. The facultiesare expected to meet the given national standards toensure educational comparability.
Global relevance of curricular transparency becomesclear in view of increasing mobility of medical doctors.Many national competency-based frameworks define aScholar role and certain scientific skills for medical doctors,but with considerable differences (Hautz et al. 2016).Unifying the national educational standards in these issueshas failed so far (Marz et al. 2013). Furthermore,
implementation of these components appears highly chal-lenging within already overloaded curricula, and facultycommitment to this differs strongly nationally and inter-nationally. In contrast, some medical faculties (e.g. Dukeand Stanford) already provide excellent scientific andresearch trainings for students (Laskowitz et al. 2010).
To the best of our knowledge, the methodologicalapproach to analyze and visualize multi-centric mappingdata is a new way to gain promising information about med-ical curricula. It supports local curricular diagnosis andbenchmarking with other programs. The results suggest thebeneficial impact on increasing transparency, gaining (exter-nal) reference data and reducing costs through sharing ofresources.
The MERlin-database allows conducting curricular map-ping of varying granularity: Differentiated course data arereported primarily at sub-competency level; the lowest levelof objectives is only addressed in marking objective thatare taught. This concept has proven its practicability inmedical workplace reality earlier (Lammerding-Koeppelet al. 2017b). It is accepted by mappers and allows provid-ing and processing large quantities of relevant data in aclear and efficient manner. In the following findings, under-lying causes and implications will be discussed.
Local program profiles and chances of improvement
The present mapping data from Germany demonstratefeasibility of achieving science and research-related learningoutcomes up to high competency levels. The results showthat the faculties explicitly cover NKLM research-relatedobjectives to a wide range, but at various degrees. Everyprogram highlights a few selected objectives from differentsub-competencies as a unique feature. However, whencomparing objectives either between reformed and trad-itional faculties or across all the four programs, only veryfew significant differences are detected. Regarding
Table 4. Competency level achievement after 5 years of study. Sub-competencies are specified in abbreviated form and NKLM code numbers (for full-textwording see Table 3). Competency level: 1 = knowledge/understanding/basic skills; 2 = applied knowledge and skills in training; 3 = competency in practice(3a = supervised, 3b = independent). Color-code: light gray=matching; dark gray=outperforming; white=underperforming.
ReferenceFaculties competency level
NKLM subcompetency NKLM level Ref-1 Ref-2 Trad-1 Trad-2
Chap. 6: Scholar 6.1.1 Mastering principles of learning in CME 3b 3b 3b 3b 3a6.1.2 Designing an adequate learning process 3b 3b 3b 3b 3b6.2.1 Evaluating critically scientific sources 3b 3b 3a 3b 3b6.2.2 Collecting and evaluating evidences 3a 3b 3b 3b 3b6.2.3 Integrating critically reflected evidences in medical
practice2 3a 3a 3b 2
6.3.1 Applying principles of learning in medical/health careeducation
2 3a 1 3b –
6.3.2 Applying principles of learning in patient education 2 3a 2 3b 3a6.3.3 Reflecting and assessing teaching-learning situations 3a 3b 3b 3b 3b6.4.1 Internalizing principles of scientific research 2 3b 3b 2 26.4.2 Recognizing ethical and legal principles in scientific
research3a 3a 3b 2 2
6.4.3 Contributing to creation of new knowledge 3a 3b 3a 3b 3aChap. 14a: Medical Scientific
Skills14a.1.1 Collecting and assessing evidence for a medical
problem3a 3b 3b 3b 3b
14a.1.2 Using evidence for decisions in daily practice 3a 3a 3a 3b 3b14a.2.1 Developing a research question 3a 3b 3b 3a 3a14a.2.2 Planning and conducting a scientific study 3a 3b 3a 3a 3a14a.2.3 Applying appropriate statistical methods 3a 3b 3a 3a 214a.3.1 Presenting and discussing scientific results 3a 3b 3a 3b 3a14a.3.2 Disseminating scientific findings 3a 3b 3b 3a 3a
CME: continuing medical education.
170 M. LAMMERDING-KOEPPEL ET AL.
competency level, the faculties widely meet the levels asrequired or even to a higher degree. Only few issues areunderperformed.
The profiles provide curricular transparency and enablefaculties to identify missing, shortcoming and/or (unin-tended) overrepresented objectives. Thus strengths, weak-nesses and local preferences, as well, become evident.However, it should be kept in mind that the extent of sub-competency performance in courses (“weighting”) mighthave diverse explanations. Facing high values, it has to beconsidered, e.g. whether redundancies and requestedresources are reasonable, or whether they are based onrandomly grown structures or systematically planned. Incase of low values, it may be considered e.g. whether it isbased on an informed decision or has passed unnoticedhitherto, or whether it should be corrected or remainunchanged. Anyhow, noticeable findings need to be clari-fied in detail by further curricular data analysis.
This study exclusively focuses on courses addressing theScholar and research-related objectives explicitly. Research-related content from other chapters and implicitly taughtcontent are excluded. Moreover, it is to be expected thatdiverse disciplines address this content, often without refer-ring to each other or to NKLM objectives and withoutrespecting an officially planned curriculum. This is oftendue to a lack of transparency (Harden 2001; Harden 2013).Including curricular and assessment mapping data as well,the faculties have the potential for transparent, systematic-ally structured programs promoting the didactic concept ofconstructive alignment (Biggs 1996). In literature, longitu-dinal approaches and early exposure of undergraduates toscientific and research-oriented topics are recommended;they foster students’ basic research capabilities and stimu-late their interest in the multifaceted role of the clinicianscientist (Roberts 2010; Laidlaw et al. 2012; vanSchravendijk et al. 2013; Giesler et al. 2016). Analyzing thecurricular status at different milestones will support toensure longitudinal competency development.
Benchmarking and implications
Our methodological approach allows curricular diagnosticin light of three perspectives. Comparison to any nationalcompetency-based framework may indicate status of cur-ricular alignment and local characteristics, as discussedabove. Benchmarking to other programs reveals a faculty’sposition in a group of others (compare Figure 1; Table 4).This can be conducted either in a sense of competitiveranking or for collaborative exchange to learn from eachother. Benchmarking may also be used in order to analyzeone’s performance against selected competitive institutions(e.g. traditional versus reformed curricula). With this inmind, benchmarking supports local data interpretation. Itmay stimulate outcome-driven discussions between stake-holders, including faculty members and students. Hence,curricular mapping is regarded as the starting point foreffective and efficient program development (Uchiyamaand Radin 2009; Zelenitsky et al. 2014; Rawle et al. 2017).
Internationally, mobility of medical graduates is increas-ing. However, medical curricula vary considerably betweencountries because of strong cultural and contextual differ-ences (van Schravendijk et al. 2013; Hautz et al. 2016).
Political and legal pressure is high to ensure that thesegraduates meet the requirements of practice. Consequently,an urgent need has been come up for consensus in learn-ing outcomes at all stages of training, including research-related competencies, at least in Europe (Marz et al. 2013).The challenge now is to promote further local and inter-national discussions on these topics. Cross-university map-ping projects and benchmarking studies promise to openexchange for new ideas and best practice examples andmight reveal relevant data for (international) programaccreditation.
Limitations
The descriptive and retrospective nature of this study is aninherent limitation. Furthermore, the curriculum mapreflects subjective perspectives of some representatives of adiscipline and might not reflect the full teaching reality. Inaddition, previous multi-center studies on medical teachers’understanding of medical doctor’s roles and concurrencewith official definitions revealed that their understanding ofseveral roles clearly differs from the given definitions(Ringsted et al. 2006; Griewatz et al. 2016; Renting et al.2016). This was particularly observed for the Scholar whichled to negative comments and resistance and might havenegatively influenced the mapping results.
The timing of our mapping (after 5 years of study) corre-sponds most largely to the milestone of graduation aftersix years of study. The final practical year is clearly clinicallyfocused, often completed outside the context of universityand has only very limited educational research compo-nents. Thus, our mapping results appear comparable to thecore learning outcomes of the two Bologna cycles Bachelorand Master together (Marz et al. 2013).
Conclusions and perspective
The four faculties reveal a great variety in curricular weight-ing of science and research-related content as well as inconsistency of faculties’ agreement. Alignment with theundergraduate competency framework is transparent. Anyinstitution may evaluate its commitment for an adequatecurricular link to Medical Scientific Skills and research by acomparison to a national competency framework.
Our approach can be applied to any other competencyof interest. It enables evaluation of curricular strengths andweaknesses and offers benchmarking opportunities andguidance for improvement. Recently eight more Germanmedical faculties have joined the curriculum-mapping pro-ject. It is expected that a meaningful dataset emerges lead-ing to evidence-based and sophisticated discussions onstatus quo, and on how the faculties should proceed locallyas well as nationally. It will be necessary to keep track oftheir further developments in order to ensure topicality ofthe database.
The presented data are indicators from the perspectiveof the taught curriculum. Students’ and administrative per-spectives are promising sources for control (Harden 2001).A critical and revealing insight into a local program and itseffectiveness is of high relevance for program reputation,resource economy and accreditation issues, as well. Using
MEDICAL TEACHER 171
the mapping data in quality assurance justifies the timeand effort invested in the mapping process by the faculty.
Acknowledgments
The authors would like to thank Dr. Amir Yousef (Tuebingen) for hisexcellent support with the statistical analysis, and Julia Eckel for herhelpful assistance with the data collection in Mannheim. We also wishto express our thanks to the entire MERlin Group for stimulating dis-cussions, to the deans of study and teaching as well as the staff of allthe deans’ offices of student affairs for their eminent support of themapping project. Particularly, we wish to thank all teaching coordina-tors and those responsible for modules for their energetic participationin the mapping process and the discussions. They contributed import-ant information and gave impulses for optimizing the instruments andthe process.
Ethical considerations
This research was performed in accordance with the Declaration ofHelsinki. The study does not involve any human subjects, includingresearch on identifiable human material and data. The study wasapproved by the deans of studies of all participating Faculties ofMedicine.
Disclosure statement
The authors report no conflict of interest. The authors alone areresponsible for the content and writing of the article.
Funding
The MERlin-Project (Phase I: 2012–2016, Phase II: 2017–2020) is fundedby the Federal Ministry of Education and Research of Germany (BMBF;reference numbers: 01PL12011A, 01PL17011A). The joint project of theMedical Faculties of Baden-Wuerttemberg (Freiburg, Heidelberg,Mannheim, Tuebingen, Ulm) is focused on medical education researchwithin a cooperative, statewide teaching network regarding compe-tency-based education and assessment.
Notes on contributors
Maria Lammerding-Koeppel, MD, is an anatomist with a Master’sdegree in Medical Education by Bern University, Switzerland. She isDirector of the Competence Centre for University Teaching in MedicineBaden-Wuerttemberg at the Faculty of Medicine, University ofTuebingen.
Olaf Fritze, Dr. rer. nat., is a biotechnologist, research associate at theCompetence Centre for University Teaching in Medicine Baden-Wuerttemberg at the Faculty of Medicine, University of Tuebingen.
Marianne Giesler, PhD, is a psychologist and business administrationgraduate. She is Director of the Competence Centre for Evaluation inMedicine Baden-Wuerttemberg at the Faculty of Medicine, Universityof Freiburg.
Elisabeth Narciss, MD, is research associate at the Dean’s Office ofStudent Affairs and coordinator of final year education at theCompetence Centre of the Final Year Baden-Wuerttemberg, MedicalFaculty Mannheim, University Heidelberg.
Sandra Steffens, MD, is a urologist, an Adjunct Professor for Urologyand responsible for the faculty development program for medicalteachers in Hannover Medical School in Lower Saxony, Germany.
Annette Wosnik, MD, is Head of the Dean’s Office of Student Affairs atthe Faculty of Medicine, University of Tuebingen.
Jan Griewatz, M.A., is a pedagogue, research associate and deputyhead at the Competence Centre for University Teaching in MedicineBaden-Wuerttemberg at the Faculty of Medicine, University of
Tuebingen. He is responsible for the faculty development program formedical teachers in Baden-Wuerttemberg.
Glossary
Curriculum mapping: According to Harden (2001), curriculummapping means the visual representation of a whole curricu-lum as an elaborate combination of many curricular facets (e.g.learning outcomes, course formats and content, and assess-ment). A curriculum map makes the scope, sequence and ele-ments of a curriculum as well their relationship transparent andaccessible to stakeholders (e.g. students, teachers, curriculumdevelopers, managers, profession, politics, and public). In thisway, it provides a basis for dialog and exchange between thesegroups about the actual curricular status and coordinateddevelopment, reflecting the overall objectives of the faculties.Online tools enhance the reach and effectiveness of curriculummapping.
ORCID
Maria Lammerding-Koeppel http://orcid.org/0000-0001-6980-9308Olaf Fritze http://orcid.org/0000-0002-3825-3703Sandra Steffens http://orcid.org/0000-0002-7478-3920Jan Griewatz http://orcid.org/0000-0002-9731-3171
References
Biggs J. 1996. Enhancing teaching through constructive alignment.High Educ. 32:347–364.
Camp R. 1989. The search for industry best practices that lead tosuperior performance. New York (NY): Productivity Press.
Fischer MR, Bauer D, Mohn K. NKLM-Projektgruppe. 2015. Finallyfinished! national competence based catalogues of learningobjectives for undergraduate medical education (NKLM) anddental education (NKLZ) ready for trial. GMS J Med Educ.32:Doc35.
Fischer MR, Fabry G. 2014. Thinking and acting scientifically: indispens-able basis of medical education. GMS J Med Educ. 31:Doc24.
Frank JR, editor. 2005 The CanMEDS phycisians competency frame-work. Better standards. Better physicians. Better care. Ottawa(Canada): The Royal College of Physicians and Surgeons.
Frenk J, Chen L, Bhutta ZA, Cohen J, Crisp N, Evans T, Fineberg H,Garcia P, Ke Y, Kelley P. 2010. Health professionals for a new cen-tury: transforming education to strengthen health systems in aninterdependent world. Lancet. 376:1923–1958.
Fritze O, Boeker M, Gornostayeva M, Narciss E, Wosnik A, Lammerding-Koeppel M. 2014. Competence-oriented curriculum mapping – aweb-based database as a powerful tool for curriculum development.Poster presented at: Annual AMEE Conference 2014; Milano, Italy.[accessed 2017 Sep 14]. http://www.amee.org/getattachment/f5f472e4-0df2-4941-9850-32a459fd705c/AMEE-2014-Abstract-Book.pdf.
Giesler M, Boeker M, Fabry G, Biller S. 2016. Importance and benefitsof the doctoral thesis for medical graduates. GMS J Med Educ.33:Doc8.
Griewatz J, Wiechers S, Ben-Karacobanim H, Lammerding-Koeppel M.2016. Medical teachers’ perception of professional roles in theframework of the German National Competence-Based LearningObjectives for Undergraduate Medical Education (NKLM) – a multi-centre study. Med Teach. 38:1157–1165.
Griffin MF, Hindocha S. 2011. Publication practices of medical studentsat British medical schools: experience, attitudes and barriers to pub-lish. Med Teach. 33:e1–e8.
Harden RM. 2001. AMEE guide no. 21: Curriculum mapping: a tool fortransparent and authentic teaching and learning. Med Teach.23:123–137.
172 M. LAMMERDING-KOEPPEL ET AL.
Harden RM. 2013. Curriculum planning and development. In: Dent JA,Harden RM, editors. A practical guide for medical teachers. 4th ed.2013. London (UK): Elsevier; p. 8–15.
Hautz SC, Hautz WE, Feufel MA, Spies CD. 2016. What makes a doctora scholar: a systematic review and content analysis of outcomeframeworks. BMC Med Educ. 16:119.
German Rectors‘Conference (HRK). 2016. On quality assurance for doc-toral degrees in medicine. Recommendation by the 21th GeneralAssembly of the HRK on 8th November 2016 in Mainz. Berlin: HRK.[accessed 2017 Sep 14]. https://www.hrk.de/fileadmin/_migrated/content_uploads/Empfehlung_MV_Zur_Qualitaetssicherung_der_Promotion_in_der_Medizin_08112016_EN.pdf.
Laidlaw A, Aiton J, Struthers J, Guild S. 2012. Developing research skillsin medical students: AMEE Guide No. 69. Med Teach. 34:754–771.
Lammerding-Koeppel M, Giesler M, Gornostayeva M, Narciss E, WosnikA, Zipfel S, Griewatz J, Fritze O. 2017a. Monitoring and analysis ofthe change process in curriculum mapping compared to thenational competency-based learning objective catalogue for under-graduate medical education (NKLM) at four medical faculties. Part I:Conducive resources and structures. GMS J Med Educ. 34:Doc7.
Lammerding-Koeppel M, Giesler M, Gornostayeva M, Narciss E, WosnikA, Zipfel S, Griewatz J, Fritze O. 2017b. Monitoring and analysis ofthe change process in curriculum mapping compared to thenational competency-based learning objective catalogue for under-graduate medical education (NKLM) at four medical faculties. Part II:Key factors for motivating the faculty during the process. GMS JMed Educ. 34:Doc6.
Laskowitz DT, Drucker RP, Parsonnet J, Cross PC, Gesundheit N. 2010.Engaging students in dedicated research and scholarship duringmedical school: the long-term experiences at Duke and Stanford.Acad Med. 85:419–428.
Lawson McLean A, Saunders C, Velu PP, Iredale J, Hor K, Russell CD.2013. Twelve tips for teachers to encourage student engagement inacademic medicine. Med Teach. 35:549–554.
Mabvuure NT. 2012. Twelve tips for introducing students to researchand publishing: a medical student’s perspective. Med Teach.34:705–709.
Macleod MR, Michie S, Roberts I, Dirnagl U, Chalmers I, Ioannidis JPA,Al-Shahi Salman R, Chan A-W, Glasziou P. 2014. Biomedical research:increasing value, reducing waste. Lancet. 383:101–104.
Marz R, Dekker FW, Van Schravendijk C, O'Flynn S, Ross MT. 2013.Tuning research competences for Bologna three cycles in medicine:report of a MEDINE2 European consensus survey. Perspect MedEduc. 2:181–195.
Nationaler Kompetenzbasierter Lernzielkatalog Medizin [NationalCompetency-based Learning Objective Catalogue] (NKLM). 2015.
[Internet] Berlin: MFT Medizinischer Fakultaetentag derBundesrepublik Deutschland. [accessed 2017 Sep 14]. http://www.nklm.de.
Nature Editorial. 2015. Ills of the system. Nature. 527:7.Nikkar-Esfahani A, Jamjoom AA, Fitzgerald JE. 2012. Extracurricular par-
ticipation in research and audit by medical students: opportunities,obstacles, motivation and outcomes. Med Teach. 34:e317–e324.
Rawle F, Bowen T, Murck B, Hong R. 2017. Curriculum mapping acrossthe disciplines: differences, approaches, and strategies. CELT.10:75–88.
Renting N, Dornan T, Gans ROB, Borleffs JCC, Cohen-Schotanus J,Jaarsma ADC. 2016. What supervisors say in their feedback: con-struction of CanMEDS roles in workplace settings. Adv Health SciEduc. 21:375–387.
Ringsted C, Hansen TL, Davis D, Scherpbier A. 2006. Are some of thechallenging aspects of the CanMEDS roles valid outside Canada?Med Educ. 40:807–815.
Roberts T. 2010 Catch them early. Med Teach. 32:193–194.Scottish Deans’ Medical Education Group. 2008. The Scottish doctor:
learning outcomes for the medical undergraduate in Scotland: afoundation for competent and reflective practitioners. [accessed2017 Sep 14]. http://www.scottishdoctor.org/resources/scotdoc3.pdf.
Spencer AL, Brosenitsch T, Levine AS, Kanter SL. 2008. Back to thebasic sciences: an innovative approach to teaching senior medicalstudents how best to integrate basic science and clinical medicine.Acad Med. 83:662–669.
Stigler FL, Duvivier RJ, Weggemans M, Salzer HJ. 2010. Health profes-sionals for the 21st century: a students’ view. Lancet.376:1877–1878.
Uchiyama KP, Radin JL. 2009. Curriculum mapping in higher education:a vehicle for collaboration. Innov High Educ. 33:271–280.
van Schravendijk C, M€arz R, Garcia-Seoane J. 2013. Exploring the inte-gration of the biomedical research component in undergraduatemedical education. Med Teach. 35:e1243–e1251.
Working Group of the Swiss Medical Faculties. 2008. Swiss catalogueof learning objectives for undergraduate medical training. 2nd ed.[accessed 2017 Sep 14]. http://sclo.smifk.ch/downloads/sclo_2008.pdf.
German Council of Science and Humanities (WR). 2014. Empfehlungenzur Weiterentwicklung des Medizinstudiums in Deutschland aufGrundlage einer Bestandsaufnahme der humanmedizinischenModellstudieng€ange. Dresden: WR. [accessed 2017 Sep 14]. http://www.wissenschaftsrat.de/download/archiv/4017-14.pdf.
Zelenitsky S, Vercaigne L, Davies NM, Davis C, Renaud R, Kristjanson C.2014. Using curriculum mapping to engage faculty members in theanalysis of a pharmacy program. Am J Pharm Educ. 78:139.
MEDICAL TEACHER 173
