CuRe Task Group 4 Informatics, Information Literacy and ... · PDF fileI2L/EBM Task Group Final Report 1(November 22, 2011) CuRe Task Group 4 Informatics, Information Literacy and

I2L/EBM Task Group Final Report (November 22, 2011) 1

CuRe Task Group 4 Informatics, Information Literacy and Evidence-based Medicine

I2L/EBM Executive Summary

Task group #4 was charged with determining how to incorporate information literacy, informatics, critical appraisal, and evidence-based medicine training elements into the new UGME curriculum. Skill in these domains is essential to support clinical practice following graduation and promote life-long learning capabilities. Task Group members took up the challenge fully. To begin, definitions for the three key terms, informatics and information literacy (I2L), and evidence-based medicine (EBM), were determined by consensus with the recommendation that these concepts be united under the umbrella term „Information Science‟. Five questions guided the work of the group:

1. How are information literacy, informatics, critical appraisal, and evidence-based medicine taught within the current curriculum?

2. Are there current published recommendations addressing these topics? 3. What is happening nationally/internationally in these domains? 4. How, where, when and by whom can these topics be best integrated into a new curriculum? 5. How should these topics be evaluated?

An environmental scan established the challenges and realities confronting medical educators in 2011 including the:

Explosion of new information

Digitization of information

Undeniable impact of the Internet, wireless technology and social networking

Emergence of a new generation of learners

Advent of new instructional technologies

Accelerated pace of technological change

Change in the public‟s expectations of their physicians

Overflowing medical curricula and associated lack of enthusiasm for introducing biomedical informatics education.

Aspects of the hidden curriculum were identified and labeled as „Myths & Realities‟. Of note, students enter medical school without a fully formed set of information literacy skills particularly in relation to their ability to assess the quality of information. This is compounded by students‟ self-assessment of their abilities, which may be incongruent with that of external expert observers. Further, faculty infrequently model the literacy skills expected of our medical students owing for some, to a personal lack of knowledge and skill regarding informatics, information literacy and evidence-based medicine and for others, a paucity of teaching strategies or opportunity to teach and integrate these domains into the practice of clinical medicine. Current curriculum content was assessed through an OPAL keyword search conducted on 30 terms representing I2L/EBM. Fourteen hours out of a total 1841 Pre-clerkship hours are spent on I2L/EBM and statistics activities (lecture, self-directed learning, tutorials), of which 90% is delivered in Med I/Block I and never revisted. ITC has 1 hour (new this year) and Clerkship rotations include three isolated assignments, one OB/GYN online EBM critical appraisal assignment, one Family Medicine quality improvement project, and one Multispecialty Rotation community health sciences report on a public health question. There are no informatics objectives in the curriculum. The curricular hours are disconnected. The content does not appear at meaningful points over the four years. Vertical and horizontal integration is completely absent, thus the relevance and applicability of this material is unclear to medical students.


An online survey of our PGY 1&2 residents (N=12/56) who trained at out-of-province medical schools and our undergraduate medical students (N=45) revealed a significant number of UM students learn I2L/EBM through trial and error. Active learning was seen as a “best way” to learn this material and role modeling by professors was considered an important learning strategy. An extensive literature search coupled with a review of Canadian Medical Schools (from websites and personal contact) was completed. Strategies for teaching I2L/EBM in various stages of development were found at the Universities of Alberta, BC, Calgary, Saskatchewan and Dalhousie. From all sources, overall general recommendations are clear:

1. Address information literacy, informatics and evidence-based medicine topics in the curriculum a. Introduce concepts, supporting knowledge and skills at points in the UGME curriculum that

will ensure progressive learning and maximum relevance 2. Embed information science throughout the curriculum rather than offering as a short course

a. Fully integrate information science into preclinical and clinical courses rather than as a stand-alone module within any course

b. Reinforce information competencies throughout the four-year program and in residency 3. Provide ongoing faculty development in I2L/EBM

a. Include „information science‟ faculty as part of the teaching team 4. Create an atmosphere that encourages and supports faculty to model the desired behaviors

a. Accord appropriate faculty status within the educational environment of the institution 5. Examine students for competency in these skills

a. Assess information science competencies as a routine part of student evaluation of any course

Opportunities for teaching, assessment and evaluation abound. Some examples include:

1. Pre-test incoming students‟ skill level and offer remedial online courses for those identified as having gaps

2. Assign Discovery Projects in pre-clerkship, link, clerkship and finishing course periods. Example examinable projects include having the student:

a. Choose a medical topic of interest for a selection, search the literature on that topic, decide on relevant articles, write-up an annotation for each article using citation management software. Answer a written exam question on that topic using and citing their annotated literature.

b. Create and present the contents of a focused Patient Education Portfolio (medical condition selected from a faculty-generated list) based on reviewing and selecting specific websites, patient oriented literature and items from the tertiary medical literature

c. Participate in a team systematic review of the literature following the Cochrane Handbook methodology and present to colleagues

3. Journal clubs - integrated with real-time clinical and academic activities including oversight by faculty experts who mentor, offer didactic support, teach the use of structured critical appraisal tools, and facilitate interactive discussion of clinical applicability

4. Online self-directed modules for core topics such as privacy, data security, information management, and clinical decision support

5. Library laboratory sessions and workbook activities with information science OSCEs to assess for higher level literacy skills

6. Introduction of one or two I2L/EBM objectives into every small group case-based problem solving session, for completion and discussion in the group setting

7. A Clerkship EBM Bootcamp/workshop (for example, using the Oxford model) on Critical Appraisal 8. Creation of a portfolio of PICO questions (patient/intervention/comparison/outcome)– one per

Clerkship rotation, generated from a patient exposure with a search of secondary and tertiary literature under faculty mentorship and application to patient care, evaluating the outcome

9. Working with an EBM preceptor for every clerkship rotation who teaches an academic half day early in the rotation, followed by oversight of the student‟s generation, approach to answering and application of knowledge to a clinical question rising from a real patient encounter


10. Post CaRMS finishing course to include learning how to apply the process of critical inquiry to patient care with presentation in a journal club or grand rounds format

11. Working in an inter-professional team (possibly within an educational EMR) to solve a healthcare system issue, incorporate a new practice into a healthcare setting, or manage change in a health service using real-life examples identified from experiences in clinical placements, a project that takes a more experiential holistic approach to evidence-based practice in order to prompt a change that makes clinical care more evidence-based.

Faculty development was seen as critical to the success of creating and adopting curricula in information science. A key issue identified by this Task Group was the disconnect between what is/will be (in the new curriculum) expected of students and the faculty practices they now observe with respect to application of information science knowledge, skill and judgment in the classroom setting and at the bedside. The role of information science training and its importance in the curriculum must be publically acknowledged and supported (including necessary infrastructure) at the level of the Dean‟s office in the Faculty of Medicine. Faculty arrive with great variability in their knowledge of, and skill level in I2L/EBM and need to be supported to self assess and build skills where required. The TG recommends the design, development and conduct of three distinct faculty-training modules:

1. Foundation information science course, mandatory for ALL UM Medical Faculty 2. Information literacy knowledge and skills course for pre-clinical small group & tutorial preceptors 3. EBM and critical appraisal course for clinical clerkship preceptors

This would allow uniformity and consistency in achieving the proposed information science competencies (see Appendix 1) and foster connections between faculty and students. The development of these „courses‟ should begin immediately so that faculty skills are well in place before the start of the new curriculum. Attention must be given to hiring educational technology personnel and acquiring software and hardware in support of reaching faculty in rural and community clinical placements. In addition, the Dean‟s office and UGME division should identify, support and provide resources for designated faculty stewards in information literacy, informatics and evidence-based medicine to oversee delivery of curricular elements, provide leadership in these areas and foster faculty collaboration locally and across distributed learning sites. The Task Group deliverables provided in this report include:

1. Information Science goals for the graduating physician 2. I2L/EBM competency based learning objectives (Appendix 1) 3. Recommendations for enablers to support an I2L/EBM curriculum 4. A detailed bibliography to support recommendations and learning objectives (Appendix 2).


CuRe Task Group 4: Informatics, Information Literacy, and Evidence-based Medicine

Committee Members

Judy Littleford Co-chair [email protected]

Tania Gottschalk Co-chair [email protected]

Karen Howell Project Manager – Faculty of Medicine [email protected]

Sadeesh Srinathan EBM – Critical appraisal [email protected]

Clare Ramsey EBM – Critical appraisal [email protected]

Diamond Kassum Informatics [email protected]

Jocelyn Advent OPAL – curriculum management [email protected]

Greg Van de Mosselaer Web instruction, web applications [email protected]

Brenda Stutsky CPD – Computer applications [email protected]

Dean Bell Information literacy [email protected]

Ken Zimmer Information literacy [email protected]

Timo Gosselin Med II student [email protected]

Elizabeth Berg Resident [email protected]

Mandate

1. To develop strategies and approaches to ensure the incorporation of information literacy, informatics, critical appraisal, and evidence-based medicine training elements into the new UGME curriculum, designed to support clinical practice following graduation and further life-long learning capabilities based upon:

A review of current curricular content in OPAL

A survey of PGY1s and PGY2s presently at UM who graduated from other Canadian medical schools

A survey of other Canadian and international medical school curricula by a review of websites, online documents, direct communication, and published articles

Review Academic Medicine September 2010 supplement 85(9) pages S615-643

A comprehensive review of the literature since 2001 using key terms: informatics, information literacy, evidence-based medicine, critical appraisal, electronic health or medical record, telemedicine/health, computerized physician order entry, clinical decision support, point of care tools, citation management, current awareness strategies, mobile technologies, critical appraisal, computer-based tools, e-learning, Web 2.0, medical education, curriculum, digital literacy, teaching and learning methodologies

Reading and selecting key and supportive documents from the literature review

Brainstorming amongst group members

In order to answer the following questions: i. How are information literacy, informatics, critical appraisal, and evidence-based medicine taught within the

current curriculum? a. Is the exposure comprehensive? b. Evaluated appropriately? c. Supported by various teaching modalities such as small group learning, case-based learning, or self-

directed study? ii. Are there current published recommendations addressing these topics? iii. What is happening nationally/internationally in these domains? iv. How could these topics be best integrated into a new curriculum, for example, separate horizontal course,

integrated into problem-based learning? a. Who would teach them? b. What would the curriculum elements look like? c. What teaching methodologies could/should be used?

v. How would you evaluate these topics in a new curriculum?

2. Develop principles and/or actual content depending upon the inclination of the group. All members of this group participated fully in the mandate.

mailto:[email protected]

mailto:[email protected]


Introduction Medical educators are facing new challenges and realities (Robin et al, 2011) including:

1. An explosion of new information 2. Digitization of information 3. A new generations of learners 4. The emergence of new instructional technologies 5. An accelerated pace of technological change

The transmission and organization of facts and information has changed forever owing to the undeniable impact of the Internet and wireless technology, social networking applications, the development of mobile devices and the processes of digital storage and retrieval. Medical education today should be less about accumulating facts that are a keystroke away and more about evaluating the veracity of the information and developing powers of interpretation and judgment (Mills, 2011). With medical curricula already overflowing, there has been a lack of enthusiasm for introducing biomedical informatics education (Shortliffe, 2010). Despite recommendations that health informatics be included in core undergraduate medical education activities, it has not penetrated most curricula in Canadian medical schools (Hurley et al, 2009). Yet, the call to introduce formal competency-based informatics, information literacy and evidence-based curricula has steadily gone from a quiet rumble to a mighty roar over the past decade (Stead et al, 2011). As noted by Shortliffe (2010):

Medical students need to learn about MEDLINE searching, the value and role of bibliographic databases as opposed to Google searches, the role of order-entry systems, electronic health records, regional data exchanges, telemedicine, and other important but current issues in the application of informatics in health and medicine. However, students also need to understand key underlying notions such as data privacy and confidentiality, conceptual errors in judgment, flaws in information and how to anticipate and deal with them, reasoning methods and opportunities for bias, principles of effective interactions between human beings and technology, and the role of human factors and usability research in the creation of effective, efficient, and accepted information aids. Students do not need to become computer programmers nor do they need to design systems, but they do need to be informed consumers who understand both the power and vulnerabilities of the tools they will be using in their practices.

The challenge is to blend such topics effectively into the existing medical education environment and to sustain understanding of these topics throughout residency training and beyond. There are curricular design obstacles, given the burgeoning number of relevant topics for future physicians, as well as challenges in identifying or recruiting faculty members who can teach such topics effectively and in a way that sustains a sense of relevance to students who are focused on their future clinical roles. Biomedical informatics is not a topic that is optimally taught in a single course during the preclinical years but rather should be blended into the 4-year curriculum with evolving topics and the use of clinical examples and challenges to motivate and direct the grasp of informatics concepts.

Librarians, informaticians, and evidence-based medicine advocates along with their respective professional organizations address the concepts of information literacy, informatics (I2L), and evidence-based medicine (EBM) using different but overlapping terminology, often proposing independent undergraduate curricular solutions. Exploration of the literature and interdisciplinary discussion highlights the need for consensus in defining I2L and EBM, terms that are often used interchangeably. Each represents an important but distinct facet of the life-long learning skills required by physicians.


Definitions used by the Task Group for this report Informatics Biomedical informatics is the interdisciplinary field that studies and pursues the effective uses of biomedical data, information, and knowledge for scientific inquiry, problem solving and decision-making, motivated by efforts to improve human health (Shortliffe, 2010; Mantas et al, 2010; Gardner et al, 2009). Information literacy IL is an understanding and set of abilities enabling individuals to recognize when information is needed and to have the capacity to locate, evaluate and use this information effectively. Information literacy training provides practitioners with tools to approach medical literature in an organized, efficient manner and to locate, evaluate, and use information effectively to accomplish a specific purpose. Information literacy forms the basis of lifelong learning. Competency in this area is broader in scope than skillful use of technological tools. An individual may be computer literate and skilled at working in an electronic environment but may not necessarily be information literate. The American Library Association defines IL as a set of abilities requiring individuals to „„recognize when information is needed and to have the ability to locate, evaluate, and use the needed information effectively.‟‟ (Demczuk, Gottschalk, & Littleford, 2009; Association of College and Research Libraries, 2000) Evidence-based medicine EBM is the conscientious, explicit, and judicious use of current best evidence in making decisions about the care of individual patients. Evidence-based clinical practice requires integration of individual clinical expertise and patient preferences with the best available external clinical evidence from systematic research, and consideration of available resources (Sackett et al, 1996). The process of evidence-based medicine and critical appraisal involves systematically examining the literature to judge its trustworthiness, and its value and relevance in a particular context, then applying the best available evidence gained from the scientific method to clinical decision making (Burls, 2009). We recommend these concepts be defined then united under the umbrella term, Information Science. Information Science Educational Goals Competency in all three domains of medicine, basic science, clinical science, and information science will be important in the education of future physicians in order to:

1. Better prepare physicians for the changing behaviors of patients, who are increasingly Internet-savvy, informed, networked/connected and questioning.

2. Provide physicians with tools to use in their interactions with patients to help them become more accountable for their health (empowerment).

3. Ensure physicians recognize the benefits of using I2L/EBM to improve the quality of interventions, health care delivery (locally and remotely), and the organization of health care systems and to advance global health initiatives.

4. Motivate physicians to develop expertise in using I2L and EBM in order to assume responsibility for their own continuing professional development.

5. Instill the notion that I2L/EBM skill is mandatory because the volume and complexity of knowledge has outstripped the ability of health professional to function optimally without the support of information management tools (Karsenti & Charlin, 2008; Kwankam, 2004).

Myths and Realities (The Hidden Curriculum) Myth #1 - Students of the net generation, also referred to as digital natives or millennials, enter medical school with a fully formed set of information literacy skills Reality – This statement is not supported by many studies that have evaluated the information literacy skills of medical students. Chen, Safdar, & Nagy (2011) sums up as follows:


We cannot assume incoming medical students already possess core competencies (fundamental skills that enable physicians to rapidly adapt to new technologies and to access, sort and synthesize information quickly) simply by being products of the digital age, thus standardized training should be mandatory. Myth #2 – Students are a good judge of their own skills in I2L/EBM Reality – Students‟ self-assessment of abilities are often not congruent with those of external expert observers. Recently published research addressing health information literacy and necessary competencies indicate the majority of students believe their research skills are good to excellent yet many are unable to conduct advanced information searches, judge the trustworthiness of health-related websites and articles and differentiate between various information resources. Students‟ self-reports may not be an accurate predictor of their actual health information competencies (Ivanitskaya et al, 2006). Myth #3 – A student will absorb a global understanding of what constitutes their professional body of literature by osmosis and, as a result, access appropriate evidence-based resources Reality – Speed and convenience influence the information seeking behaviour of digital natives (Connaway, Dickey, Radford, 2011). Google and Wikipedia are used by used by 80% and 70% of physicians respectively for both clinical decisions and medical education (Hughes et al, 2009). There is widespread concern about the risk of obtaining poor quality information with Web 2.0 use. Heilman (2011) notes, for example, that of the top 100 most viewed medical articles in Wikipedia only 24% were deemed of high quality: for the WikiProject Medicine portion as a whole this was less than 1%. Myth #4 – Faculty model information literacy and evidence-based medicine skills in clinical practice Reality – A recent (Maggio, Posley, and Fabbro, 2008) online survey of 15 faculty mentors (selected via a competitive process) at Stanford University School of Medicine revealed:

29% do not know what information literacy means or what skills are associated with IL 64% consider themselves information literate but feel they do not have the skills to teach others this domain 57% resort to decision-making with whatever information they already have when they hit a roadblock while searching for evidence-based answers 36% ask a colleague for assistance

Myth #5 - Informatics competencies are not applicable during preclinical courses and there is no place in clerkships to teach them Reality – Informatics competencies empower students to adapt to the growing reality of the influence of informatics on patient safety, team-based medical practice, and evidence-based care.


Answers to Key Questions 1. How are information literacy, informatics, critical appraisal, and evidence-based medicine

taught within the current curriculum? Is the exposure comprehensive? Evaluated appropriately? Supported by various teaching modalities such as small group learning, case-based learning, or self-directed study? An OPAL keyword search was conducted on 30 terms representing I2L/EBM. Since full-text searching of instructional materials in OPAL is not possible, we must assume that results retrieved are accurate reflections of the current curriculum. Current curricular elements include: Pre-Clerkship Medical Special

MD032 Orientation Tour to Library

MD033 Lecture – Connecting to Best Medical Information (1 hour)

MD034 2 Hour Library Lab Workshop – PubMed Training (2 hours)

Population Health

PH030, 31, 34, 35 – Using the Medical Literature I, II, III, IV (5 hours)

PH006, 008, 10 – Research Design I,II, III (3 hours)

PH026, 033 – Evaluating treatment I, II (2 hours)

PH036 – Translational Research (1 hour) Combines self-directed learning, lecture, and small-group tutorial

Ob/Gyn journal article critical appraisal assignment

ITC Introduction to Medical Student Toolkit and mobile devices in Library Lab (1 hour) Introduction to SBGH Physician Order Entry System in Computer Lab (3 hours) Clerkship Obstetrics and Gynecology rotation – one online EBM critical appraisal assignment Family Medicine rotation – address a quality improvement issue Multispecialty rotation – written paper researching a current public health question

Fourteen formal pre-clerkship instructional hours are spent on I2L/EBM out of a total of 1841 hours (LCME Medical Education Database 2009-2010). 90% of this content is delivered in Med I/Block I and never revisited. There are no informatics objectives in the curriculum. Information literacy and informatics skills are not formally examined in Pre-Clerkship. Clerkship has two isolated assignments and no curricular thread In the present state, the curricular hours are disconnected. There is a lack vertical and horizontal integration. Content does not appear at meaningful points over the four years, thus the relevance and applicability of the material is unclear to medical students.

2. Are there current published recommendations addressing these topics? Yes. These key articles summarize what is considered to be the most contemporary approach to information science:


Association of College and Research Libraries (2000). Information literacy competency standards for higher education. Accessed November 15, 2011 at: http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm

Campbell, C., Silver, I., Sherbino, J., Cate, O. T., & Holmboe, E. S. (2010). Competency-based continuing professional development. Medical teacher, 32(8), 657-62.

Contemporary issues in medicine--medical informatics and population health: report II of the Medical School Objectives

Project. (1999). Academic medicine : journal of the Association of American Medical Colleges, 74(2), 130-41. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10065054

Dawes, M., Summerskill, W., Glasziou, P., Cartabellotta, A., Martin, J., Hopayian, K., Porzsolt, F., et al. (2005). Sicily statement on evidence-based practice. BMC medical education, 5(1), 1.

Gardner, R. M., Overhage, J. M., Steen, E. B., Munger, B. S., Holmes, J. H., Williamson, J. J., & Detmer, D. E. (2009).

Core content for the subspecialty of clinical informatics. Journal of the American Medical Informatics Association : JAMIA, 16(2), 153-7.

Hunt, F., & Birks, J. (2004). Best Practices in Information Literacy. portal: Libraries and the Academy, 4(1), 27-39. doi:10.1353/pla.2004.0010

Hurd, P. D. (1998). Scientific literacy: New minds for a changing world. Science Education, 82(3), 407-416.

Iannuzzi, P. (2000). Information Literacy Competency Standards for Higher Education. Community & Junior College Libraries, 9(4), 63-67.

Mantas, J., Ammenwerth, E., Demiris, G., Hasman, A., Haux, R., Hersh, W., Hovenga, E., et al. (2010). Recommendations of the International Medical Informatics Association (IMIA) on Education in Biomedical and Health Informatics. First Revision. Methods of information in medicine, 49(2), 105-120.

Stead, W. W., Searle, J. R., Fessler, H. E., Smith, J. W., & Shortliffe, E. H. (2011). Biomedical informatics: changing what

physicians need to know and how they learn. Academic medicine : journal of the Association of American Medical Colleges, 86(4), 429-34.

3. What is happening nationally and internationally in these domains?

To answer this question, the Task Group: 1. Conducted a literature review 2. Reviewed websites of Canadian medical schools and contacted a select number of individual faculty

in medicine and library science 3. Surveyed PGY1 and PGY2 residents who graduated from other medical schools and are pursuing

postgraduate studies at UM. Literature Review A comprehensive bibliography of the references for each domain is attached. Task Group members reviewed and selected the articles listed. In terms of informatics it is clear that medical schools must (Triola et al, 2010):

Teach core biomedical informatics competencies that address health information technology, including explaining electronic medical record systems and computerized provider order entry systems and their role in patient safety

Describe the research uses and limitations of a clinical data warehouse

Present the concepts and importance of information system interoperability

Explain the ways clinical information systems can fail

http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm


Some medical schools have created portals that give medical students an opportunity to use and experiment with electronic medical records (Joe et al, 2009; Hart, Newton, & Boone, 2010). AFMC in partnership with Canada Health Infoway has initiated a project on eHealth curriculum and eLearning (http://chec-cesc.afmc.ca/community/afmc-infoway-physician-in-training-e-health-curriculum-e-learning). In terms of information literacy there are studies showing that courses and tandem teaching from librarians and physicians helps students to (Chen et al, 2009; O'Dwyer & Kerns, 2011):

1. Do better literature searches to answer questions 2. Use higher quality sources to gather evidence 3. Appropriately cite the literature in their written work more often

In general though the literature contains small studies with relatively short follow-up periods and no meaningful long-term outcome data. It is clear that medical schools need to (Ilic & Forbes, 2010; Brown & Nelson, 2003; Gruppen, Rana & Arndt, 2005; Porter et al, 2010; Slawson & Shaughnessy, 2005; Shurtz, 2009):

Address information literacy topics in the curriculum

Embed information literacy throughout the curriculum rather than offering as a short course

Provide ongoing faculty development in information literacy

Crete an atmosphere that encourages faculty to model the desired behaviors

Examine students for competency in these skills

Dawes et al (2005) have authored the Sicily statement on evidence-based practice, an international consensus paper that outlines the minimum standard educational requirements for an evidence-based medicine curriculum.

Several methods have been used to integrate EBM teaching into undergraduate medical curricula. Some novel ideas include a seminar approach using students as the teachers (Weberschock et al, 2005)), a six-week full-time course linking EBM with ethics and the management of change in health services (Rhodes et al, 2006), use of handheld devices during clerkship rotations to guide EBP at the bedside (Lam et al, 2004), submission of structured EBM report prompted by a clinical case (Thomas & Cofrancesco, 2001), and computer assisted self-directed learning (Bradley et al, 2005; Bradley et al, 2005) Most reported curricular strategies used a combination of a few lectures, followed by small group tutorials where the emphasis was on generating clinical questions. In some circumstances journal clubs were incorporated. In general, online instruction was uncommon but appeared to be successful when combined with mentorship to review a student‟s progress. Integrating EBM practice throughout medical school with distribution of content over multiple years is important. There is debate about the effectiveness of various EBM teaching and learning activities, and a lack of consensus as to best educational methods. Empirical and theoretical evidence suggests interactive and clinically integrated teaching and learning activities constitute the best educational approach (Khan & Coomarasamy, 2006). One of the main challenges for educators is how to assess competency in evidence-based practice (Ilic, 2009; Shaneyfelt et al, 2006). Review of Canadian Medical Schools Hurley et al (2009) as part of Dalhousie‟s curriculum renewal process conducted a survey of Canadian medical schools and found no medical school had a formal curriculum in health informatics. Our Task Group reviewed websites of Canadian medical schools in search of new developments since the Hurley report. A select number of individuals in Faculties of Medicine and Library Science were contacted. University of Alberta has 2 online courses called “Health Information Literacy” and “Health Evidence Literacy”. The latter is for the Faculties of Medicine and Dentistry. HIL focuses on informatics topics and includes six modules. Curriculum is integrated horizontally and vertically into the first three years. It is


competency-based and interactive. It moves through five levels (learner, reader, recorder, interpreter, and researcher) and includes short self-assessment quizzes and assignments. HEL combines elements of information literacy and evidence-based medicine. It includes four horizontally integrated courses timed throughout the four years using the core textbook “Users Guide to the Medical Literature”. Further reinforcement sessions are offered in academic half-days during clerkship. HEL has two required assignments, which are revisited again and again with different emphases over four years: a personal evidence project and a group evidence project. There are course and comprehensive examinations. An optional third course that includes clinical problem-solving challenges not addressed in small or large group teaching is made up of exclusively self-directed learning modules. University of British Columbia is currently in a curriculum renewal process. E-health, EBM, and technology content are new topics. The current curriculum contains ~15 hours of information literacy content integrated across all four years and delivered in lecture format, hands-on sessions, small group sessions with tutors, online assignments/tutorials, and formal evaluation organized by a librarian and clinician. University of Calgary estimates 15 to 30 hours of information literacy and informatics topics taught in a longitudinal population health course, “Healthy Populations” (Year 1) and “Applied EBM” (Year 2), by librarians and clinicians. The focus is primarily on evidence-based medicine and critical appraisal. Sample topics include: “how to read a journal article”, “how to incorporate EBM in practice”, and “how to search an online database”. Some workshops are given at key times focusing on how to integrate technology into the patient/doctor relationship (EMR) and software that facilitates learning (UpToDate). There is a faculty workshop for teaching, learning, and practicing EBM. University of Saskatchewan teaches a total of 11 hours of content distributed between year 1 and year 3 including such topics as: understanding the relevance of information literacy, developing a research question, and mobile computing devices in practice. There is an assignment in year 1 and in Clerkship rotations. About 15-20 hours of classroom time with most of the content taught in small groups plus a few lecture sessions. One unique project assigns a condition/disease to a student. They are asked to select someone in their life and pretend that person has the condition and the student becomes their primary resource for information. The student must create a portfolio of evidence-based information for the patient by searching databases, local resources, and patient education websites. Dalhousie’s curriculum renewal website indicates work in I2L/EBM is underway by two separate committees: health informatics (Dr. K. Hurley) and evidence-based practice (Dr. R. Whyte). The Curriculum Renewal Symposium documents are available at: http://symposium.medicine.dal.ca/material.htm Three three-hour informatics cases have been integrated into the curriculum: patient safety, chronic illness management and transitions of care. McGill underwent curriculum review in 2003 (according to their website) and implemented their changes over the ensuing three years to include EBM and related IL skills taught in second year. Their course consists of two two-hour lectures, four two-hour small group sessions, and one 1.5-hour e-classroom workshop. Students are required to hand in an EBM assignment. A “Mastering Medical Information Course” is taught in first year by librarians and includes a one-hour lecture and three-hour workshop. University of Western Ontario has an 18-hour elective block called health informatics in addition to a few hours of teaching in the first year. University of Ottawa has two sessions in their Introduction Unit titled “Introduction to the Technical Environment” and “Introduction to E-Learning”. In the remainder of pre-clerkship there are six additional sessions: “Introduction to EBM e-literacy”, “EBM Harm”, “Literature Searching”, “EBM Therapy”, “EBM Diagnostic Tests”, and “EBM Prognosis”. All are small group sessions and are integrated with contemporaneous medical subject matter. Each clerkship rotation deals with critical appraisal apropos the subject matter being discussed on the educational day for that rotation.

http://symposium.medicine.dal.ca/material.htm


Survey of PGY1 and PGY2 Residents and UM undergraduate medical students An online survey was created via Survey Monkey to obtain feedback regarding I2L/EBM teaching methods and distributed via email, with a one week response deadline, to all UM undergraduate medical students and all first and second year residents who completed their undergraduate education at a Canadian university other than Manitoba, and are completing their residency at UM. Number of respondents:

45 undergraduate medical students

12 residents (Surveys sent to 56) = 21.4% response rate PGME respondents graduated from 11 different universities including: Manitoba, Saskatchewan, Calgary, Alberta, UBC, UWO, McMaster, Ottawa, McGill, Memorial, and Ross (Caribbean). No respondents from NOSM, Sherbrooke, Dalhousie, or Queens. When asked how the student/resident learned/is learning about informatics, the top 5 of 10 possible responses included:

Response Resident Medical student

1. Through trial and error on my own 58% 89%

2. Through sessions taught by library staff 75% 71%

3. By completing my medical school assignments 58% 64%

4. By having one or more informatics classes in medical school 58% 58%

5. By actually doing it in clinical practice 83% -

When asked how the student/resident learned/is learning about information literacy, the top 5 of 10 possible responses included:



2. Through sessions taught by library staff - 51%

3. By completing my medical school assignments - 58%

4. By having one or more informatics classes in medical school 58% 51%

5. By actually doing it in clinical practice 75%

6. Through role modeling by professors 58%

When asked how the student/resident learned/is learning about EBM and critical appraisal, the top 5 of 10 possible responses included:


1. By having > 1 EBM/Critical Appraisal classes in med school 83% 84%


3. Through role modeling by professors 58% -

4. By completing my medical school assignments 75% 80%

5. Through online learning methods 58% -

6. By actually doing it in clinical practice 75% -

When asked what are the BEST WAYS to learn about informatics, information literacy, and EMB/critical appraisal, the top 5 of 19 possible responses included:


1. By having one or more classes in medical school 83% 69%

2. By having concepts incorporated throughout the curriculum 75% 73%

3. Through role modeling by professors 66% -

4. By being able to work through various self-directed modules 66% -

5. By actually doing it in clinical practice 66% 51%

We concluded from reviewing the survey results that:

A significant number of UM students learn I2L/EBM through trial and error

Active learning is seen as a “best way” to learn I2L/EBM


Role modeling by professors is an important learning strategy

4. How do you see these topics being best integrated into a new curriculum, for example, separate horizontal course, integrated into problem-based learning? Who would teach them? What would the curriculum elements look like? What teaching methodologies could/should be used?

5. How would you evaluate these topics in a new curriculum?

To begin answering question 4 and 5, we took a poll of the Task Group members by asking the following

question – If you could request one important I2L/EBM fix for the UGME curriculum, what would it be? Here are their

answers:

Teach, model and evaluate I2L/EBM across all four years in a spiral fashion

Place equal emphasis on Basic Science, Clinical Science & Information Science

Ensure sufficient integration and repetition in what is taught so the skills become second nature

Reinforce the concept that lifelong competencies in I2L/EBM are essential for professional practice

Include evaluation of I2L/EBM competencies in UGME examinations and assessments

Introduce I2L/EBM early and often and continue throughout PGME Curriculum Frameworks Task Group members wanted our I2L/EBM instruction to be congruent with current recommendations for how medical curricula should be delivered. We selected four curricular frameworks that resonated with Task Group members (AAMC, 2007; Harden, 2006; Hodges, 2010, O‟Connell, 2009), then proceeded to discuss answers to questions 4 and 5. Hodges (2010) lists four goals and three premises for medical education: Goals: 1. Standardize learning outcomes and individualize the learning process. 2. Integrate formal knowledge and clinical experience. 3. Develop habits of inquiry and innovation. 4. Focus on professional identity formation. Premises: 1. Learning is progressive and developmental.

a. Curricular structure connects formal and experiential knowledge and integrates longitudinally. b. Pedagogies accommodate individual learning needs; there should be longitudinal mentorship and

oversight and opportunities to direct one‟s own learning. c. The system of assessment should be an integrated one; progress should be tracked in multiple

domains. 2. Learning is participatory.

a. The curriculum sequences activities, allows contributions to patient care, and gives explicit attention to communication and collaboration.

b. There should be guiding and coaching, clear roles and responsibilities, and feedback. c. There should be performance-based assessment in a variety of domains, both individual and

collective. 3. Learning is situated and distributed.

a. Problems and experiences should be encountered multiple times in different contexts; there should be a focus on approaching problems, using information, and making visible hidden knowledge.

b. There should be guided observation and reflection, inquiry, and discovery. Discussions make assumptions transparent and congruent with the field.

c. Assessment includes the assessment of the use of resources, of social networks, and of the individual‟s contribution to collective performance.


O‟Connell (2009) compares his SPICES model to the traditional alternative:

SPICES Traditional approach

Student-centered Teacher-centered

Problem based Information gathering

Integrated Discipline based

Community based Hospital based

Electives with a core Uniform

Systematic Apprenticeship

Harden (2006) contained some guiding principles for medical education:

Effective feedback

Repetitive practice

A range of difficulty

Multiple learning strategies

Clinical variation

A controlled learning environment

Individualized learning

Lastly, a critical document reviewing educational technologies in medical school curricula provides expert guidance on when and how to employ technology during the medical education process (AAMC, 2007). Key content from this document includes:

A comparison of the advantages and disadvantages of various educational technologies

A listing of educational technologies best suited to accomplishing educational goals

Ten instructional multi-media principles guiding application of multimedia components in educational interventions to optimize student learning

Ten characteristics of high fidelity medical simulation that promote successful learning

A suggested instructional design framework

In summary, with respect to I2L/EBM explicitly, McGowan et al‟s approach (1998) itemizes most cogently the points considered in our group‟s discussion:

Concepts and supporting knowledge and skills need to be introduced at a point in the UGME curriculum that will ensure maximum relevance with both the educational process as well as the competency objectives defined for future practice.

Information Management instruction needs to be fully integrated into preclinical and clinical courses and not presented as a stand alone module within any course no matter how relevant the context of the presentation.

Course information competencies need to be reinforced throughout the four-year program and in residency where appropriate.

Assessment of the information management competencies needs to be frequent and handled as routine part of the student evaluation of any course.

Information faculty need to be part of the teaching team and accorded appropriate faculty status within the educational environment of the institution.

The three domains of Information Science naturally flow in sequence from learning and developing higher-level skills in information literacy, being introduced to biomedical informatics, and gaining knowledge of evidence-based medicine and skill in critical appraisal. This novice to expert evolution is paralleled over the years in medical school by the transition from theoretical patients introduced in pre-clerkship classroom activities, to the simulated clinical environment and finally to the experience of actual patients encountered in


clerkship clinical activities. The real questions of evidence-based practice and future ongoing professional development require real patients. The approach to teaching medicine is changing and will inevitably incorporate more technology, more self-directed online learning, more interprofessional exposure, and access to a more global perspective on best practice. Expanding our conception of teaching to incorporate these elements will incrementally enhance our educational enterprise but the core of our mandate – the human connection and rapport between faculty as teachers and scholars and our students as learners – must be preserved. Faculty Development A disconnect between students and faculty with respect to information science is a key issue identified by this Task Group. Our student members summarized the sentiments of their peers in answer to the question “Do my teachers think I2L/EBM is important in their lives and how do they model it?” as follows:

There is great variability in faculty knowledge about I2L/EBM

There is great variability in practical application

These skills are rarely taught or assessed and as a result seem to be the product of a variable path of self-discovery

Teachers usually don't take the time to reveal their own progress in learning, how they discover, evaluate and sort information or how I2L/EBM is of value to them in clinical practice and ongoing professional development

Faculty rarely critique students‟ I2L/EBM skill set in the classroom setting or on the wards

Faculty tend not to question students‟ sources of information nor recommend alternatives

A proportion of faculty are technologically naïve

There is lack of clarity regarding institutional commitment and resource allocation to I2L/EBM

It is difficult to practice I2L/EBM at the bedside without mobile devices and wireless connectivity The TG recommends the design, development and conduct of three distinct faculty-training modules:

4. Foundation information science course, mandatory for ALL UM Medical Faculty 5. Information literacy knowledge and skills course for pre-clinical small group & tutorial preceptors 6. EBM and critical appraisal course for clinical clerkship preceptors

This would allow uniformity and consistency in achieving the proposed information science competencies (see Appendix 1) and foster connections between faculty and students. The development of these „courses‟ should begin immediately so that faculty skills are well in place before the start of the new curriculum. Attention must be given to hiring educational technology personnel and acquiring software and hardware in support of reaching faculty in rural and community clinical placements. In addition, the Dean‟s office and UGME division should identify, support and provide resources for designated faculty stewards in information literacy, informatics and evidence-based medicine to oversee delivery of curricular elements, provide leadership in these areas and foster faculty collaboration locally and across distributed learning sites. Of interest, is a faculty development program - the Educators-4-Care at Stanford University (http://med.stanford.edu/e4c/, a Master Teacher/Faculty Mentor system led by Manitoba alumnus Dr. Charles Prober - that embeds amongst other things, an IL/EBM component into their faculty development program (Maggio and Posley, 2011). Background: The E4C Program was established to enhance the development of medical students as skilled and compassionate physicians. It provides a formal curriculum aimed to foster the development of some of our core values – Compassion, Advocacy, Responsibility, and Empathy – from the beginning and throughout medical school. Beginning in 2008, each incoming medical student was matched with an Educators-4-CARE faculty (15 faculty selected through a competitive application process), who serve as teacher, mentor, and colleague for the duration of the student‟s time at the School of Medicine. Faculty dedicate 2.5 days per week to the

http://med.stanford.edu/e4c/


program, teaching and guiding five to six students per class year in the following ways:

During the pre-clerkship years, precept students once per week in the Practice of Medicine (POM) course, cultivating students‟ acquisition and refinement of patient communication skills, physical examination skills, clinical reasoning, and professionalism

During the clerkship years, continue to provide guidance for students‟ bedside clinical skills and professionalism through semi-monthly Doctoring with CARE

Provide student mentoring and regular, periodic feedback throughout medical students‟ tenure at Stanford

Work with others to ensure that all Stanford medical students graduate with mastery of core clinical skills

Write letters of reference as requested

Collaborate with POM course directors, MD Advising Program, and Office of Medical Education leadership to assist in students‟ academic and professional development

Participate in student milestone events and celebratory gatherings Prior to the start of the E4C program, Maggio, Fabbro and Posley‟s (2009) online survey of E4C faculty revealed: 29% do not know what information literacy means or what skills are associated with IL 64% consider themselves information literate but feel they do not have the skills to teach others this domain 43% feel they find too much information when searching for evidence-based information 57% resort to decision-making with whatever information they already have when they hit a roadblock searching for evidence-based answers, 36% ask a colleague for assistance. As a result of the survey an IL/EBM faculty development plan was implemented to:

Provide targeted training for faculty, merging the concepts of IL and EBM

Define an information literate physician

Provide increased opportunities for faculty to practice and model IL skills

Offer training that fosters faculty skills in providing helpful IL feedback

Integrate IL/EBM skills into their medical school curriculum

Link student and faculty experience as closely as possible According to Chen (2011):

If medical schools are serious about teaching informatics, they need to spend resources to train medical educators on how to better incorporate the use of information technology into their treatments of the basic and clinical sciences. Without this step, we end up in the situation we are in today, in which many tout the need for informatics training, but few institutions have even taken the basic steps of declaring educational objectives geared toward this end.

In other words, we can‟t expect anything from students that is not taught and modeled by faculty (Thangaratinam et al, 2009). Established programs for training faculty in EBM exist at McMaster and Oxford Universities. Likewise informatics exposure and training is available online through the AMIA i10x10 program and in a blended fashion through the National Institute of Health Informatics in Canada (A 3-day bootcamp, followed by monthly webinars for 6 months). No such formal resource exists for information literacy, however there is local expertise amongst the NJM Health Sciences Librarian faculty, who are available to create a local solution for faculty training and development.

Teaching and Evaluation

http://med.stanford.edu/md/advising/

http://ome.stanford.edu/


There are many different examples of teaching strategies and evaluation methods in the medical education literature. To seamlessly weave information science into the new UGME curriculum, the framework and timing of its elements must be established. In general, the Task Group recommends adopting a mix of face-to-face, online, and blended learning strategies. Strategies can be loosely divided over the four years as follows: Discovery Projects Students submit one project in each of Preclerkship, Link and Clerkship periods (three in total). Example ideas:

Research Project with Exam – choose from a selection of medical topics, find relevant articles, and write up annotations for each article using citation management software. A final exam in which students answer a question on their chosen topic using and citing their annotated literature.

Patient Education Portfolio – create an educational portfolio for a family member with a medical problem evaluating and selecting patient-oriented articles, websites, books, and other tertiary literature

Team Systematic Review – generate a question and conduct systematic review of the literature following the Cochrane Handbook methodology. Present results to colleagues.

After Acceptance but Before First Day of Medical School Consider a pre-test to determine incoming students‟ information science knowledge and skill level. Offer remedial online courses for those identified as having gaps. Pre-clerkship –Information Literacy focus Focus on mastery of higher-level information literacy skills using assignments, exam questions, and information science OSCEs as part of the evaluation. Link period between pre-clerkship & clerkship – Informatics focus Example ideas:

Online self-directed modules covering selected core topics such as privacy, data security, information management, and clinical decision support. Successful completion required prior to clinical exposure

Mini lecture series to include for example telemedicine, PACS, pharmacogenomics

Formal introduction to Canada Health Infoway, EMRs and EPRs. Specific introduction to locally used systems. Training on simulated EMRs.

Clerkship – Evidence-Based Medicine focus Example ideas:

EBM preceptor for every clerkship rotation who oversees the delivery of a workshop early in the rotation then supervises the construction of an answerable question from the clerks actual patient exposure, the subsequent search for an answer in the literature, and the clerk‟s follow through with application to patient care.

Journal clubs - integrated with real-time clinical and academic activities. Oversight by faculty experts who mentor, offer didactic support, teach the use of structured critical appraisal tools, and facilitate interactive discussion of clinical applicability.

Mid-clerkship EBM bootcamp – two-day workshop (such as the Oxford model) on Critical Appraisal

Portfolio of clinical questions – one clinical question per Clerkship rotation, generated from a patient exposure. Search secondary and tertiary literature to determine the answer under faculty mentorship.


Post CARMS Example ideas:

Finishing course to include learning how to: o Apply the process of critical inquiry to patient care o Present at journal club o Present a grand rounds topic o Use presentation software and embed multimedia

Work in an interprofessional team to solve a healthcare system issue, incorporate a new practice into a healthcare setting, or manage change in a health service using real-life examples identified from experiences in clinical placements i.e. taking a more experiential holistic approach to EBP to prompt change that makes clinical care more evidence-based.

6. Information Science objectives (see Appendix 1)

The domains of informatics, information literacy and evidence-based medicine are both unique and complimentary. By developing the objectives in parallel and integrating like concepts together, we believe the Objectives document represents a more comprehensive approach to the study of information science and the required competencies than previously outlined in the published literature. Clarification of terminology will be essential for interdisciplinary communication and collaboration. The objectives will need to be organized into a matrix format (laddered approach, deliverables, milestones, timetable for evaluation and self-evaluation) that extends across all four years in accordance with an adult skill acquisition model such as the one proposed by Dreyfus (2004). The teaching and learning strategies used to deliver the material and test for mastery of the competencies will depend on decisions made by the Curriculum Renewal Steering Committee regarding the percentage blend of time-based versus outcome-based modeling in the new curriculum (Hodges, 2010). Hunt & Birks (2004), in their article, „Best Practices in Information Literacy‟ suggest the following task list be utilized when developing new curricula. Determine:

i. What are the aims and objectives? Task completed ii. What content should be included? iii. How should the content be organized? iv. What educational strategies should be adopted? v. What teaching methods should be used? vi. How should assessment be carried out? vii. How should details of the curriculum be communicated? viii. What educational environment or climate should be fostered? ix. How should the process be managed?


Appendix I - I2L/EBM Competency-based Learning Objectives (October 28th, 2011) Information science education must be an integral component in modern medical curricula, as important as training in basic and clinical sciences, to ensure graduates have the necessary tools for successful practice in today‟s world.

Students must have certain basic computer skills before they can develop I2L/EBM competencies. These skills should be assessed at the start of medical school and deficiencies should be addressed early in the first year. Basic computer literacy includes the ability to:

a) Access the Internet using a web browser and search using a search engine b) Send, receive, forward, and add attachments to email c) Use a standard word processing program to create, format and edit a document including inserting tables and

graphics d) Organize files and folders e) Save, print, and copy files

I2L/EBM objectives and associated competencies for Medical Education

1. Define evidence-based medicine

2. Describe the five-step process of evidence-based practice as defined in the Sicily Statement on Evidence-based Practice:

Translation of uncertainty to an answerable question

Systematic retrieval of best evidence available

Critical appraisal of evidence for validity, clinical relevance, and applicability

Application of results in practice

Evaluation of performance 3. Acknowledge that proficiency in information literacy is central to every physician‟s work in order to:

a) Recognize the need for medical information in a particular circumstance (for example, related to a patient

encounter or classroom learning, following consultation with colleagues/teachers, or via exposure to other sources like working groups, news media, and social networking)

b) Define and articulate the nature and extent of this information need, demonstrating the ability to:

i. Identify key concepts and terms that describe the information need

Translate colloquial English into medical terminology using: o An online medical dictionary o MeSH o Snomed CT o DSM IV o Metathesaurus – UMLS

Explain what controlled vocabulary is and why it is used

Find definitions of medical terms or abbreviations using: o An online medical dictionary o „Define:‟ feature of Google o MeSH

ii. Recognize that health information divided into various categories:

Patient Care o Clinical presentation/Diagnosis/Etiology/Therapy/Prognosis o Standards of Care/Guidelines o Prevention/Screening o Patient safety

Public and Population Health

Professionalism/Ethics/History

Legislation/Regulations/Policies/Health Systems

Professional Associations

Patient Education/Consumer Health


iii. Explore general medical information sources to increase familiarity with a new topic as needed in

the Pre-Clerkship years:

Acknowledge that Wikipedia is not a professional medical resource

Confine searching in Google to best medical evidence using: o Medical terminology o Advanced search feature o Phrase searching using quotation marks o Tilde & minus symbols o Boolean operators and parentheses

Access online evidence-based general information using tools in the Medical Student Toolkit (available at http://libguides.lib.umanitoba.ca/medicalstudent):

o Medline Plus o FirstConsult o eMedicine from WebMD o University of Maryland Medical Encyclopedia o Merck Manual

Navigate and locate materials in a variety of formats (for example, audiovisual resources, reference textbooks) in the physical Library via (Orientation/Amazing Library Race – Block 1):

o Library online catalogue o Pre-Clerkship Reserve textbooks list o Call numbers of the NLM Classification Scheme o Librarians at the Reference and Research Services desk o “Ask a Librarian” online chat services

iv. Employ online multi-media resources to support learning in:

Basic Medical Sciences

Clinical Skills

Information Sciences

v. Differentiate between various evidence-based resources or tools that might be helpful to address a patient care/clinical case information need in the physical library and online in the Medical Student Toolkit. Select the most appropriate resources for accessing the needed information.

Online textbooks, point of care tools or review articles published in journals for an overview of the general diagnosis and treatment of common medical conditions

o First Consult o Access Medicine o UpToDate o BMJ clinical evidence o PubMed limiting to Review

Medical calculator

High-quality online drug databases to access pharmacological information including class, mechanism of action, pharmacokinetics, contraindications/warnings, adverse effects, dosing/indications, and interactions

o Lexi-Comp o Natural Standard (herbs and supplements) o eCPS (drug availability in Canada)

Laboratory Tests o Clinical Laboratory Tests Normal Values (MCCQE) o Diagnostic Services of Manitoba Lab Manual o Access Medicine online Pocket Guide to Diagnostic Tests, 5e

Procedure Videos o NEJM videos in clinical medicine including: hand hygiene, pulse oximetry,

blood pressure measurement, diagnosing otitis media – otoscopy & cerumen removal, clinical evaluation of the knee, basic splinting techniques, peripheral intravenous cannulation , arterial puncture for blood gas analysis , ultrasound guided IJ vein cannulation

o Procedures consult


o Multimedia section of AccessSurgery & AccessMedicine

Clinical Practice Guidelines/Best Evidence/Systematic Reviews o CMA Infobase o AHRQ National Guidelines Clearinghouse o PubMed search limiting to Practice Guidelines or Systematic Reviews o Meta-search engines [TRIP database, SUMsearch] o Cochrane library o health-evidence.ca

vi. Recognize that smart phones and tablets offer physicians a platform to:

Access information to assist with timely patient care during a doctor-patient interaction o Recognize the inviolability of the physician-patient relationship; follow rules of

professional etiquette when using a device/technology to assist in patient care o Keep current with new apps and new developments in mobile technologies

using the blog, iMedicalApps.com o Download and use medical apps available through the NJMHS library o Explore medical apps available through the iTunes Store; share useful apps

with colleagues

Complete administrative tasks over the course of the day o Use apps that facilitate work productivity (email, calendar, contacts, telephone

and SMS o Download and populate a feed reader (e.g. Google Reader) with

news/blog/journal RSS feeds of interest

vii. Access websites available in the Medical Student Toolkit that explain the theory and practice of evidence-based medicine and supply critical appraisal tools for evaluating journal articles

Oxford CEBM

CEBM Toronto

JAMA Evidence

BMJ How to read a paper series

viii. Differentiate between various resources or tools that might be helpful to address a public health or population health need in the physical library and online in the Medical Student Toolkit

Statistics Canada

Canadian Institute for Health Information (CIHI)

Public Health Agency of Canada

Manitoba Health

US Centers for Disease Control and Prevention

ix. Differentiate between various resources or tools that might be helpful to address a patient education or consumer health need in the physical library and online in the Medical Student Toolkit

Health Canada

Medline Plus

UpToDate for Patients

Recognized evidence-based encyclopedias available in Library‟s Consumer Health Collection or at any bookstore:

o Harvard Medical School Family Health Guide o CMA Complete Home Medical Guide

x. Differentiate between various resources or tools that might be helpful to address a patient safety

issue in the physical library and online in the Medical Student Toolkit

Canadian Patient Safety Institute (CPSI)

Institute for Healthcare Improvement (IHI)

c) Formulate clinical questions

i. Differentiate between background and foreground questions


ii. Ask background questions (who, what when, where, why, how) to gain an initial

understanding of the topic and find clarifying information that will help in the formulation

of a foreground question

iii. Employ the PICO method to create a focused foreground clinical question

iv. Identify the domain (diagnosis and screening, prognosis, therapy [single trials, systematic

reviews], harm/etiology) to which the foreground question belongs

d) Search for information to address a foreground question, demonstrating the ability to:

i. Describe the differences between the results of a search using a general Web search engine (Google, Yahoo) and PubMed

ii. List the limitations of Google Scholar compared to PubMed in terms of finding/searching for current evidence-based information

iii. Construct and implement an effectively-designed search strategy in PubMed iv. Demonstrate a field search using Single Citation Matcher (author, title, journal, date) in

PubMed and indicate when it is appropriate to search a particular field v. Perform a PubMed search using appropriate commands (for example, logical [Boolean]

operators), in a manner that reflects understanding of medical language, terminology and the relationships among medical terms and concepts

vi. Refine search strategies to improve relevance and completeness of retrieved items (for example, use limits, controlled vocabularies [MESH], related articles feature)

vii. Assess the quantity, quality, and relevance of the search results to determine whether alternative information retrieval systems (using different user interfaces and search engines [Scopus, EMBASE, Web of Science, Cochrane Library, Google Scholar], with different command languages, protocols, and search parameters) or investigative methods should be utilized

e) Retrieve information online or in person using a variety of methods, demonstrating the ability to: i. Identify and link to full-text electronic documents available from the University of Manitoba

“virtual” library ii. Use “Get It @ UML” to link to full-text articles in PubMed, SCOPUS, EMBASE, Web of

Science, Cochrane Library iii. Bookmark the University of Manitoba Libraries proxy bookmarklet into Favorites Toolbar i. Link to full-text documents found in a variety of Internet locations (for example, Google,

Google Scholar, a professional association website, a publisher website) using the proxy bookmarklet

ii. Create a personal library using a citation management tool (Papers, Mendeley) iii. Download citations or documents into a personal citation management tool iv. Seek the assistance a librarian for large or systematic literature reviews where complex search

strategies and a more sophisticated platform (EndNote, Reference Manager, or Refworks) may be required to manage, manipulate and store large data sets.

v. Set up an account with Loansome Doc to request articles not held by the Library vi. Make requests for document delivery via Loansome Doc vii. Expand your search strategy to include materials available from professional associations,

community resources, practitioners and experts using:

Formal written consultation

Person to person contact (for example, telephone, email, telemedicine)

Letter writing

Interviews

Site visits

Surveys

f) Distinguish peer-reviewed from non peer-reviewed (non scholarly/popular) material

g) Evaluate the source of non-peer reviewed material (for example, web pages, newspapers, magazines) according to currency, relevance, authority, accuracy distinguishing:

i. Evidence from propaganda ii. Probability from certainty iii. Data from assertions iv. Rational belief from superstition


v. Science from folklore

h) Evaluate peer-reviewed information, demonstrating the ability to: i. Distinguish between primary, secondary, and tertiary literature ii. Distinguish between systematic reviews, meta-analyses, randomized control trials, evidence-

based guidelines, cohort studies, case series, non-systematic reviews of the literature, case reports, expert opinion

iii. Navigate the Oxford CEBM, CEBM Toronto, and/or JAMA Evidence websites to locate tools for critical appraisal

iv. Critically appraise a journal article using recognized evidence-based medicine techniques (for example, BMJ How to read a paper series)

v. Identify the factors that influence the accuracy, authority, timeliness, and validity of research/clinical information

vi. Search ClinicalTrials.gov to verify registration of a clinical trial vii. Identify quality, bias and errors in research methodology

i) Organize and manage a personal database of biomedical literature j) Edit incomplete bibliographic meta-data of articles stored in a personal database of biomedical literature

k) Filter and reconcile information of value to clinical practice, demonstrating the ability to:

i. Investigate differing viewpoints encountered in the literature, take steps to reconcile the

differences within your value system, and determine whether to incorporate or reject

viewpoints encountered

ii. Weigh conflicting information from several sources, extracting and selecting information that provides the best evidence for the topic

l) Determine if the original information need has been satisfied, if the initial query should be revised/refined

or whether additional information is needed

m) Revise the information need, question or search strategy as required and incorporate supplementary

concepts as necessary

n) Summarize the main ideas to be extracted from the information gathered for presentation in a professional

forum

4. Validate your understanding and interpretation of information retrieved when appropriate to do so, through

discourse with other individuals, subject-area experts, and/or practitioners, demonstrating the ability to: a) Participate in classroom, ward round, small group and other discussions b) Participate in forums designed to encourage discourse on the topic (for example, bulletin boards, chat

rooms, OPAL forums) c) Seek out expert opinion using a variety of communication techniques d) Integrate new information with previous information or knowledge

5. Inform medical decision-making with a combination of problem-solving skill; evidence-based practice; recognition

of a patient‟s unique biology, values, and circumstances; and, accumulated wisdom and experience within the context of the healthcare system and available resources

6. Enact evidence-based medicine in the course of patient activities

7. Keep a journal or log of activities related to the information seeking, evaluating, and communicating process, reflecting on past successes, failures, and alternative strategies

8. Exhibit ethical information habits, demonstrating the ability to: a) Make decisions based on evidence, when such is available, rather than opinion b) Demonstrate an understanding of copyright, fair use of copyrighted material, intellectual property c) Follow UM Copyright: Fair Dealing Guidelines (accessed April 11, 2011 at:

http://www.umanitoba.ca/admin/vp_admin/ofp/legal/index.html as they apply to copying of paper and electronic documents

d) Acknowledge that creators retain copyright, while allowing liberal use of their work using Creative Commons and open access resources

http://www.umanitoba.ca/admin/vp_admin/ofp/legal/index.html


e) Explain what constitutes plagiarism, employ practices to avoid plagiarism and represent work attributable to others appropriately

f) Follow UM Cheating, Plagiarism, and Fraud guideline (accessed April 12, 2011 at: http://www.umanitoba.ca/student/resource/student_advocacy/cheating_plagiarism_fraud.html

g) Properly cite and acknowledge information

Recognize the format of an information source (for example, book, chapter in a book, journal article, web page, interview)

Demonstrate awareness of the different citation styles frequently used in Medicine (Uniform, Vancouver, American Psychological Association)

Use the AMA style guide and Uniform style to write and cite

Identify verbatim material that can then be appropriately quoted

Obtain and post permission granted notices as needed for copyrighted material h) Make a distinction between privacy, confidentiality, and security of health information i) Comply with the law and regulations of Manitoba‟s Personal Health Information Act (PHIA) available at:

http://www.gov.mb.ca/health/phia/law.html j) Protect and respect the confidentiality of private information obtained from patients, colleagues, other

individuals and, from the medical record 9. Access and use information, information equipment and technology ethically and legally, following regulations and

institutional policies, demonstrating the ability to: a) Maintain a healthy skepticism about the quality and validity of information, being aware that information

can be lost or become corrupted b) Take appropriate preventative action to mitigate lost or corrupted information

Routinely employ backup procedures for personal and institutional data

Use anti-virus software

Notify computer network personnel to report suspected problems c) Employ network security procedures

Choose and maintain confidential, robust passwords

Log off network, electronic medical record, and dispensing systems upon completion of tasks d) Use the security-directed features of an information system to protect the integrity of electronic data e) Be aware there are policies that govern human subjects research

10. Employ professionalism with respect to personal and social presence in online environments

a) Analyze and monitor your online presence using standard Internet search engines to determine whether the particulars conveyed are consistent with the professional profile expected of a physician.

b) Remove (if possible) unbecoming information and going forward, consider every action online as permanent

c) Keep online personal profiles in secure and non-secure environments accurate, honest, and up to date. Use privacy controls to manage the personal parts of your online profile

d) Choose appropriate language, avoid libelous comments and defamation of character, and comply with federal and provincial personal health information and privacy legislation in all electronic communications

11. Retrieve patient-specific information from a clinical information system (for example, EMR) a) Demonstrate the ability to display, integrate, and incorporate selected subsets of the information about a

given patient into clinical decision making b) Identify and locate, when possible, crucial pieces of missing clinical information, and determine when it is

appropriate to act on incomplete information c) Integrate evidence-based sources of medical knowledge with the facts of a specific clinical case d) Make critical use of embedded decision support (point of care tools, online textbooks, online drug

resources, and advisories) accessible from a computer-based patient record e) Recognize these tools have power and vulnerabilities

12. Select and utilize information resources for professional education, demonstrating the ability to:

a) Employ instructional technologies

Podcasts

Webcasts/screencasts

Videoconferencing

Webinars b) Use online instructional tools, including tutorials and simulators

http://www.umanitoba.ca/student/resource/student_advocacy/cheating_plagiarism_fraud.html

http://www.gov.mb.ca/health/phia/law.html


c) Use online self assessment tools d) Interact using online collaboration and social networking tools e) Navigate professional websites for CME, licensing, and practice guidelines f) Develop a strategy for keeping current with advances in medicine using either email or RSS feeds or both

Describe the purpose of a RSS feed reader (for example, Google Reader)

Subscribe to a RSS feed from a journal(s), medical blog(s), or PubMed

Subscribe to an eToC alert from a journal

Create an account, develop a search strategy on a topic of interest, and save it to MyNCBI in PubMed in order to receive an email alert

13. Select and utilize information resources for patient education, demonstrating the ability to:

a) Navigate and evaluate consumer health-oriented websites b) Recognize prejudice, deception, manipulation and advertising

c) Identify downloadable patient-oriented educational materials d) Write a patient information prescription

14. Effectively employ written, electronic and oral communication, demonstrating the ability to: a) Use software to create visual materials that effectively support oral presentations (for example,

PowerPoint) b) Create a handout that includes graphics and tables for use in teaching or patient education c) Collaborate across multiple sites using email, discussion lists, instant messaging, videoconferencing, blogs,

wikis, and related communication technologies d) Cite the electronic communications policies of the local, provincial, and national organizations and

relevant medical professional associations e) Use proper etiquette in all forms of electronic, written, verbal, and face-to-face communication f) Access the elements and correct syntax of a citation for a wide range of sources (for example, webpage,

executive summary report, journal article, conference proceedings) using the AMA style guide g) Communicate clearly and with a style that supports the purposes of the intended audience h) Use a thesaurus and spell checker i) Utilize supplementary equipment (for example, audio or visual equipment) to enhance delivery of ideas

15. Use an electronic personal and clinical scheduling system

16. Explain the term “biomedical informatics” and be aware of the evolving role of this discipline in health, health care,

public health, and biomedical research 17. Compare and contrast the roles of various individuals in the health information technology workforce

18. Recognize the role of telemedicine in the delivery of healthcare

a) Classify the different types of telemedicine and their efficacy as shown in clinical studies b) Compare the technological capabilities and limitations of telehealth and videoconferencing systems in

patient care c) List the advantages and disadvantages of medical care delivered via telehealth d) Use telehealth etiquette to improve the quality of patient interactions

19. Describe the management of images in clinical settings, including the use of PACS systems 20. Recognize the evolving role of translational bioinformatics in the provision of “personalized” clinical care (for

example pharmacogenomics, genomics)

21. Explain what is meant by the terms electronic health record (EHR), electronic medical record (EMR) and personal health record (PHR)

22. Characterize the concepts of computerized provider order entry (CPOE), Clinical Decision Support tools, and

automated Reminders, Prompts & Alerts as they relate to the EHR 23. Recognize how information technology (CPOE, Clinical Decision Support tools, Reminders, Prompts & Alerts) can

be used to reduce medical errors and to develop, implement, and monitor compliance with clinical pathways and other forms of patient care protocols

a) Explain the process of CPOE and reflect on the challenges to its use


b) Distinguish the different types of clinical decision support tools and consider their utility and limitations in clinical practice

c) Give examples of preventative medicine strategies amenable to automated reminders and alerts in the EMR of an individual patient

d) Outline how CPOE, clinical decision support tools and automated electronic reminders/alerts impact patient safety

24. Maintain awareness of the activities of Canada Health Infoway and Manitoba eHealth to foster and accelerate the

development and adoption of electronic health record systems (EHR) that use compatible standards and employ interoperable communications technologies

a) In EHR design, know that compatible standards and interoperability refer to the unified and agreed upon pre-selection of common data identifiers, medical terminology, nomenclature & semantics, indexing & retrieval methodology, natural language processing of clinical text, checks & balances to ensure valid, accurate, quality data entry, and message exchange protocols for all creators/users of the system, along with issues of privacy, confidentiality and security

b) Explain the importance of standards and interoperability of clinical data c) Give examples of information system interoperability d) Observe the roles and responsibilities of medical associations, healthcare associations, government,

consumer groups and vendors in designing and implementing an EHR/EMR/EPR e) Identify the essential functions of an electronic health record (EHR) and the barriers to its use f) Acknowledge the enabling potential of an EHR system to evaluate aspects of health care delivery,

including clinical workflow analysis/process redesign to coordinate patient care, improve efficiencies, promote safety and reduce costs

g) Compare and contrast the terms EHR (birth to death record), electronic medical record (EMR – clinic, office, primary care record) and electronic patient record (EPR – institution record)

h) Compare and contrast the practice of medicine, including physician interaction with patients, documentation, patient/results follow up and office management in an electronic system versus a more traditional paper-based system

i) List the types of information contained in a typical EMR and EPR j) Elucidate the potential benefits of storing a patient‟s medical information electronically k) What types of information should be collected in order to improve a patient‟s care in terms of screening,

prevention, interventions, teaching, follow up, monitoring outcomes, continuity of care l) List the key attributes of an EMR and EPR that might improve its usability (for example layout,

organization of data, use of drop down lists, auto completion of entries, report generation) m) Identify the basic tenets of biomedical computing to enable optimal selection of hardware, software,

network connections, and personnel training for implementation of an EMR in a primary care office setting

n) Acknowledge that best practices for documentation in an EHR are not yet developed and dialogue between health professionals is ongoing on this topic

25. Define personal health record (PHR), identify possible practicable components of a PHR, describe its interface with

the EHR/EMR/EPR, and its value in promoting personal health

26. Define medical coding as the transformation of narrative descriptions of diseases, injuries, and healthcare procedures occurring in a patient‟s medical record into numeric or alphanumeric designations (code numbers). Enumerate common uses of coded health-related data

27. Recognize physicians have a responsibility to accurately capture narrative descriptions of diseases, injuries, and

healthcare procedures in their written and dictated notes in order to provide quality data for coding, statistical analysis and subsequent use in meeting public health and population health needs

28. Recognize how aggregated electronic clinical information can be used to determine health care service planning,

evaluate health care quality and make transformational change to the health care system, the purpose of which is to enable measurable improvement population health (improved clinical outcomes) [so-called meaningful use of the EHR]

29. Describe the uses and limitations of a clinical data warehouse as a research tool 30. Enumerate the ways clinical information systems can fail


31. Maintain awareness of changes in federal, provincial, and local legislation and health care policy that impact clinical practice using the following websites:

Manitoba Health

Health Canada

Winnipeg Regional Health Authority Intranet


Appendix 2: Bibliography Informatics and Informatics Teaching Methodologies Aiyer, M. K., Dorsch, J. L., Mintz, M., Torre, D. M., Thomas, M. R., Papp, K. K., & Durning, S. (2009). Medical informatics in the internal medicine clerkship: Results of a national survey. Medical Teacher, 31(8), e351-5. Bloomrosen, M., & Detmer, D. E. (2010). Informatics, evidence-based care, and research; Implications for national policy: A report of an American Medical Informatics Association health policy conference. Journal of the American Medical Informatics Association, 17(2), 115-123. Borycki, E. M., Armstrong, B., & Kushniruk, A. W. (2009). From prototype to production: Lessons learned from the evolution of an EHR educational portal. AMIA Symposium.AMIA Symposium, 2009, 55-59. Borycki, E. M., Kushniruk, A. W., Joe, R., Armstrong, B., Otto, T., Ho, K., Frisch, N. (2009). The university of victoria interdisciplinary electronic health record educational portal. Studies in Health Technology and Informatics, 143, 49-54.

Breil, B., Fritz, F., Thiemann, V., & Dugas, M. (2010). Multidisciplinary education in medical informatics--a course for medical and informatics students. Studies in Health Technology and Informatics, 160(Pt 1), 581-584. Buckeridge, D. L., & Goel, V. (2002). Medical informatics in an undergraduate curriculum: A qualitative study. BMC Medical Informatics and Decision Making, 2, 6. Bulik, R. J., & Shokar, G. S. (2010). Integrating telemedicine instruction into the curriculum: Expanding student perspectives of the scope of clinical practice. Journal of Telemedicine and Telecare, 16(7), 355-358. Burgun, A., Darmoni, S., Duff, F. L., & Weber, J. (2006). Problem-based learning in medical informatics for undergraduate medical students: An experiment in two medical schools. International Journal of Medical Informatics, 75(5), 396-402. Chatterley, T., & Chojecki, D. (2010). Personal digital assistant usage among undergraduate medical students: Exploring trends, barriers, and the advent of smartphones. Journal of the Medical Library Association : JMLA, 98(2), 157-160.

Chen, M., Safdar, N., & Nagy, P. (2011). Should medical schools incorporate formal training in informatics? Journal of Digital Imaging : The Official Journal of the Society for Computer Applications in Radiology, 24(1), 1-5.

Gardner, R. M., Overhage, J. M., Steen, E. B., Munger, B. S., Holmes, J. H., Williamson, J. J., & Detmer, D. E. (2009).

Core content for the subspecialty of clinical informatics. Journal of the American Medical Informatics Association : JAMIA, 16(2), 153-7.

Gjerde, C. L., Pipas, C. F., & Russell, M. (2004). Teaching of medical informatics in UME-21 medical schools: Best practices and useful resources. Family Medicine, 36 Suppl, S68-73. Hart, J. K., Newton, B. W., & Boone, S. E. (2010). University of Arkansas for medical sciences electronic health record and medical informatics training for undergraduate health professionals. Journal of the Medical Library Association : JMLA, 98(3), 212-216. Hurley, K. F., Taylor, B., Postuma, P., & Paterson, G. (2009). What are Canadian medical students learning about health informatics? Unpublished manuscript. Jaspers, M. W., Gardner, R. M., Gatewood, L. C., Haux, R., & Evans, R. S. (2007). An international summer school on health informatics: A collaborative effort of the Amsterdam medical informatics program and IPhiE--the international partnership for health informatics education. International Journal of Medical Informatics, 76(7), 538-546. Joe, R. S., Kushniruk, A. W., Borycki, E. M., Armstrong, B., Otto, T., & Ho, K. (2009). Bringing electronic patient records into health professional education: Software architecture and implementation. Studies in Health Technology and Informatics, 150, 888-892. Keenan, C. R., Nguyen, H. H., & Srinivasan, M. (2006). Electronic medical records and their impact on resident and medical student education. Academic Psychiatry : The Journal of the American Association of Directors of Psychiatric Residency


Training and the Association for Academic Psychiatry, 30(6), 522-527.

Krause, N. D., Roulette, G. D., Papp, K. K., & Kaelber, D. (2006). Assessing medical informatics confidence among 1st and 2nd year medical students. AMIA Annual Symposium Proceedings, 989. Kushniruk, A. W., Borycki, E. M., Armstrong, B., Joe, R., & Otto, T. (2009). Bringing electronic patient records into health professional education: Towards an integrative framework. Studies in Health Technology and Informatics, 150, 883-887. Kwankam, S.Y. (2004). What e-Health can offer. Bull World Health Organ. October; 82(10): 800–802. Liaw, S. T., & Gray, K. (2010). Clinical health informatics education for a 21st century world. Studies in Health Technology and Informatics, 151, 479-491. Lungeanu, D., Tractenberg, R. E., Bersan, O. S., & Mihalas, G. I. (2009). Towards the integration of medical informatics education for clinicians into the medical curriculum. Studies in Health Technology and Informatics, 150, 936-940. Mantas, J., Ammenwerth, E., Demiris, G., Hasman, A., Haux, R., Hersh, W., . . . IMIA Recommendations on Education Task Force. (2010). Recommendations of the international medical informatics association (IMIA) on education in biomedical and health informatics. first revision. Methods of Information in Medicine, 49(2), 105-120. Mills, B. (2011). The force of technology in the transformation of education. Educause Review, 46(6)(November/December), 8-9. Shaw, N. (2010). Medical education & health informatics: Time to join the 21st century? Studies in Health Technology and Informatics, 160(Pt 1), 567-571. Shortliffe, E. H. (2010). Biomedical informatics in the education of physicians. JAMA : The Journal of the American Medical Association, 304(11), 1227-1228. Stead, W. W., Searle, J. R., Fessler, H. E., Smith, J. W., & Shortliffe, E. H. (2011). Biomedical informatics: Changing what physicians need to know and how they learn. Academic Medicine : Journal of the Association of American Medical Colleges, 86(4), 429-434.

Triola, M. M., Friedman, E., Cimino, C., Geyer, E. M., Wiederhorn, J., & Mainiero, C. (2010). Health information technology and the medical school curriculum. The American Journal of Managed Care, 16(12 Suppl HIT), SP54-6. Information Literacy and Information Literacy Teaching Methodologies Association of College and Research Libraries. Information literacy competency standards for higher education. Accessed November 15, 2011 at: http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm Bobish, G. (2010). Participation and pedagogy: Connecting the social web to ACRL learning outcomes. The Journal of Academic Librarianship, Brown, J. F., & Nelson, J. L. (2003). Integration of information literacy into a revised medical school curriculum. Medical Reference Services Quarterly, 22(3), 63-74.

Buckley, S., Coleman, J., Davison, I., Khan, K. S., Zamora, J., Malick, S., . . . Sayers, J. (2009). The educational effects of portfolios on undergraduate student learning: a Best Evidence Medical Education (BEME) systematic review. BEME Guide No. 11. Medical Teacher, 31(4), 282-298. Chen, H. C., Tan, J. P., O'Sullivan, P., Boscardin, C., Li, A., & Muller, J. (2009). Impact of an information retrieval and management curriculum on medical student citations. Academic Medicine : Journal of the Association of American Medical Colleges, 84(10 Suppl), S38-41. Connaway, L. S., Dickey, T. J., & Radford, M. L. (2011). "If it is too inconvenient I'm not going after it:" Convenience as a critical factor in information-seeking behaviors. Library and Information Science Research, 33(3), 179-190.

http://www.ala.org/ala/mgrps/divs/acrl/standards/informationliteracycompetency.cfm


Contemporary issues in medicine--medical informatics and population health: report II of the Medical School Objectives

Project. (1999). Academic medicine : journal of the Association of American Medical Colleges, 74(2), 130-41. Retrieved from http://www.ncbi.nlm.nih.gov/pubmed/10065054

Craig, E. (2007). Developing online information literacy courses for NHS Scotland. Health Information and Libraries Journal, 24(4), 292-297. Cullen, R., Clark, M., & Esson, R. (2011). Evidence-based information-seeking skills of junior doctors entering the workforce: An evaluation of the impact of information literacy training during pre-clinical years. Health Information and Libraries Journal, 28(2), 119-129. Dadich, A. (2010). From bench to bedside: Methods that help clinicians use evidence-based practice. Australian Psychologist, 45(3), 197-211. Deenadayalan, Y., Grimmer Somers, K., Prior, M., & Kumar, S. (2008). How to run an effective journal club: A systematic review. Journal of Evaluation in Clinical Practice, 14(5), 898-911.

Demczuk, L., Gottschalk, T., & Littleford, J. (2009). Introducing information literacy into anesthesia curricula. Canadian Journal of Anaesthesia , 56(4), 327-335. Ellaway, R., & Masters, K. (2008). AMEE guide 32: E-learning in medical education part 1: Learning, teaching and assessment. Medical Teacher, 30(5), 455-473. Grunwald, T., & Corsbie-Massay, C. (2006). Guidelines for cognitively efficient multimedia learning tools: Educational strategies, cognitive load, and interface design. Academic Medicine, 81(3) Heilman, J. (2011). Why we should all edit Wikipedia. UBC Medical Journal, September 3(1), 32-33. Hughes, B., Joshi, I., Lemonde, H., & Wareham, J. (2009). Junior physician's use of Web 2.0 for information seeking and medical education: a qualitative study. International Journal of Medical Informatics, 78(10), 645-655. Hull, D., Pettifer, S. R., & Kell, D. B. (2008). Defrosting the digital library: bibliographic tools for the next generation web. PLoS Computational Biology, 4(10). Hunt, F., & Birks, J. (2004). Best practices in information literacy. Portal, 4(1), 27-39.

Hurd, P. D. (1998). Scientific literacy: New minds for a changing world. Science Education, 82(3), 407-416.

Iannuzzi, P. (2000). Information Literacy Competency Standards for Higher Education. Community & Junior College Libraries, 9(4), 63-67.

Ivanitskaya, L., Boyle, I., & Casey, A. M. (2006). Health information literacy and competencies of information age students: Results from the interactive online research readiness self-assessment (RRSA). Journal of Medical Internet Research, 8(2) Kaplowitz, J. R., & Yamamoto, D. O. (2001). Web-based library instruction for a changing medical school curriculum. Library Trends, 50(1), 47-57. Karsenti, T., & Charlin, B. (2008). Information and Communication Technologies (ICT) in Medical Education and Practice: The Major Challenges. Revue Internationale Des Technologies En Pédagogie Universitaire, 5(2), 68-81. Karshmer, E., & Bryan, J. E. (2011). Building a first-year information literacy experience: Integrating best practices in education and ACRL IL competency standards for higher education. Journal of Academic Librarianship, 37(3), 255-266. Kingsley, K. V., & Kingsley, K. (2009). A case study for teaching information literacy skills. BMC Medical Education, 9, 7. Lasserre, K. E., Foxlee, N., Kruesi, L., & Walters, J. (2011). Health sciences librarians' research on medical students' use of information for their studies at the medical school, university of Queensland, Australia. Medical Reference Services Quarterly, 30(2), 141-157.


Lawrence, J. C., & Levy, L. S. (2004). Comparing the self-described searching knowledge of first-year medical and dental students before and after a MEDLINE class. Medical Reference Services Quarterly, 23(1), 73-81.

Luo, L. (2010). Web 2.0 integration in information literacy instruction: An overview. The Journal of Academic Librarianship, 36(1), 32-40. Lynn, V. A. (2010). Foundations of database searching: Integrating evidence-based medicine into the medical curriculum. Medical Reference Services Quarterly, 29(2), 121-131. Massis, B. E. (2011). Information literacy instruction in the library: Now more than ever. New Library World, 112(5), 274-277. Masters, K., & Ellaway, R. (2008). e-learning in medical education guide 32 part 2: Technology, management and design. Medical Teacher, 30(5), 474-489. McCord, G., Smucker, W. D., Selius, B. A., Hannan, S., Davidson, E., Schrop, S. L., Albrecht, P. (2007). Answering questions at the point of care: Do residents practice EBM or manage information sources? Academic Medicine, 82(3), 298-303. Monheit, D. F. (2011). Evaluating health information web sites for credibility. Journal of Hospital Librarianship, 11(1), 39-44. Nicholas, D., Rowlands, I., Clark, D., & Williams, P. (2011). Google generation II: Web behaviour experiments with the BBC. Aslib Proceedings: New Information Perspectives, 63(1), 28-45. O'Dwyer, L., & Kerns, S. C. (2011). Evolution of an information literacy curriculum for third-year medical students. Medical Reference Services Quarterly, 30(3), 221-232. Park, S., Mardis, L. A., & Ury, C. J. (2011). I've lost my identity - oh, there it is... in a style manual: Teaching citation styles and academic honesty. Reference Services Review, 39(1), 42-57. Phua, J., & Lim, T. K. (2007). Use of traditional versus electronic medical-information resources by residents and interns. Medical Teacher, 29(4), 400-402. Porter, J. A., Wolbach, K. C., Purzycki, C. B., Bowman, L. A., Agbada, E., & Mostrom, A. M. (2010). Integration of information and scientific literacy: Promoting literacy in undergraduates. CBE Life Sciences Education, 9(4), 536-542. Rowlands, I., Nicholas, D., Williams, P., Huntington, P., Fieldhouse, M., Gunter, B., Tenopir, C. (2008). The google generation: The information behaviour of the researcher of the future. Aslib Proceedings: New Information Perspectives, 60(4), 290-310. Saranto, K., & Hovenga, E. (2004). Information literacy - what it is about? literature review of the concept and the context. International Journal of Medical Informatics, 73(6), 503-513. Sastre, E. A., Denny, J. C., McCoy, J. A., McCoy, A. B., & Spickard, A.,3rd. (2011). Teaching evidence-based medicine: Impact on students' literature use and inpatient clinical documentation. Medical Teacher, 33(6), e306-12.

Schardt, C. (2011). Health information literacy meets evidence-based practice. Journal of the Medical Library Association : JMLA, 99(1), 1-2. Schwartz, B. (2011). The introduction of a medical informatics course into a medical school curriculum. Medical Reference Services Quarterly, 30(1), 74-82. Schwartz, M. D., Dowell, D., Aperi, J., & Kalet, A. L. (2007). Improving journal club presentations, or, I can present that paper in under 10 minutes. Evidence Based Medicine, 12(3), 66. Seago, B., Gyore, R., & Williams, S. (2007). Assessment of the integration of AAMC medical informatics objectives into the medical school curriculum. Medical Reference Services Quarterly, 26(2), 1-14. Shurtz, S. (2009). Thinking outside the classroom: Providing student-centered informatics instruction to first- and second-year medical students. Medical Reference Services Quarterly, 28(3), 275-281.


Spring, H. (2010). Health professionals of the future: Teaching information skills to the google generation. Health Information and Libraries Journal, 27(2), 158-162. Spring, H. (2010). Theories of learning: Models of good practice for evidence-based information skills teaching. Health Information and Libraries Journal, 27(4), 327-331. Stead, W. W., Searle, J. R., Fessler, H. E., Smith, J. W., & Shortliffe, E. H. (2010). Biomedical informatics: Changing what physicians need to know and how they learn. Academic Medicine : Journal of the Association of American Medical Colleges, Trevena, L. (2011). WikiProject medicine: Could become a trusted resource if it is assisted not shunned. BMJ, 343(7817) Whipple, E. C., Richwine, M. W., Kaneshiro, K. N., & Brahmi, F. A. (2009). Teaching first-year medical students where to go first: Connecting information needs to e-resources. Medical Reference Services Quarterly, 28(2), 180-186. Wood, J. A., Smigielski, E. M., & Haynes, G. (2007). Case-based approach for improving student MEDLINE searches. Medical Education, 41(5), 510-511. Evidence-Based Medicine and EBM Teaching Methodologies Aiyer, M. K., & Dorsch, J. L. (2008). The transformation of an EBM curriculum: A 10-year experience. Medical Teacher, 30(4), 377-383. Aronoff, S. C., Evans, B., Fleece, D., Lyons, P., Kaplan, L., & Rojas, R. (2010). Integrating evidence based medicine into undergraduate medical education: Combining online instruction with clinical clerkships. Teaching and Learning in Medicine, 22(3), 219-223. Barnett, S. H., Kaiser, S., Morgan, L. K., Sullivant, J., Siu, A., Rose, D., Stagnaro-Green, A. (2000). An integrated program for evidence-based medicine in medical school. The Mount Sinai Journal of Medicine, New York, 67(2), 163-168. Bordley, D. R., Fagan, M., & Theige, D. (1997). Evidence-based medicine: A powerful educational tool for clerkship education. The American Journal of Medicine, 102(5), 427-432.

Bradley, P., & Humphris, G. (1999). Assessing the ability of medical students to apply evidence in practice: The potential of the OSCE. Medical Education, 33(11), 815-817. Bradley, P., Oterholt, C., Herrin, J., Nordheim, L., & Bjorndal, A. (2005). Comparison of directed and self-directed learning in evidence-based medicine: A randomised controlled trial. Medical Education, 39(10), 1027-1035. Bradley, P., Oterholt, C., Nordheim, L., & Bjorndal, A. (2005). Medical students' and tutors' experiences of directed and self-directed learning programs in evidence-based medicine: A qualitative evaluation accompanying a randomized controlled trial. Evaluation Review, 29(2), 149-177. Burls, A. (2009). What is critical appraisal? Retrieved November 2, 2011, from http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/What_is_critical_appraisal.pdf Burrows, S. C., & Tylman, V. (1999). Evaluating medical student searches of MEDLINE for evidence-based information: Process and application of results. Bulletin of the Medical Library Association, 87(4), 471-476. Davis, J., Chryssafidou, E., Zamora, J., Davies, D., Khan, K., & Coomarasamy, A. (2007). Computer-based teaching is as good as face to face lecture-based teaching of evidence based medicine: A randomised controlled trial. BMC Medical Education, 7, 23. Dawes, M., Summerskill, W., Glasziou, P., Cartabellotta, A., Martin, J., Hopayian, K. Second International Conference of Evidence-Based Health Care Teachers and Developers. (2005). Sicily statement on evidence-based practice. BMC Medical Education, 5(1), 1. Dorsch, J. L., Aiyer, M. K., & Meyer, L. E. (2004). Impact of an evidence-based medicine curriculum on medical students' attitudes and skills. Journal of the Medical Library Association : JMLA, 92(4), 397-406. Dorsch, J. L., Jacobson, S., & Scherrer, C. S. (2003). Teaching EBM teachers: A team approach. Medical Reference Services

http://www.medicine.ox.ac.uk/bandolier/painres/download/whatis/What_is_critical_appraisal.pdf


Quarterly, 22(2), 107-114.

Ellis, P., Green, M., & Kernan, W. (2000). An evidence-based medicine curriculum for medical students: The art of asking focused clinical questions. Academic Medicine : Journal of the Association of American Medical Colleges, 75(5), 528. Feldstein, D. A., Mead, S., & Manwell, L. B. (2009). Feasibility of an evidence-based medicine educational prescription. Medical Education, 43(11), 1105-1106. Finkel, M. L., Brown, H. A., Gerber, L. M., & Supino, P. G. (2003). Teaching evidence-based medicine to medical students. Medical Teacher, 25(2), 202-204. Ghali, W. A., Saitz, R., Eskew, A. H., Gupta, M., Quan, H., & Hershman, W. Y. (2000). Successful teaching in evidence-based medicine. Medical Education, 34(1), 18-22. Gruppen, L. D., Rana, G. K., & Arndt, T. S. (2005). A controlled comparison study of the efficacy of training medical students in evidence-based medicine literature searching skills. Academic Medicine : Journal of the Association of American Medical Colleges, 80(10), 940-944. Guyatt, G. (2008). Users' guides to the medical literature: a manual for evidence-based clinical practice (2nd ed.). New York: McGraw-Hill Medical. Guyatt, G., Cairns, J., Churchill, D., Cook, D., Haynes, B., Hirsh, J., . . . Detsky, A. (1992). Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, 268(17), 2420-2425.

Harris, J., Kearley, K., Heneghan, C., Meats, E., Roberts, N., Perera, R., & Kearley-Shiers, K. (2011). Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Medical Teacher, 33(1), 9-23. Holloway, R., Nesbit, K., Bordley, D., & Noyes, K. (2004). Teaching and evaluating first and second year medical students' practice of evidence-based medicine. Medical Education, 38(8), 868-878. Hurwitz, S. R., & Slawson, D. C. (2010). Should we be teaching information management instead of evidence-based medicine? Clinical Orthopaedics and Related Research, 468(10), 2633-2639. Ilic, D. (2009). Assessing competency in evidence based practice: Strengths and limitations of current tools in practice. BMC Medical Education, 9, 53. Ilic, D., & Forbes, K. (2010). Integrating evidence based medicine into the medical curriculum: Barriers, enablers and implementation strategies. Medical Teacher, 32(5), 443-444. Ilic, D., & Forbes, K. (2010). Undergraduate medical student perceptions and use of evidence based medicine: A qualitative study. BMC Medical Education, 10, 58. Ismach, R. B. (2004). Teaching evidence-based medicine to medical students. Academic Emergency Medicine : Official Journal of the Society for Academic Emergency Medicine, 11(12), e6-10. Johnston, J. M., Leung, G. M., Tin, K. Y., Ho, L. M., Lam, W., & Fielding, R. (2004). Evaluation of a handheld clinical decision support tool for evidence-based learning and practice in medical undergraduates. Medical Education, 38(6), 628-637. Khan, K. S., & Coomarasamy, A. (2006). A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Medical Education, 6, 59. Kim, S., Willett, L. R., Murphy, D. J., O'Rourke, K., Sharma, R., & Shea, J. A. (2008). Impact of an evidence-based medicine curriculum on resident use of electronic resources: A randomized controlled study. Journal of General Internal Medicine, 23(11), 1804-1808. Lai, N. M., & Nalliah, S. (2010). Information-seeking practices of senior medical students: The impact of an evidence-based medicine training programme. Education for Health (Abingdon, England), 23(1), 151. Lai, N. M., & Teng, C. L. (2009). Competence in evidence-based medicine of senior medical students following a


clinically integrated training programme. Hong Kong Medical Journal = Xianggang Yi Xue Za Zhi / Hong Kong Academy of Medicine, 15(5), 332-338.

Lam, W. W., Fielding, R., Johnston, J. M., Tin, K. Y., & Leung, G. M. (2004). Identifying barriers to the adoption of evidence-based medicine practice in clinical clerks: A longitudinal focus group study. Medical Education, 38(9), 987-997. Mahoney, J. F., Cox, M., Gwyther, R. E., O'Dell, D. V., Paulman, P. M., & Kowlowitz, V. (2004). Evidence-based and population-based medicine: National implementation under the UME-21 project. Family Medicine, 36 Suppl, S31-5. McMahon, G. T., & Dluhy, R. G. (2006). Impact of a novel evidence-based medicine curriculum. Medical Education, 40(5), 464-465. Nieman, L. Z., Cheng, L., & Foxhall, L. E. (2009). Teaching first-year medical students to apply evidence-based practices to patient care. Family Medicine, 41(5), 332-336. Ramani, S. (2006). Twelve tips to promote excellence in medical teaching. Medical Teacher, 28(1), 19-23. Rhodes, M., Ashcroft, R., Atun, R. A., Freeman, G. K., & Jamrozik, K. (2006). Teaching evidence-based medicine to undergraduate medical students: A course integrating ethics, audit, management and clinical epidemiology. Medical Teacher, 28(4), 313-317. Rosenberg, W. M., Deeks, J., Lusher, A., Snowball, R., Dooley, G., & Sackett, D. (1998). Improving searching skills and evidence retrieval. Journal of the Royal College of Physicians of London, 32(6), 557-563. Sackett, D. L., Rosenberg, W. M., Gray, J. A., Haynes, R. B., & Richardson, W. S. (1996). Evidence based medicine: What it is and what it isn't. BMJ (Clinical Research Ed.), 312(7023), 71-72. Sandars, J., Siddiqi, K., Walsh, K., Richardson, J., Ibison, J., & Maxted, M. (2010). An undergraduate education package on evidence-based medicine: Some NICE lessons. Medical Education, 44(5), 511-512. Scherrer, C. S., Dorsch, J. L., & Weller, A. C. (2006). An evaluation of a collaborative model for preparing evidence-based medicine teachers. Journal of the Medical Library Association : JMLA, 94(2), 159-165.

Schilling, K., Wiecha, J., Polineni, D., & Khalil, S. (2006). An interactive web-based curriculum on evidence-based medicine: Design and effectiveness. Family Medicine, 38(2), 126-132. Shaneyfelt, T., Baum, K. D., Bell, D., Feldstein, D., Houston, T. K., Kaatz, S., Green, M. (2006). Instruments for evaluating education in evidence-based practice: A systematic review. JAMA : The Journal of the American Medical Association, 296(9), 1116-1127. Slawson, D. C., & Shaughnessy, A. F. (2005). Teaching evidence-based medicine: Should we be teaching information management instead? Academic Medicine : Journal of the Association of American Medical Colleges, 80(7), 685-689. Srinivasan, M., Weiner, M., Breitfeld, P. P., Brahmi, F., Dickerson, K. L., & Weiner, G. (2002). Early introduction of an evidence-based medicine course to preclinical medical students. Journal of General Internal Medicine, 17(1), 58-65. Taheri, H., Mirmohamadsadeghi, M., Adibi, I., Ashorion, V., Sadeghizade, A., & Adibi, P. (2008). Evidence-based medicine (EBM) for undergraduate medical students. Annals of the Academy of Medicine, Singapore, 37(9), 764-768. Tai, B. C., & Koh, W. P. (2008). Does team learning motivate students' engagement in an evidence-based medicine course? Annals of the Academy of Medicine, Singapore, 37(12), 1019-1023. Thangaratinam, S., Barnfield, G., Weinbrenner, S., Meyerrose, B., Arvanitis, T. N., Horvath, A. R., Khan, K. S. (2009). Teaching trainers to incorporate evidence-based medicine (EBM) teaching in clinical practice: The EU-EBM project. BMC Medical Education, 9(1) Thomas, A., Saroyan, A., & Dauphinee, W. D. (2011). Evidence-based practice: A review of theoretical assumptions and effectiveness of teaching and assessment interventions in health professions. Advances in Health Sciences Education, 16(2), 253-276. Thomas, P. A., & Cofrancesco, J.,Jr. (2001). Introduction of evidence-based medicine into an ambulatory clinical


clerkship. Journal of General Internal Medicine, 16(4), 244-249.

Tudiver, F., Rose, D., Banks, B., & Pfortmiller, D. (2009). Reliability and validity testing of an evidence-based medicine OSCE station. Family Medicine, 41(2), 89-91. Tumbo, J. (2010). Improving assessment of evidence-based medicine skills in medical students. Medical Education, 44(5), 510-511. Van Dijk, N., Hooft, L., & Wieringa-De Waard, M. (2010). What are the barriers to residents' practicing evidence-based medicine? A systematic review. Academic Medicine, 85(7), 1163-1170. Wadland, W. C., Barry, H. C., Farquhar, L., Holzman, C., & White, A. (1999). Training medical students in evidence-based medicine: A community campus approach. Family Medicine, 31(10), 703-708. Weberschock, T. B., Ginn, T. C., Reinhold, J., Strametz, R., Krug, D., Bergold, M., & Schulze, J. (2005). Change in knowledge and skills of year 3 undergraduates in evidence-based medicine seminars. Medical Education, 39(7), 665-671. Welch, H. G., & Lurie, J. D. (2000). Teaching evidence-based medicine: Caveats and challenges. Academic Medicine : Journal of the Association of American Medical Colleges, 75(3), 235-240. West, C. P., & McDonald, F. S. (2008). Evaluation of a longitudinal medical school evidence-based medicine curriculum: A pilot study. Journal of General Internal Medicine, 23(7), 1057-1059. Zebrack, J. R., Anderson, R. C., & Torre, D. (2005). Enhancing EBM skills using goal setting and peer teaching. Medical Education, 39(5), 513-514. Zwolsman, S. E., Wieringa-De Waard, M., Hooft, L., & Van Dijk, N. (2011). Measuring evidence-based medicine knowledge and skills. the dutch berlin questionnaire: Translation and validation. Journal of Clinical Epidemiology, 64(8), 928-930. Faculty Development

Cartwright, C. A., Korsen, N., & Urbach, L. E. (2002). Teaching the teachers: Helping faculty in a family practice residency improve their informatics skills. Academic Medicine : Journal of the Association of American Medical Colleges, 77(5), 385-391. Curran, C. R. (2008). Faculty development initiatives for the integration of informatics competencies and point-of-care technologies in undergraduate nursing education. The Nursing Clinics of North America, 43(4), 523-33. Dorsch, J. L., Jacobson, S., & Scherrer, C. S. (2003). Teaching EBM teachers: A team approach. Medical Reference Services Quarterly, 22(2), 107-114. Dorsch, J. L., Jacobson, S., & Scherrer, C. S. (2003). Teaching EBM teachers: A team approach. Medical Reference Services Quarterly, 22(2), 107-114. Harris, D. L., Krause, K. C., Parish, D. C., & Smith, M. U. (2007). Academic competencies for medical faculty. Family Medicine, 39(5), 343-350.

Maggio, L., Fabbro, K., & Posley, K. (2009). The Road Not Traveled: Evidence-Based Practice and Information Literacy. AAMC Western Group on Educational Affairs, Stanford University. Maggio, L. A., & Posley, K. A. (2011). Training the trainers: Teaching clinician educators to provide information literacy skills feedback. Journal of the Medical Library Association : JMLA, 99(3), 258-261. Nicholson, L. J., Warde, C. M., & Boker, J. R. (2007). Faculty training in evidence-based medicine: improving evidence acquisition and critical appraisal. The Journal of Continuing Education in the Health Professions, 27(1), 28-33. Ramani, S. (2006). Twelve tips to promote excellence in medical teaching. Medical Teacher, 28(1), 19-23. Robinson, M. A. (2003). Issues and strategies for faculty development in technology and biomedical informatics. Advances in Dental Research, 17, 34-37.


Robin, B. R., McNeil, S. G., Cook, D. A., Agarwal, K. L., & Singhal, G. R. (2011). Preparing for the changing role of instructional technologies in medical education. Academic Medicine : Journal of the Association of American Medical Colleges, 86(4), 435-439. Scherrer, C. S., Dorsch, J. L., & Weller, A. C. (2006). An evaluation of a collaborative model for preparing evidence-based medicine teachers. Journal of the Medical Library Association : JMLA, 94(2), 159-165. Curriculum Design Key Documents AAMC Institute for Improving Medical Education. (2007). Effective use of educational technology in medical education - colloquium on educational technology: Recommendations and guidelines for medical educators. Washington, DC: Association of American Medical Colleges. Abrahamson, S. (1978). Diseases of the curriculum. Journal of Medical Education, 53(12), 951-957. Bland, C. J., Starnaman, S., Wersal, L., Moorehead-Rosenberg, L., Zonia, S., & Henry, R. (2000). Curricular change in medical schools: How to succeed. Academic Medicine: Journal of the Association of American Medical Colleges, 75(6), 575-594.

Campbell, C., Silver, I., Sherbino, J., Cate, O. T., & Holmboe, E. S. (2010). Competency-based continuing professional development. Medical teacher, 32(8), 657-62.

Dreyfus, S. E. (2004). The five-stage model of adult skill acquisition. Bulletin of Science, Technology and Society, 24(3), 177-181. Frenk, J., Chen, L., Bhutta, Z. A., Cohen, J., Crisp, N., Evans, T., Zurayk, H. (2010). Health professionals for a new century: Transforming education to strengthen health systems in an interdependent world. Lancet, 376(9756), 1923-1958. Harden, R. M. (1986). Ten questions to ask when planning a course or curriculum. Medical Education, 20(4), 356-365. Harden, R. M. (2006). Trends and the future of postgraduate medical education. Emergency Medicine Journal : EMJ, 23(10).

Hodges, B. D. (2010). A tea-steeping or i-Doc model for medical education? Academic Medicine : Journal of the Association of American Medical Colleges, 85(9 Suppl), S34-44. McGowan, J., Raszka, W., Light, J., Magrane, D., O'Malley, D., & Bertsch, T. (1998). A vertical curriculum to teach the knowledge, skills, and attitudes of medical informatics. Proceedings/AMIA Annual Symposium. AMIA Symposium, 457-461. O‟Connell, H. P. (2009). Spicing up medical education: Six cornerstones in teaching medical students today. Student BMJ, 17, b2390. Skochelak, S. E. (2010). A decade of reports calling for change in medical education: What do they say? Academic Medicine: Journal of the Association of American Medical Colleges, 85(9 Suppl), S26-33. Souza, K. H., Kamin, C., O'Sullivan, P., Moses, A., & Heestand, D. (2008). Organizational models of educational technology in U.S. and canadian medical schools. Academic Medicine: Journal of the Association of American Medical Colleges, 83(7), 691-699.