View
2
Download
0
Category
Preview:
Citation preview
F A C U L T Y O F M E D I C I N E
Educational Technology Integrated Environmental Scan
Educational Technology
Integrated Environmental Scan
Summary of Notable Findings Report1
Submitted: September 13, 2010
Prepared by: Educational Technology Integrated Environmental Scan Project Team
of the UBC Faculty of Medicine’s Technology Enabled Learning group
1 The UBC Faculty of Medicine and the TEL-Educational Technology Integrated Environmental Scan project team
thank all participating institutions for their generous contribution of time, experience, and insight. We have
endeavoured to capture and present all information as accurately as possible and sincerely apologize for any
inadvertent misrepresentation of information that may appear in this report.
Educational Technology Integrated Environmental Scan
Prepared by Technology Enabled Learning, Faculty of Medicine, UBC Page i
September 13, 2010
Table of Contents
Executive Summary ............................................................................................................................1
1 Background .................................................................................................................................2
1.1 Purpose ...................................................................................................................................... 2
2 Scan Methodology .......................................................................................................................3
2.1 Institution Selection .................................................................................................................. 3
2.2 Data Capture and Analysis ........................................................................................................ 3
3 System Trends Summaries ...........................................................................................................3
3.1 Curriculum Management Tools ................................................................................................. 3
3.1.1 Curriculum Management Tool Adoption Drivers ...................................................... 4
3.2 Learning Objects Repositories ................................................................................................... 5
3.3 Online Health Education Platform and Open Educational Resources ...................................... 6
3.3.1 Online Health Education Platforms ........................................................................... 6
3.3.2 Trends in Open Educational Resources in Medical Education ................................... 6
3.4 Learning Management Systems ................................................................................................ 7
3.4.1 Learning Management System Adoption Drivers ..................................................... 7
4 Summary of Notable Findings .......................................................................................................8
4.1 Commercial Systems Integrated with In-House Developed Systems ....................................... 9
4.2 Open Source Systems Integrated with In-House Developed Systems ...................................... 9
4.3 Multiple Integrated Commercial Systems ............................................................................... 10
4.4 Comprehensive Systems Developed In-House ........................................................................ 11
5 Conclusions and Next Steps ........................................................................................................ 11
Appendix A: CMTs by Software......................................................................................................... 13
Appendix B: LMSs by Software ......................................................................................................... 15
Appendix C: System Integration by Institution .................................................................................. 17
Appendix D: Other Commercial Systems ........................................................................................... 21
Agresso Curriculum Management System (ACMS) ........................................................................... 21
OpalQM ............................................................................................................................................. 21
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 1 of 21
September 13, 2010
Executive Summary
As part of the UBC Faculty of Medicine’s (FoM) curriculum renewal process, Technology Enabled
Learning developed an Educational Technology Strategy Report that outlined five investment
recommendations. This report, which summarizes results of an environmental scan, is a first step to
achieving objectives of three of the investment recommendations: curriculum management, learning
objects, and online health education platform. The scan was designed to obtain qualitative information
on educational technology used by other medical education institutions to meet curricular needs. Semi-
structured, one-hour interviews were conducted with 32 targeted institutions (a 57% response rate of
the 56 contacted institutions).
Results show that 13% (n=4) of respondents use commercial systems alone to meet their needs. 16%
(n=5) of respondents use integrated in-house developed solutions. Remaining respondents (71%) use a
combination of in-house developed and/or open source products with commercial products.
Respondents that have developed in-house systems and integrated them with commercial systems have
met curriculum mapping and management requirements to varying degrees of success. They tend to be
satisfied but face integration and customizability limitations. Furthermore, respondents indicated that
cost and access to support can be variable.
Respondents that have integrated in-house developed systems with open source systems report similar
strengths and weaknesses as above. While respondents found open source systems to be readily
customizable, factors such as cost and support availability can be difficult to anticipate.
A minority of institutions adopted multiple commercial systems. This has the advantage of predictable
costs and timelines associated with procurement, implementation, and support. However, most
respondents were not able to fully satisfy the need to map learning objectives to learning objects and
activities in a way that is meaningful to curriculum design due to limited customizability.
Scan respondents that developed or adopted integrated solutions containing curriculum management
tool (CMT), learning management system (LMS), and learning object repository (LOR) functionality
reported the highest levels of satisfaction based on the need to link learning objectives, objects, and
activities. These systems were cited as being fully integrated, customizable, and able to provide single-
authentication and role-based access through a portal. Based on user satisfaction, this option is worth
investigating further. However, due to the added complexity of integration, there is risk associated with
implementation timelines and costs which should be examined in detail.
Following this external scan, the next steps towards fulfilling the investment recommendations outlined
in the Educational Technology Strategy Report are to conduct an internal needs assessment to
determine specific requirements from a CMT, LOR, and online health education platform (OHEP) to
conduct a product comparison analysis, test a shortlist of systems and make a recommendation for the
implementation of one or more technologies. This report will supplement the outputs of these next
steps by providing strengths and weaknesses of solution types for the UBC FoM to consider.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 2 of 21
September 13, 2010
1 Background
The University of British Columbia (UBC) Faculty of Medicine (FoM) is currently undergoing curriculum
renewal. To support this process, the Technology Enabled Learning (TEL) group developed an
Educational Technology Strategy Report that outlines five investment recommendations:
1. Synchronous Communication Platform: Continue to develop and implement synchronous
educational technology to support and expand distributed medical education.
2. Curriculum Management: Implement tools that support the process of curriculum renewal,
curriculum mapping, and the implementation of an integrated curriculum management
repository.
3. Learning Objects: Provide tools that support the modularization of educational resources to be
reused in various instructional contexts. These tools will link to the curriculum management
system.
4. Educational Technology Tools: Create mechanisms to support, implement, and integrate
educational technology tools (e.g. virtual patients, simulators, ePortfolios) into medical
education.
5. Online Health Education Platform: Create an online gateway for web-based learning resources
for UBC Health Professions education.
These recommendations are driving several projects, three of which are relevant here: Curriculum
Management, Learning Objects, and Online Health Education Platform. The charters for each of these
projects contain an environmental scan as the preliminarily step. These environmental scan components
were combined for efficiency and to ensure that information regarding integrated systems was
captured.
1.1 Purpose
The goal of the Educational Technology Integrated Environmental Scan was to obtain more information
about technology solutions that other institutions are using and how well those met their curricular
needs. While this environmental scan was the first step in the aforementioned charters, future
recommendations and decisions will also be informed by upcoming activities such as an internal needs
assessment and a formal technology options assessment, which will align needs with feasible solutions.
These additional activities, in addition to the scan results contained herein, are vital to inform solution
recommendations and decision making scheduled to take place by March, 2011.
This report is intended to present notable findings from the scan in a way that will inform the UBC FoM’s
technology recommendations and decision making process. To achieve this, the report categorizes
solutions, identifies reasons for adoption, summarizes important strengths and weaknesses associated
with solution types, and reports levels of satisfaction where available.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 3 of 21
September 13, 2010
2 Scan Methodology
The environmental scan was designed to obtain relevant information from educational technology
representatives at target Faculties, Schools, and Colleges of Medicine within higher education
institutions in North America, the United Kingdom, and Australia. This was not designed as a
quantitative study, rather a qualitative investigation of a selected sample of institutions.
2.1 Institution Selection
Using principles from comparative research, institutions both similar and different from the UBC FoM
were selected. Many institutions were selected based on same accreditation requirements, while other
jurisdictions were selected to ensure that additional, creative solutions were also identified. The person
most directly responsible for the entire suite of educational technologies at the medical school, or
alternatively, the dean responsible for undergraduate medical education was invited to participate in an
interview. Invitations for phone or videoconference interviews were sent by email to 56 institutions. 10
Canadian, 18 US, 1 UK and 3 Australian medical schools were interviewed for a total of 32 interviews
and a 57% response rate. Interviews were conducted between July 5th and August 16th, 2010.
2.2 Data Capture and Analysis
The environmental scan was conducted using a semi-structured interview methodology using a standard
set of guiding questions. Each interview lasted approximately one hour. The data was analyzed
qualitatively to identify trends and capture the complexity of the experiences presented during the
interviews. A qualitative thematic analysis was performed on the interview transcriptions. Information
regarding reasons for adoption and level of satisfaction was parsed into themes. Examples in the
analysis were selected for breadth, showing both originality of solutions and commonly occurring
solutions and problems. Satisfaction levels were determined based on the combination of interviewees’
observations on faculty and student satisfaction as well as satisfaction from administrative program
levels and implementation perspectives. Satisfaction cannot be seen as representative of all
stakeholders at the institution, but represent a valid perspective on the experience of implementing
these solutions.
3 System Trends Summaries
This section provides an overview of approaches that medical education institutions are taking with
regards to curriculum management tools (CMT), learning object repositories (LOR), online health
education platforms (OHEP), and learning management systems (LMS). A summary table of system
integration by institution can be found in Appendix C and discussion of the integration of these systems
is in the Notable Findings Section.
3.1 Curriculum Management Tools
According to the UBC FoM’s chartered investment recommendations, the goal of curriculum
management is to “increase educational effectiveness and facilitate the accreditation process for
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 4 of 21
September 13, 2010
undergraduate medical education by: increasing consistency in educational delivery; improving the
integration between interdependent courses; and increasing teaching focus in areas where more
learning is required.”2
The UBC FoM intends to realize this goal by making the relationships between curricular objectives and
educational activities and resources more explicit and transparent by researching and implementing a
CMT that allows for integration with an LOR and LMS.
78% (n=25) of scan respondents indicated that they meet their curriculum management objectives with
some form of an electronic CMT. Of this, 18 (56% of total) respondents developed their own solution. 12
(38% of total) of these developed a system in-house and 6 (19% of total) rely on a combination of
relational databases and/or spreadsheets to manage curriculum. Some of the in-house developed
systems are available for adoption and customization by other institutions3.
13% (n=4) of respondents indicated that they are able to meet curriculum management needs with
commercial systems. Of those, two are currently using, and one is implementing, the same system
currently in use by the UBC FoM (one45). The fourth is using WEAVEOnline.
Most respondents upload learning objectives to the Curriculum Management and Information Tool
(CurrMIT), provided by the American Association of Medical Colleges (AAMC). Only two (6%) rely on
CurrMIT alone to map and manage objectives.
For more detailed information regarding CMTs including reported strengths and weaknesses, please see
Appendix A.
3.1.1 Curriculum Management Tool Adoption Drivers
Two primary CMT adoption drivers are:
• A robust centralized search feature, and
• The ability to have program-wide visibility into objective coverage without administrative
overload.
Also, similar to UBC, eight (25%) respondents adopted a CMT in response to needs identified through a
curriculum renewal process. Respondents tend to favour systems that are customizable, contain
versatile applications, and can be integrated with an LMS. Please see Figure 1 for commonly reported
CMT adoption drivers.
2 UBC FoM Educational Technology Strategy Investment Recommendation: Curriculum Management project
charter, June 11, 2010. 3 The University of California San Francisco developed a CMT called Ilios. It has been adopted and customized by
the University of California San Diego and the University of Arizona Tucson.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 5 of 21
September 13, 2010
3.2 Learning Objects Repositories
The goal of the learning objects investment recommendation is “to increase the quality of existing and
emerging learning content for medical education and to improve the efficiency of instructional design
for faculty members by providing mechanisms to share, annotate, and access reusable learning
objects.”4 To achieve this, the UBC FoM intends to implement a Learning Objects Repository (LOR) to
enhance storage of and access to learning objects in a way that is integrated with the CMT.
Learning objects are resources that can be used for learning, education, or training. The term commonly
applies to documents and presentations that are often digital and web-based, and is increasingly used to
describe multimedia learning modules.
The majority (94%; n=30) of scan respondents use either their CMT or LMS as their LOR. Of the 21
respondents (66%) that use their LMS to house learning objects, only two are able to map learning
objectives back to objects, which is an important function to the UBC FoM. To achieve this, they
developed their own LMS or CMT.
With one exception, all scan respondents that developed comprehensive in-house solutions to meet
curriculum and learning management needs also use this same solution as an LOR. The institution with
an LOR separate from its CMT/LMS manages learning objects using digital asset management software,
developed in-house and released as open source5. This allows it to manage and tag learning objects
4 UBC FoM Educational Technology Strategy Investment Recommendation: Learning Objects project charter, June
11, 2010. 5 The University of Sydney Medical School’s MediaBank
0 2 4 6 8 10
require robust searchability of CMS content
curriculum renewal
to increase perspective of objectives coverage
customization requirement
spreadsheets and database are sufficient
report generation
versatility of applications
need to integrate the CMS with the LMS
online access
desire to tracking teaching hours
Figure 1: Most Common Factors Influencing the Selection of a CMT
Number of Comments
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 6 of 21
September 13, 2010
separately from any particular system, but still provide a link directly to objects from within the in-house
developed CMT/LMS6.
Other institutions employ commercial LORs such as Blackboard or have adapted SharePoint to achieve
LOR functionality. Institutions with lecture recording or other video storage needs tend to store these in
a separate video server, unless their system is able to store video7.
Highlighting the importance of integration across systems, institutions that reported storing learning
objects in an LMS not integrated with the CMT tend to manage learning objects within course folders. In
these cases, learning object management often then becomes the responsibility of faculty or staff
members who administer course components. Learning objects stored in this kind of folder structure are
rarely categorized, tagged with metadata or expiry dates and are often re-submitted to the system. This
creates inefficiencies, prevents comprehensive inventory scanning, reduces search efficacy, and does
not enable linking learning objectives to objects.
3.3 Online Health Education Platform and Open Educational Resources
3.3.1 Online Health Education Platforms
The goal of the OHEP investment recommendation is “to create the conditions for an enhanced learning
experience by improving coherence, delivery, and access to web-based learning resources and tools for
the undergraduate medical education program and UBC health professions education.”8 The UBC FoM
intends to achieve this through implementing a consolidated access point for web-based teaching and
learning tools and educational resources that is scalable beyond medical education to include other
health professions education.
Less than half of scan respondents (43%) host a portal that aggregates links to all relevant resources. Of
these, the majority do not seamlessly integrate resources under single sign-on functionality.
One respondent provides an integrated, single sign-on portal for sharing learning resources with
students, trainees, and faculty from registration to retirement. The system creates a convenient way to
access all learning resources which can be grouped and accessed through filters that focus on an
individual’s appropriate learning communities, along with relevant resources.9
3.3.2 Trends in Open Educational Resources in Medical Education
There is a growing initiative to freely share learning objects with other faculties, departments, or
institutions. These types of shared resources are called Open Educational Resources. In this context,
offering open educational resources permits those who are not enrolled in the medical program, and
6 The University of Sydney Medical School developed COMPASS to meet curriculum and learning management
needs 7 In-house developed systems such as HOMER and Ilios are capable of storing videos
8 UBC FoM Educational Technology Strategy Investment Recommendation: Online Health Education Platform
project charter, June 11, 2010 9 The system referenced here is the University of Alberta Medical School’s HOMER
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 7 of 21
September 13, 2010
who do not have access to the medical program’s curriculum, to search for and access those educational
resources that have been shared by the author(s) in a centralized repository.
Scan respondents are slightly more likely to share specific learning objects, such as virtual patients,
through national or international repositories. However, two respondents have means for faculty to
voluntarily contribute learning objects to an open educational resource repository supported by the
medical school itself.10
Commonly cited reasons for not engaging in an open educational resources initiative is the lack of clarity
around intellectual property rights and enforcement. Also, the availability of the popular MedEdPORTAL
amongst other national and international peer-reviewed resource repositories is another reason why
most medical schools have not created their own open resource repository.
One possibility for overcoming intellectual property issues is to only enable access to educational
resources by medical school alumni and other health profession education faculties within the same
university. However, this does not achieve the same results as a fully public repository.
3.4 Learning Management Systems
An LMS is a software application, usually web-based, that facilitates the administration and
management of learning programs and course delivery by providing access to schedules, learning
objects, assessment and reporting tools, and other collaborative communication tools.
Among scan respondents, 44% (n=14) use a commercial LMS, 53% (n=17) use an in-house developed or
open source system, and one respondent11 uses an in-house developed system in combination with a
commercial system to meet LMS functionality.
In general, respondents using commercial LMSs are less satisfied than respondents using in-house
developed or open source systems. Please see Appendix B for a list of LMSs, including some systems
that are serving as substitute LMSs, with reported strengths and weaknesses.
3.4.1 Learning Management System Adoption Drivers
The most commonly reported LMS adoption drivers are:
• The degree to which the system is customizable and interoperable with other systems (planned
or in place), and
• The degree to which the calendar is suitable to displaying the institution’s curriculum.
Respondents that use a commercial LMS frequently reported that the calendar interface was
not suitable for displaying their particular curriculum.
10
The University of Michigan Medical School’s Open.Michigan (https://open.umich.edu/) which has now expanded
to include 10 faculties within the university; and Tufts University School of Medicine’s OpenCourseWare
(http://ocw.tufts.edu/) 11
University of Melbourne Medical School
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 8 of 21
September 13, 2010
Other reported important considerations are a central search function and availability of central support
from the university IT unit. Please see Figure 2 for commonly reported LMS adoption drivers.
4 Summary of Notable Findings
The environmental scan uncovered four primary categories of solutions used by medical education
institutions to achieve curriculum mapping, learning object management, and delivery objectives:
1. Commercial systems integrated with in-house developed systems
2. Open source systems integrated with in-house developed systems
3. Multiple integrated commercial systems
4. Comprehensive systems developed in-house
No single system uncovered during the environmental scan enables an institution to completely fulfill
objectives related to curriculum management, learning objects, and online access as outlined in the UBC
FoM Educational Technology Strategy Report.
Nevertheless, scan respondents achieve similar objectives, or sub-sets of these objectives using system
combination solutions. With combination solutions, the degree of compatibility, integration, and
customizability of disparate systems determines the extent to which an institution is able to achieve
functional cohesion among learning objectives, objects, activities, and outcomes. Where data was
available, for each solution category, we have included information related to strengths, weaknesses,
customizability, integration, and general cost and support considerations. Please see Appendix C for a
summary of system integration by institution.
0 2 4 6 8 10 12
customizability/ interoperability with other systems
calendar suitability to the medical curriculum
central Searchability
system is centrally support by university IT unit
ease of use
implemented to link resources with objectives
nothing suitable on the market for medical education
robustness of features
low implementation Cost
contractual support available for this system
Figure 2: Most Common Factors Influencing the Selection of an LMS
Number of comments
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 9 of 21
September 13, 2010
4.1 Commercial Systems Integrated with In-House Developed Systems
A lack of commercially available products to meet their needs led several scan respondents to develop
custom sub-systems to meet a given functional need that was not available through commercial
software. These in-house developed systems are often integrated with commercial systems to cover
required functionality. Interestingly, the most commonly cited solution was an in-house developed CMT
integrated with a commercial LMS.
Respondents often reported that a main strength of this approach is that an institution can create a CMT
that is customizable and adaptable to its particular curriculum design. Like comprehensive systems built
in-house, custom developed systems, such as CMTs created to meet a particular set of requirements,
are amendable and can be customized to support renewal of curriculum structure or to include a variety
of functional features as needed by the curriculum. Some institutions have also integrated scheduling,
consent tracking, teaching efforts tracking, and learning object management into their in-house
developed CMT. Depending on the degree of integration with other commercial tools, this type of
solution may permit institutions to map curricular objectives with learning objects, events, and
outcomes. Finally, because this type of solution leverages a commercial component, the long-term cost
is more transparent than developing a comprehensive, in-house solution.
This type of blended solution presents challenges related to the way in which information is displayed
and accessed. While system integration can permit curricular objectives to be mapped to learning
objects from a planning and management perspective, respondents indicated that the content from the
CMT is often displayed in the commercial LMS in ways that are not be appropriate to the curriculum.
Specifically, respondents indicated that commercial LMSs are often unable to display objectives in a
calendar view that can accommodate longitudinal, spiral, or integrated nature of some curriculum
models. Further, while a single point of authentication is sometimes possible, users may still need to be
given direction regarding which information to look for in each of the systems. Commercial products
offer varying levels of adaptability and system customization that may hinder full integration with an in-
house developed system. Finally, venders’ level of customer support varies and does not extend to in-
house developed systems that are integrated with their product.
Institutions which have developed noteworthy solutions which integrate CMTs and other sub-systems
with commercial systems include the University of New South Wales Faculty of Medicine, the University
of Vermont College of Medicine, the University of Pittsburgh School of Medicine, and the University of
Colorado Medical School. Memorial University of Newfoundland Faculty of Medicine is adopting a
similar approach as part of its curriculum renewal program.
4.2 Open Source Systems Integrated with In-House Developed Systems
Some scan respondents elected to pair an in-house developed system with an open source system
(instead of a commercial system as described above). As described in the previous section, these
institutions developed in-house sub-systems to meet a functional need that was not available through
other systems. Many strengths and weaknesses of integrating an in-house developed system with an
open source system are similar to those of integrating an in-house system with a commercial system.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 10 of 21
September 13, 2010
However, one of the key differences is that open source systems tend to be more customizable than
commercial systems. Furthermore, open source software enables the institution to participate in an
open source community which contributes to the pool of system enhancements over time.
An important consideration with open source LMSs is that costs associated with implementing,
customizing, and supporting these systems can be difficult to anticipate. Furthermore, while support is
available for some, it is not consistently available for all and can be difficult to obtain.
Currently, no single open source system exists which can fulfill curriculum and learning object
management functionality. Two institutions that have developed or adopted in-house systems and
integrated them effectively with an open source LMS include the University of California San Francisco
School of Medicine and the Northern Ontario School of Medicine. The University of Arizona Tucson
Medical School has adopted an in-house CMT (which is soon to become open source) from another
institution and integrated it with their own in-house developed LMS.
4.3 Multiple Integrated Commercial Systems
The next category is solutions achieved through integration of multiple commercial systems. This is
currently the approach the UBC Faculty of Medicine has taken to address its curriculum and learning
management needs. Four scan respondents employ a similar strategy.
As with other categories, no single commercial product was found which can meet curriculum mapping
and management, and learning object delivery and management functionality. The main strength of
commercial systems is the ability to plan and forecast costs and timelines associated with acquisition,
implementation, and support with greater confidence. This has significant benefits for budgeting and
scheduling. Furthermore, commercial systems can be implemented more quickly because development
time is not an issue.
The commercial applications discovered through this scan generally support only limited customization
and often present both functional and interoperability issues. This lack of interoperability among
commercial systems leads to the inability to efficiently and consistently share information across
systems. Low degrees of customizability result in the inability to change system features or make
adjustments to more closely align with a given institutions curriculum design and/or renewal process.
Respondents often cited the inability to reflect longitudinal, spiral, integrated, or case-based medical
curricula as a key drawback of commercial CMTs. Furthermore, multiple commercial systems often
imply multiple log-in requirements. Finally, commercial system licensing costs can be high and
sometimes involve multi-year contracts with vendors, reducing future flexibility.
Of the limited number of commercial CMTs available, one system identified through internet research,
but not used by any scan respondents was Agresso Curriculum Management System (ACMS). Please see
Appendix D for information on functionality. This product may be worth investigating further.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 11 of 21
September 13, 2010
4.4 Comprehensive Systems Developed In-House
A lack of commercially available products that enable institutions to meet their objectives has led some
institutions to develop solutions in-house. Comprehensive systems developed in-house are all-in-one
solutions which are created and fully maintained by one institution or a small consortium. These may or
may not be available under an open source license, but are often free to other institutions by
arrangement.
By nature, these types of systems can be built to maximize flexibility and customizability. They enable
medical schools to illustrate the link among objectives, learning objects, and delivery thereof in
meaningful ways by facilitating all mapping, planning, management, and learning activities within one
system. Furthermore, they contain scheduling systems and calendar interfaces that are customizable to
a given medical school’s curriculum model, whether it be longitudinal or spiral in nature, for example.
These systems also have central search capabilities, which simplify the user experience. Finally, in-house
systems are generally able to tailor role-based access profiles to the specific institution’s needs and be
adapted quickly to changes in the curriculum.
The primary drawback of developing a comprehensive system is the large development effort and high
ongoing support costs associated with an in-house development team. Nevertheless, institutions that
developed or adopted all-in-one systems that contain CMT, LMS, and LOR functionality, reported the
highest levels of satisfaction.
Interestingly, one respondent12 is using TUSK, developed in-house by Tufts University, for no licensing
fee. This approach can mitigate some cost and time issues associated with development.
Scan respondents that have developed noteworthy comprehensive systems include the University of
Calgary Faculty of Medicine, the University of Alberta Faculty of Medicine and Dentistry, the University
of California San Diego Medical School, Tufts University School of Medicine, and the University of Sidney
Medical School.
Another custom-developed system discovered through online research is OpalQM. The University of
Manitoba Faculty of Medicine developed this system which has now been commercialised. According to
online marketing material, this system has much of the same functionality as other all-in-one systems13.
For more information, please see Appendix D. As this system was not discovered as part of the
environmental scan, it may be worth investigating further.
5 Conclusions and Next Steps
Scan respondents that developed or adopted integrated solutions containing CMT, LMS, and LOR
functionality reported the highest levels of satisfaction based on the need to link learning objectives,
objects, and activities. These systems were cited as being fully integrated, customizable, and able to
12
The University of Hawaii JABSOM 13
http://www.opalqm.com/Pages/home.aspx
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 12 of 21
September 13, 2010
provide single-authentication and role-based access through a portal. Based on user satisfaction, this
option is worth investigating further. However, due to the added complexity of integration, there is risk
associated with implementation timelines and costs which should be examined in detail.
Respondents that have integrated in-house developed systems with commercial systems have achieved
mapping and management requirements to varying degrees of success. Institutions that have taken this
approach tend to be satisfied but also face limitations of partial integration and challenges related to the
inability to customize the commercial system to match curriculum design. Furthermore, obtaining
support can be expensive and challenging for some product types.
Respondents that have integrated in-house developed systems with open source systems report similar
strengths and weaknesses as above. While respondents found open source systems to be readily
customizable, factors such as cost and support availability can be difficult to anticipate.
A minority of institutions adopted multiple commercial systems. This has the advantage of predictable
costs and timelines associated with procurement, implementation, and support. However, most
respondents were not able to fully satisfy the need to map learning objectives to objects and activities in
a way that is meaningful, given particularities of curriculum design. Further, product licensing costs can
be high for many commercial products and supply contracts can be restrictive.
This environmental scan focused on gathering experiences presented by medical education institutions
external to the UBC FoM. The next steps towards fulfilling the investment recommendations outlined in
the Educational Technology Strategy Report are to conduct an internal needs assessment to determine
specific requirements from a CMT, LOR, and OHEP, to conduct a product comparison analysis, test a
shortlist of systems and make a recommendation for the implementation of one or more technologies.
This report will supplement the outputs of these next steps by providing strengths and weaknesses of
solution types for the UBC FoM to consider.
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 13 of 21
September 13, 2010
Appendix A: CMTs by Software
CMT Type Medical Schools
Number of
Respondents
Using System
Key Strengths Key Disadvantages
cMap Developed
In-house
University of
Toronto 1
Robust searchability of content; search results
displayed by tiers of relevancy; tracking by
thread/metadata; Suitability to the Medical
Curriculum
none reported
Compass Developed
In-house
University of
Sydney 1
Role-based access and customizable views of content;
Easy to use; Friendly interface; customizable and
flexible; Appropriate to the medical curriculum;
Robust scheduling and calendar system that is suitable
to the medical curriculum; Centralized Search; Links
resources with objectives;
Integrated with assessment systems that evaluates
question performance and links results back to
objectives; Great reporting system
Currently does not include
social networking tools
Curriculum
Management
System at U.
of Vermont
Developed
In-house
University of
Vermont 1
Tracks by thread/metadata; tracks teaching hours;
suitable to the medical curriculum; podcast consent
and schedule management
none reported
CURE Developed
In-house
University of
California San
Diego
1
Centralized Search; Searchable at granular levels;
Tracks faculty teaching efforts; Links resources with
objectives; Customizable and flexible
none reported
CurrMIT Provided by
the AAMC
University of
Hawaii; Indiana
University;
2 Ability to compare to other institutions; Good basic
report generation
Data not comprehensive;
System not functionally
rich or up-to-date enough
to rely on.
Spreadsheets
/ Database
solutions
Developed
In-house
Memorial
University;
University of
Washington;
University of
Melbourne; McGill
University; Johns
Hopkins University
5 high degree of customization
Solution is not functionally
rich; interoperability
problems with other
systems
eMed Developed
In-house
University of New
South Wales 1
Searchable, can be adjusted to program needs; very
effective in management of curriculum and reporting;
allows students to monitor own progress; integrates
multiple systems: curriculum map, graduate exit
competencies, timetables, portfolios for performance,
and learning project portfolios; clinical placements,
group activities. Calendar is downloadable with mobile
technology. eMed can produce reports for tutor
performance, ex. how many quizzes they have
marked.
$80-130 K per year on
development of eMED
Entrada
Developed
In-house;
open-source
University of
Calgary 1
Customizable and flexible; Appropriate to the medical
curriculum; Robust scheduling and calendar system
that is suitable to the medical curriculum; Centralized
Search; Links resources with objectives; Low Cost;
Integrated with curriculum mapping system;
Sophisticated search; Strong administrative system;
Responsive open source community
none reported
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 14 of 21
September 13, 2010
CMT Type Medical Schools
Number of
Respondents
Using System
Key Strengths Key Disadvantages
HOMER Developed
In-house
University of
Alberta 1
Customizable and flexible; Sophisticated search;
Role-based access and customizable views of content;
Integrated with curriculum mapping system;
Appropriate to the medical curriculum; Robust
scheduling and calendar system that is suitable to the
medical curriculum; Integrates with EMR system; Links
resources with objectives
Feature over-load;
Increasingly complex
navigation
Ilios v1
Developed
In-house;
v2 Open-
source in
2011
University of
California San
Francisco;
University of
Colorado;
University of
Arizona Tucson
3
Robust search by mesh terms, tags, discipline, course,
faculty member. Stores learning objects, tracks
teaching hours; allows tagging of learning objects with
expiration dates; highly customizable.
Ilios v1 does not include a
student focused interface
or student tools for
objective tracking. Does
not provide a taxonomy
layer to curriculum
mapping; inadequate
reports/analytics
MyLearning
Online
Developed
In-house
Northern Ontario
School of
Medicine
1 highly customizable, strong report generation
features, tracking by thread/metadata none reported
Navigator Developed
In-house
University of
Pittsburgh 1
Robust search; simple to use interface; robust object
repository; automatic upload to CurrMIT
Needs a code upgrade to
increase operating speed
one45 Commercial
Dalhousie
University;
University of
Saskatchewan;
University of
Western Ontario
& Windsor
3
Includes scheduling and logging features; appropriate
for basic objective mapping; easy implementation;
online; searchable
Does not integrate well
with other products; no
major customization is
possible; vendor is slow to
changes; features are
basic; no visual map
display of objective
coverage; system
performance issues
TUSK Developed
In-house Tufts University 1
Customizable and flexible; Sophisticated search; Role-
based access and customizable views of content;
Integrated with curriculum mapping system;
Appropriate to the medical curriculum; Robust
scheduling and calendar system that is suitable to the
medical curriculum; Centralized Search; Suggested
keyword search function; Links resources with
objectives; Contractual support available for this
system; Low implementation costs (based on
University of Hawaii's experience who adopted the
system from Tufts).
4 FTE Programmers
provide support for TUSK
Weave Online Commercial Charles Drew
University 1
easy to use; easy to implement; low implementation
cost; appropriate for mapping the medical curriculum none reported
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 15 of 21
September 13, 2010
Appendix B: LMSs by Software
LMS Type Medical Schools
Number of
Respondents
Using
System
Key Strengths Key Disadvantages
Angel Commercial Indiana University 1 Customizable script; interoperable with
other systems; easy to use;
Scheduling/calendar feature is not
appropriate to medical school
ArizonaMed Developed
In-house
Arizona University
Tucson; Arizona
University Phoenix
2
Customizable and flexible; Ties in multiple
systems; Robust scheduling and calendar
system that is suitable to the medical
curriculum; Calendar-based view;
Integrates widgets; Integrated with
curriculum mapping system
Possibility of customization has
increased feature requests
Blackboard Commercial
Charles Drew
University; University
of Melbourne;
University of
Colorado; University
of New South Wales;
University of
Vermont; University
of Toronto; Johns
Hopkins University;
University of Nevada
8 Easy to use; Centralized online access;
Searchable by keyword and free text
No contextual tools for learning
objects; Courses are in silos ; No self-
performance tracking tools; Difficulty
in uploading materials; Challenging
interface; Does not interoperate well
with other systems; Not suitable for
displaying objectives in meaningful
ways; Scheduling/calendar feature is
not appropriate to medical school;
Weak assignment grading tool; Weak
reporting and analytics features; Weak
assessment features; Poor support
from vendor
Catalyst Developed
In-house
University of
Washington 1
Easy to use; Friendly interface; Perfect for
faculty intimidated by Blackboard and
WebCT; Dropbox feature; Strong
communication features
No centralized search; Not as feature
rich as other systems; Weak reporting
and analytics features
Compass Developed
In-house University of Sydney 1
Role-based access and customizable views
of content; Easy to use; Friendly interface;
customizable and flexible; Appropriate to
the medical curriculum; Robust scheduling
and calendar system that is suitable to the
medical curriculum; Centralized Search;
Links resources with objectives;
Integrated with assessment systems that
evaluates question performance and links
results back to objectives; Great reporting
system
Currently does not include social
networking tools
CURE Developed
In-house
University of
California San Diego 1
Centralized Search; Searchable at granular
levels; Tracks faculty teaching efforts; Links
resources with objectives; Customizable
and flexible
none reported
DalMedix
(Coldfusion)
Developed
In-house Dalhousie University 1
Works well on a basic level; Strong
document management System is outdated
Desire2Learn Commercial Memorial University 1 none reported
Inappropriate for the medical
curriculum; Calendar is not suitable to
the medical curriculum; unable to
show objectives and activities in a
relational manner
E*Value Commercial Arizona University
Phoenix 1
Content delivery, evaluations, and portfolio
development
Not used for first and second year UG
students; not integrated with other
system
Entrada
Developed
In-house;
Open source
University of Calgary 1
Customizable and flexible; Appropriate to
the medical curriculum; Robust scheduling
and calendar system that is suitable to the
medical curriculum; Centralized Search;
Links resources with objectives; Low Cost;
none reported
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 16 of 21
September 13, 2010
LMS Type Medical Schools
Number of
Respondents
Using
System
Key Strengths Key Disadvantages
Integrated with curriculum mapping
system; Sophisticated search; Strong
administrative system; Responsive open
source community
HOMER
Developed
In-house;
Open source
University of Alberta 1
Customizable and flexible; Sophisticated
search; Role-based access and
customizable views of content; Integrated
with curriculum mapping system;
Appropriate to the medical curriculum;
Robust scheduling and calendar system
that is suitable to the medical curriculum;
Integrates with EMR system; Links
resources with objectives
Feature over-load; Increasingly
complex navigation
Moodle Open source
Northern Ontario
School of Medicine;
University of
California San
Francisco;
University of Sydney;
St. George's
University London;
University of Virginia
5
Software is customizable;
Scheduling/Calendar is suitable to the
medical curriculum; Free software;
Software supports growth; Responsive
open source community; Commercial
support is available; Easier to use than
Blackboard; Good Interface
Scheduling/Calendar is not suitable to
the medical curriculum; Lacks a central
search function; Weak in curriculum
navigation between courses; Courses
are in silos: No deep or long structure
to display continuity of courses or
objectives; New features coming out all
the time; difficult to keep track of
upgrades; Weak reporting and
analytics features
Sakai Community /
Open source
University of North
Carolina;
University of
Michigan;
Oregon Health and
Science University
3
Customizable and flexible; Calendar is
suitable for the medical curriculum;
Student group management; Easy transfer
of resources between folders; Commercial
support is available; Responsive open
source community
Interface problems with authenticating
between systems; Weak search
function; Scheduling/calendar feature
is not appropriate to medical school;
Does not provide a way for students to
track their objectives across the
program; No contextual tools for
learning objects
SharePoint Commercial University of
Pittsburgh 1
Customizable and flexible; Interoperable
with other systems; Integrates with
curriculum mapping system; Allows
students to create own team group
websites; Contractual support available for
this system
Was not designed for learning; New
versions of SharePoint are not
backwards compatible; Not
transparent in the code/difficult to
look under the hood; Not easy to
troubleshoots; not browser or platform
independent
TUSK Developed
In-house
Tufts University;
University of Hawaii 2
Customizable and flexible; Sophisticated
search; Role-based access and
customizable views of content; Integrated
with curriculum mapping system;
Appropriate to the medical curriculum;
Robust scheduling and calendar system
that is suitable to the medical curriculum;
Centralized Search; Suggested keyword
search function; Links resources with
objectives; Contractual support available
for this system; Low implementation costs
(based on University of Hawaii's experience
who adopted the system from Tufts).
4 FTE Programmers provide support for
TUSK
WebCT Commercial
McGill University;
University of
Western Ontario &
Windsor (Schulich)
2 Satisfactory search function; Student group
management
Scheduling/calendar feature is not
appropriate to medical school; No
contextual tools for learning objects;
Courses are in silos: No deep or long
structure to display continuity of
courses or objectives; No self-
performance tracking tools
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 17 of 21
September 13, 2010
Appendix C: System Integration by Institution
Curriculum Management Tool Learning Management System
CMT/LMS
Integratio
n
Learning
Object
Repository
Landing
Page
University of
Alberta
System HOMER HOMER v2
Yes HOMER Yes
Satisfaction High High
Future
Direction
Working on the next version;
incorporating all assessment and
residency training into Homer;
improving interface: more like iPad
with selectable apps.
Working on the next version;
incorporating all assessment and
residency training into Homer;
improving interface: more like iPad
with selectable apps.
University of
Arizona,
Phoenix
System None ArizonaMed and E*Value
No ArizonaMed No
Satisfaction ArizonaMed = high
E*Value = high
Future
Direction
Develop an ArizonaMed core (core
management system) and have each
campus individually customize modules
around it.
University of
Arizona,
Tucson
System Ilios v1 ArizonaMed
Yes ArizonaMed Yes
Satisfaction Medium High
Future
Direction
Phase out ASP code and match
Coldfusion which ArizonaMed is based
on.
Developing ArizonaMed v2. Launch
September 2011.
University of
California, San
Diego
System CURE CURE
Yes CURE Yes Satisfaction High High
Future
Direction
Will Improve reporting and automatic
export to CurrMIT.
University of
California, San
Francisco
System Ilios Moodle
Yes Ilios Yes
Satisfaction High Medium: weak features for medical
curriuclum but easy to use
Future
Direction
Ilios 2.0 will be released as open source
in 2011. The focus in this release is
access to students. It will interact
better and will focus on features for
students to interact with curriculum
mapping. Improved workflow
management and content review
features; course and student group
management, course calendar can be
downloaded. Improved interaction
with Moodle.
Moodle 2.0
University of
Calgary
System Entrada (Called Osler at U of C) Entrada (Called Osler at U of C)
Yes Blackboard No Satisfaction Yes Yes
Future
Direction
Charles Drew
University
System Weave Online Blackboard (non-enterprise)
No Blackboard No Satisfaction Yes Low
Future
Direction
University of
Colorado
System Ilios Blackboard
Yes Ilios No Satisfaction Medium Low
Future
Direction
Ilios 2.0 Ilios 2.0
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 18 of 21
September 13, 2010
Curriculum Management Tool Learning Management System
CMT/LMS
Integratio
n
Learning
Object
Repository
Landing
Page
Dalhousie
University
System One45 (implementing this year) DalMedix (Coldfusion)
No,
Exploring
Current:
DalMedix
(Coldfusion)F
uture:
SharePoint
Yes Satisfaction TBD Low
Future
Direction
One45 one45/Blackboard
University of
Hawaii,
JABSOM
System CurrMIT TUSK
No TUSK No Satisfaction Medium High
Future
Direction
Looking forward to CurrMIT upgrade:
Curriculum Inventory Portal
Indiana
University
System CurrMIT Angel
No Angel Yes
Satisfaction Medium Medium
Future
Direction
Ilios (exploring) Exploring other systems; Evaluating
Sakai at the moment
Johns Hopkins
University
System Spreadsheets/Databases Blackboard (non-enterprise)
No Blackboard Yes
Satisfaction High Medium (poor features but low cost
and easy to use)
Future
Direction
Expanding databases and making them
relational
University of
Nevada
School of
Medicine
System None Blackboard (Web Campus)
No Blackboard No
Satisfaction Medium
Future
Direction
Moving to Blackboard Learn
McGill
University
System Spreadsheets/Databases WebCT
No WebCT No
Satisfaction Medium Medium
Future
Direction
Exploring OPAL and Vanderbilt system Exploring OPAL and Vanderbilt system
McMaster
University
System Home-grown database None (D2L being used by central
campus)
No Alfresco Yes Satisfaction No data – still in development n/a
Future
Direction
Looking for a system that will handle
playback and tracking of SCORM
packages
University of
Melbourne
System File Maker PRO Database Blackboard & CATS
Currently
No, Yes
with
Curriculu
m
Connect
Currently
Blackboard,
Curriculum
Connect in
the future
No
Satisfaction Low Low
Future
Direction
Developing Curriculum Connect. Will
be implemented in 2011. It will be able
to store resources, be fully searchable
with a Google customized search, it will
allow faculty to search all relevant
teaching resources relating to
objectives.
Developing Curriculum Connect. Will
be implemented in 2011. It will be able
to store resources, be fully searchable
with a Google customized search, it will
allow faculty to search all relevant
teaching resources relating to
objectives.
Memorial
University
System Databases Desire2Learn
No Desire2Learn Yes
Satisfaction Low Low
Future
Direction
Putting the database online, changing
it to SQL and overlaying it with Think
Map.
Exploring other systems.
University of
Michigan
System None Sakai
No Sakai No
Satisfaction Low
Future
Direction
Yes; over the next two years, the
medical school will make some
upgrades to Sakai: enterprise search;
incorporate lecture capture;
introduced team based tools. UofM
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 19 of 21
September 13, 2010
Curriculum Management Tool Learning Management System
CMT/LMS
Integratio
n
Learning
Object
Repository
Landing
Page
will try to release Sakai 3 in 2012 or
2013. The objective is to have more
features that meet the needs of
medical schools as well.
University of
Nevada
School of
Medicine
System None Blackboard - Web Campus
No Blackboard No Satisfaction No data
Future
Direction
Moving to Blackboard Learn
University of
New South
Wales
System eMed Blackboard and eMed
No eMed Yes Satisfaction High Low
Future
Direction
University of
North Carolina
System None Sakai
NO Sakai No Satisfaction Medium
Future
Direction
Northern
Ontario
School of
Medicine
System MyLearning Online Moodle
No;
exploring
MyLearning
Online
No,
Piloting
Satisfaction High High
Future
Direction
MyCurriculum Online
Oregon Health
and Science
University
System None Sakai
No Sakai No Satisfaction High
Future
Direction
University of
Pittsburgh
System Navigator Navigator & SharePoint
Yes Navigator No Satisfaction Medium High
Future
Direction
University of
Saskatchewan
System one45 Blackboard
No Blackboard
Yes
Satisfaction High (integration is not a top priority) Low
Future
Direction
Potential for change as curriculum
renewal begins.
St. George's
University
London
System None Moodle
No Moodle No Satisfaction n/a High
Future
Direction
University of
Sydney
System Compass Compass & Moodle
Yes MediaBank Yes Satisfaction High High
Future
Direction
University of
Toronto
System cMap (SQL database) Blackboard
Yes Blackboard No Satisfaction High Low
Future
Direction
Tufts
University
System TUSK TUSK
Yes TUSK No Satisfaction High High
Future
Direction
Interface and navigation upgrade Interface and navigation upgrade;
leaning towards open source
University of
Vermont
System Curriculum Management System Blackboard Yes
Currently
Blackboard, Yes
Satisfaction High Medium
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 20 of 21
September 13, 2010
Curriculum Management Tool Learning Management System
CMT/LMS
Integratio
n
Learning
Object
Repository
Landing
Page
Future
Direction
Improving tracking and searchability
within objects
Would like to replace Blackboard with
an open source system within three
years.
SharePoint
in the future
University of
Virginia
System None X-credit (Built on Moodle). Is mainly
used for organizing learning resources.
No data No No data Satisfaction Do not know yet; still undergoing
development
Future
Direction
University of
Washington
(WWAMI)
System Spreadsheets/Database Catalyst
No Catalyst Yes Satisfaction Low Medium: weak features but easy to use
Future
Direction
exploring E*Value
University of
Western
Ontario &
Windsor
System one45 WebCT
No WebCT No Satisfaction Medium Low
Future
Direction
Educational Technology Integrated Environmental Scan
Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 21 of 21
September 13, 2010
Appendix D: Other Commercial Systems
Agresso Curriculum Management System (ACMS)
Although designed for higher education in general, this product suite offers applications including
curriculum management, student management, student engagement, and others. It also contains
document management and workflow integration modules. The curriculum management component
supports the development, approval, publication, and quality assurance of units, courses, and
programs.14
OpalQM
Another custom-developed system discovered through online research is OpalQM. The University of
Manitoba Faculty of Medicine developed this system which has now been commercialised. According to
online marketing material, this system can be used to manage and report on curriculum; map
curriculum to experiences and instructors; map instructor, facility, and student evaluations; manage
learning materials; and enhance communication, information sharing, scheduling, and reporting15. This
system was not captured as part of the environmental scan; further investigation may be warranted.
14
http://www.unit4software.co.uk/products/agresso/higher-education 15
http://www.opalqm.com/Pages/home.aspx
Recommended