Educational Technology Integrated Environmental Scanmed-fom-medit.sites.olt.ubc.ca/files/2013/06/Notable-Findings-Repor… · Prepared by: Technology Enabled Learning, Faculty of

F A C U L T Y O F M E D I C I N E

Educational Technology Integrated Environmental Scan

Educational Technology

Integrated Environmental Scan

Summary of Notable Findings Report1

Submitted: September 13, 2010

Prepared by: Educational Technology Integrated Environmental Scan Project Team

of the UBC Faculty of Medicine’s Technology Enabled Learning group

1 The UBC Faculty of Medicine and the TEL-Educational Technology Integrated Environmental Scan project team

thank all participating institutions for their generous contribution of time, experience, and insight. We have

endeavoured to capture and present all information as accurately as possible and sincerely apologize for any

inadvertent misrepresentation of information that may appear in this report.


Prepared by Technology Enabled Learning, Faculty of Medicine, UBC Page i

September 13, 2010

Table of Contents

Executive Summary ............................................................................................................................1

1 Background .................................................................................................................................2

1.1 Purpose ...................................................................................................................................... 2

2 Scan Methodology .......................................................................................................................3

2.1 Institution Selection .................................................................................................................. 3

2.2 Data Capture and Analysis ........................................................................................................ 3

3 System Trends Summaries ...........................................................................................................3

3.1 Curriculum Management Tools ................................................................................................. 3

3.1.1 Curriculum Management Tool Adoption Drivers ...................................................... 4

3.2 Learning Objects Repositories ................................................................................................... 5

3.3 Online Health Education Platform and Open Educational Resources ...................................... 6

3.3.1 Online Health Education Platforms ........................................................................... 6

3.3.2 Trends in Open Educational Resources in Medical Education ................................... 6

3.4 Learning Management Systems ................................................................................................ 7

3.4.1 Learning Management System Adoption Drivers ..................................................... 7

4 Summary of Notable Findings .......................................................................................................8

4.1 Commercial Systems Integrated with In-House Developed Systems ....................................... 9

4.2 Open Source Systems Integrated with In-House Developed Systems ...................................... 9

4.3 Multiple Integrated Commercial Systems ............................................................................... 10

4.4 Comprehensive Systems Developed In-House ........................................................................ 11

5 Conclusions and Next Steps ........................................................................................................ 11

Appendix A: CMTs by Software......................................................................................................... 13

Appendix B: LMSs by Software ......................................................................................................... 15

Appendix C: System Integration by Institution .................................................................................. 17

Appendix D: Other Commercial Systems ........................................................................................... 21

Agresso Curriculum Management System (ACMS) ........................................................................... 21

OpalQM ............................................................................................................................................. 21


Prepared by: Technology Enabled Learning, Faculty of Medicine, UBC Page 1 of 21

September 13, 2010

Executive Summary

As part of the UBC Faculty of Medicine’s (FoM) curriculum renewal process, Technology Enabled

Learning developed an Educational Technology Strategy Report that outlined five investment

recommendations. This report, which summarizes results of an environmental scan, is a first step to

achieving objectives of three of the investment recommendations: curriculum management, learning

objects, and online health education platform. The scan was designed to obtain qualitative information

on educational technology used by other medical education institutions to meet curricular needs. Semi-

structured, one-hour interviews were conducted with 32 targeted institutions (a 57% response rate of

the 56 contacted institutions).

Results show that 13% (n=4) of respondents use commercial systems alone to meet their needs. 16%

(n=5) of respondents use integrated in-house developed solutions. Remaining respondents (71%) use a

combination of in-house developed and/or open source products with commercial products.

Respondents that have developed in-house systems and integrated them with commercial systems have

met curriculum mapping and management requirements to varying degrees of success. They tend to be

satisfied but face integration and customizability limitations. Furthermore, respondents indicated that

cost and access to support can be variable.

Respondents that have integrated in-house developed systems with open source systems report similar

strengths and weaknesses as above. While respondents found open source systems to be readily

customizable, factors such as cost and support availability can be difficult to anticipate.

A minority of institutions adopted multiple commercial systems. This has the advantage of predictable

costs and timelines associated with procurement, implementation, and support. However, most

respondents were not able to fully satisfy the need to map learning objectives to learning objects and

activities in a way that is meaningful to curriculum design due to limited customizability.

Scan respondents that developed or adopted integrated solutions containing curriculum management

tool (CMT), learning management system (LMS), and learning object repository (LOR) functionality

reported the highest levels of satisfaction based on the need to link learning objectives, objects, and

activities. These systems were cited as being fully integrated, customizable, and able to provide single-

authentication and role-based access through a portal. Based on user satisfaction, this option is worth

investigating further. However, due to the added complexity of integration, there is risk associated with

implementation timelines and costs which should be examined in detail.

Following this external scan, the next steps towards fulfilling the investment recommendations outlined

in the Educational Technology Strategy Report are to conduct an internal needs assessment to

determine specific requirements from a CMT, LOR, and online health education platform (OHEP) to

conduct a product comparison analysis, test a shortlist of systems and make a recommendation for the

implementation of one or more technologies. This report will supplement the outputs of these next

steps by providing strengths and weaknesses of solution types for the UBC FoM to consider.



September 13, 2010

1 Background

The University of British Columbia (UBC) Faculty of Medicine (FoM) is currently undergoing curriculum

renewal. To support this process, the Technology Enabled Learning (TEL) group developed an

Educational Technology Strategy Report that outlines five investment recommendations:

1. Synchronous Communication Platform: Continue to develop and implement synchronous

educational technology to support and expand distributed medical education.

2. Curriculum Management: Implement tools that support the process of curriculum renewal,

curriculum mapping, and the implementation of an integrated curriculum management

repository.

3. Learning Objects: Provide tools that support the modularization of educational resources to be

reused in various instructional contexts. These tools will link to the curriculum management

system.

4. Educational Technology Tools: Create mechanisms to support, implement, and integrate

educational technology tools (e.g. virtual patients, simulators, ePortfolios) into medical

education.

5. Online Health Education Platform: Create an online gateway for web-based learning resources

for UBC Health Professions education.

These recommendations are driving several projects, three of which are relevant here: Curriculum

Management, Learning Objects, and Online Health Education Platform. The charters for each of these

projects contain an environmental scan as the preliminarily step. These environmental scan components

were combined for efficiency and to ensure that information regarding integrated systems was

captured.

1.1 Purpose

The goal of the Educational Technology Integrated Environmental Scan was to obtain more information

about technology solutions that other institutions are using and how well those met their curricular

needs. While this environmental scan was the first step in the aforementioned charters, future

recommendations and decisions will also be informed by upcoming activities such as an internal needs

assessment and a formal technology options assessment, which will align needs with feasible solutions.

These additional activities, in addition to the scan results contained herein, are vital to inform solution

recommendations and decision making scheduled to take place by March, 2011.

This report is intended to present notable findings from the scan in a way that will inform the UBC FoM’s

technology recommendations and decision making process. To achieve this, the report categorizes

solutions, identifies reasons for adoption, summarizes important strengths and weaknesses associated

with solution types, and reports levels of satisfaction where available.



September 13, 2010

2 Scan Methodology

The environmental scan was designed to obtain relevant information from educational technology

representatives at target Faculties, Schools, and Colleges of Medicine within higher education

institutions in North America, the United Kingdom, and Australia. This was not designed as a

quantitative study, rather a qualitative investigation of a selected sample of institutions.

2.1 Institution Selection

Using principles from comparative research, institutions both similar and different from the UBC FoM

were selected. Many institutions were selected based on same accreditation requirements, while other

jurisdictions were selected to ensure that additional, creative solutions were also identified. The person

most directly responsible for the entire suite of educational technologies at the medical school, or

alternatively, the dean responsible for undergraduate medical education was invited to participate in an

interview. Invitations for phone or videoconference interviews were sent by email to 56 institutions. 10

Canadian, 18 US, 1 UK and 3 Australian medical schools were interviewed for a total of 32 interviews

and a 57% response rate. Interviews were conducted between July 5th and August 16th, 2010.

2.2 Data Capture and Analysis

The environmental scan was conducted using a semi-structured interview methodology using a standard

set of guiding questions. Each interview lasted approximately one hour. The data was analyzed

qualitatively to identify trends and capture the complexity of the experiences presented during the

interviews. A qualitative thematic analysis was performed on the interview transcriptions. Information

regarding reasons for adoption and level of satisfaction was parsed into themes. Examples in the

analysis were selected for breadth, showing both originality of solutions and commonly occurring

solutions and problems. Satisfaction levels were determined based on the combination of interviewees’

observations on faculty and student satisfaction as well as satisfaction from administrative program

levels and implementation perspectives. Satisfaction cannot be seen as representative of all

stakeholders at the institution, but represent a valid perspective on the experience of implementing

these solutions.

3 System Trends Summaries

This section provides an overview of approaches that medical education institutions are taking with

regards to curriculum management tools (CMT), learning object repositories (LOR), online health

education platforms (OHEP), and learning management systems (LMS). A summary table of system

integration by institution can be found in Appendix C and discussion of the integration of these systems

is in the Notable Findings Section.

3.1 Curriculum Management Tools

According to the UBC FoM’s chartered investment recommendations, the goal of curriculum

management is to “increase educational effectiveness and facilitate the accreditation process for



September 13, 2010

undergraduate medical education by: increasing consistency in educational delivery; improving the

integration between interdependent courses; and increasing teaching focus in areas where more

learning is required.”2

The UBC FoM intends to realize this goal by making the relationships between curricular objectives and

educational activities and resources more explicit and transparent by researching and implementing a

CMT that allows for integration with an LOR and LMS.

78% (n=25) of scan respondents indicated that they meet their curriculum management objectives with

some form of an electronic CMT. Of this, 18 (56% of total) respondents developed their own solution. 12

(38% of total) of these developed a system in-house and 6 (19% of total) rely on a combination of

relational databases and/or spreadsheets to manage curriculum. Some of the in-house developed

systems are available for adoption and customization by other institutions3.

13% (n=4) of respondents indicated that they are able to meet curriculum management needs with

commercial systems. Of those, two are currently using, and one is implementing, the same system

currently in use by the UBC FoM (one45). The fourth is using WEAVEOnline.

Most respondents upload learning objectives to the Curriculum Management and Information Tool

(CurrMIT), provided by the American Association of Medical Colleges (AAMC). Only two (6%) rely on

CurrMIT alone to map and manage objectives.

For more detailed information regarding CMTs including reported strengths and weaknesses, please see

Appendix A.

3.1.1 Curriculum Management Tool Adoption Drivers

Two primary CMT adoption drivers are:

• A robust centralized search feature, and

• The ability to have program-wide visibility into objective coverage without administrative

overload.

Also, similar to UBC, eight (25%) respondents adopted a CMT in response to needs identified through a

curriculum renewal process. Respondents tend to favour systems that are customizable, contain

versatile applications, and can be integrated with an LMS. Please see Figure 1 for commonly reported

CMT adoption drivers.

2 UBC FoM Educational Technology Strategy Investment Recommendation: Curriculum Management project

charter, June 11, 2010. 3 The University of California San Francisco developed a CMT called Ilios. It has been adopted and customized by

the University of California San Diego and the University of Arizona Tucson.



September 13, 2010

3.2 Learning Objects Repositories

The goal of the learning objects investment recommendation is “to increase the quality of existing and

emerging learning content for medical education and to improve the efficiency of instructional design

for faculty members by providing mechanisms to share, annotate, and access reusable learning

objects.”4 To achieve this, the UBC FoM intends to implement a Learning Objects Repository (LOR) to

enhance storage of and access to learning objects in a way that is integrated with the CMT.

Learning objects are resources that can be used for learning, education, or training. The term commonly

applies to documents and presentations that are often digital and web-based, and is increasingly used to

describe multimedia learning modules.

The majority (94%; n=30) of scan respondents use either their CMT or LMS as their LOR. Of the 21

respondents (66%) that use their LMS to house learning objects, only two are able to map learning

objectives back to objects, which is an important function to the UBC FoM. To achieve this, they

developed their own LMS or CMT.

With one exception, all scan respondents that developed comprehensive in-house solutions to meet

curriculum and learning management needs also use this same solution as an LOR. The institution with

an LOR separate from its CMT/LMS manages learning objects using digital asset management software,

developed in-house and released as open source5. This allows it to manage and tag learning objects

4 UBC FoM Educational Technology Strategy Investment Recommendation: Learning Objects project charter, June

11, 2010. 5 The University of Sydney Medical School’s MediaBank

0 2 4 6 8 10

require robust searchability of CMS content

curriculum renewal

to increase perspective of objectives coverage

customization requirement

spreadsheets and database are sufficient

report generation

versatility of applications

need to integrate the CMS with the LMS

online access

desire to tracking teaching hours

Figure 1: Most Common Factors Influencing the Selection of a CMT

Number of Comments



September 13, 2010

separately from any particular system, but still provide a link directly to objects from within the in-house

developed CMT/LMS6.

Other institutions employ commercial LORs such as Blackboard or have adapted SharePoint to achieve

LOR functionality. Institutions with lecture recording or other video storage needs tend to store these in

a separate video server, unless their system is able to store video7.

Highlighting the importance of integration across systems, institutions that reported storing learning

objects in an LMS not integrated with the CMT tend to manage learning objects within course folders. In

these cases, learning object management often then becomes the responsibility of faculty or staff

members who administer course components. Learning objects stored in this kind of folder structure are

rarely categorized, tagged with metadata or expiry dates and are often re-submitted to the system. This

creates inefficiencies, prevents comprehensive inventory scanning, reduces search efficacy, and does

not enable linking learning objectives to objects.

3.3 Online Health Education Platform and Open Educational Resources

3.3.1 Online Health Education Platforms

The goal of the OHEP investment recommendation is “to create the conditions for an enhanced learning

experience by improving coherence, delivery, and access to web-based learning resources and tools for

the undergraduate medical education program and UBC health professions education.”8 The UBC FoM

intends to achieve this through implementing a consolidated access point for web-based teaching and

learning tools and educational resources that is scalable beyond medical education to include other

health professions education.

Less than half of scan respondents (43%) host a portal that aggregates links to all relevant resources. Of

these, the majority do not seamlessly integrate resources under single sign-on functionality.

One respondent provides an integrated, single sign-on portal for sharing learning resources with

students, trainees, and faculty from registration to retirement. The system creates a convenient way to

access all learning resources which can be grouped and accessed through filters that focus on an

individual’s appropriate learning communities, along with relevant resources.9

3.3.2 Trends in Open Educational Resources in Medical Education

There is a growing initiative to freely share learning objects with other faculties, departments, or

institutions. These types of shared resources are called Open Educational Resources. In this context,

offering open educational resources permits those who are not enrolled in the medical program, and

6 The University of Sydney Medical School developed COMPASS to meet curriculum and learning management

needs 7 In-house developed systems such as HOMER and Ilios are capable of storing videos

8 UBC FoM Educational Technology Strategy Investment Recommendation: Online Health Education Platform

project charter, June 11, 2010 9 The system referenced here is the University of Alberta Medical School’s HOMER



September 13, 2010

who do not have access to the medical program’s curriculum, to search for and access those educational

resources that have been shared by the author(s) in a centralized repository.

Scan respondents are slightly more likely to share specific learning objects, such as virtual patients,

through national or international repositories. However, two respondents have means for faculty to

voluntarily contribute learning objects to an open educational resource repository supported by the

medical school itself.10

Commonly cited reasons for not engaging in an open educational resources initiative is the lack of clarity

around intellectual property rights and enforcement. Also, the availability of the popular MedEdPORTAL

amongst other national and international peer-reviewed resource repositories is another reason why

most medical schools have not created their own open resource repository.

One possibility for overcoming intellectual property issues is to only enable access to educational

resources by medical school alumni and other health profession education faculties within the same

university. However, this does not achieve the same results as a fully public repository.

3.4 Learning Management Systems

An LMS is a software application, usually web-based, that facilitates the administration and

management of learning programs and course delivery by providing access to schedules, learning

objects, assessment and reporting tools, and other collaborative communication tools.

Among scan respondents, 44% (n=14) use a commercial LMS, 53% (n=17) use an in-house developed or

open source system, and one respondent11 uses an in-house developed system in combination with a

commercial system to meet LMS functionality.

In general, respondents using commercial LMSs are less satisfied than respondents using in-house

developed or open source systems. Please see Appendix B for a list of LMSs, including some systems

that are serving as substitute LMSs, with reported strengths and weaknesses.

3.4.1 Learning Management System Adoption Drivers

The most commonly reported LMS adoption drivers are:

• The degree to which the system is customizable and interoperable with other systems (planned

or in place), and

• The degree to which the calendar is suitable to displaying the institution’s curriculum.

Respondents that use a commercial LMS frequently reported that the calendar interface was

not suitable for displaying their particular curriculum.

10

The University of Michigan Medical School’s Open.Michigan (https://open.umich.edu/) which has now expanded

to include 10 faculties within the university; and Tufts University School of Medicine’s OpenCourseWare

(http://ocw.tufts.edu/) 11

University of Melbourne Medical School



September 13, 2010

Other reported important considerations are a central search function and availability of central support

from the university IT unit. Please see Figure 2 for commonly reported LMS adoption drivers.

4 Summary of Notable Findings

The environmental scan uncovered four primary categories of solutions used by medical education

institutions to achieve curriculum mapping, learning object management, and delivery objectives:

1. Commercial systems integrated with in-house developed systems

2. Open source systems integrated with in-house developed systems

3. Multiple integrated commercial systems

4. Comprehensive systems developed in-house

No single system uncovered during the environmental scan enables an institution to completely fulfill

objectives related to curriculum management, learning objects, and online access as outlined in the UBC

FoM Educational Technology Strategy Report.

Nevertheless, scan respondents achieve similar objectives, or sub-sets of these objectives using system

combination solutions. With combination solutions, the degree of compatibility, integration, and

customizability of disparate systems determines the extent to which an institution is able to achieve

functional cohesion among learning objectives, objects, activities, and outcomes. Where data was

available, for each solution category, we have included information related to strengths, weaknesses,

customizability, integration, and general cost and support considerations. Please see Appendix C for a

summary of system integration by institution.

0 2 4 6 8 10 12

customizability/ interoperability with other systems

calendar suitability to the medical curriculum

central Searchability

system is centrally support by university IT unit

ease of use

implemented to link resources with objectives

nothing suitable on the market for medical education

robustness of features

low implementation Cost

contractual support available for this system

Figure 2: Most Common Factors Influencing the Selection of an LMS

Number of comments



September 13, 2010

4.1 Commercial Systems Integrated with In-House Developed Systems

A lack of commercially available products to meet their needs led several scan respondents to develop

custom sub-systems to meet a given functional need that was not available through commercial

software. These in-house developed systems are often integrated with commercial systems to cover

required functionality. Interestingly, the most commonly cited solution was an in-house developed CMT

integrated with a commercial LMS.

Respondents often reported that a main strength of this approach is that an institution can create a CMT

that is customizable and adaptable to its particular curriculum design. Like comprehensive systems built

in-house, custom developed systems, such as CMTs created to meet a particular set of requirements,

are amendable and can be customized to support renewal of curriculum structure or to include a variety

of functional features as needed by the curriculum. Some institutions have also integrated scheduling,

consent tracking, teaching efforts tracking, and learning object management into their in-house

developed CMT. Depending on the degree of integration with other commercial tools, this type of

solution may permit institutions to map curricular objectives with learning objects, events, and

outcomes. Finally, because this type of solution leverages a commercial component, the long-term cost

is more transparent than developing a comprehensive, in-house solution.

This type of blended solution presents challenges related to the way in which information is displayed

and accessed. While system integration can permit curricular objectives to be mapped to learning

objects from a planning and management perspective, respondents indicated that the content from the

CMT is often displayed in the commercial LMS in ways that are not be appropriate to the curriculum.

Specifically, respondents indicated that commercial LMSs are often unable to display objectives in a

calendar view that can accommodate longitudinal, spiral, or integrated nature of some curriculum

models. Further, while a single point of authentication is sometimes possible, users may still need to be

given direction regarding which information to look for in each of the systems. Commercial products

offer varying levels of adaptability and system customization that may hinder full integration with an in-

house developed system. Finally, venders’ level of customer support varies and does not extend to in-

house developed systems that are integrated with their product.

Institutions which have developed noteworthy solutions which integrate CMTs and other sub-systems

with commercial systems include the University of New South Wales Faculty of Medicine, the University

of Vermont College of Medicine, the University of Pittsburgh School of Medicine, and the University of

Colorado Medical School. Memorial University of Newfoundland Faculty of Medicine is adopting a

similar approach as part of its curriculum renewal program.

4.2 Open Source Systems Integrated with In-House Developed Systems

Some scan respondents elected to pair an in-house developed system with an open source system

(instead of a commercial system as described above). As described in the previous section, these

institutions developed in-house sub-systems to meet a functional need that was not available through

other systems. Many strengths and weaknesses of integrating an in-house developed system with an

open source system are similar to those of integrating an in-house system with a commercial system.



September 13, 2010

However, one of the key differences is that open source systems tend to be more customizable than

commercial systems. Furthermore, open source software enables the institution to participate in an

open source community which contributes to the pool of system enhancements over time.

An important consideration with open source LMSs is that costs associated with implementing,

customizing, and supporting these systems can be difficult to anticipate. Furthermore, while support is

available for some, it is not consistently available for all and can be difficult to obtain.

Currently, no single open source system exists which can fulfill curriculum and learning object

management functionality. Two institutions that have developed or adopted in-house systems and

integrated them effectively with an open source LMS include the University of California San Francisco

School of Medicine and the Northern Ontario School of Medicine. The University of Arizona Tucson

Medical School has adopted an in-house CMT (which is soon to become open source) from another

institution and integrated it with their own in-house developed LMS.

4.3 Multiple Integrated Commercial Systems

The next category is solutions achieved through integration of multiple commercial systems. This is

currently the approach the UBC Faculty of Medicine has taken to address its curriculum and learning

management needs. Four scan respondents employ a similar strategy.

As with other categories, no single commercial product was found which can meet curriculum mapping

and management, and learning object delivery and management functionality. The main strength of

commercial systems is the ability to plan and forecast costs and timelines associated with acquisition,

implementation, and support with greater confidence. This has significant benefits for budgeting and

scheduling. Furthermore, commercial systems can be implemented more quickly because development

time is not an issue.

The commercial applications discovered through this scan generally support only limited customization

and often present both functional and interoperability issues. This lack of interoperability among

commercial systems leads to the inability to efficiently and consistently share information across

systems. Low degrees of customizability result in the inability to change system features or make

adjustments to more closely align with a given institutions curriculum design and/or renewal process.

Respondents often cited the inability to reflect longitudinal, spiral, integrated, or case-based medical

curricula as a key drawback of commercial CMTs. Furthermore, multiple commercial systems often

imply multiple log-in requirements. Finally, commercial system licensing costs can be high and

sometimes involve multi-year contracts with vendors, reducing future flexibility.

Of the limited number of commercial CMTs available, one system identified through internet research,

but not used by any scan respondents was Agresso Curriculum Management System (ACMS). Please see

Appendix D for information on functionality. This product may be worth investigating further.



September 13, 2010

4.4 Comprehensive Systems Developed In-House

A lack of commercially available products that enable institutions to meet their objectives has led some

institutions to develop solutions in-house. Comprehensive systems developed in-house are all-in-one

solutions which are created and fully maintained by one institution or a small consortium. These may or

may not be available under an open source license, but are often free to other institutions by

arrangement.

By nature, these types of systems can be built to maximize flexibility and customizability. They enable

medical schools to illustrate the link among objectives, learning objects, and delivery thereof in

meaningful ways by facilitating all mapping, planning, management, and learning activities within one

system. Furthermore, they contain scheduling systems and calendar interfaces that are customizable to

a given medical school’s curriculum model, whether it be longitudinal or spiral in nature, for example.

These systems also have central search capabilities, which simplify the user experience. Finally, in-house

systems are generally able to tailor role-based access profiles to the specific institution’s needs and be

adapted quickly to changes in the curriculum.

The primary drawback of developing a comprehensive system is the large development effort and high

ongoing support costs associated with an in-house development team. Nevertheless, institutions that

developed or adopted all-in-one systems that contain CMT, LMS, and LOR functionality, reported the

highest levels of satisfaction.

Interestingly, one respondent12 is using TUSK, developed in-house by Tufts University, for no licensing

fee. This approach can mitigate some cost and time issues associated with development.

Scan respondents that have developed noteworthy comprehensive systems include the University of

Calgary Faculty of Medicine, the University of Alberta Faculty of Medicine and Dentistry, the University

of California San Diego Medical School, Tufts University School of Medicine, and the University of Sidney

Medical School.

Another custom-developed system discovered through online research is OpalQM. The University of

Manitoba Faculty of Medicine developed this system which has now been commercialised. According to

online marketing material, this system has much of the same functionality as other all-in-one systems13.

For more information, please see Appendix D. As this system was not discovered as part of the

environmental scan, it may be worth investigating further.

5 Conclusions and Next Steps

Scan respondents that developed or adopted integrated solutions containing CMT, LMS, and LOR

functionality reported the highest levels of satisfaction based on the need to link learning objectives,

objects, and activities. These systems were cited as being fully integrated, customizable, and able to

12

The University of Hawaii JABSOM 13

http://www.opalqm.com/Pages/home.aspx



September 13, 2010

provide single-authentication and role-based access through a portal. Based on user satisfaction, this

option is worth investigating further. However, due to the added complexity of integration, there is risk

associated with implementation timelines and costs which should be examined in detail.

Respondents that have integrated in-house developed systems with commercial systems have achieved

mapping and management requirements to varying degrees of success. Institutions that have taken this

approach tend to be satisfied but also face limitations of partial integration and challenges related to the

inability to customize the commercial system to match curriculum design. Furthermore, obtaining

support can be expensive and challenging for some product types.

Respondents that have integrated in-house developed systems with open source systems report similar

strengths and weaknesses as above. While respondents found open source systems to be readily

customizable, factors such as cost and support availability can be difficult to anticipate.

A minority of institutions adopted multiple commercial systems. This has the advantage of predictable

costs and timelines associated with procurement, implementation, and support. However, most

respondents were not able to fully satisfy the need to map learning objectives to objects and activities in

a way that is meaningful, given particularities of curriculum design. Further, product licensing costs can

be high for many commercial products and supply contracts can be restrictive.

This environmental scan focused on gathering experiences presented by medical education institutions

external to the UBC FoM. The next steps towards fulfilling the investment recommendations outlined in

the Educational Technology Strategy Report are to conduct an internal needs assessment to determine

specific requirements from a CMT, LOR, and OHEP, to conduct a product comparison analysis, test a

shortlist of systems and make a recommendation for the implementation of one or more technologies.

This report will supplement the outputs of these next steps by providing strengths and weaknesses of

solution types for the UBC FoM to consider.



September 13, 2010

Appendix A: CMTs by Software

CMT Type Medical Schools

Number of

Respondents

Using System

Key Strengths Key Disadvantages

cMap Developed

In-house

University of

Toronto 1

Robust searchability of content; search results

displayed by tiers of relevancy; tracking by

thread/metadata; Suitability to the Medical

Curriculum

none reported

Compass Developed

In-house

University of

Sydney 1

Role-based access and customizable views of content;

Easy to use; Friendly interface; customizable and

flexible; Appropriate to the medical curriculum;

Robust scheduling and calendar system that is suitable

to the medical curriculum; Centralized Search; Links

resources with objectives;

Integrated with assessment systems that evaluates

question performance and links results back to

objectives; Great reporting system

Currently does not include

social networking tools

Curriculum

Management

System at U.

of Vermont

Developed

In-house

University of

Vermont 1

Tracks by thread/metadata; tracks teaching hours;

suitable to the medical curriculum; podcast consent

and schedule management

none reported

CURE Developed

In-house

University of

California San

Diego

1

Centralized Search; Searchable at granular levels;

Tracks faculty teaching efforts; Links resources with

objectives; Customizable and flexible

none reported

CurrMIT Provided by

the AAMC

University of

Hawaii; Indiana

University;

2 Ability to compare to other institutions; Good basic

report generation

Data not comprehensive;

System not functionally

rich or up-to-date enough

to rely on.

Spreadsheets

/ Database

solutions

Developed

In-house

Memorial

University;

University of

Washington;

University of

Melbourne; McGill

University; Johns

Hopkins University

5 high degree of customization

Solution is not functionally

rich; interoperability

problems with other

systems

eMed Developed

In-house

University of New

South Wales 1

Searchable, can be adjusted to program needs; very

effective in management of curriculum and reporting;

allows students to monitor own progress; integrates

multiple systems: curriculum map, graduate exit

competencies, timetables, portfolios for performance,

and learning project portfolios; clinical placements,

group activities. Calendar is downloadable with mobile

technology. eMed can produce reports for tutor

performance, ex. how many quizzes they have

marked.

$80-130 K per year on

development of eMED

Entrada

Developed

In-house;

open-source

University of

Calgary 1

Customizable and flexible; Appropriate to the medical

curriculum; Robust scheduling and calendar system

that is suitable to the medical curriculum; Centralized

Search; Links resources with objectives; Low Cost;

Integrated with curriculum mapping system;

Sophisticated search; Strong administrative system;

Responsive open source community

none reported



September 13, 2010

CMT Type Medical Schools

Number of

Respondents

Using System


HOMER Developed

In-house

University of

Alberta 1

Customizable and flexible; Sophisticated search;

Role-based access and customizable views of content;


Appropriate to the medical curriculum; Robust

scheduling and calendar system that is suitable to the

medical curriculum; Integrates with EMR system; Links

resources with objectives

Feature over-load;

Increasingly complex

navigation

Ilios v1

Developed

In-house;

v2 Open-

source in

2011

University of

California San

Francisco;

University of

Colorado;

University of

Arizona Tucson

3

Robust search by mesh terms, tags, discipline, course,

faculty member. Stores learning objects, tracks

teaching hours; allows tagging of learning objects with

expiration dates; highly customizable.

Ilios v1 does not include a

student focused interface

or student tools for

objective tracking. Does

not provide a taxonomy

layer to curriculum

mapping; inadequate

reports/analytics

MyLearning

Online

Developed

In-house

Northern Ontario

School of

Medicine

1 highly customizable, strong report generation

features, tracking by thread/metadata none reported

Navigator Developed

In-house

University of

Pittsburgh 1

Robust search; simple to use interface; robust object

repository; automatic upload to CurrMIT

Needs a code upgrade to

increase operating speed

one45 Commercial

Dalhousie

University;

University of

Saskatchewan;

University of

Western Ontario

& Windsor

3

Includes scheduling and logging features; appropriate

for basic objective mapping; easy implementation;

online; searchable

Does not integrate well

with other products; no

major customization is

possible; vendor is slow to

changes; features are

basic; no visual map

display of objective

coverage; system

performance issues

TUSK Developed

In-house Tufts University 1

Customizable and flexible; Sophisticated search; Role-

based access and customizable views of content;


Appropriate to the medical curriculum; Robust

scheduling and calendar system that is suitable to the

medical curriculum; Centralized Search; Suggested

keyword search function; Links resources with

objectives; Contractual support available for this

system; Low implementation costs (based on

University of Hawaii's experience who adopted the

system from Tufts).

4 FTE Programmers

provide support for TUSK

Weave Online Commercial Charles Drew

University 1

easy to use; easy to implement; low implementation

cost; appropriate for mapping the medical curriculum none reported



September 13, 2010

Appendix B: LMSs by Software

LMS Type Medical Schools

Number of

Respondents

Using

System


Angel Commercial Indiana University 1 Customizable script; interoperable with

other systems; easy to use;

Scheduling/calendar feature is not

appropriate to medical school

ArizonaMed Developed

In-house

Arizona University

Tucson; Arizona

University Phoenix

2

Customizable and flexible; Ties in multiple

systems; Robust scheduling and calendar

system that is suitable to the medical

curriculum; Calendar-based view;

Integrates widgets; Integrated with

curriculum mapping system

Possibility of customization has

increased feature requests

Blackboard Commercial

Charles Drew

University; University

of Melbourne;

University of

Colorado; University

of New South Wales;

University of

Vermont; University

of Toronto; Johns

Hopkins University;

University of Nevada

8 Easy to use; Centralized online access;

Searchable by keyword and free text

No contextual tools for learning

objects; Courses are in silos ; No self-

performance tracking tools; Difficulty

in uploading materials; Challenging

interface; Does not interoperate well

with other systems; Not suitable for

displaying objectives in meaningful

ways; Scheduling/calendar feature is

not appropriate to medical school;

Weak assignment grading tool; Weak

reporting and analytics features; Weak

assessment features; Poor support

from vendor

Catalyst Developed

In-house

University of

Washington 1

Easy to use; Friendly interface; Perfect for

faculty intimidated by Blackboard and

WebCT; Dropbox feature; Strong

communication features

No centralized search; Not as feature

rich as other systems; Weak reporting

and analytics features

Compass Developed

In-house University of Sydney 1

Role-based access and customizable views

of content; Easy to use; Friendly interface;

customizable and flexible; Appropriate to

the medical curriculum; Robust scheduling

and calendar system that is suitable to the

medical curriculum; Centralized Search;

Links resources with objectives;

Integrated with assessment systems that

evaluates question performance and links

results back to objectives; Great reporting

system

Currently does not include social

networking tools

CURE Developed

In-house

University of

California San Diego 1

Centralized Search; Searchable at granular

levels; Tracks faculty teaching efforts; Links

resources with objectives; Customizable

and flexible

none reported

DalMedix

(Coldfusion)

Developed

In-house Dalhousie University 1

Works well on a basic level; Strong

document management System is outdated

Desire2Learn Commercial Memorial University 1 none reported

Inappropriate for the medical

curriculum; Calendar is not suitable to

the medical curriculum; unable to

show objectives and activities in a

relational manner

E*Value Commercial Arizona University

Phoenix 1

Content delivery, evaluations, and portfolio

development

Not used for first and second year UG

students; not integrated with other

system

Entrada

Developed

In-house;

Open source

University of Calgary 1

Customizable and flexible; Appropriate to

the medical curriculum; Robust scheduling

and calendar system that is suitable to the

medical curriculum; Centralized Search;

Links resources with objectives; Low Cost;

none reported



September 13, 2010

LMS Type Medical Schools

Number of

Respondents

Using

System


Integrated with curriculum mapping

system; Sophisticated search; Strong

administrative system; Responsive open

source community

HOMER

Developed

In-house;

Open source

University of Alberta 1

Customizable and flexible; Sophisticated

search; Role-based access and

customizable views of content; Integrated

with curriculum mapping system;

Appropriate to the medical curriculum;

Robust scheduling and calendar system

that is suitable to the medical curriculum;

Integrates with EMR system; Links

resources with objectives

Feature over-load; Increasingly

complex navigation

Moodle Open source

Northern Ontario

School of Medicine;

University of

California San

Francisco;

University of Sydney;

St. George's

University London;

University of Virginia

5

Software is customizable;

Scheduling/Calendar is suitable to the

medical curriculum; Free software;

Software supports growth; Responsive

open source community; Commercial

support is available; Easier to use than

Blackboard; Good Interface

Scheduling/Calendar is not suitable to

the medical curriculum; Lacks a central

search function; Weak in curriculum

navigation between courses; Courses

are in silos: No deep or long structure

to display continuity of courses or

objectives; New features coming out all

the time; difficult to keep track of

upgrades; Weak reporting and

analytics features

Sakai Community /

Open source

University of North

Carolina;

University of

Michigan;

Oregon Health and

Science University

3

Customizable and flexible; Calendar is

suitable for the medical curriculum;

Student group management; Easy transfer

of resources between folders; Commercial

support is available; Responsive open

source community

Interface problems with authenticating

between systems; Weak search

function; Scheduling/calendar feature

is not appropriate to medical school;

Does not provide a way for students to

track their objectives across the

program; No contextual tools for

learning objects

SharePoint Commercial University of

Pittsburgh 1

Customizable and flexible; Interoperable

with other systems; Integrates with

curriculum mapping system; Allows

students to create own team group

websites; Contractual support available for

this system

Was not designed for learning; New

versions of SharePoint are not

backwards compatible; Not

transparent in the code/difficult to

look under the hood; Not easy to

troubleshoots; not browser or platform

independent

TUSK Developed

In-house

Tufts University;

University of Hawaii 2

Customizable and flexible; Sophisticated

search; Role-based access and

customizable views of content; Integrated

with curriculum mapping system;

Appropriate to the medical curriculum;

Robust scheduling and calendar system

that is suitable to the medical curriculum;

Centralized Search; Suggested keyword

search function; Links resources with

objectives; Contractual support available

for this system; Low implementation costs

(based on University of Hawaii's experience

who adopted the system from Tufts).

4 FTE Programmers provide support for

TUSK

WebCT Commercial

McGill University;

University of

Western Ontario &

Windsor (Schulich)

2 Satisfactory search function; Student group

management

Scheduling/calendar feature is not

appropriate to medical school; No

contextual tools for learning objects;

Courses are in silos: No deep or long

structure to display continuity of

courses or objectives; No self-

performance tracking tools



September 13, 2010

Appendix C: System Integration by Institution

Curriculum Management Tool Learning Management System

CMT/LMS

Integratio

n

Learning

Object

Repository

Landing

Page

University of

Alberta

System HOMER HOMER v2

Yes HOMER Yes

Satisfaction High High

Future

Direction

Working on the next version;

incorporating all assessment and

residency training into Homer;

improving interface: more like iPad

with selectable apps.

Working on the next version;

incorporating all assessment and

residency training into Homer;

improving interface: more like iPad

with selectable apps.

University of

Arizona,

Phoenix

System None ArizonaMed and E*Value

No ArizonaMed No

Satisfaction ArizonaMed = high

E*Value = high

Future

Direction

Develop an ArizonaMed core (core

management system) and have each

campus individually customize modules

around it.

University of

Arizona,

Tucson

System Ilios v1 ArizonaMed

Yes ArizonaMed Yes

Satisfaction Medium High

Future

Direction

Phase out ASP code and match

Coldfusion which ArizonaMed is based

on.

Developing ArizonaMed v2. Launch

September 2011.

University of

California, San

Diego

System CURE CURE

Yes CURE Yes Satisfaction High High

Future

Direction

Will Improve reporting and automatic

export to CurrMIT.

University of

California, San

Francisco

System Ilios Moodle

Yes Ilios Yes

Satisfaction High Medium: weak features for medical

curriuclum but easy to use

Future

Direction

Ilios 2.0 will be released as open source

in 2011. The focus in this release is

access to students. It will interact

better and will focus on features for

students to interact with curriculum

mapping. Improved workflow

management and content review

features; course and student group

management, course calendar can be

downloaded. Improved interaction

with Moodle.

Moodle 2.0

University of

Calgary

System Entrada (Called Osler at U of C) Entrada (Called Osler at U of C)

Yes Blackboard No Satisfaction Yes Yes

Future

Direction

Charles Drew

University

System Weave Online Blackboard (non-enterprise)

No Blackboard No Satisfaction Yes Low

Future

Direction

University of

Colorado

System Ilios Blackboard

Yes Ilios No Satisfaction Medium Low

Future

Direction

Ilios 2.0 Ilios 2.0



September 13, 2010


CMT/LMS

Integratio

n

Learning

Object

Repository

Landing

Page

Dalhousie

University

System One45 (implementing this year) DalMedix (Coldfusion)

No,

Exploring

Current:

DalMedix

(Coldfusion)F

uture:

SharePoint

Yes Satisfaction TBD Low

Future

Direction

One45 one45/Blackboard

University of

Hawaii,

JABSOM

System CurrMIT TUSK

No TUSK No Satisfaction Medium High

Future

Direction

Looking forward to CurrMIT upgrade:

Curriculum Inventory Portal

Indiana

University

System CurrMIT Angel

No Angel Yes

Satisfaction Medium Medium

Future

Direction

Ilios (exploring) Exploring other systems; Evaluating

Sakai at the moment

Johns Hopkins

University

System Spreadsheets/Databases Blackboard (non-enterprise)

No Blackboard Yes

Satisfaction High Medium (poor features but low cost

and easy to use)

Future

Direction

Expanding databases and making them

relational

University of

Nevada

School of

Medicine

System None Blackboard (Web Campus)

No Blackboard No

Satisfaction Medium

Future

Direction

Moving to Blackboard Learn

McGill

University

System Spreadsheets/Databases WebCT

No WebCT No

Satisfaction Medium Medium

Future

Direction

Exploring OPAL and Vanderbilt system Exploring OPAL and Vanderbilt system

McMaster

University

System Home-grown database None (D2L being used by central

campus)

No Alfresco Yes Satisfaction No data – still in development n/a

Future

Direction

Looking for a system that will handle

playback and tracking of SCORM

packages

University of

Melbourne

System File Maker PRO Database Blackboard & CATS

Currently

No, Yes

with

Curriculu

m

Connect

Currently

Blackboard,

Curriculum

Connect in

the future

No

Satisfaction Low Low

Future

Direction

Developing Curriculum Connect. Will

be implemented in 2011. It will be able

to store resources, be fully searchable

with a Google customized search, it will

allow faculty to search all relevant

teaching resources relating to

objectives.

Developing Curriculum Connect. Will

be implemented in 2011. It will be able

to store resources, be fully searchable

with a Google customized search, it will

allow faculty to search all relevant

teaching resources relating to

objectives.

Memorial

University

System Databases Desire2Learn

No Desire2Learn Yes

Satisfaction Low Low

Future

Direction

Putting the database online, changing

it to SQL and overlaying it with Think

Map.

Exploring other systems.

University of

Michigan

System None Sakai

No Sakai No

Satisfaction Low

Future

Direction

Yes; over the next two years, the

medical school will make some

upgrades to Sakai: enterprise search;

incorporate lecture capture;

introduced team based tools. UofM



September 13, 2010


CMT/LMS

Integratio

n

Learning

Object

Repository

Landing

Page

will try to release Sakai 3 in 2012 or

2013. The objective is to have more

features that meet the needs of

medical schools as well.

University of

Nevada

School of

Medicine

System None Blackboard - Web Campus

No Blackboard No Satisfaction No data

Future

Direction

Moving to Blackboard Learn

University of

New South

Wales

System eMed Blackboard and eMed

No eMed Yes Satisfaction High Low

Future

Direction

University of

North Carolina

System None Sakai

NO Sakai No Satisfaction Medium

Future

Direction

Northern

Ontario

School of

Medicine

System MyLearning Online Moodle

No;

exploring

MyLearning

Online

No,

Piloting

Satisfaction High High

Future

Direction

MyCurriculum Online

Oregon Health

and Science

University

System None Sakai

No Sakai No Satisfaction High

Future

Direction

University of

Pittsburgh

System Navigator Navigator & SharePoint

Yes Navigator No Satisfaction Medium High

Future

Direction

University of

Saskatchewan

System one45 Blackboard

No Blackboard

Yes

Satisfaction High (integration is not a top priority) Low

Future

Direction

Potential for change as curriculum

renewal begins.

St. George's

University

London

System None Moodle

No Moodle No Satisfaction n/a High

Future

Direction

University of

Sydney

System Compass Compass & Moodle

Yes MediaBank Yes Satisfaction High High

Future

Direction

University of

Toronto

System cMap (SQL database) Blackboard

Yes Blackboard No Satisfaction High Low

Future

Direction

Tufts

University

System TUSK TUSK

Yes TUSK No Satisfaction High High

Future

Direction

Interface and navigation upgrade Interface and navigation upgrade;

leaning towards open source

University of

Vermont

System Curriculum Management System Blackboard Yes

Currently

Blackboard, Yes

Satisfaction High Medium



September 13, 2010


CMT/LMS

Integratio

n

Learning

Object

Repository

Landing

Page

Future

Direction

Improving tracking and searchability

within objects

Would like to replace Blackboard with

an open source system within three

years.

SharePoint

in the future

University of

Virginia

System None X-credit (Built on Moodle). Is mainly

used for organizing learning resources.

No data No No data Satisfaction Do not know yet; still undergoing

development

Future

Direction

University of

Washington

(WWAMI)

System Spreadsheets/Database Catalyst

No Catalyst Yes Satisfaction Low Medium: weak features but easy to use

Future

Direction

exploring E*Value

University of

Western

Ontario &

Windsor

System one45 WebCT

No WebCT No Satisfaction Medium Low

Future

Direction



September 13, 2010

Appendix D: Other Commercial Systems

Agresso Curriculum Management System (ACMS)

Although designed for higher education in general, this product suite offers applications including

curriculum management, student management, student engagement, and others. It also contains

document management and workflow integration modules. The curriculum management component

supports the development, approval, publication, and quality assurance of units, courses, and

programs.14

OpalQM

Another custom-developed system discovered through online research is OpalQM. The University of

Manitoba Faculty of Medicine developed this system which has now been commercialised. According to

online marketing material, this system can be used to manage and report on curriculum; map

curriculum to experiences and instructors; map instructor, facility, and student evaluations; manage

learning materials; and enhance communication, information sharing, scheduling, and reporting15. This

system was not captured as part of the environmental scan; further investigation may be warranted.

14

http://www.unit4software.co.uk/products/agresso/higher-education 15

http://www.opalqm.com/Pages/home.aspx

Documents

Educational Technology Integrated Environmental Scanmed-fom-medit.sites.olt.ubc.ca/files/2013/06/Notable-Findings-Repor… · Prepared by: Technology Enabled Learning, Faculty of