DEPLOYING MISSION CRITICAL

LMS USING OPEN SOURCE

SOFTWARE

Leigh Jin

San Francisco State University

OMG Standards for FOSS Governance Workshop

Santa Clara, California

December 11, 2013

OUTLINE

Introduction

Research Questions

Methodology

Research Findings

Discussions and Conclusions

INTRODUCTION

Open Source Learning Management System – an

attractive alternative: as institutions of higher

education confronting budget cuts and financial

crisis

Low cost (+): no licensing fee

Flexibility (+): open source code means

customizability

Deploying and maintenance cost (-)

Hardware

People with Technical Skills

INTRODUCTION: NACUBA SURVEY

NACUBA survey (2004) finds members perceive

open source software as a viable alternative to

commercial software due to:

Open standards and interoperability with other

application systems (61%)

Freedom to modify the code (58%)

Software designed by and for the industry (58%)

Lower cost of ownership (55%)

NA

CU

BA

refe

rs to N

atio

nal A

ssocia

tion

of

Colle

ge a

nd

Un

iversity

Bu

siness O

fficers

OSS DEVELOPMENT VS. IMPLEMENTATION

OSS Development: two assumptions:

Developers are users with “itch to scratch”

OSS movement is unique because of the way software is being

developed

OSS Implementation: Fitzgerald and Kenny

(2003) study OSS implementation in an Ireland hospital leads to cost

savings of €13 million over five years

Free access to source code is not the primary motivation for

adoption

INTRODUCTION OSS IMPLEMENTATION:

BENEFITS VS. BARRIERS:

OSS Implementation

Benefits • Low licensing cost

• Open standard and interoperability, avoid vender

“lock-in”

• Freedom to modify the code

• Greater choice and control for end-users

• Innovation

Barriers • Lack of documentation and support service from the

OSS provider

• Technical skills required to implement, customize,

and maintain the software

• Steep learning curve and end user training and

support

CASE STUDY RESEARCH QUESTIONS

How OSS LMS is implemented at an University

in California

The motivation, benefits, and challenges

associated with adopting open source LMS in the

public education sector

How its impacts are perceived by the

stakeholders ?

INTRODUCTION: IMOODLE

IMPLEMENTATION FRAMEWORK

OSS LMS Community

•Frequent releases on weekly/monthly basis

•Global audience: 40079 sites from 203 countries

•186 Moodle developers (with write access) from all over the world

University Deployment

•New release per semester

•Designed specifically to suit the local needs for a University in California

•Developed and supported by Academic Technology Group within the university

A Course Using

iMoodle

•Contents are updated throughout the semester

•Features of iMoodle are selectively chosen to meet the pedagogy needs for a specific class

•Designed and managed by the instructor of the class

OSS IMPLEMENTATION: STAKEHOLDER

VIEWS

Technologists

Ideology towards Common Good

Software Development Methodology

Security & Risk Management

Adoption of Innovation

Total Cost of Ownership

Educators

Constructivist Learning Environments

Online Pedagogy

Commercial LMS Limitations

Easy to Use

Administration

Cost Saving

Strategic Benefits

User Satisfaction

Policy

METHODOLOGY

Interpretive Case Study

Participant Observation

Longitudinal Study 2008-2013

Interviews with

Administrators

IT Staffs

Faculty

Focus Group study with 40 students (college of

business)

IMOODLE IMPLEMENTATION MOTIVATION:

Limitations of Commercial LMS

10-12 years experience with Blackboard

Earliest adopters of version 6 on 2002

We had a class that went over 1000

students…Blackboard forced us to break a class into

two sections

We ended up having blackouts, we had 13 hours

outage during the finals week…

we find that they are not on our timeline… they said

we would fix it in six months, when we release that

new version. But for us, we need the answer now…

Officially move to Moodle on 2007, currently having 26,000

(out of 30,000) students uses iMoodle (customized Moodle) in

their classes.

IMOODLE INITIAL IMPLEMENTATION:

SCALABILITY AND STABILITY ISSUES

“We have some major slow time, where people would login

and it would take a minute for the page to load…”

“We found that there are a number of issues that cause this,

part of it was some of the ways that people were using the

tool, part of it was how it was configured, part of it was our

hardware…”

“but when you have 1,500 students in one class, with the

same deadline of midnight on Wednesday, and they are all

taking a quiz, we would see 16,000 quiz attempts in a 12

hours period, nobody else in the whole [Moodle] community

has seen this.”

ADMINISTRATOR’S TAKE ON OSS

“So I didn’t have anything to do with that decision. I came from a

really structured project environment for a proprietary software…”

“There wasn’t a project plan that I’m used to seeing, there wasn’t a

scope of the project…”

“What is this leap of faith into this open source technology

without this documentation?”

“That scared me, I’m still scared. And I don’t know if I would have

made the decision to go open source given the situation that I came

into.”

UPGRADING IS AN ISSUE

“The tipping point I think was 1.8… based on the manpower that

we have and the resources, we can't test this out thoroughly.

But the [manager] really wanted to upgrade to 1.8, and it was just

a nightmare”

“The automated process would not complete, we were manually

downloading the files for each course, manually takes a 1.5

courses and make it become 1.8 compatible and uploaded into 1.8

and restore it…There were thousands of courses there. And our

team stayed until 11pm.”

WHEN TO UPGRADE ?

“First of all there were no procedures in place to define what should be the

process of maintaining and upgrading these systems…”

“Everybody said, well, this is open source and when an open-source bug is

fixed we implement it and we keep going…”

“But you have to follow the natural timeline for your business

processes, which in this case is the academic process. The academic

process has a beginning of the academic year and an ending of the

academic year...”

“So every academic year we would do a major change. Minor changes

like bug fixes we would allow if it’s something critical, but we wouldn’t go

from a Moodle, let’s say, 1.84 to 1.92 or something. Only on an annual

basis, where there would be major changes.”

ISSUE OF TESTING

“They had no testing, but for ongoing work you have to decide

when you are going to implement changes, test them and then put

them into production. So you have to distinguish between

development, testing and production…”

So our second task was to instill a process for upgrades and

testing and we hired student assistants to write test case

scenarios.”

ISSUE OF VIRTUALIZATION

“You’ve got to remember, we’ve got Linux upgrades, we’ve got Apache

upgrades, PHP upgrade, and on top of that you’ve got the MySQL. So all

four components plus Moodle are changing. It's a lot of different pieces

that have to be synchronized…”

“So what we would do is to create a virtual system which consisted of the

Linux operating system, that was the platform, Apache, PHP and MySQL

of a particular version, married to a particular Moodle version with its

content database. And we would create a new virtual system for every

new semester. So we would do this on an annual basis…”

Once the concept of virtual machine is accepted, then it becomes a much

more stable environment and also a scalable environment…”

PROJECT MANAGEMENT SKILLS IS

NECESSARY

“It doesn’t matter whether it’s open-source or private or in an academic

institution, it has to be treated like any other piece of software that

goes into production…”

“And these people don’t know the ABCs of project management. That’s the

problem, they grow up as developers, from developers they became sort of

supervisors and they don’t have any skills in project management…”

“Open source gets hurt more because they don’t have the discipline

of project management…”

“I took what we taught in project management class and implemented it

step-by-step. This is what had to be done…”

THE MOODLE COMMUNITY

“The Moodle core team members, it’s a very interesting mix of younger

developers who always want to just try to implement new features

without thinking about usability, without thinking about

scalability or how easy it can be maintained…”

“Also, it is very different from Linux kernel, where most of the developers

really know the code and also they are the users of the code. The Moodle

community is different in the way that most of the users might be

teachers and students and they might not know all the technical

details…”

“So whoever will be paying the Moodle core team a lot of money and say,

‘hey, you do this,’ they would just do it. And after they do it, most likely the

newly written code will be added into the core code and got released despite

that it might not be really helpful for most of the university…”

CONTRIBUTING BACK TO MOODLE

COMMUNITY

“The way that I have kind of established my own voice is to volunteer a lot

in the community. Either help out with testing or with documentation or

with reports and things like that…”

“We have taken ownership of certain modules or blocks as they are called.

So now we maintain the particular QuickMail and Gradebook Plus for

the whole community. So when an upgrade happens, we test them to make

sure they works with the upgrade and make any changes necessary and

then release them back to the community…”

ESTABLISHING RELATIONSHIP WITH THE

MOODLE CORE TEAM

“Because of our size, right now I think we have 26,000 out of the 30,000

students use it for one or more classes, some of the things that we’ve been

experiencing that other campuses have not…”

“We’ve been in communication with the core team: here are some issues

with scalability that, when [an university] gets big, they are going to face

these things too so it’s to your advantage to kind of learn from us and make

the fixes… “

“They already recognized us as [a national leader]. They’ve had me

be the keynote speaker last year. We co-organized the Moodle conference

this year. And then people call me from around the country to ask

questions. And some from Canada. I’ve had one from Israel. Santa

Barbara. All over the place.”

“In 2008 Moodle conference, the core team actually came to our campus to

visit every one of us, and that’s because they recognized us.”

CONTRIBUTIONS TO EDUCATION

COMMUNITY

“When I consult with different campuses, I give them strategies for finding

answers: find a small group of local campuses that are either a little bit

ahead of you or at the same place [in Moodle implementation] and get their

phone numbers. So I said, I can be one of those phone numbers…”

“We have the Quick Guides project which is how-to documents for people,

to learn how to use iMoodle. And because of this consortium with other

Moodle campuses, we modified the quick guides tool to allow multiple

campuses to use the same information…”

“We worked with [some community colleges], so we gave them all of our

training materials and our Moodle course for instructors which has lots of

pedagogical information and technical how-tos… And then a month later

they said we made podcasts from all of your documents would you like

those?”

BENEFITS TO CAMPUS

Revenue Generation: “We could be hired by other campuses with zero

programmers to make a widget. We can provide help desk for a campus

that has no help desk. We can bring money in through support,

through development or even through consulting. We can start

charging for me to go to campuses or something.”

Curriculum Innovation: “we have been able to encourage many more

faculty to move beyond just storing their documents for students to

download. Now they are engaging students in different types of activities

like discussion forums or self-assessment quizzes or chats or wikis…

Administrative Use: “we have over 250 organizations and committees

using iMoodle for non-coursework. We have HR who wants to use it for all

their training sessions. I’m trying to push the defensive driver training

because I don’t want to go to that room for three hours...”

Other OSS Projects on Campus: I think the campus has gained a whole lot

more in terms of those commonalities that we have across projects than

just paying a fee to some service provider to provide us with a service.

TECHNOLOGIST: WHY WE NEED TO

CONTRIBUTE BACK

“Because the more you customize, the next time you want to upgrade, when

you’re into a new version of Moodle, you can’t just upgrade to what the new

Moodle community says is Moodle. You’ve got all these dependencies and

customizations that you have to make sure they work. Now you have to re-

customize all of these things.”

ADMINISTRATOR’S TAKE ON

CONTRIBUTION OUTSIDE OF CAMPUS

“In the early days, I participated in Moodle forum for outreach reasons, we

encountered a lot of stability problems and we really need help. So it

started for work purposes, not for personal gain. But later on, people

noticed that I gained too much visibility in Moodle community, so now we

had a policy that doesn’t allow forum participation during office hours, so

I have to use my spare time to do it instead.”

“Speaking of revenue, one of our staff members is extremely generous and

was making a lot of visits to other campuses to spread the word of Moodle.

And you can put a price on that. At $300 a day it looks like you just gave

away $4000 worth of work. And some of our needs on our own campus are

not being met. So I said, we have to stop that, and if they want you to come

then we will set up a costing structure for what we will charge internally

for consultations and what we charge externally.”

STAFF RETENTION ISSUE

“We lost seven people in seven weeks this summer. And the other

manager and I, who oversees this group, we talked about having

posttraumatic stress disorder because when anybody would ask to meet

with us privately, we were afraid that they were going to resign…”

“I mean, one person that left was making $27,000 a year when I came in.

And I got him up to $50,000 but then he still left for a $72,000 a year job

which is what his market rate is. He deserves to make $70,000.”

“I think that a lot of the people who left had a certain loyalty to [the

manager who left]. And I think that is interesting to see the wave that is

still hanging on that I'm really fighting to keep are a wave that came in

with [the new manager]. So I think there is a certain loyalty there.”

OPEN SOURCE LMS: MOTIVATIONS AND

BENEFITS

Flexibility and ability to customize our own

implementation is important”

Usability and Accessibility

Technical innovation and rich features

Pedagogical innovation and sharing

“I have seen just as many contributions to the moodle

community pedagogically derived, so people say “this

is how I am teaching math with this tool”, nobody

asks them to supply that, they just happy to share.”

Ideology and Value

Leadership role and Reputation

CHALLENGES

Technical skills and know-how

Technical support related issues

Upgrades

Scalability

Testing

Staff Retention

User Training and Education

Unrealistic expectations from users

“Concerns people have are not only cost related

issues, but the safety net, if something goes wrong,

they want to yell at somebody else, now when

something goes wrong, they yell at me…”

COST OF OWNERSHIP

Cost-: Free licensing

Cost+: Pay for hosting and other costs associated

with customization and support

Cost-: Engaging other local campus who are

interested in adopting Moodle, establishing

collaborative relationships

Cost-, Revenue+: developing expertise

surrounding Moodle support, potential consulting

and revenue generating opportunities

ACTIVELY PARTICIPATING IN MOODLE

COMMUNITY

At developer level

Contributing and maintaining the “quickmail” block

Maintaining tools like “gradebook+” in the previous

release

At system administrator level

Communicating with core teams regarding scalability

issues

Training, support and documentation level

Making contributions to forum discussions and

documentation

FROM MOODLE TO IMOODLE

Initially, upgrades following Moodle release cycle caused stability and scalability issues

Measures taken to solve the problem: Carefully evaluate features of Moodle and customize it to

meet needs of a specific campus

Establish testing environment, thoroughly test new build before releasing to the campus

Adjust the upgrade cycle of iMoodle to match the academic semester

Virtualization, instead of attempting to upgrade all courses across all semesters, isolating upgrades within each semester

More effective project management, focusing resources to stabilize and support the system

As the result, the current iMoodle implementation has been quite stable, no major outages and disruptions

FROM IMOODLE TO THE CLASSROOM –

THE FACULTY PERSPECTIVE

iMoodle is very effective and easy to set up comparing to

Blackboard

Faculty computer skills vary across the campus

Faculty don’t have time to attend training sessions

System stability is really important

Do we really need new upgrade every semester ?

Online course is a whole different beast to support:

“my department benefit a lot when I get 1300 students in my online

class, but AT people are not paid for supporting me. I think online

course should be treated differently because in this case, the

technology is the classroom”

FROM IMOODLE TO CLASSROOM:

STUDENTS PERSPECTIVE

iMoodle is easy and intuitive to learn and use

How effective iMoodle is really depends on how instructor engage and set up the course Biggest complaint: instructors do not access students work on

time

The instructor uses iMoodle actively tend to be better instructors, students tend to perform better in the class

Students do expect instructor to be the problem solver when they encounter technical problems

If a feature is available in iMoodle, students do expect instructors to use them

Feature Requests: Deadline for instructor update grades?

Automatic notification and reminding emails before homework due

Who is logged in now feature, dynamically forming work groups

POLICIES AND GUIDELINES

Service Level Agreements

Let campus know this is what we can provide now with our

limited staff and what we can do , and then this is what we ask

you to do to help us out.

Incentives for faculty to make effort in curriculum

innovation: Tenure and Promotion policy

Providing venues for faculty to learn and share best

practices of using technology effectively in the classroom

Providing effective and fair funding mechanism to support

technology innovation