1 Supporting Sakai: Lessons from the University of Michigan Sean DeMonner Jeff Ziegler Usability Support and Evaluation Lab

1

Supporting Sakai:Lessons from the University of Michigan

Sean DeMonner

Jeff Ziegler

Usability Support and Evaluation Lab

2Usability, Support & Evaluation Lab, Digital Media Commons, University of Michigan

What we’d like to share with you

• Brief history, scale and scope of the implementation

• Support group structure• Descriptive statistics related to supporting a

large scale Sakai implementation• Support metrics and satisfaction indicators• Common issues, and strategies for addressing

them• Best practices in support and lessons learned


Overview - History

• Sakai in use at U-M for about three years

• Successfully migrated from legacy CMS to “CTools” CLE in Summer 2005

• Currently running Sakai 2.1.2


Overview - Scale

• Total existing sites10,760 Course sites

7,217 Project sites

• Winter 06 Statistics2,806 Course sites in use

44,740 unique users

12,700 daily users (avg)

4,190 concurrent users at peak


Overview - Scope• Email and phone support available during

standard business hours

• Remote email support WeekendsEvenings (when warranted)

• 24x7 system availabilityexcept maintenance windows


Usage Statistics - NetTracker


Daily Usage Pattern


Weekly Usage Pattern


Support Structure

• Two-tiered Tech Support Model1st Tier – 90 hours/wk temporary staff

• Front line technical issues

2nd Tier - 1.5 FTE professional staff• Policy issues• Issues escalated by 1st Tier

• Training and Documentation1.25 FTE professional staff

• .5 Staff Training• .75 Help Documentation


Support Structure

• Quality Assurance30 hours/wk temporary staff

• Local Unit SupportCTools Affiliate program

Department IT staff

Sites computing consultants


User Training

• Ongoing, with a pre-term peak

• Instructor focused

• Group/department presentations preferred

• Low-stakes student introduction

• Coordination with release timing desirable


Documentation

• Do not assume “no one reads the docs”16% of faculty & 7% of students list docs as “most effective way to get help”

• Include docs in localization effort

• Occasional request for manualOnline and task based PDF “chapters”

• Flash visual tutorials are being implemented


Quality Control

• Internal staff tests local instanceRisk based testing

Test integration points with campus systems

Test across campus browser pool

1-2 weeks preferred; semi-formal reports; go/no-go meetings

Cross-test w/Sakai

• Operations Group runs load testingEvolution of protocol

Automated load testing


Tech Support Performance Metrics

• Number of support requestsEmail (91%)

Phone (9%)

• Response Time

• Number of touches

• Satisfaction data from annual survey

• Daily and weekly status reports


Issue Tracking

• FootPrints Electronic queuing mechanism

Feeds metrics gathering and reporting

Issue classification:


Support Requests - Email

LS&A mandate

Fall site creation Winter site

creation


Support Requests - Email W06


Response Time• Service level goal: 2 hours or less

• Business hours constrain this goal

• *On average* response of 15 min. or less

• Peak times see increases in wait times

• Daily “clear the queue” push


Touches Per Ticket

N= 5518; does not include site creations

These numbers should see some improvement due to “up front” requests for user info


2006 Support Survey - Instructors

• 1,357 respondents

• 33% have contacted support

• 18% report email is “most effective way” they get help

• Instructors are “very satisfied” with support services (4.21 out of 5 quality rating)


2006 Support Survey - Students

• 2,485 respondents

• 15% have contacted support

• 36% report “keep trying on my own” is the most effective way to get help

• Students are “moderately satisfied” with Support Services (3.7 out of 5 quality rating)


2006 Support Survey - Comparison


Support Request by Feature


Status Reports

• Daily and weekly communications


Voice of the Customer

• Customer feedback informs organization:


Voice of the Customer

• Policy Issues often come through Support:


Common Issues

• Sites created before registrar data is available

• Maintenance windows

• Large files & upload/download problems

• Site reuse vs. creating new


Common Issues

• Integration with immature or unsupported services/browsers

• Lack of Auditing Tools

• Distributed product delivery (e.g. Identity management, File systems, Registrar data)


Common Issues

• Managing large resource collections

• Managing peak support loads


Lessons Learned

• Training is key

• Low-stakes introduction to tools

• Use a ticket tracking system Multi-tiered support staffing works well

• Hire and train Tier 1 staff early (July for Fall)


Lessons Learned• Distributed support model (Affiliates in units)

• Escalation of Issues / Communication Flow

• Advisory committee(s) / Policy making bodies

• Embrace the workaround

• Establish Support Accounts and use Jira• Combine MOTD with Known Issues


Lessons Learned

• Know browser targets and test local instance• Migration never ends• “Eat your own dogfood” • Changing expectations of online systems

There’s no substitute for talking with customers, attending trainings, etc.

• Support is the ear of the organization and should “have a spot at the table”


Questions?• Sean DeMonner [email protected]

• Jeff Ziegler [email protected]


What else?

• FootPrints demo

• NetTracker demo

• Selecting support staff


Abstract• Sakai has been in use at the University of Michigan for the past 3 years, serving tens

of thousands of customers. Hear the in and outs of supporting a large scale Sakai deployment from the folks who answer the phones and respond to the email.

Description: This presentation will review the experiences of the team supporting Sakai at the University of Michigan, including:

- brief history, scope and scale of the implementation- support group structure & relations with other groups (Development, Operations, QC)- descriptive statistics related to supporting a large scale Sakai implementation- common issues and strategies for addressing them- best practices in support and lessons learned- support metrics and satisfaction indicators

Attendees will come away with quantitative and qualitative information related to supporting a large scale Sakai implementation.

Documents

1 Supporting Sakai: Lessons from the University of Michigan Sean DeMonner Jeff Ziegler Usability Support and Evaluation Lab