Embed Size (px)
Citation preview
Deploying Office 365 At UC Merced
Nick Dugan and Todd Van ZandtUC Merced Information Technology
UCCSC 2013
Introductions
Nick Dugan, Senior Systems Administrator [email protected]
Todd Van Zandt, Director of Academic Technology and User Services [email protected]
Agenda
Background Selection and Pilot Process Infrastructure Deployment Migration User Support Next Steps and Conclusions
Background
UC Merced Opened 2005 Population in Fall 2012
5,800 students 140 faculty 1,000 staff
10,000 students by 2020
Background
Technology ~8,000 active accounts ~10,000 alumni / former student
accounts Lots of infrastructure from ~2005
Sun Messaging ServerOracle CalendarSPARC Hardware
Background
Sun -> Oracle == $$$ SPARC hardware refreshes Maintenance and license costs
Oracle Calendar EOL Heavy use of Outlook with Oracle Calendar connector Incompatible with Outlook 2010 Client software not supported in Mac OS X Mountain
Lion Email Growing Pains
Quotas, Compromised accounts, SMTP blacklisting Aging webmail client
Background
Minimally Utilized Windows Infrastructure Identity Management: Sun Waveset LDAP: Sun Directory Server
Active Directory Fed from IDM Lab support Printing NAS Authentication Minimal account attributes populated
Selection and Pilot Process
Selection Committee Convened November 2011, charged with
developing functional requirements and recommending a solution
Representatives from faculty, staff, and student populations
Locally hosted options ruled out Microsoft vs. Google Work completed March 2012
Selection and Pilot Process
Microsoft vs. Google No clear consensus among committee
members Subjective and objective scoring gave a slight
advantage to Office 365 Both better than current solution High level of satisfaction with both High satisfaction with Lync among staff Final decision left to IT after consideration
of committee report and analysis of costs
Selection and Pilot Process
IT Review March – May 2012 Microsoft reduced price of A2 plan to $0 Evaluated technical requirements for
implementation Reviewed committee input Spoke with other UC CIOs and IT
organizations Final recommendation for Office 365
delivered to Chancellor and Provost
Infrastructure Deployment
Consultant Engagement Decision made to employ professional
services for tenant deployment and migration support
Interviewed and vetted approx. 10 companies in July 2012
Selected Cloudbearing, initial engagement Sept. 2012
Infrastructure Deployment
Active Directory Enhancements Domain Controllers in four physical locations
(Merced x2, Berkeley, Fresno)
ADFS Servers 2x ADFS, 2x ADFS proxy, load balanced
across two physical locations
DirSync Server HA deployment not possible
Infrastructure Deployment
Why ADFS and DirSync? Apprehension about pushing passwords to
the cloud IDM Already fed Active Directory Infrastructure vs. custom development
trade-offs made sense
UC Merced AD ID
Microsoft Federation Gateway
Wpadfs01169.236.80.174
Wpadfs02169.236.11.108
Wpdirsync01169.236.80.175
Wpadfsp0110.15.4.26
Wpadfsp0210.15.4.27
mail2169.236.253.81
Wpfread01192.35.211.235
MX Record
lpvmg02.ucmerced.edulpvmg01.ucmerced.edumg03.ucmerced.edu
F5 BIG-IP 1600 NLBadfs.ucmerced.edu
Mail flow between on-premise and cloud mail hosts
Inbound Mail Flow during pilot phase
Recipient synchronization from on-premises to Office 365
AD FS authorization traffic
Internal Domainucmerced.edu
Object Custom Filter Sync User Attribute Extension15=0365 OU=0365 (O365 Groups Container) OU=People (User Container)
UC Merced AD FS DeploymentSeptember 2012
Office 365
Auth Provisioning
MailClients
On-site bulk mailerMailman
IDM
ActiveDirectory
DirSync
PowerShell
ADFS
Shibboleth
Portal
Infrastructure Deployment
Lesson – Time is everything ADFS is sensitive to clock skew ADFS Proxies are not domain-joined by
design No automatic clock sync with DCs
Small time discrepancies lead to authentication outages Proxies need reliable NTP
Infrastructure Deployment
Identity Management Integration Sun Waveset (Java) on Solaris EOL, Replacement in 2014 Office 365 == PowerShell IDM provision/deprovision events now trigger
licensing actions on a remote PowerShell server
Lots of waiting built into process due to (un)timeliness of events in o365 cloud
Account provisioning went from real-time to “an hour or two”
Infrastructure Deployment
Shibboleth / SSO integration Goal: Never see the ADFS sign-on page,
OWA integration into campus portal
Infrastructure Deployment
Shibboleth configured as a relying party in ADFS
ADFS Home Realm Discovery page modified to bypass IdP selection box
ADFS Logout handler modified to ignore SAML errors and redirect to global logout page
We will help [email protected]
Infrastructure Deployment
Dynamic Mailing Lists Sun Messaging does this well Office 365 does this… differently.
Limited to default attributes We have dynamic lists based on lots of attributes we
don’t push to AD Interim Solution: maintain Sun Messaging Server
at lists.ucmerced.edu Distribution lists in o365 forward to @lists for
expansion and ultimate delivery back to o365 Long-term Solution: Next-Gen IDM with proper
group management
Migration - Email
Approach and timing1. Early adopters, opt-ins (early-mid
December)2. Faculty and Staff by affiliation/geographic
location (Jan 2-20)3. Students alphabetically (Jan 14-25)4. Mailing lists (Feb 2013)5. Alumni (March-April 2013)
Migration – Email
Technical considerations Respecting user settings
Forwarding Vacation messages (didn’t do)
Messages that won’t migrate (>25 MB) Identify and notify
Migration - Email
Technical Process Set DirSync flag in AD so accounts are pushed to
Cloud Assign licenses using PowerShell (not too early)
wait Copy forwarding settings using PowerShell Change LDAP routing address
[email protected] Set migration password Migrate mail Remove migration password
Migration – Email
Tool #1: Office 365 IMAP Migration Tool Pros:
Free Fast (not subject to the same bandwidth
throttling as non-MS tools) Easy
Cons: ZERO reporting/logging. Only provides a count of “skipped” messages
for each account – not which messages or why they were skipped.
Migration - Email
Lesson: Never trust a tool that has no logging.
Half-way through faculty/staff migration, number of skipped messages went through the roof Hundreds or thousands on some accounts,
zero on others All attempts at resolution failed
Session/resource issue with local IMAP server? – No
Too many concurrent migrations? – No Microsoft able to provide an explanation or
resolution? – No Panic? – Yes
Migration – Email
Migration Tool #2 – MigrationWiz Contract with Cloudbearing provided up to
300 mailbox licenses Pros:
Excellent Reporting Capabilities Excellent Support Easy to incorporate into migration workflow
Cons: $11/mailbox == $$$$ Subject to Microsoft inbound traffic throttling
Migration - Email
Lesson: Ask what other people are doing first.
Migration - Email
Migration Tool #3: imapsync http://imapsync.lamiral.info/ Pros:
$50 (might as well be free) Excellent support Great reference from UCSB
Cons: Required a bit of work to incorporate into our
migration workflow Subject to Microsoft inbound traffic throttling
Migration - Email
Schedule, bandwidth, timings “Just in time” mailbox provisioning was
difficult Start data migration by 10pm, run until
done Largest migration batch was ~1,100
accounts 40 simultaneous connections Generally finished by 8:00am
Migration - Email
Lesson: Your carefully constructed process will experience its worst breakdown the day you migrate the Chancellor.
Story
Migration – Email
Jan 3, 2013 – Migration of Chancellor’s office 8:00am – Departmental account identified
that did not provision successfully (sAMAccountName >20 chars)
10:00am – Sysadmin attempts to rectify problem by deleting and then re-provisioning account to Active Directory
Sysadmin accidentally deletes ou=People instead of individual account
Migration – Email
Recovery First thought – pull the (virtual) network
plug on one of the DCs before changes propagate Too late
Second thought – shut down DirSync server NOW Preserved accounts and mailboxes in cloud
tenant ADFS non-functional, so as auth tokens expired,
users lost access Third thought – Reprovision from IDM?
No IDM group management, share permissions would be lost
Migration - Email
Recovery Decision made to restore from tape and
rebuild AD Backups were good, but VM restore process
was failing 3:00pm – Support case with Symantec Overnight – work case with Symantec, upgrade
server and client software Jan 4 morning – single DC restored Windows sysadmins rebuild AD forest Jan 5 9am – Full functionality restored
Migration – Email
Lessons: AD Recycle Bin would have made this a
non-issue Requires 2008 R2 domain functional level
UCM was at 2003
Verify backups (AND restores) Prepare for the worst when migrating
campus executives
Migration - Calendar
Cloudbearing contract included migration of calendar data through third party (CalMover)
All calendar data migrated to Office 365 at the same time Thursday, Jan 17 5:00pm – Oracle Calendar
database extracted and uploaded to CalMover
Friday, Jan 18 – receive PSTs from CalMover Fri-Sun – import PSTs to Office 365 Monday Jan 22 – Office 365 official UCM
Calednar
Migration - Calendar
Issues, side effects Modification or cancelation of migrated
meetings would not notify attendees Contacts, tasks, and other non-calendar
data was excluded from import Resource delegates, sharing and booking
permissions did not migrate
Migration – Calendar
Timing, speed, historical events Cloudbearing recommended limiting
historical data to 12 months PST import could be very slow, might not finish
over weekend window Individuals needing complete historical data
would be accommodated through other means of import/export
Application/service updates made the import much quicker than anticipated Allocated 3 days, completed in 6 hours
User Support
Campus Communication Pilot Support Migration Support Documentation Training Client Issues
User Support - Communication
Campuswide announcements News articles MSOs Info sessions
Record and make available Inform about
Benefits Cons / Changes (more important) Process , expectations
Direct emails to users at the time of migration
User Support - Communication
Website Large Banner to draw attention O365 project section included
Migration schedule Configuration Guides
List by device, not just service or client Training & Getting Started Guides Explanation of what is coming (large mailboxes,
mobile support) , but also limitations and differences Update all old information pages, aside from
Config Guides
User Support - Pilot
Dedicated a technician to assist with installs Did not provide a lot of help Asked pilot group to review draft documentation IT Student Employees Engaged Resident Assistants (additional
support)
Lessons Learned Needed to have created formal testing plans Needed to have been more engaged in support
User Support - Migration
Get Extra Help! We used temps, technically savvy dept staff, and IT
students and staff Dedicated 2 (of 5) Desktop Support technicians to this
effort once migrations started 2 DSS departed as we were starting
Hired 4 temps to help with migration appointments First used them to create and finalize documentation for various
client set ups Followed up on problems, after migrations completed for the
day Identify technical leads in departments and target for
early adoption Student Technology Consultants Interested IT staff
User Support - Migration
Temps: The Bad: Not as engaged, less reliable, had to
replace two The Good: Easy to hire, able to release when
not needed, trained for the future, targeted at O365 tasks
Suggestions: Interview more so you have extras “on-hand”; Have projects lined up for them in downtime; Keep an eye on them
User Support - Migration
Coordinated with Depts (opt-in) Took a LOT of effort
Difficulties with scheduling 60-120 per group;
Only so many depts on a day Plus additional one off early adopters Coordinating postponements for out of office
users Allowed us to test our processes and
improve documentation
User Support - Migration
Informed Depts for mass moves Chose by location; focused support to an area Tables in the lobbies (rarely used) Self-help documentation was good
Morning appts for high profile groups Most done within 4-6hrs Everyone wanted help immediately
OWA was an option Many users (faculty and lecturers) not present
User Support - Migration
Staff Very interested; needed help and assistance Many Windows Outlook users
Remove Outlook Oracle Connector Reattach Personal Folders and Archives Import Address Books and Autocomplete lists
Upgrade to Office 2010 (became too lengthy) Faculty
Many were out of office Many self-configured their clients
User Support - Migration
Students Mostly web access users so just adapted to using OWA
Or still forwarded their email to Gmail, etc. Switched webmail in Campus Portal if migrated Did not read email to understand process; if went to
direct to MyMail link Migration of lots of students took longer for data to copy,
new mailbox looked empty except for new Offered trainings, did not attend More critical for Grad students (instruction and
research) Finally moved them en masse; should’ve planned that from
start
User Support - Documentation
Documentation Configuration of Clients
Outlook, Thunderbird, MacMail, Mac Outlook What about calendar for Oracle Calendar Client
users What about Lightning, Mac Calendar?
Configuration of Mobile Devices Use of new systems (OWA)
User Support - Documentation
Documentation Didn’t do well documenting for Outlook
2007 2010 Didn’t do well documenting Outlook
calendar for old Oracle Calendar client users
Documenting the use of Oracle Calendar web client for early adopters who were Outlook users
User Support - Training
Training Pre-migration
What to expect, limitations and differences, overview of OWA and Mac/PC Outlook
Post-migration OWA Calendar Lync iOS
Dept or Individual Request Specific questions
User Support – Client Issues
Calendar Mac Outlook & Outlook 2007
Viewing & Sharing calendars Acting on calendars in Mac Outlook (selected vs. checked)
Client differences for viewing multiple calendars Overlay, Side-by-side, Both
Setting delegates; only Outlook client or powershell Needed to have done more testing with delegates and
room calendars to understand needs and functionality The wonderful subculture of Exec Administrative Assistants Breaking bad processes established by deficiencies in
previous Calendar Adding Holidays
User Support – Client Issues
Email Forwarding to multiple users Away Messages when Forwarding is set Creation of Distribution Groups (limiting) Attachment / Message Size limits Limits on Mass Mailing to unique accounts MacMail – sync stops Thunderbird – archive issue iOS – disappearing messages Outlook – Unread Email and Sync logs & Modification
Resolution Recovering deleted emails and calendar appointments
User Support – Client Issues
OWA Different on mobile browsers Settings are limited on mobile
Lync Mac client
Login issues Features not as robust
Considerations for default presence; All visible or All offline? All visible – encourages use of enterprise IM All offline – Faculty and Administrators not public Can’t separate students easily if all on the same tenant
What’s Next
“Wave 15” Service Upgrade Back-end upgrade to 2013 Office products Significant web UI changes
Just in time to make our new documentation obsolete
Some time between now and end of year Self-service web portal for resource
management and calendar delegation Changing booking options for resources Additional means of managing calendar
permissions for non-Outlook users
Conclusions
What have we gained? Modern messaging and calendaring
platform 25x quota increase Enterprise IM Proper mobile device support +1 critical enterprise service (ADFS)
Conclusions
Dollars and cents 1 FTE freed to work on other projects +6 Windows servers (VM) -3 SPARC servers $17,000 annual hardware/software
maintenance Money we won’t spend
Hardware refreshes (servers, SAN) Ongoing (increasing) Oracle licensing costs
Conclusions
What have we lost? Control
Upgrades and changes (announced and otherwise) happen when they happen
Outages
