Embed Size (px)
DESCRIPTION
Blackboard's Transition from Enterprise to Cloud Software through DevOps How do you implement DevOps in a software company that has 16 years of established culture and processes? What if this organization is the industry leader and has everything to lose by changing? Over the last two years, Blackboard has gone through an enormous change, from a company delivering enterprise software once every 18 months to one on the verge of delivering Cloud enabled education software through continuous deployment. My presentation will talk about the triumphs and challenges of taking a group entrenched in years of legacy to a new vision of faster delivery of high quality software.
Citation preview
Keep Your Head in the
Clouds Blackboard’s Transition from Enterprise to Cloud Software Through DevOps
David AshmanChief Architect, Cloud Architecture@davidbashman
● The industry leader in educational technology and services
● 17 years young● Privately held● $450M revenue at time of going private● Headquarters in Washington, DC
● 17 year old codebase with roots in Perl● Millions of lines of Java code (Hybrid Java/Perl for
many years)● 7 development/operations offices, worldwide● 700 people between development, testing and
operations (All products)● ~3000 virtual machines across 1000 clients in
hosting● ~8PB of content and data storage
Blackboard Learn
We are a horse.
● 6+ month lead times● Technical debt● High update failure rates● Poor communication paths from development to
operations● Poor feedback loops from operations to
development
And like so many of you...
But through the art of DevOps...
We are a horse.
We are a horse.better
^
● Automation● Cloud Infrastructure● Culture
What changed?
Automation
LOC
# of code commits
Problem
24-36 Hours
Mike McGarr
LOC
# of code commits
Problem
LOC
Wat
# of code commits
Problem
LOC
# of code commits
15-30 Minutes24-36 Hours
But wait... There’s more!Functional Acceptance Testing
Value
5 minutes
CreateTicket
Commit/Build/
Integrate
15 minutes
5 minutes
Problem!
Code Test SuiteAnalyze Failed Tests
4 hours 1 hour 1 hour
5 minutes
36hours
24 hours
70 hours total time
6 hours (value) = 9% Efficient
Waste (wait)
AssignTicket
15 minutes
6 hours
DevelopmentTesting
(2-5 days)
Commit
TicketProject
Management
3-6+ days
100s of Failures
60% script issues(invalid tests, out of sync with functionality)
30% data/environment issues(data left in the database or filesystem)
7% pre-existing issues
3% newly discovered issues
Problem
● Elongated testing cycles (3+ months)● Longer time to market● Reduced visibility to our development team● Long delays between coding and fixing● Too much noise distracting us from product
improvements
The Problem
● TDD● Fully automated acceptance testing pipeline● < 30m acceptance test feedback● New testing approaches (Jasmine, Protractor,
RESTAssured)
Test Automation
6+ month lead time
1-2 week lead time
Cloud Infrastructure
OperationsProduct Development
Development Test Production
!= !=
● Snowflakes - No two environments were alike● Completely different deployment methods
o Manual development buildso Automated installs in testingo Gold master images in production
● Completely different deployment architectureso Windows development machineso Linux VMs in testingo Linux clusters in production
Deployment Environments
Development Operations
● Everything, from the ground up, is automated.● Orchestration in code - stored in SCC, executed
in ALL deployment environments.● On demand provisioning for developers.
o Still develop on their laptops, but can now test in a real deployment environment.
Learn in the Cloud
● OpenStack in our data centers● Abstract cloud fabric to gain benefits in both
clouds● Centralized, standardized Chef automation● Increased visibility for development
o Monitoring, APM, statsd, centralized logging
BbCloud
Culture
“QA is responsible for defining the testing strategy.”
“QA is responsible for checking quality when a feature is done.”
“Unit testing is not enough to verify a feature.”
OperationsDevelopment
Executive Buy-in
X
● Development teams deploying their production environments.
● Developers solving operational issues in production.
● Open feedback loops on operational issues.● Data-driven decisions based on that feedback.
What we’ve achieved
● Understand the impact that cloud computing can have on DevOps even in an “enterprise” company.
● Understand the cost models comparing cloud computing and traditional hosting.
● Frame your pitch to the executive team to make material changes in your company culture.
I might be able to help...
● Effective testing strategies for traditionally manual tests (UI, etc).
● Applying DevOps strategies for shipped-to-premise software.
I’d like to learn more about...
