Embed Size (px)
Citation preview
Keep Your Head in the Clouds
Blackboard’s Transition from Enterprise to Cloud
Software Through DevOps
David Ashman
Chief Architect, Cloud Architecture
@davidbashman
● The industry leader in educational
technology and services
● 17 years young
● Privately held
● $450M revenue at time of going private
● Headquarters in Washington, DC
● 17 year old codebase with roots in Perl
● Millions of lines of Java code (Hybrid Java/Perl for many
years)
● 7 development/operations offices, worldwide
● 700 people between development, testing and
operations (All products)
● ~3000 virtual machines across 1000 clients in hosting
● ~8PB of content and data storage
Blackboard Learn
We are a horse.
● 6+ month lead times
● Technical debt
● High update failure rates
● Poor communication paths from development to
operations
● Poor feedback loops from operations to development
And like so many of you...
But through the art of DevOps...
We are a horse.
We are a horse.^
● Automation
● Cloud Infrastructure
● Culture
What changed?
Automation
# of code commits
24-36 Hours
Mike McGarr
# of code commits
# of code commits
# of code commits
15-30 Minutes24-36 Hours
But wait... There’s more!
Functional Acceptance Testing
Value
5 minutes
Create
Ticket
Commit/
Build/
Integrate
15 minutes
5
minutes
Problem!
Code Test Suite
Analyze
Failed
Tests
4 hours 1 hour 1 hour
5
minutes
36
hours
24
hours
70 hours total time
6 hours (value)= 9% Efficient
Waste (wait)
Assign
Ticket
15 minutes
6
hours
DevelopmentTesting
(2-5 days)
Commit
TicketProject
Management
3-6+ days
100s of Failures
60% script issues(invalid tests, out of sync with functionality)
30% data/environment issues(data left in the database or filesystem)
7% pre-existing issues
3% newly discovered issues
Problem
● Elongated testing cycles (3+ months)
● Longer time to market
● Reduced visibility to our development team
● Long delays between coding and fixing
● Too much noise distracting us from product
improvements
The Problem
● TDD
● Fully automated acceptance testing pipeline
● < 30m acceptance test feedback
● New testing approaches (Jasmine, Protractor,
RESTAssured)
Test Automation
6+ month lead time
1-2 week lead time
Cloud Infrastructure
OperationsProduct Development
Development Test Production
!= !=
● Snowflakes - No two environments were alike
● Completely different deployment methodso Manual development builds
o Automated installs in testing
o Gold master images in production
● Completely different deployment architectureso Windows development machines
o Linux VMs in testing
o Linux clusters in production
Deployment Environments
Development Operations
● Everything, from the ground up, is automated.
● Orchestration in code - stored in SCC, executed in ALL
deployment environments.
● On demand provisioning for developers.o Still develop on their laptops, but can now test in a real
deployment environment.
Learn in the Cloud
● OpenStack in our data centers
● Abstract cloud fabric to gain benefits in both clouds
● Centralized, standardized Chef automation
● Increased visibility for developmento Monitoring, APM, statsd, centralized logging
BbCloud
Culture
“QA is responsible for defining the testing strategy.”
“QA is responsible for checking quality when a feature is
done.”
“Unit testing is not enough to verify a feature.”
OperationsDevelopment
Executive Buy-in
● Development teams deploying their production
environments.
● Developers solving operational issues in production.
● Open feedback loops on operational issues.
● Data-driven decisions based on that feedback.
What we’ve achieved
● Understand the impact that cloud computing can have
on DevOps even in an “enterprise” company.
● Understand the cost models comparing cloud
computing and traditional hosting.
● Frame your pitch to the executive team to make
material changes in your company culture.
I might be able to help...
● Effective testing strategies for traditionally manual tests
(UI, etc).
● Applying DevOps strategies for shipped-to-premise
software.
I’d like to learn more about...
