Upload
brendan-obra
View
226
Download
0
Embed Size (px)
Citation preview
Are we there yet?
Continuous Delivery @Comcast
Brendan O’Bra, Principal Engineer (brendan_o’[email protected])Twitter: @brendanobra LinkedIn: https://www.linkedin.com/in/brendanobra
September 17th, 2015Scala Bay Meetup
We are more than just cable TV…
Internet Phone On Demand
Home Security
Business Services (Internet)
Next Gen TV (IP Video)
Internet TV Apps (On
X1)
TV Content Production
(NBC)
Movies (Universal)
3
Almost everything at “National Scale”
• 10’s of Datacenters across US • Traffic can be “surge-ish”• “Carrier Grade” Customer Expectations• Software Upgrades • Millions of Devices• Cloud heterogeneity is important• 1000’s of VMs running at any time, across
all clouds
Continuous Delivery @Comcast
4
In the past… Many humans, some machines MUCH time
Continouos Delivery @Comcast
5
Now: One Human, many machines much LESS time
Continouos Delivery @Comcast
6
Let’s go FAST
Market was clearly changing, with competition delivering some cool stuff. Time to market became a stronger driver We did some light reading:
7
And then drank gallons of Continuous Delivery
8
And then… Epiphany!
Immutable Servers!
9
We arrived at a system that consisted of:
Puppet
App Config
Hiera
App Automation
Openstack
VMWare
Route53
Git
Cloud-init
10
Which can also be viewed like this…
11
Feedback from initial attempts at Continuous Delivery
• It needs a GUI
• Why is it so hard?
• What are all those moving parts there? WTF?!
• write the GUI with whatever you want!
12
Gumby!Your elastic friend in cloud
13
V1 : Play + Akka in Scala
• We made an app, in Play! GUI was in Angular, REST layer -> Biz Layer in Play/Akka (one dude want to learn Akka/Play/Scala, other dude wanted to learn AngularJS). Gumby was born!
• It was a great experiment, it worked in production, and actually solved a real problem(s)
14
Gumby: Spray + Akka to scale the Comcast Cloud
• Then.. The folks that that make X1 heard about this “gumby” thing and told us they wanted to use it.. That’s great, but it was only a science experiment….
• Time to rewrite in Spray, and Go Big• Spray allowed up to really clean things up, and because it was so fast, we
could DOS any cloud api we encontered ;)
15
Gumby: Current Architecture (Monolithic)
Workflow (Akka)
VSphere
Openstack
EC2
REST(Spray)Client
16
Fun Facts about Gumby
• Can deploy ~400 vms per dc / hour• Currently about 21000 lines of scala• Typically does 2-3 full national deploys/week• Deploys 60% of national X1 backend footprint
(4000 VMs)• Will deploy 100% of national X1 backend footprint
by Q1 2016• Is transitioning to microservices• Can deploy itself• David Bolene: “One Big Side Effect”