Tectonic Summit 2016: Ticketmaster's Public Cloud & Kubernetes Strategy

Ticketmaster - CoreOS Tectonic Summit 2016

COREOS TECTONIC SUMMITDECEMBER 12, 2016

JUSTIN DEAN● SVP, Platform & Technical Operations● ~1.75 Years at Ticketmaster● Passionate about building high

performance organizations ● Nerdy about automating my beer &

BBQ pipeline (see PitmasterPi on github)

OUR STORY● About Ticketmaster● Our Journey● Large Enterprise Challenges &

Lessons Learned● Why Kubernetes● CoreOS Partnership● Up Next

● Publicly Traded Company (LYV)● $7.6B Revenue● $25B in GTV (Gross Transaction Value)●Top 5 eCommerce site

ABOUT USHISTORY● 1976 - Founded at Arizona State University● 1996 - Ticketmaster.com launched● 2010 - Live Nation and Ticketmaster join forces to

power live experiences

We power unforgettable moments of joy!

Concerts, Sports, Arts & Theater, Small Venues & Clubs

TECH COMPLEXITY

● Every era of software, many not ready for containers and cloud

● 1970s: Custom VMS OS on Emulated VAX (The Host)

● 2000s: Xen Cloud, Big-Iron Filers, NFS, custom built infrastructure

PRE-MODERN TECHNOLOGY Tech Museum

TECH SCALE● 21 Ticketing Systems and over 250

unique products● 1,400+ people in Product & Tech org● Custom Private Cloud with over 22,000

VMs across 7 global data centers● Over 15,000+ network endpoints across

the world (Venues, Arenas, Kiosks, etc)● Over 60% VM growth in last year

1 BILLION MACHINES!!*

*Not really :)

Onsales = Black Friday every day!● Huge spikes / demand for

tickets● Global company = across time

zones● Limited inventory (Beyonce

Tickets!)● Multiple sales channels

0 to 150M transactions in minutes! That’s a spike of >8 GBps !!!!!

Self Inflicted DDOS-as-a-Business

BIG SCALE, BIG CHALLENGES

COMPETITION

● Market leader with huge surface area ● Competitors of every size and shape ● Speed and agility are absolutely key● Scale and complexity of 40-year old business make rapid changes very

COMPETITIVE LANDSCAPE

TO RECAP...

Public company / market pressure / highly competitive landscape

Legacy tech, not ready for containers

Tech debt with high interest rates

Huge scale and complexity

Black Friday every day

GETFASTER!

SIMPLIFY OUR PLATFORMMore Revenue and

Market Share

Better Products & Features

Deliver Products Faster

Autonomous Product Teams

Simplify Our Platform

OUR JOURNEY

Self-d

Transf

ormati

Autono

mous D

elivery

Public

rnetes

2013 2016 2017WE ARE HERE

SELF-DISRUPTIONSe

Transf

ormati

Delivery

Public

rnetes

LEAN TRANSFORMATION● Laser focused on highest priorities● Created 65+ cross-functional delivery teams● Eventually all roads led to “blocked by ops”● Got faster at developing; did not get faster at delivering

Self-d

isrupti

Delivery

Public

rnetes

AUTONOMOUS DELIVERY TEAMS● Moved application support teams out of TechOps and into the

product teams directly● Embedded Systems Engineers into product delivery teams

(closer to truly “cross-functional”)● Self-Service Tools: Surge towards getting teams out of the ops

business● Self-Sufficient businesses (build it, run it, own it, optimize it,

monetize it)

Self-d

isrupti

Transf

ormati

Delive

Public

rnetes

Microbusiness

TRANSFORMATION INSIGHTSRealized our ability to innovate is dampened by our overly complex software factory:

30-50%Of development time spent

moving code around ($60M-$90M problem)

150Custom-built

ways to release products (often

manually)

~50%Incidents were preventable; mostly self-

inflicted

Self-d

isrupti

Transf

ormati

Delivery

Public

rnetes

PUBLIC CLOUD

Self-d

isrupti

Transf

ormati

Delivery

c Clou

rnetes

Vehicle for deep introspection of every product

Immediate access to infrastructure as APIs

Forcing function to modernize all products to cloud native standard (all the *.-ilities)

Public Cloud = Huge carbon filter

CLOUD ENABLEMENT TEAM ● Small team of experts dedicated to developing:

▪ Future state architecture▪ Path to Public Cloud▪ Cloud Native Solution Patterns ▪ Cure us of our on-prem addiction (NFS, Always scaled, HW reliance, SW trees,

etc)● Provide Self-Service tooling and documentation for those solutions ● Enable teams to:

▪ Raise their tech maturity▪ Containerize and retool their app ▪ Migrate themselves to the cloud

Self-d

isrupti

Transf

ormati

Delivery

c Clou

rnetes

CLOUD ENABLEMENT METHOD7 “Simple” Steps:1. Containerize your app; use CoreOS2. Terraform your infrastructure3. Instrument everything, rich telemetry - no SSH or RDP! 4. Use synthetic monitoring to understand the health of your product5. Security, security, security6. Design shared-nothing architecture (no NFS)7. Build for availability - no single points of failure

Self-d

isrupti

Transf

ormati

Delivery

c Clou

rnetes

READY TO ROLL● Highly skilled team ● Modern new stack architecture● Comprehensive DIY toolkit/software● 1,000+ pages of detailed documentation and solution patterns

Self-d

isrupti

Transf

ormati

Delivery

c Clou

rnetes

Everybody has a plan until they get punched in the

face.- Mike Tyson

LEARNINGS

Public Cloud

Learn the API's/Primitives, Learn to build Infrastructure,Learn to code it in Terraform

Programmatic Checkout Page

65,000 permutations on how to

use AWS service

offerings =

64,999 ways to get

it wrong

Rich set of Primitives and API's

100's of Devs, different tech stacks

LEARNINGS SUMMARY● Huge learning curve● Hard to manage distributed systems at scale● Wrong people to build & optimize infrastructure (across 100+

teams)● Baking purchasing decisions into distributed terraform code is BAD

...Spending too much time writing software to deploy software

instead of writing software to make money

SOLUTION: CONTAINER ORCHESTRATION● Abstract complexities of infrastructure from development teams,

including how to:▪ Design▪ Deploy▪ Purchase▪ Optimize

● Allows us to easily manage distributed systems at scale

WE CHOSE KUBERNETES● Kubernetes started organically appearing all over our company● Ahead of other container management platform and rapidly

improving● Amazing community with hockey-stick velocity ● Kubernetes APIs and primitives are sweet!

▪ Iteration time is seconds VS minutes▪ Automated rollbacks▪ Scaling and self-healing are much faster than ASG’s

● Kubernetes gets us much better utilization of our EC2 instances● Successfully used it to solve a major stability issues

OPENTSDB ON KUBERNETES● Critical system for application monitoring

▪ 500k metrics per second● Large queries during ticketing sales were DDOS’ing OpenTSDB

services● Kubernetes pod health checks detect this and restart the failed

containers● Kubernetes primitives took a service that required hand holding to

something that manages itself● Learning Moment! A reboot from an automated OS upgrade

required manual intervention

SIMPLIFICATION WITH KUBERNETES

Public Cloud

Public CloudKubernetes cluster optimized by Cluster Ops team

Kubernetes APIs / abstraction

Homogenized deployments via Kubernetes

$ $$ $

KUBERNETES PROJECTGOAL: Deploy a Ticketmaster product into a production-grade Kubernetes cluster and equip team with the skills required to support its operation.

● Fully-remote team of 6● Tons of work!

▪ How many clusters to build?▪ Which architecture is right for us?▪ How should we deploy and test the cluster?▪ Which networking option to use inside of AWS?

QUESTIONS● Kubernetes @scale best practices and pitfalls

▪ Kubernetes @Ticketmaster Roadmap: − Documented Reference Architecture specific to

Ticketmaster based on all the below that includes answers to any below questions. We need a documented roadmap for the team to start building based on Apprenda Experience/Reference architecture.

▪ Guidance on what goes in K8S and what should not (if anything)

▪ What have we missed? What didn’t we ask?▪ Best practices around secrets; how do companies manage this

at scale? Risks, alternatives, etc.?▪ Kubernetes upgrades, possible w/o downtime?▪ Insight on cloud primitives that are not K8S managed

(Lambda, S3, SQS, KMS, RDS, etc….). What are other companies doing here? Are some of these on the K8S roadmap to orchestrate? Are these resources managed by “clusterops”, or do delivery teams self-build outside the k8s workflow? This is called the K8s service catalog

▪ What do they recommend for configuring containers within kubernetes

▪ How do they recommend granting iam roles to containers● Kubernetes cross-domain (AWS/onprem/other cloud) insight

▪ Good idea? Possible pitfalls?▪ How to front end AWS and Onprem so we can dynamically run

HOT aws expensive stuff on onprem behind the scenes▪ Cross AWS region?▪ If we run Kubernetes in Equinix, how do they recommend

logging into ECR with Kubernetes● Cluster Networking

▪ What do they recommend for loadbalancer services in aws▪ Overlay networking▪ Software defined firewall▪ Best ecosystem components (calico vs x, etc)

● Team / Operations▪ How do engineering teams interactive with the cluster, kubectl

on their laptops? Probably not▪ How long do they see it taking to build enough knowledge for

production support of k8s▪ Insight on other companies K8S support models (what does

ops do, what does devops do, what are the governance models)

▪ Understanding of Implications on chargeback in AWS. How much effort goes into tagging and reporting on ephemeral resources (containers) that move around on AWS primitives (EC2 instances)

● CET (cloud enablement team)▪ How to marry it into our CET strategy, specifically Terraform▪ Help on rollout strategy. Start working in context with early

adopter enthusiastic teams asap OR wait until we have it more ‘operationally mature’. Both tactics have merit, help us think through the strategy here.

● Persistent storage, period. ▪ Torus, Ceph, EFS, NFS, Gluster, portworx ; pros / cons▪ Databases (large/shared) on k8s?▪ Other persistent workloads: elastic, cache, message bus, etc..

● Ongoing Apprenda Engagement▪ Information regarding their consulting offerings/ prices/

models of engagement. On prem team? Support team? Customized kubernetes solutions and maintenance.

▪ Connect us to peer group in Kubernetes space● Should we just leverage Tectonic? ● Archtics (massive legacy windows/powerbuilder/sybase/rdp over internet

to sports teams) Help● Prometheus help

overlay networking?

Calico?

Flannel?

VPC networking? Canal? cluster ops

Linkerd?

how many etcd

nodes?

Terraform vs Kube

Prometheus?

24/7 support

COMMUNITY ENGAGEMENT● Spent time with CoreOS, Kelsey

Hightower, Apprenda● Attended conferences● Hosted Meetups● Joined SIGs● Joined

MILESTONESSimple Kubernetes

cluster

Operationalize Kubernetes

Enterprise Ready / HA Kubernetes Cluster

Address consumability by appsOn-call production support

First customers golive on Kubernetes

Expand!

WORK BEGINS...BUT● Continued to identify new questions● Had not figured out operational support● Needed enterprise-level features (auth)● Needed answers based on experience; not theory● Needed to accelerate implementation

STRATEGIC PARTNERSHIP

MILESTONES✔ Simple Kubernetes

cluster

Operationalize Kubernetes

Enterprise Ready / HA Kubernetes Cluster

Address consumability by appsOn-call production support

First customers golive on Kubernetes

Expand!

WHY TECTONIC● Vanilla upstream Kubernetes - No lock in● Immediate enterprise level confidence● Supported reference architecture (instead of DIY)● Recommendations on operational practices, service provider

integration, third party add-ons, etc. ● Production Go-Live Support● Automatic OS Updates! *Bummer, no more fun upgrade projects!

COREOS PARTNERSHIP● Providing input on Tectonic roadmap● Influence the roadmap for things that REALLY matter to Enterprises● Jointly solve Enterprise + Web Scale challenges● Help foster the Enterprise Kubernetes community

NEW TICKETMASTER WEB PLATFORM ON K8SBefore:

● Semi-manual stack creation, bespoke cloudformation + python boto scripts = 20+ mins to deploy

● Low Confidence

Now: ● K8S + Tectonic, fully

automated = 60 second app updates

● High Confidence● Unlocked Daily Delivery

Culture

LET THE MAKERS MAKE

● We have an amazing company of Makers, Creators, Visionaries

● We must create the space for them to innovate and deliver great solutions to the market

RECAP● Use Kubernetes to abstract infrastructure

complexities● Have a cluster ops team do the

optimization voodoo; not everyone else● Stop wasting effort writing software to

deploy software ● Let the Makers Make! Give time and

mindshare back to your most valuable asset (your people) to do what they do best: Make Things!

TICKETMASTER KUBERNAUTSStop by and say hi during the break!

&Join us at the Sysdig/CoreOS/Ticketmaster

party tonight!Food, drinks, LIVE BAND!!

Justin Dean Kraig Amador Abe Ingersoll Bindi BelangerJean-François Nadeau

justin.dean@ticketmaster.com@justinmdean

Tectonic Summit 2016: Ticketmaster's Public Cloud & Kubernetes Strategy

Technology

Agenda - Meetupfiles.meetup.com/14055662/2016.05.21.3.IBM.Austin Summit...• OpenStack, Magnum, CoreOS, Kubernetes… –Unleash the Power of IaaS •Containerized OpenStack –Faster

Tectonic Summit 2016: CoreOS Tectonic on AWS

Kubernetes intro public - kubernetes meetup 4-21-2015

Kubernetes Immersion

Tectonic Summit 2016: Kubernetes 1.5 and Beyond

KUBERNETES FOR JAVA DEVELOPER · 2019-09-27 · CoreOS Tectonic Kubernetes Operations (kops) RASPI CLUSTER 4 Raspberry Pi 3 Wiﬁ Router 6 Port USB charger 32 GB SD-Cards Costs: ~

Running Cloud Native Applications on DigitalOcean Kubernetesassets.digitalocean.com/...digitalocean-kubernetes.pdf · Kubernetes and DigitalOcean Kubernetes Kubernetes, initially

Monitoring Kubernetes with Prometheus (Kubernetes Ireland, 2016)

Kubernetes: Up and Running - Tectonic by CoreOS · This Preview Edition of Kubernetes: Up and Running, ... software started eating the world. ... based on a template, are

Endocode Kubernetes Meetup: Architecture Patterns for Microservices in Kubernetes

Tectonic Summit 2016: Betting on Kubernetes

Kubernetes intro public - kubernetes user group 4-21-2015

Tectonic Summit 2016: Managing 6 PB of Earth Imagery with Kubernetes

Kubernetes Ingress Services by VMware NSX® Advanced Load ... · Kubernetes/OpenShift Cluster Integration using Avi Kubernetes Operator Avi Kubernetes Ingress Services can be used

Pegasus Workflows on OLCF - Summit · . 6. Kubernetes: Pods • A. Pod. is the basic execution unit of a Kubernetes application • Pods represent processes running on the cluster

Continuously Delivered Scalable Geoservices on Kubernetes · 2019 Esri Developer Summit Palm Springs -- Presentation, 2019 Esri Developer Summit Palm Springs, Continuously Delivered

Tectonic Summit 2016: Multi-Cluster Kubernetes: Planning for Unknowns

Tectonic Summit 2016: Brandon Philips, CTO of CoreOS, Keynote

Infrastructure Design for Kubernetes · Kubernetes 1.9 Kubernetes 1.10 Kubernetes 1.11 Kubernetes 1.12 December 2017 March 2018 June 2018 September 2018 Kubernetes 1.13 December 2018

Setting up Kubernetes with tectonic