Shuen-Huei Guan, KKBOX
2015-12-01
AWS re:Invnet 2015 Recap Migrate the Mission Critical Application to
AWS Cloud
Who am I?
Shuen-Huei (Drake) Guan / @drakeguan Sr. Principal Engineer, Video.KKBOX Volunteer, PyCon Taiwan
What to Expect from the Session
• How we migrate a video-on-demand service. • What we benefited from AWS. • What we saw in AWS re:Invent 2015. • What’s next then.
VideoPass
• Video-on-demand (OTT) service, built as on-premises platform.
• 1M subscription users. • The scale of commercial contents is huge.
• More than 10k videos, ~= 2,000,000 minutes.
• Content encoding capacity is constrained.
VideoPass Migration Plan
• 1yr Development. • Integration of 4+2 vendors. • Nearly seamless. No obvious service shutdown. • 2-staged migration.
What We Pursue…
• Built on cloud from 1st day. • Just one AWS DevOps Engineer. • 4 dynamic environments:
develop, testing, staging, and production. • Scalable infrastructure, storage and bandwidth. • Powerful content encoding capability. • Standardized format: MPEG-DASH, HLS, CENC, …
Typical Architecture
• Entry points: EC2 + ELB + Auto Scaling group.
• Valuable Data: Multi-AZ RDS + ElasticSearch.
• Media warehouse: S3 + CDN.
• Future-proof: wrapped API access to AWS.
Orchestrating the execution of workflows for media streaming service and even more
- PyCon APAC 2015 - https://github.com/KKBOX/mass�
Gains from AWS
• Cooperation with AWS team to survive the migration. - IEM (Infrastructure Event Management) - Enterprise Support.
• Really insane encoding capability. - 3,000 videos in just one day. - done by 800 encoders.
• We just hired one AWS DevOps!
Data Center Migration
• GAM303 - Riot Games: Migrating Mountains of Big Data to AWS
• DAT205 - Introduction to AWS Database Migration Service
• ISM201 - Preparing for Migration: Developing a Roadmap to AWS for 7,700 Systems and 5 Global Data Centers
• … (Data center of KKBOX music streaming?)
Real-time Streaming Data
• BDT403 - Best Practices for Building Real-time Streaming Applications with Amazon Kinesis
• ARC310-APAC - Amazon.com: Solving Amazon's Catalog Contention and Cost with Amazon Kinesis
• BDT205 - Your First Big Data Application on AWS • … (Real-time user behaviors analysis?)
Microservices
• ARC201 - Microservices Architecture for Digital Platforms with AWS Lambda, Amazon CloudFront and Amazon DynamoDB
• ARC303 - Pure Play Video OTT: A Microservices Architecture in the Cloud
• ARC309 - From Monolithic to Microservices: Evolving Architecture Patterns in the Cloud
• ... (Developers >> Researchers >> DevOps)
Cost Optimization
• ARC302 - Running Lean Architectures: How to Optimize for Cost Efficiency
• ISM402 - Cost Optimization at Scale • STG311 - AWS Storage Gateway: Simple, Secure, Cost-
Effective Backup and Archive in the Cloud • … (Cost optimization is always that hot even after
re:Play party!)
AWS Lambda (2.0)
• Python++ • 1 minute -> 5 minutes • Scheduling • Events
• S3 • Kinesis • SWF • CloudWatch Log
Before AWS re:Invent, We Thought
• Cost optimization: - lower IT labor - flexible infrastructure
• Fully-managed scalability (IaaS): - Auto-Scaling group - ELB - VPC
• Well support: - Enterprise support
After AWS re:Invent, We Thought
• Cost optimization: - development faster, then deployment - quit/change even faster if necessary
• Brainless scalability (PaaS): - serverless microservices - Lambda + API Gateway
• Deeper support: - continuous mutual cooperation - more KKBOX staffs to AWS re:Invent!
KKBOX Video
• Initiated in 2014. • Team built from 10 staff. • 1yr development for the whole system. • Preparing for next challenges.