Mailiao Refinery, Taiwan May 31, 2016
Troy Toman @troytoman
Managing 6PB of Earth Imagery with Kubernetes
Dongtaijinai’er Salt Lake, China MAY 3, 2016
You think you have cloud problems? Let me show you cloud problems
Uluru, Australia, DEC 2, 2015
and some Kubernetes
SPACE!
AGENDA
• Planet’s Mission 1• Where we started• Why Kubernetes• Where we are going
To image the whole world every day, making change visible, accessible and actionable.
CAPE TOWN, SOUTH AFRICAMARCH 9, 2016
MONITORING PORT ACTIVITY
CAPE TOWN, SOUTH AFRICAMARCH 14, 2016
MONITORING PORT ACTIVITY
CAPE TOWN, SOUTH AFRICAMARCH 15, 2016
MONITORING PORT ACTIVITY
SAN GABRIEL MOUNTAINSJUNE 4, 2016
SAN GABRIEL MOUNTAINSJUNE 4, 2016
SAN GABRIEL WILDFIRES
VEGETATION INDEX MAP
VEGETATION
bare sparse densemoderate
SAN GABRIEL WILDFIRES
FIRE RETARDENT LINES ARE IN RED
JUNE 22, 2016SAN GABRIEL MOUNTAINS
JUNE 22, 2016SAN GABRIEL MOUNTAINS
FALSE COLOR COMOPOSITE
REVEALS BURN SCARS
Muir Woods & Mt. Tamalpais, California, USA DEC 23, 2015
OUR APPROACH
HISTORICAL APPROACH • Tasking• Low coverage
• Weeks to gain access
• Expensive
PLANET APPROACH • Monitoring• Global coverage• Daily online delivery
• Affordable
13 Builds in only 3 years
BUILD 1 APR 2012
BUILD 6 APR 2013
BUILD 13 JUN 2015
AGILE AEROSPACE
12+SATELLITES
3 - 5M RESOLUTION
4-BAND IMAGERY
10 GROUND STATIONS
4SITES
10,000IMAGES PER DAY
APIFOR INTERNAL DATA
PIPELINE
100SOF VIRTUAL MACHINES
TBsof STORAGE
The early days …
Early approach
• One service team• 2 basic services
• Storage• Jobs
• Everything packaged as debs• Deployed on VMs w/Ansible• Everything tested/deployed as one
Planet’s Evolving Infrastructure
JobsWorker Spot
Instances
Imagery Storage
Storage Elastic Search
Planet API v0
100+SATELLITES
3 - 5M RESOLUTION
4-BAND IMAGERY
26 GROUND STATIONS
9SITES
10+TBPER DAY
EVERY DAY
APIFOR INTERNAL DATA
PIPELINE AND PLATFORM ACCESS,
CHANGE DETECTION, BASE MAPS,
MOSAICS, and MORE
10,000sOF VIRTUAL MACHINES
PBsof STORAGE
The very near future …
PLATFORMAUTOMATED DATA
PIPELINE
DATA AND FUNCTIONAL EXPLOSION!
MOSAIC & IMAGE
PROCESSING
ANALYTICS CONTAINER
API – RESTFUL & JSON
PlanetScope~150 Mkm2/DAY
RapidEye~6 Mkm2/DAY
Processing Engine
Ingestion API WEB TILES
API
CATALOG
SCENES, MOSAIC, TILE APIANALYTICS ANALYSIS API
ARCHIVE GUIFIND, DOWNLOAD
DATA VIA WEB BROWSER
TOOLSSAMPLE CODE
3RD PARTY APPLICATIONS
More to come …
Landsat 8
Large Scale, Dynamic Environment
We’ve Seen Change
• Growing number of service teams• Dozens of services
• Different scales• Different maturity• Different markets
• Unintentional monolith and/or unmanageable fringe services
• Infrastructure as impediment
How our software world is changing• Containers
• Flexible per service packaging• Faster, simpler, consistent deploys
• Kubernetes for orchestration• Dynamic reallocation of resources• Isolate development teams from infra
changes and each other• Couple service changes
where necessary
Planet’s Evolving Infrastructure
JobsWorker Spot
Instances
Imagery Storage
Storage Elastic Search
Planet API v0
Planet API v1
Infra
Apps
Planet Kubernetes Tooling
• Klondike• Bootstraps k8s on AWS for Planet• https://github.com/planetlabs/klondike• Focused on infrastructure deployment
• Planet-Express• Environments, Services, Deployments, Secrets• Template driven manifests and artifacts
kubectl create -f build/<platform>/<service>/ kubectl replace -f build/<platform>/<service>/
• Focused on services development and deployment
Key Learnings
• Kubernetes increases our flexibility and responsiveness
• Move can be evolutionary• Tooling is leveraged across teams• More good stuff to come• More change on the way - better
prepared to exploit opportunity
London Array Wind Farm, United Kingdom, APR 17, 2016
See Change. Change the World.
(applies to software and infrastructure too!) [email protected]
@troytoman
London Array Wind Farm, United Kingdom, APR 17, 2016