Skynet vs Planet of Apes, duel!

Adrien Blind, Société Générale - @adrienblindLaurent Grangeau, Sogeti - @laurentgrangeauLudovic Piot - @lpiot

With great help from J. Petazzoni - @jpetazzo& David Chung - @dchungsf

Overall Concept

VS

An independant, automated, hybrid, self-healing, multi-arch “Skynet” Docker CaaS, aimed at being merely indestructible: will it survive the Apes’ attack?

Independant & agnostic cluster

● Almost all cloud vendors provide their own complete, ready-to-use Docker CaaS stacks : Azure ACS, AWS ECS, Google Container Engine

● You may create your own docker CaaS, leveraging on several providers at a time : your DC, cloud vendors…

● Benefits: less lock-in, autonomy, migration, elasticity, cloud arbitration ...

A secured Cluster● Control plane:

secure by design

● Data plane:encrypted overlays(with --opt encrypted)

Provisioning: InfraKit● Setup bootstrap node with CloudFormation● Start InfraKit to check cluster desired state and spawn up the

different instances○ Get static assets from URL○ Check the principal groups.json○ Check the groups.json of the different plugins (AWS, GCP)○ Check the manager.json and the worker.json

● Deploy skynet script !

Provisioning: InfraKit● For IoT devices (RPi), it has an experimental

feature : remoteboot● Start a DHCP server, a iPXE and a TFTP server● Can startup LinuxKit image

InfraKit: focus on selector plugin● InfraKit can leverage multi-cloud

(here AWS and GCP)● It takes care of the resilience of the

cluster at the nodes level● Enables scaling cluster up and down

InfraKit: focus on selector plugin"Instance": {"Plugin": "selector/spread","Properties": {

"aws/ec2-instance": { "region": "eu-central-1a", "type": "t2.micro" }, "gcp/compute": { "MachineType": "n1-standard-1", "Disks":[{ "Boot": true, "SizeGb": 60, "Image": "https://www.googleapis.com/[...]/ubuntu-1404-trusty-v20161205", "Type": "pd-standard", "AutoDelete": false, "ReuseExisting": true }], "Scopes": [ "https://www.googleapis.com/[...]/logging.write" ] }

}}

aws-eu-central

swarm configs

Group RPC API

Client

gcpaws

gcp-eu-central

50% 50%

Self-healing with InfraKit●InfraKit takes care of the desired state of the cluster

●When instances are terminated, new ones are spawned

Self-healing and Scalability2 separate stages of resilience & scalability:● At containers/service level● At cluster/swarm level

App architecture design is crucial to benefit from this form of resiliency:● Foster microservice principles: stateless, take care of app

discovery, data...

Fight! Chaos Monkey

● Originally worked on Netflix’s Chaos monkey, but:○ Built for prod, not demo (kills instances every 24h!)○ Tightly coupled with target apps delivery tools (Spinnaker)

● Created our own Chaos monkey that terminates nodes every 3 min○ Enables us to test infrastructure’s resilience○ Packaged in a Docker image

Demo time!

LinuxKit: on-purpose OS

● Small is beautiful

● Immutable:Rebuild vs Upgrade

LinuxKit: on-purpose images● Create a linuxkit.yml with Docker inside

○ - name: docker

○ image: docker:17.09.0-ce-dind

● Build the LinuxKit image○ moby build -output raw linuxkit.yml && linuxkit push aws -

bucket skynet-linuxkit -timeout 1200 linuxkit.raw● Repeat for each cloud provider

● Start the AMI on each provider● The linuxkit/metadata package gets the public SSH key and writes

it at boot start

LinuxKit: drawbacks● We cannot use “cloud-init” due to LinuxKit’s immutable nature

● How can we inject the manager IP address and Swarm token?○ Bake the token in the LinuxKit image○ Expose token with “infrakit util fileserver”, then:

“docker swarm join --token $(curl http://leader:port/token)”○ Add a “cloud-init” style container for LinuxKit

Hybridation: get best of breed!Rise of Windows & HyperVContainers

ARM: end of the x86 hegemony?(Tiny servers, IoT…)

Create hybrid clusters to create cross-tech stacks

Back on SkynetCloud (x86) & physical (RPi) nodes

Terminator containerized app on RPi only:● Forwards sensors data to skynet brain● Gathers instructions

Skynet containerized brain everywhere:● Stores information (mocked w/ Consul)

Should you destroy every DC on earth, terminators should continue to interact

Hub & registries support various images architectures since Jan. 2016● adrienblind/consul-amd64 → built for amd64● hypriot/rpi-consul → built for arm

Manifest lists are “meta-images” containing references to (arch,image) pairs● adrienblind/consul → references both images above

Given a manifest, Docker will automatically pick theappropriate image

Multi-arch images

Not clear ?

Multi-arch imagesmanifest-tool : CLI tool implementing client side of Docker registry v2.2 API for interacting with manifest objects in a registry conforming to that specification (https://github.com/estesp/manifest-tool)

./manifest-tool --username user --password password inspect hypriot/rpi-consulhypriot/rpi-consul: manifest type: application/vnd.docker.distribution.manifest.v1+json

Digest: sha256:53563136358cc966e850b577008084bbeb246bf53126a0c631c47c5271586717Architecture: arm

OS: linux# Layers: 0

Multi-arch imagesmultiarch-consul.yaml :image: adrienblind/consul:latestmanifests:

-Image: adrienblind/consul-

amd64:latestplatform:architecture: amd64os: linux

-image: hypriot/rpi-consul:0.6.4platform:architecture: armos: linux

Now push the manifests list on the Hub:./manifest-tool --username user --password password push from-spec multiarch-consul.yaml

Multi-arch images./manifest-tool --username user --password password inspect adrienblind/consulName: adrienblind/consul (Type: application/vnd.docker.distribution.manifest.list.v2+json)Digest: sha256:7c718aeb9e98319466e00d70511275dd2bf3d31da789044b4ad50617a94ad64d* Contains 2 manifest references:1 Mfst Type: application/vnd.docker.distribution.manifest.v1+json1 Digest: sha256:82b48d7135ee2ddc22c524ec2359a4114870d3963e9701d51a31f936cea90e071 Mfst Length: 529881 Platform:1 - OS: linux1 - OS Vers:1 - OS Feat: []1 - Arch: amd641 - Variant:1 - Feature:1 # Layers: 0

(continuation)2 Mfst Type: application/vnd.docker.distribution.manifest.v1+json2 Digest: sha256:1d96180da7455ad590d9e6a7434180aa544d3960ac867cda569f5a0613ab28022 Mfst Length: 132552 Platform:2 - OS: linux2 - OS Vers:2 - OS Feat: []2 - Arch: arm2 - Variant:2 - Feature:2 # Layers: 0

Multi-arch imagesdocker service create --name multiarchconsul --replicas 3 adrienblind/consul

ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTSg4ztbaz4vkxf multiarchconsul.1 adrienblind/consul:latest vm Running Running 6 seconds ago

xu9e4ytdvc6u multiarchconsul.2 adrienblind/consul:latest pi1 Running Running 4 seconds ago

r3rpughvfm1w multiarchconsul.3 adrienblind/consul:latest pi2 Running Running 2 seconds ago

Docker CLI PR 138 will embed everything natively (https://github.com/docker/cli/pull/138)● By Christy L. Norman Perez● docker manifest create|annotate|push

Conclusion

Conclusion● InfraKit is awesome for provisioning & self-healing infrastructure● LinuxKit is awesome for immutability● Multi-arch images are great for seamless IT automation● Docker gets beyond containers, enabling to automate IT leveraging

DevOps principles

● Infinit.sh storage: not addressed yet (stay tuned!)

Beyond the talk● Replace the Raspberry Pis with Boston Dynamics robots● Merge with LAWs principles (Lethal Autonomous Weapons)● Decentralize network (Navy works on Disruption-Tolerant Mesh Nets)● Mix some AI capabilities to replace our simplistic Consul brain

● … and you’ll get real Terminators. Oups!

We’ll be back...

Appendix - Original pitch

Two self-managed Docker clusters deployed on public clouds and fight each other in a ruthless battle. One has been designed to resist any form of threat. The other one's only aim is to destroy the first one. Who's going to win?Through this fantasy, we'll first cover all the technologies concretely used to set up the platforms and run the battle (linuxkit, infrakit, & swarm mode, and even raspberry devices among others), while we'll step back in the second part to address the subsequent architecture stakes involved : reliability, scalability, edge computing, immutability, micro-services, hybridation, distributed storage will hold no secret for you ! Most of all, you'll understand the importance of the synergies implied between the platform's and the app's design to achieve such a result.