View
208
Download
0
Category
Preview:
Citation preview
@slallema
Migrating 100% of your production services to containers
Simon LallemandSystem engineer
Over 30 million members and growing
1.5 million+joining each month
We are in 22 countries
150+ bare-metal servers200+ services3000+ running containers
?
How did we get there ?
(Pre)HistoryThe platform
evolution
It all started 10 years ago...
Web hosting
Dedicated server
1 rack
3 racks
5 racks
Industrialization eraStep 1
Step 1 : Installation and configuration of the servers
Chef
Foreman
Collins
Industrialization eraStep 2
Step 2 : Hardware uniformization
Limit the models of servers
Buy one rack at a time
Engineers don’t go to the DC
Industrialization eraStep 3
Step 3 : Abstract the hardware
Different workloads on one server
VM ? no.
=> Containers !
Containers : from the choice to the prod
Containersas container runtime
Dock... no, rkt !
Experimenting with rkt in early 2015
Taskforce to create the workflow
Workshops to validate
A lot of iterations
No existing tools: we made our own !
80% of the prod migrated in november 2015
The new infrastructure
On 100% of our new servers
CoreOS : easier to install and maintain
Perfect OS to run containers
CoreOSstable
rktas container runtime
we rkt !
Everything must run in containers !
(One of?) The biggest rkt deployment in production
We like the simplicity, stability and the modularity
?
How to build a container image ?
App containerspecification
A way of packaging
We like the appc approach because - it’s an open specification - it supports PODs - it’s easier to build tools around this spec than with docker images
But we also watch closely the OCI...
dgrContainer build and runtime tool
Harmonize the way we build
Quick build
One way of doing things
Easy to understand for newcomers
As little code replication as possible
Templating at container start
A good integration with rkt
github.com/blablacar/dgr
dgr : build directory of an ACI
├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl
A standardized structureEase maintenance and teamwork
Inspired by config managementSeparation of templates, attributes and scripts.
dgr : ACI manifest
name: redis:0.1aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf
Simpler manifest formatFill only what is important
YAML <3
One process per ACIComposition is done only with POD
dgr : ACI manifest
name: elasticsearch:2.3.3aci: app: exec: - /bin/elasticsearch dependencies: - debian - java
Use of dependenciesCompositionLighter images
dgr : runlevel build
├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl
Scripts executed in the containerBuild from inside the container with all the dependencies
dgr : runlevel build
#!/bin/bash
apt-get install -y redis-server
Install packages and stuffFor debian based containers you can install packages using apt-get for instance.
dgr : runlevel builder
name: aci-go-synapsebuilder: dependencies:
- example.org/aci-go- example.org/aci-git
aci: app: exec: - /bin/go-synapse
Build from outside of the containerDedicated builder image with go & git
dgr : runlevel builder
#!/dgr/bin/busybox sh
git clone https://github.com/blablacar/go-synapse
cd go-synapse
go build
cp dist/go-synapse ${ROOTFS}/bin
Builder scriptClone, build and copy binary to target ACI
Only /bin/go-synapse in final ACI
dgr : runlevel builder
name: example.org/gentoo-lighttpdbuilder: dependencies: - example.org/gentoo-stage4aci: dependencies: - example.org/base app: exec: [ /usr/sbin/lighttpd, -D ]
Other exampleBuilder with gentoo’s stage4
dgr : runlevel builder
#!/dgr/bin/busybox sh
emerge -v lighttpd
Install packages from outsideUse emerge to install your app and its dependencies.
Lighttpd with dependencies only in the final ACI
dgr : templates & attributes
├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── build│ └── install.sh└── templates └── etc └── redis └── redis.conf.tmpl
Render configuration filesTemplates stored in the aci
Default attributes stored in the aci
Overridable when used as dependencies
Overridable by environment var
dgr : templates & attributes
# templates/etc/redis.conf.tmpldaemonize noport {{ .redis.port }}…
# attributes/redis.ymldefault: redis: port: 6379
dgr : runlevel prestart
├── aci-manifest.yml├── attributes│ └── redis.yml├── runlevels│ └── prestart-late│ └── 10-init-db.sh└── templates └── etc └── redis └── redis.conf.tmpl
Initialize containerprestart-early and prestart-late scripts
Before and after templating
Initialize environment before exec
dgr : testing
├── aci-manifest.yml├── attributes├── runlevels├── templates└── tests └── wait.sh └── my_cool_tests.bats
TestingBats as default tester
wait.shWait for service to be ready
dgr : testing
#!/dgr/bin/bats -x
@test "Nginx should be running" { run bash -c "ps -aux | grep nginx" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/sbin/nginx" ]]}
@test "Nginx should listen on port: 80" { run bash -c "netstat -peanut | grep nginx" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":80" ]] [[ "${lines[0]}" =~ "nginx" ]]}
dgr : subcommands
~ # dgr init
~ # dgr try
~ # dgr build
~ # dgr test
~ # dgr install
~ # dgr push
?
What about PODs ?
dgr : POD manifest
name: example.org/pod-myapp:1pod: apps: - dependencies: - example.org/nginx:1.10
- dependencies: - example.org/php:7.0 - example.org/myapp:42 app: exec: [/usr/sbin/php5-fpm, -F]
Grouping ACIACIs as a single unitSame execution context
Use dependencies (again)Create dedicated ACIs for the pod
Pin versionCan override attributes/templates for pod context
dgr : run a POD
$ rkt run \ example.org/pod-myapp_aci-nginx:1 \ example.org/pod-myapp_aci-php:1
?
How to contextualize a POD ?
dgr : run a POD
# attributes/redis.ymloverride: redis: maxmemory: 4GB
TEMPLATER_OVERRIDE={“redis”:{“maxmemory”: “4GB”}…}
Override attributesInject run context attributes as environment var
dgr : run a POD with systemd
[Service]ExecStart=/usr/bin/rkt run \ --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
ggnManage services in fleet clusters
Pods Services and Environments
Different services that use the same pods
Environments (1 per DC for prod + dev…)
Abstraction of fleet commands
github.com/blablacar/ggn
INSERTLOGOHERE
ggn
├── prod-dc1│ └── attributes│ └── services│ └── redis-quota│ │ └── attributes│ │ └── service-manifest.yml│ │ └── unit.tmpl│ └── redis-rating├── preprod│ ...
Single directory treeDescribes the servicesThe environmentsVersioned in git
ggn : service manifest
containers: - aci.blbl.cr/pod-redis:3nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3
Define the servicePod image and versionNodes composing the service
ggn : run a POD
$ ggn prod-dc1 redis-quota update
$ ggn prod-dc1 redis-quota journal -f
github.com/blablacar/ggn
How do pods talk to each other ?
Service discovery
with nerve & synapseAirBnB’s smartstack
Dynamic topology of services
Scalability requires service discovery
We used smartstack of AirBnB
Since then we improved it and rewrote it in go
github.com/blablacar/go-nervegithub.com/blablacar/go-synapse
Service discovery/database/node1
XX
/database
?
What’s next ?
rktnetesOrchestrate all the things !
+
Questions?
@BlaBlaCarTechBlaBlaTech.com
@slallema
Thanks!@BlaBlaCarTech
BlaBlaTech.com
Recommended