Upload
justin-carter
View
55
Download
0
Embed Size (px)
Citation preview
Local testing of Containerized Distributed Systems
Strange Group Meetup, Justin Carter (@bodymindarts)
Berlin, Oct. 27. 2016 Stark & Wayne
Delmo
Delmo is a tool to test systems running within multiple docker containers
https://github.com/bodymindarts/delmo
30 staff, 7 countries
PaaS experts
Cloud Foundry experts
Plattforms
Dingo PostgreSQL
Delmo
http://www.slideshare.net/KarlIsenberg/cloud-surfing-kubernetes-on-mesos
The sophistication of our platform bounds the granularity of decomposition in our
systems
(also means more complexity to setup locally)
How much of the cloud stack do we want to run locally (if any)?
Platform on dev workstation? Dev env in the cloud?
Apps decoupled from platform?
Contract Driven Tests / Mocks add a lot of code to maintain.
Dealing with deps?
Do we need them?
Cloud dev workflowHow much of the stack do we want to run?
How do we want to run our deps?
How do we automate this and integrate into an efficient workflow?
When you can deploy 1000s of apps, you’d better be able to deploy 1000s
of dbs!
Platform consideration
PostgreSQL in Docker
Clustered with streaming replication
Continuously backed up
Automatic failover
Dingo PostgreSQL
https://github.com/dingotiles/dingo-postgresql-release
Dingo PostgreSQL Architecture
H A P r o x y
Deployed by BOSH
Client 1
Client 2
Critical FeaturesRe-configuration of PostgreSQL
Restart PostgreSQL
Streaming backups to S3
Changing Docker Image
Test independently of BOSH
Test smaller system boundary
Build and test containers locally
Goals
Postgres Leader
Postgres Replica
Example TestWill failover work?
Dingo PG
HAProxy
Query
Leader
etcd
AssertDid data get replicated?
Did reconfiguration / promotion work?
Are connections being accepted?
Given: System definition
When: Some failure
Then: Assert expected Behaviour
Test Case
Delmo
Declare system with docker-compose
Declare test cases
Run tests (locally or remotely)
patroni1
Docker Compose
patroni2
etcd
System Tasks
Tasks are packaged bash scripts that execute commands against the running system
System
Tasks
Cache
Delmo Config
Meta
Tests
Tasks
Initialise
Start SystemRun TasksCleanup
Summary
MoreRun on a remote docker-machine
Run tasks in parallel
delmo-concourse image (for executing in CI/CD)
Next Steps
Better error output / validation
Network failure / partitioning
Support for other backends (bosh)?
Questions?
Justin Carter @bodymindarts github.com/bodymindarts github.com/bodymindarts/delmo