moovel aws summit Frankfurt 2016

Using AWS to Build a Large ScaleDockerized Microservices ArchitectureDr. Oliver Wahlen moovel Group GmbHFrankfurt, 30. Juni 2016

The moovel Group GmbH

Our vision is an ecosystem that• simplifies mobility now• solves the mobility challenges of the future

Mobility as a Service

Urban areas will become more crowded:• Today 54% of people live in cities In 2050 there will be 66%• More people on less space• Driving and parking will become more regulated• High demand for new transportation concepts

Autonomous driving will become reality:• Taxi and car sharing business will change completely• Cost efficiency similar to public transport but faster• Owning a car becomes less relevant

The future will be Mobility as a Service

Entwicklung 3: Autonomes Fahren

Wir verändern die Art, wie wir uns fortbewegen.

We are building apps, widgets and services that integrate existing transportation providers

Benefits for the user:• Only one registration: no need to remember passwords

or provide payment data several times• Transparency on mobility alternatives• Intermodal routing

Benefits for the partners:• Get new users from other mobility providers• Get new users from different geographic regions

moovel Technology



• Scalability up to millions of users

• Future prove: Easy to modify components and update technologies

• Usable on global scale

• Minimal operational effort

• Teams can develop independently

Design Goals of Our System Architecture



Avoid Vendor Lock-In vs. Reduced Operational Effort

Services currentlyused by moovel

Entwicklung 3: Autonomes FahrenWhat and Why Docker?

Docker Containers• Provide application(s) with a virtual operating system• Isolate applications from the host OS and other docker containers

(e.g. Files, Network)• Are self contained run on any machine without other installs• Are simple to build images reference a base image

Docker Registry• Stores docker images produced by the build system• Provides access control and maintenance of tags• Dockerhub (Docker Inc.) vs. Amazon EC2 Container Registry

Entwicklung 3: Autonomes FahrenWhat and Why EC2 Container Service (ECS)?

ECS is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster of Amazon EC2 instances.– AWS documentation

Technology similar to Kubernetes on Google Container Enginebut tailored for AWS and fully managed

Terminology• ECS cluster: Group of EC2 instances that allow docker execution• ECS container instance: EC2 instance with ECS container agent• ECS task: The running docker container on an instance• ECS task definition: docker image, CPU, mem, port requirements, …• ECS service: number of parallel tasks, scalability, health checks, …

Entwicklung 3: Autonomes FahrenWhat and Why Microservices?

http://martinfowler.com/articles/microservices.html

• Services independently deployable Continuous Deployment

• Scalable on service level Specific to service needs:

CPU, memory

• Services have firm boundaryallows for: different languages per service different teams per service group evolution of logic and technology

ECS Cluster

EC2 Instance

Docker Task= ECS Task

Web Service

Entwicklung 3: Autonomes FahrenConway’s Law And Microservices: A Good Fit

Organizations which designsystems [...] are constrained toproduce designs which are copiesof the communication structuresof these organizations

– Melvin Conway 1967

Product Owner

App-Developers

Backend-Devs

SearchSquad

BookSquad

UserSquad

PaySquad

moovel product dev is organized in squads (feature teams):• with a long term business vision• ability to build and modularize mobile apps / componets• build their own set of microservices• choose the technology that best fits their needs

Entwicklung 3: Autonomes FahrenContinuous Integration with Docker and AWS

Developer

code push to

SaaS Build SystemVersion Control

triggers dockerfile.tgz

S3 Bucket

Build Path

SaaS Build System Dockerhub

Deployment triggers:develop branch: automaticmaster branch: manually

Deployment Path

docker imagepush to

start update

Cloudformation Elastic ContainerServices: dev/prod

update docker

Entwicklung 3: Autonomes Fahrenmoovel System Architecture

moovel VPC

Client App

External ELB• TLS termination

Kong API Gateway• reverse proxying• LUA plugins

Consul• service discovery• client load balancing• service configuration

synchronous REST call

ECS microserviceSNSTopic

SQSQueue

asynchronouslypublish changes

logging Kinesis ELK Stack

Entwicklung 3: Autonomes FahrenLimitations of Existing Solution

• Difficult to realize dynamic port allocation Currently each service has a fixed port

• Two levels of scaling introduce additional complexity: Cluster nodes = No. of EC2 instances ECS Tasks = No. of running Docker containers for a service had to introduced a custom metric taskless instance count

• Service discovery with Consul: Automates node and service registry Provides hierarchical key value store with ACL for configuration Consul is a very critical component Must be operated and maintained

• Kong API gateway: Extremely flexible and powerful nginx based reverse proxy Requires cassandra DB Must be operated and maintained

• Elasticsearch Logstash Kibana (ELK): Do not use it for long term data storage!

Entwicklung 3: Autonomes FahrenCheck-up of Design Goals

Scalability moovel runs 80 services on 21 ECS clusters (dev+prod)

Future prove Exchanging microservice technology is easy We have microservices in Java, Node.js, Python, Kotlin, …

Global Scale The world is currently operated from eu-west-1 Pub/Sub is good starting point for multi region deployment

Minimal Operational Effort Technology is operated by 10 squads Currently no DevOps team needed: squads build it and run Deployments are easy, frequently done and without downtime

Teams can develop independently Technology is operated by 10 squads Microservices, REST and Pub/Sub are a good basis UI decomposition in app is difficult: modularization ongoing

Thank You