Wrapp: Transitioning to ECS

Containers @ WrappPowered by AWS EC2 Container Service

JudeD’SouzaSolutionsArchitect@WrappPhone:+46767085740Email: [email protected]

AboutMe…q JudeD’Souza– Stockholm,Swedenß Karachi,Pakistan

[email protected]

q MS– DistributedSystemsKTHRoyalInstituteofTechnology,Stockholm,Sweden

q [email protected]

q Passionforbuildinglargescaledistributedsystems

AboutWrapp…Wrapp deliverspersonaloffersandrewardsbasedonwhereyoushop,dineandthebrandsyoufollow.

q Microservice Architecture

q Infrastructureq 13ECSClustersq 50+Instancesq 90+Services

q TechnologyStackq OperatingSystem- Ubuntuq Consul,Registrator,AWS(EC2,ECS,Lambda),Docker,HAProxyq Monitoring– Sensu,Riemannq Logs- Logentries andSumoq Languages- Go,Python,Java

Agenda• ArchitecturebeforeECS

– ServiceOrchestration– ServiceDiscovery– ServiceDeployments– ServiceAvailability&Scalability– ContainerRegistry

• TransitiontowardsECS– Microservice Architecture@Wrapp

• WhyECS?

• NextChallenges

• QA

Howwedidit…BeforeECS(1/3)

ServiceOrchestrationq Viarunlist configs

Oninstanceboot:q Determineautoscaling groupq Pullrunlist forautoscaling groupq Runcontainersspecifiedinconfigq Addthesetosupervisor

misc ops api

miscAutoscaling group

opsAutoscaling group

api Autoscaling group

PULLRUNLISTCONFIGS

SampleRunlist configurationfile

ü ServiceOrchestrationServiceDeploymentsServiceAvailability&ScalabilityContainer Registry

ServiceDeployments

q Find instanceswhereserviceisrunning(viaserf)

q SSHintotheseinstances

q Dorollingdeploy (Terminatecontainer,Launchcontainerwithnewversion)

misc ops api


- misc-i-abc 10.0.0.1 rewards/http/9090- misc-i-def 10.0.1.1 rewards/http/9090- misc-i-ghi 10.0.2.1 rewards/http/9090- ....- ....

Sampleserfdataforservices

SERFCLUSTER


apiAutoscaling group


ü ServiceOrchestrationü ServiceDeploymentsServiceAvailability&ScalabilityContainer Registry

ServiceAvailabilityq Supervisord

ServiceScalabilityq Serviceitselfcan'tscaleq Scaleonly byscalingautoscaling group

misc ops api


SERFCLUSTER


apiAutoscaling group

misc


ContainerRegistryq Hostingourowncontainerregistryq Backedbys3

ü ServiceOrchestrationü ServiceDeploymentsü ServiceAvailability &Scalabilityü Container Registry

TransitiontoECS

ServiceOrchestrationq Runlistsq SupervisorServiceDeploymentsq SupervisorServiceAvailabilityq SupervisorServiceScalabilityq EC2AutoscalingContainerRegistryq Self-hosting

ServiceDiscoveryq Serfq HAProxy

ServiceDiscoveryq Consulq Registratorq HAProxy

ServiceOrchestrationServiceDeploymentsServiceAvailabilityServiceScalability (withlambda)

AWSContainerRegistry

+Servicestats(cpu +memory)

+Servicemonitoring+ServiceProvisioning

WrappMicroserviceArchitecture

PUBLICSUBNET

PRIVATESUBNET

ElasticLoadBalancer

api api api

Microservices

ZONEA ZONEB ZONEC

api.wrapp.com DNS

Opsweb– ToolingaroundcontrollingourInfrastructure

ClusterOverview


ServicesOverview


InstancesOverview

ServiceScalability(ConceptfromAmazon)

misc


ScalingAmazonECSServicesAutomaticallyUsingAmazonCloudWatchandAWSLambda:https://aws.amazon.com/blogs/compute/scaling-amazon-ecs-services-automatically-using-amazon-cloudwatch-and-aws-lambda/

CloudWatch AlarmSNSTopic AWSLambda

EC2ContainerService

ECSInstances

TriggerCloudWatch alarmfrom ECSMetrics(CPU/

Memory)

TriggerLambdafunctionfrom SNSTopic

+1DesiredCount inService

Service(CPU/Memory)Provisioning

api

emails

rewardsauth

notifications

eventbus

TypicalECSInstance

ü MemoryallocationbyService

ü AllocateCPUUnitsbyService

ü EfficientResourceutilization(Container placement)

ContainerLogging

TypicalECSInstance

q Logspout

q Rsyslog

q LogEntries

Logentries:https://logentries.comLogspout: https://github.com/gliderlabs/logspout

logspout

rsyslog

ContainerslaunchedviaECS

LogrouterforDocker

Forwardslogstologentries

WhyECS?

q OurExperience

q Generally goodservice (ECSisoK)

q Greatsupport

q Wedon’tneed tohosttheseservicesourselvesonAWS

q Alwaysimprovingtomaketheirservicesbetterandbetter

ü Supportsbuilt-inservicediscovery,loadbalancingandrouting

ü WeaveFlux:Supportsbuilt-inservicediscovery,loadbalancingandrouting

ü DockerSwarm:Supportsorchestratingcontainers

ü Fleet - ContainerOrchestration

api

8080

Consulserver-b

Consulserver-a

Consulserver-c

users

8082

auth

8081

users

8082

Registrator Registrator Registrator

ConsulAgentConsulAgentConsulAgent

Key/valuestore

Servicecatalog

10.0.0.1 10.0.0.2 10.0.0.3

ServiceDiscovery@Wrapp(1/2)HAConsulCluster

ServiceDiscovery@Wrapp(2/2)

Consulserver-b

Consulserver-a

Consulserver-c

users

8083

Registrator

ConsulAgent

10.0.0.3

HAProxy

auth

8082

ConsulTemplate

frontend users-inbind users:80mode httpdefault_backend users

backend usersserver Server-a 10.0.0.1:8080 …server Server-c 10.0.0.3:8080 …

####

frontend auth-inbind auth:80mode httpdefault_backend auth

backend authserver Server-a 10.0.0.1:8081 …server Server-c 10.0.0.3:8081 …

>> curl users/v1/users/1{"name": "jude""gender": "male"

}

HAConsulCluster

DNSHack(192.168.x.x)

NextChallenges(1/2)– Shortcomingsperceived…

Built-in(likekube-proxy,weave-proxy)?q ServiceDiscoveryq ServiceRoutingq ServiceLoadbalancing

Approach1:ELBPerservice?q Costly- 90+services,70+ELBs

Approach2:SingleELBforallservices?ü ServiceRoutingworksü ServiceLoadbalancingworksq Nohealthcheckperserviceq NowaytoreturnHTTP503

(butpossibleviahaproxy httpcheck option)

Benefits:q RemovestoolslikeConsulandRegistrator

ElasticLoadBalancer

8080 8081 8082

ECSBoto3:create_service()

Route53: ELB.dns = elb-services

frontend users-inbind users:80mode httpdefault_backend users

backend usersserver users elb-services:8080 …####

>> curl users/ping

SINGLEELBFORMULTIPLESERVICESAPPROACH

NextChallenges(2/2)– Shortcomingsperceived…

Ø Cron schedulingq Setupecs taskstorunoncron schedule

Ø ECSEventstreamq Actionables oncertainevents(whendeploycompletes)

Ø ECSContainerLogging

Ø OptionallyIntegrateECSClusterwithEC2Autoscaling Group

Summary

q TheWrapp ArchitecturebeforeECS

q TransitiontoECS

q ServiceDiscovery@WrappwithConsul,Registrator andHAProxy

q OurperceivedshortcomingsfromECSq Built-inServiceDiscoveryq Cron schedulingonTasksq Eventstream

Questions

?

Contact:JudeD’SouzaEmail:[email protected]:+46767085740Twitter:Wrapp Tech@wrapp_tech