M M / D D / Y Y
YOUR T ITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Operationalizing Docker at ScaleLessons from running microservices in production
V e n k a t Thiruvengadam, ZenefitsM a x i m e P e t a z z o n i , S i g n a l F x
Introductions
Venkat Thiruvengadam• Principal Engineer @ Zenefits
• Creator of Project Duplo, a hosting platform for microservices
Maxime Petazzoni• Software Engineer @ SignalFx
• Creator of MaestroNG, a container orchestrator for Docker environments
M M / D D / Y Y
YOUR T ITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Micro-services @ Zenefits
Micro-services@ZenefitsThiruvengadamVenketesan
PrincipalEngineer
ZenefitsEngineering
• 200Developers• MonolithDjango ApplicationonMySQLBackend• TransitioningtoSOA
• Dozensofservices,fewHundredcontainers• HostedinAWS(Noonpremiseservers)• Duplo:Self-serviceHostingplatformforMicroservices
• NoDirectAWSaccessfordevelopers• NodedicatedDevops
MonolithvsMicroservices
• Monolith• Advantages
• SingleCodeBase.Easytoread/debug/deploy• Nointer-servicecontracts,compatibilityorauthenticationconcerns.• Easytogetstarted.Greatforsmallteams.
• Disadvantages• Singlepointoffailure• Slowiterationwithlargeteams(Managedbycentralopsteam)• Singleprogramminglanguage&technologypattern
• Microservices• Independentdeploymentandhencefasteriteration• DistributedFunctionality,nosinglepointoffailure• ProgrammingLanguage&technologyagnostic
Micro-services:TheDarkSide
ContainerManagement
ServiceDiscovery
ConfigurationManagement
Authentication
ManagementComplexityw/oright
tools
AWSResourceManagement
ResourceUtilization Monitoring
FindYourTippingPoint
Microservices
Monolith
TeamandProductSize
Productivity
@zenefits 3years,>millionlinesofcodeand200devs,wedecidedtomovetoMicroservices
OriginalapproachtoMicro-services@Zenefits
Developers
InfraAdmin
1
TranslatetoAWSConfiguration2
ApplyConfigurationusingTerraform\Otherautomation
3
SaveInfrastructureconfigurationlikeVPC,Subnet,SecurityGrp,IAM,EC2
4
5
SaveApplicationconfigurationlikenamecredentialsinappcode;builddocker image
6
7
DeploytoECS8
- Authenticationtokens,servicediscovery,configurationmanagementfollowsimilarpattern- Infrabecomesbottleneck.Notscalablewithlargenumberservicesandusers
Anonself-serviceinfrastructuredefeatsthepurpose
IGetSelf-service,DoIneedMulti-Tenancy?
Multi-Tenant:Eachapplication(orservice)isatenant.Usershaveaccesstoonlytheirrespectiveservices
Multi-tenancy(appteam)isnecessaryifyourorganizationneeds:- Resourceandsecurityisolationbetweenservices- Isolationofconfigurationmistakes- Audittrails- Varyingresourceprivilegesbetweentenants- VaryingDevelopersecurityawareness- Largenumberofdevelopers
Notenancy
TeamSize
Mis-config andSecurityRisksMulti-tenancy
How can I do CICD in Duplo?
Let’sgiveyouAWSaccess,createyourservice
WhatisthisSecuritygroup,subnet,IAM?Whydoesmydjangoappneedthis?Willallow*work?
Hmm…..
ShallwecreateanAWSusernameforeveryone?
WeNeedaHostingPlatform
TranslatetoAWSConfiguration
2
- ApplyConfigurationusingAWSandDockerAPIs- InjectconfigurationinappENV&discoveryservice
3
ApplicationrequirementsandDockerImageviaUI/API
1
Developers
Admin
- SelfService.Deployandmanageat-will.- Multi-tenant.Eachserviceorapplicationteamhavetheirownaccounts,resourcepool,IAMrole,SGetc.- ServicesisolatedexceptexplicitAPIinterfaces.- ProgrammaticInfrastructure,consistentconfigurationgeneration
- CreatebaseinfraConfigurationinAWSusingTerraform- InstallDuplo.Setupstaticbaseinfrastructurepolicies.- CreatestaticTenantpoliciesandlimitsa.k.a.“Plans”
0
DUPLO
DEMO:DEPLOYINGAMICRO-SERVICE
DuploMicro-servicesPlatform
DUPLO
AWSOrchestrationandabstraction
ContainerManagement
CI/CD
EcosystemOrchestrationAuthentication
ConfigurationManagement
ServiceDiscovery
HowDuploWorks
BaseInfrastructure:VPC,Subnets,NAT,AdminSecurityGroups,DNSDomainName,SSLWildcharCertTerraform
DUPLOAdminPortalNewServiceRegistration
IAMRole
SecurityGroup
Autocreateperservice
SecurityGroup
IAMRoleNewServiceRegistration
AwsResources
TenantPortal
DUPLO
CreatedbytenantactionsAws
Resources
DeployandManageService
TheNextStep:ContinuousIntegrationandDelivery
DuplocandeployimageswithdesiredAWSResources.It’sselfservice.WhydoIneedCI\CD?
• OfficialBuildImagesasagainstLaptoptests
• Testsuitesthataredocumented,enforcedwithpasscriterion
• Integrationenvironmentforintegrationtestswithotherservices
• DeploymentPipelineDevsandboxtostagetoprod.
WhatisspecialaboutaCI\CDsolutionforMicroservices,canIjustuseJenkins?
- MonolithCI/CDisonlyaboutcodecorrectnessandbuildartifacts- TheycanruninanInfratopologyseparatethantherealdeployment
- Beyondcode,MicroservicesCI/CDisaboutreplicatingtherealdeploymentinfratopology- ItneedstotesttheinterfaceoftheservicewithitscomplexinfrastructurelikeIAM,Securitygroupsetc.
- ServicesCI/CDneedstointegrateandbuildontopofthehostingplatform
CI\CDDEMO:GITPRToDeployment
KatKit:CI/CDExtensiontoDuplo
• After-allbuildscanbeaMicroservice too,theyarejustshortlived!Let’sreuseDuplotolaunchandterminatebuilds.
• Let’sbuildaworkfloworchestrationontopofduplo forCI/CD.
• WegetCI/CDandHostingPlatform“integration”forfree
SUMMARY
• Determinethetippingpoint.
• SelfserviceisP0.Determineifmulti-tenancy(isolation)isnecessary
• Hostingplatformisnecessary
• AWSbyitselfisnotasufficientsolution,needahigherlayerplatformtoprovideabstraction,containermanagementandCI/CD
• CI/CDsolutionhastointegratewiththehostingplatformandtestitsinteractionwiththeservicecode.
M M / D D / Y Y
YOUR T ITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
Q&A
For more information
• Duplo Platform• https://engineering.zenefits.com/2016/03/duplo-aws-orchestration-container-
management-and-paas-for-microservices/
• https://engineering.zenefits.com/2016/07/duplo-cicd-extensions-for-microservices/
• https://www.linkedin.com/pulse/duplo-vs-kubernetes-mesos-thiruvengadam-venketesan
• Monitoring Docker containers• https://signalfx.com/blog/monitoring-docker-containers-take-get-started/
• https://signalfx.com/blog/monitoring-docker-containers-signalfx-monitors-containerized-infrastructure/
• https://signalfx.com/blog/monitoring-docker-at-scale-with-signalfx/
M M / D D / Y Y
YOUR T ITLE HERE
P R E P A R E D F O R :
P L A C E L O G O
H E R E
THANK YOU!
S IGN UP FOR A TR IAL AT:
s i gna l f x . com