DevOps and the Cloud: All Hail the Developer King
DevOps: Microservices, containers, platforms, tooling... Oh yeah, and people
Daniel Bryant @danielbryantukSteve Poole@spoole167
On the Previous Episode of Devoxx UK2014 Moving to DevOps: Easy, Hard or Just Plain Terrifying
DevOps is about extending agility across your IT org
Breaking down the silos is vital
The business needs to react to the industry changeDevOps, cloud and containers
TodayMicroservices are (operationally/conceptually) distributed systems
The application/infrastructure platform is still not fully baked
Think Safety firstSecurity, networking cyber criminals
DevOps is (still) all about the organisation, people and processes
I (we) am the one who knocks
Steve PooleIBM Developer
@spoole167Daniel BryantChief (Mad) Scientist, OpenCredo
@danielbryantuk
Making Java Real Since Version 0.9Open Source Advocate DevOps Practitioner (whatever that means!)Driving Change
Biz-dev-QA-opsLeading change in organisationsExperience of Docker, k8s, Go, JavaInfoQ, DZone, Voxxed contributor
Part 1- Painful Lessonshttps://www.flickr.com/photos/sarahmstewart/
Steves section 1 5
All I hear is microservicesIn computing, microservices is a software architecture style in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These services are small, highly decoupled and focus on doing a small task, facilitating a modular approach to system-building.https://en.wikipedia.org/wiki/Microservices
MicroservicesTurn applications into small, independent, highly decoupled, modular services
https://www.flickr.com/photos/daikrieg/
You want to make my life more complicated?
https://www.flickr.com/photos/tahini/Wheres the problem?
browserApp A V1.0Databasebrowser
YouYour CustomersserverData Centre
Simple 1 server setup. Everyones happy no Ops in sight9
browserLoad balancerApp A V2.0App AV2.0App BV1.0Databasebrowser
Database
Ops
Add a new application and server setup. Everyones happy now load balancing required etc Ops team show up.. Youre a bit more stressed
10
browserLoad balancerApp A V2.0App AV2.0App BV1.0Databasebrowser
Database
browserApp A V2.0App AV2.0App BV1.0browser
Scaling up multiple instances of the applications , data replication, bigger load balancer more stress
11
browserLoad balancerApp A V2.0App AV2.0App BV1.0Database
Database
browserApp A V2.0App AV2.0App BV1.0browser
browserbrowserbrowserbrowser
Load balancer
Database
Now move some of the workload to a different location (say the cloud) more work, more cross location calls security issues, failing systems + much more stress to simply keep It12
browserLoad balancerApp A V2.1App AV2.1App BV1.0Database
Database
browserApp A V2.1App AV2.1App BV1.0browser
browserbrowserbrowserbrowser
Load balancer
Database
Throw h In an application update with scheme change 13
A simple upgrade or a major impact?
lost revenue or going out of business?
https://www.flickr.com/photos/24151087@N00/What lessons have we learnt?Sharing data stores sounds like it saves effort but introduces cohesion between applicationsBig-bang versioning of applications means putting existing unchanged use cases at risk Scaling is challenging when you try to duplicate whole systems Infrastructure its much more important than we initially realized
Part 2 - Build Your Own Platform?
Adrian Cockcrofts (@adrianco) Thoughts
http://wikibon.com/wp-content/uploads/container_implementations.png
Technology Choices
08/06/2016@danielbryantuk
Whats Wrong with PaaS?
Core FeaturesContinuous deployment
Health checks
Logging
Monitoring
www.opencredo.com/2015/10/31/javaone-building-a-microservice-development-ecosystem-video
LoggingThe Log: What every software engineer should know about real-time data's unifying abstraction
10 Tips for Proper Application Logging
ElasticSearch-Logstash-Kibana (ELK)Buffer/proxy log sending orMount directory into container
08/06/2016@danielbryantuk
MonitoringPushSpring Boot actuator e.g. InfluxDbExporter
Pull E.g. Telegraf (TICK), Prometheus
InfluxDB vs prometheus vs graphite vs opentsdb
Information radiators - GrafanaAggregate vs individual
08/06/2016@danielbryantuk
https://github.com/codecentric/spring-boot-starter-batch-web/blob/master/src/main/java/de/codecentric/batch/metrics/InfluxdbMetricsExporter.java
23
Aggregation: Sick Cattle, Not Sick Pets
Distributed TracingCorrelation: github.com/daniel-bryant-uk/correlation-id-asyncMDC logging: logback.qos.ch/manual/mdc.html OpenZipkin: github.com/openzipkin 08/06/2016@danielbryantuk
Looking Inside the Container
Common Java / Docker IssuesNo disk space for docker loggingIncrease disk space (move logs to mount)
Restricting resources to only Xmx memory limitSet memory limit = Heap (Xmx) + Metaspace + JVM
Security or crypto issues as /dev/random limited in containers-Djava.security.egd=file:/dev/urandom
See Chris Bateys The JVM and Docker talk here at 15:00 today
Debugging Tools
Javajstat, jstack, jmap5 things you didnt know
OSTop, htop, ps, free, df h, vmstat,iostat /proc filesystem meminfo and vmstat not cgroup aware!Use sysdigwww.joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x
Problems?Rob Ewaschuks Philosophy on Alerting
Brendan Greggs USE method check utilization, saturation, and errors.
DevOps TroubleshootingKyle Rankin
08/06/2016@danielbryantuk
Part 3 Safety first
https://www.flickr.com/photos/miriamdelirium/
Tooling whats left to do?
Dynamic DevelopmentCapacity Predefined static VMs LPARs etc OpenStack Cloud(s) Docker CloudInfrastructure as CodeChef, Puppet, UCD OSInfraOn Prem Data CentresCloud ProvidersSoftLayer / Amazon etcConfigContainerized Applications Continuous AvailabilityMesos etcDeployPipelineBlock Architecture of Hybrid CloudDev SaaS Primary AudienceGIT / Jenkins / Junit .Selenium, Jmeter ApplicationDIY
Dynamic DevelopmentCapacity Predefined static VMs LPARs etc OpenStack Cloud(s) Docker CloudInfrastructure as CodeChef, Puppet, UCD OSInfraOn Prem Data CentresCloud ProvidersSoftLayer / Amazon etcConfigContainerized Applications Continuous AvailabilityMesos etcDeployPipelineBlock Architecture of Hybrid CloudDev SaaS Primary AudienceGIT / Jenkins / Junit .Selenium, Jmeter Other static hosts (BYOD) DIYApplicationDIY
Dynamic DevelopmentCapacity Predefined static VMs LPARs etc OpenStack Cloud(s) Docker CloudInfrastructure as CodeChef, Puppet, UCD OSInfraOn Prem Data CentresCloud ProvidersSoftLayer / Amazon etcConfigContainerized Applications Continuous AvailabilityMesos etcDeployPipelineBlock Architecture of Hybrid CloudDev SaaS Primary AudienceGIT / Jenkins / Junit .Selenium, Jmeter Other static hosts (BYOD) DIYCompliance / SecurityContainersVM ImagesApplicationDIY
Dynamic DevelopmentCapacity Predefined static VMs LPARs etc OpenStack Cloud(s) Docker CloudInfrastructure as CodeChef, Puppet, UCD OSInfraOn Prem Data CentresCloud ProvidersSoftLayer / Amazon etcConfigContainerized Applications Continuous AvailabilityMesos etcDeployPipelineBlock Architecture of Hybrid CloudDev SaaS Primary AudienceGIT / Jenkins / Junit .Selenium, Jmeter Other static hosts (BYOD) DIYCompliance / SecurityContainersVM ImagesApplicationDIY
You do understand about security and compliance right?https://www.flickr.com/photos/adulau/
Wheres your data?Legal restrictions on data locationVary by country even within the EU. Different rules depending on types of data
You already know this?
But now youre putting the data in the cloud.
DO YOU understand where its goingCan you control / manage / audit the situation?
Now its your problem.
Having fun finding tools to help
Hows your security knowledge?Again now its your problem
Your code is running in the cloudYou created the services & the containersAre they secure?How do you test?Are you sure?Are those web services youre buying secure?
How much do you know about networking?Not enough
Cyber crime is big business you will get targeted.
https://www.flickr.com/photos/61423903@N06/
Organized Cybercrime is the most profitable type of crime Cybercrime is estimated to be worth 445 Billion Dollars a Year
In 2013 the United Nations Office on Drugs and Crime (UNODC) estimated globally the illicit drug trade was worth 435 Billion Dollars
Guess which one has the least risk to the criminal?Guess which is growing the fastest?Guess which one is the hardest to prosecute?
Guess which one is predicted to reach 2100 Billion Dollars by 2019?
Talk to your Ops teamThey are your best friends.They know about security and networking. You need to know tooThey know (some) of the answersIts a whole new domain for you Its not a new problem for themTime to learn
Part 4 - Sharing is Caring
The Results of the Survey Are InPuppet Labs 2015 State of DevOpsAvailable: puppetlabs.com/2015-devops-report
Accelerates deploymentHigh performers 30x more deploysCode committed to production 200x faster
Prevents failures and streamlines recoveryHigh performers 60x fewer failuresRecovery 168x faster
DevOps Topologies (Bad)DevOps A
