Upload
daniel-bryant
View
1.055
Download
0
Embed Size (px)
Citation preview
DevOps: Microservices, containers, platforms, tooling... Oh yeah, and people
Daniel Bryant @danielbryantuk
Steve Poole@spoole167
On the Previous Episode of Devoxx UK…
• 2014 “Moving to DevOps: Easy, Hard or Just Plain Terrifying”
• DevOps is about extending agility across your IT org
• Breaking down the silos is vital
• The business needs to react to the industry change– DevOps, cloud and containers
Today
• Microservices are (operationally/conceptually) distributed systems
• The application/infrastructure ‘platform’ is still not fully baked
• Think “Safety first”– Security, networking cyber criminals
• DevOps is (still) all about the organisation, people and processes
I (we) am the one who knocks…Steve Poole
IBM Developer
@spoole167
Daniel Bryant
Chief (Mad) Scientist, OpenCredo
@danielbryantuk
Making Java Real Since Version 0.9
Open Source Advocate
DevOps Practitioner (whatever that means!)
Driving Change
“Biz-dev-QA-ops”
Leading change in organisations
Experience of Docker, k8s, Go, Java
InfoQ, DZone, Voxxed contributor
Part 1- Painful Lessons…
https
://w
ww
.flic
kr.c
om/p
hoto
s/sa
rahm
stew
art/
All I hear is microservices…
“In computing, microservices is a software
architecture style in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These
services are small, highly decoupled and focus on doing a small task,
facilitating a modular approach to system-building.”
https://en.wikipedia.org/wiki/Microservices
Microservices
Turn applications into small, independent, highly decoupled, modular services
https://www.flickr.com/photos/daikrieg/
You want to make my life more complicated?
https
://w
ww
.flic
kr.c
om/p
hoto
s/ta
hini
/
Where’s the problem?
browserApp A V1.0
Database
browser
You
Your Customers
server
Data Centre
browserLoad balancer
App A V2.0
App AV2.0
App BV1.0
Database
browserDatabase
Ops
browserLoad balancer
App A V2.0
App AV2.0
App BV1.0
Database
browserDatabase
browser
App A V2.0
App AV2.0
App BV1.0
browser
browser
Load balancer
App A V2.0
App AV2.0
App BV1.0
Database
Database
browser App A V2.0
App AV2.0
App BV1.0
browser
browser
browser
browser
browser
Load balancer
Database
browser
Load balancer
App A V2.1
App AV2.1
App BV1.0
Database
Database
browser App A V2.1
App AV2.1
App BV1.0
browser
browser
browser
browser
browser
Load balancer
Database
A simple upgrade or a major impact?
lost revenue or going out of business?
https
://w
ww
.flic
kr.c
om/p
hoto
s/24
1510
87@
N00
/
What lessons have we learnt?
Sharing data stores sounds like it saves effort but introduces cohesion between applications
Big-bang versioning of applications means putting existing unchanged use cases at risk
Scaling is challenging when you try to duplicate whole systems
Infrastructure – it’s much more important than we initially realized
Part 2 - Build Your Own Platform?
Adrian Cockcroft’s (@adrianco) Thoughts
http://wikibon.com/wp-content/uploads/container_implementations.png
Technology Choices
05/03/2023 @danielbryantuk
What’s Wrong with PaaS?
Core Features
• Continuous deployment
• Health checks
• Logging
• Monitoring
www.opencredo.com/2015/10/31/javaone-building-a-microservice-development-ecosystem-video
05/03/2023 @danielbryantuk
Logging• “The
Log: What every software engineer should know about real-time data's unifying abstraction”
• “10 Tips for Proper Application Logging”
• ElasticSearch-Logstash-Kibana (ELK)– Buffer/proxy log sending or…– Mount directory into container
05/03/2023 @danielbryantuk
Monitoring• Push
– Spring Boot actuator e.g. InfluxDbExporter
• Pull – E.g. Telegraf (TICK), Prometheus
• InfluxDB vs prometheus vs graphite vs opentsdb
• Information radiators - Grafana– Aggregate vs individual
Aggregation: Sick Cattle, Not Sick Pets
05/03/2023 @danielbryantuk
Distributed Tracing
• Correlation: github.com/daniel-bryant-uk/correlation-id-async• MDC logging: logback.qos.ch/manual/mdc.html • OpenZipkin: github.com/openzipkin
Looking Inside the Container
Common Java / Docker Issues• No disk space for docker logging– Increase disk space (move logs to mount)
• Restricting resources to only Xmx memory limit– Set memory limit = Heap (Xmx) + Metaspace + JVM
• Security or crypto issues as /dev/random limited in containers– -Djava.security.egd=file:/dev/urandom
• See Chris Batey’s “The JVM and Docker” talk here at 15:00 today
Debugging Tools
• Java– jstat, jstack, jmap– “5 things you didn’t know”
• OS– Top, htop, ps, free, df –h,
vmstat,iostat – /proc filesystem meminfo and
vmstat not cgroup aware!– Use sysdig
www.joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x
05/03/2023 @danielbryantuk
Problems?
• Rob Ewaschuk’s “Philosophy on Alerting”
• Brendan Gregg’s USE method – “check utilization, saturation, and errors.”
• “DevOps Troubleshooting”– Kyle Rankin
Part 3 – Safety first
https
://w
ww
.flic
kr.c
om/p
hoto
s/m
iriam
delir
ium
/
Tooling – what’s left to do?
Dynamic DevelopmentCapacity
Predefined static VM’s LPARs etc
OpenStack Cloud(s)
Docker Cloud
Infrastructure as CodeChef, Puppet, UCD …
OS
Infra On Prem Data Centres
Cloud ProvidersSoftLayer / Amazon etc
Config Containerized Applications
Continuous AvailabilityMesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev
SaaS
Primary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
✔
✔
✔
✔
✔
Application DIY
Dynamic DevelopmentCapacity
Predefined static VM’s LPARs etc
OpenStack Cloud(s)
Docker Cloud
Infrastructure as CodeChef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc
Config Containerized Applications
Continuous AvailabilityMesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev
SaaS
Primary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static hosts (‘BYOD’)
DIY
Application DIY
Dynamic DevelopmentCapacity
Predefined static VM’s LPARs etc
OpenStack Cloud(s)
Docker Cloud
Infrastructure as CodeChef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc
Config Containerized Applications
Continuous AvailabilityMesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev
SaaS
Primary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static hosts (‘BYOD’)
DIY
Compliance / Security ContainersVM Images
Application DIY
Dynamic DevelopmentCapacity
Predefined static VM’s LPARs etc
OpenStack Cloud(s)
Docker Cloud
Infrastructure as CodeChef, Puppet, UCD …
OS
Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc
Config Containerized Applications
Continuous AvailabilityMesos etc
Deploy
Pipeline
Block Architecture of Hybrid Cloud Dev
SaaS
Primary Audience
GIT / Jenkins / Junit …. Selenium, Jmeter…
Other static hosts (‘BYOD’)
DIY
Compliance / Security ContainersVM Images
Application DIY
You do understand about security and compliance right?
https
://w
ww
.flic
kr.c
om/p
hoto
s/ad
ulau
/
Where’s your data?
Legal restrictions on data locationVary by country even within the EU. Different rules depending on types of data
You already know this?
But now you’re putting the data in the cloud.
DO YOU understand where its goingCan you control / manage / audit the situation?
Now it’s your problem.
Having fun finding tools to help
How’s your security knowledge?• Again – now it’s your problem
– Your code is running in the cloud– You created the services & the containers– Are they secure?– How do you test?– Are you sure?– Are those web services you’re buying secure?
• How much do you know about networking?– Not enough…
• Cyber crime is big business – you will get targeted. https://www.flickr.com/photos/61423903@N06/
“Organized Cybercrime is the most profitable type of crime”
• Cybercrime is estimated to be worth 445 Billion Dollars a Year
• In 2013 the United Nations Office on Drugs and Crime (UNODC) estimated globally the illicit drug trade was worth 435 Billion Dollars
• Guess which one has the least risk to the criminal?• Guess which is growing the fastest?• Guess which one is the hardest to prosecute?
• Guess which one is predicted to reach 2100 Billion Dollars by 2019?
Talk to your Ops team
• They are your best friends.– They know about security and networking. • You need to know too
– They know (some) of the answers• It’s a whole new domain for you
– It’s not a new problem for them• Time to learn
Part 4 - Sharing is Caring
The Results of the Survey Are In…
• Puppet Labs 2015 State of DevOps– Available: puppetlabs.com/2015-devops-report
• Accelerates deployment– High performers 30x more deploys– Code committed to production 200x faster
• Prevents failures and streamlines recovery– High performers 60x fewer failures– Recovery 168x faster
DevOps Topologies (Bad)
• DevOps Anti-Types– Dev and Ops– DevOps Silo– No Ops Needed– Tools Team– Sysadmin– Embedded Ops
http://web.devopstopologies.com/@matthewpskelton
DevOps Topologies (Good)• DevOps Team Topologies
– Dev+Ops– Shared Ops– Ops as IaaS– DevOps-as-a-Service– Teamp DevOps Team– DevOps Evangelists– SRE Team– Container-Driven– DB Capability
http://web.devopstopologies.com/@matthewpskelton
The ’Spine Model’ – The Right Conversations• Effective conversations make for effective
collaboration
• People get stuck in a dilemma where equally plausible options are available
• “Going up the Spine” breaks deadlock
• It’s a TOOL Problem– As a species, we have always been Tool users
and makers. – We use _____ to get our work done http://spinemodel.info/explanation/introduction/
DevOps != Tooling• PRACTICES before Tools
– Decide on the Practices that the tools are there to support– We do _____ to create value
• PRINCIPLES before Practices– Decide on the Principles to measure those Practices against.– We leverage _____ to change the system
• VALUES before Principles– Make as explicit as possible the Values at play in the system.– We optimise for _____
• NEEDS before Values– It all starts at Needs. Why does this system exist in the first
place?– We are here to satisfy _____
http://spinemodel.info/explanation/introduction/
05/03/2023 @danielbryantuk
When Things (Inevitably) Go Wrong
05/03/2023 @danielbryantuk
Optimise for Learning (and Feedback)
www.infoq.com/news/2015/06/too-big-to-fail
Part 5 – Conclusions
Key Messages• Microservices force you to know about distributed systems
– You need to learn how to design and run applications in a new way
• The application/infrastructure ‘platform’ stack is still not fully baked– Essentials are CI/CD, health checks, logging and monitoring
• Think “Safety first”– It’s a wild world out there - security, networking cyber criminals
• Make your ops team your best friend– DevOps is all about the organisation, people and processes
Thanks – Questions?
Daniel Bryant@danielbryantuk
Steve Poole@spoole167
Bonus: Containers Are Not Immutable (By Default)
• Containers can be as susceptible to configuration drift as VMs/bare metal– This surprises many people
• Suggestions– docker run --read-only– docker run --tmpfs /tmp
• Gareth Rushgrove’s CraftConf talk– http://www.ustream.tv/recorded/86186490