DevoxxUK 2016: "DevOps: Microservices, containers, platforms, tooling... Oh yeah, and...

DevOps: Microservices, containers, platforms, tooling... Oh yeah, and people

Daniel Bryant @danielbryantuk

Steve Poole@spoole167

On the Previous Episode of Devoxx UK…

• 2014 “Moving to DevOps: Easy, Hard or Just Plain Terrifying”

• DevOps is about extending agility across your IT org

• Breaking down the silos is vital

• The business needs to react to the industry change– DevOps, cloud and containers

Today

• Microservices are (operationally/conceptually) distributed systems

• The application/infrastructure ‘platform’ is still not fully baked

• Think “Safety first”– Security, networking cyber criminals

• DevOps is (still) all about the organisation, people and processes

I (we) am the one who knocks…Steve Poole

IBM Developer

@spoole167

Daniel Bryant

Chief (Mad) Scientist, OpenCredo

@danielbryantuk

Making Java Real Since Version 0.9

Open Source Advocate

DevOps Practitioner (whatever that means!)

Driving Change

“Biz-dev-QA-ops”

Leading change in organisations

Experience of Docker, k8s, Go, Java

InfoQ, DZone, Voxxed contributor

Part 1- Painful Lessons…

https

://w

ww

.flic

kr.c

om/p

hoto

s/sa

rahm

stew

art/

All I hear is microservices…

“In computing, microservices is a software

architecture style in which complex applications are composed of small, independent processes communicating with each other using language-agnostic APIs. These

services are small, highly decoupled and focus on doing a small task,

facilitating a modular approach to system-building.”

https://en.wikipedia.org/wiki/Microservices

Microservices

Turn applications into small, independent, highly decoupled, modular services

https://www.flickr.com/photos/daikrieg/

You want to make my life more complicated?

https

://w

ww

.flic

kr.c

om/p

hoto

s/ta

hini

/

Where’s the problem?

browserApp A V1.0

Database

browser

You

Your Customers

server

Data Centre

browserLoad balancer

App A V2.0

App AV2.0

App BV1.0

Database

browserDatabase

Ops

browserLoad balancer

App A V2.0

App AV2.0

App BV1.0

Database

browserDatabase

browser

App A V2.0

App AV2.0

App BV1.0

browser

browser

Load balancer

App A V2.0

App AV2.0

App BV1.0

Database

Database

browser App A V2.0

App AV2.0

App BV1.0

browser

browser

browser

browser

browser

Load balancer

Database

browser

Load balancer

App A V2.1

App AV2.1

App BV1.0

Database

Database

browser App A V2.1

App AV2.1

App BV1.0

browser

browser

browser

browser

browser

Load balancer

Database

A simple upgrade or a major impact?

lost revenue or going out of business?

https

://w

ww

.flic

kr.c

om/p

hoto

s/24

1510

87@

N00

/

What lessons have we learnt?

Sharing data stores sounds like it saves effort but introduces cohesion between applications

Big-bang versioning of applications means putting existing unchanged use cases at risk

Scaling is challenging when you try to duplicate whole systems

Infrastructure – it’s much more important than we initially realized

Part 2 - Build Your Own Platform?

Adrian Cockcroft’s (@adrianco) Thoughts

http://wikibon.com/wp-content/uploads/container_implementations.png

Technology Choices

05/03/2023 @danielbryantuk

What’s Wrong with PaaS?

Core Features

• Continuous deployment

• Health checks

• Logging

• Monitoring

www.opencredo.com/2015/10/31/javaone-building-a-microservice-development-ecosystem-video

05/03/2023 @danielbryantuk

Logging• “The

Log: What every software engineer should know about real-time data's unifying abstraction”

• “10 Tips for Proper Application Logging”

• ElasticSearch-Logstash-Kibana (ELK)– Buffer/proxy log sending or…– Mount directory into container

05/03/2023 @danielbryantuk

Monitoring• Push

– Spring Boot actuator e.g. InfluxDbExporter

• Pull – E.g. Telegraf (TICK), Prometheus

• InfluxDB vs prometheus vs graphite vs opentsdb

• Information radiators - Grafana– Aggregate vs individual

Aggregation: Sick Cattle, Not Sick Pets

05/03/2023 @danielbryantuk

Distributed Tracing

• Correlation: github.com/daniel-bryant-uk/correlation-id-async• MDC logging: logback.qos.ch/manual/mdc.html • OpenZipkin: github.com/openzipkin

Looking Inside the Container

Common Java / Docker Issues• No disk space for docker logging– Increase disk space (move logs to mount)

• Restricting resources to only Xmx memory limit– Set memory limit = Heap (Xmx) + Metaspace + JVM

• Security or crypto issues as /dev/random limited in containers– -Djava.security.egd=file:/dev/urandom

• See Chris Batey’s “The JVM and Docker” talk here at 15:00 today

Debugging Tools

• Java– jstat, jstack, jmap– “5 things you didn’t know”

• OS– Top, htop, ps, free, df –h,

vmstat,iostat – /proc filesystem meminfo and

vmstat not cgroup aware!– Use sysdig

www.joyent.com/blog/linux-performance-analysis-and-tools-brendan-gregg-s-talk-at-scale-11x

05/03/2023 @danielbryantuk

Problems?

• Rob Ewaschuk’s “Philosophy on Alerting”

• Brendan Gregg’s USE method – “check utilization, saturation, and errors.”

• “DevOps Troubleshooting”– Kyle Rankin

Part 3 – Safety first

https

://w

ww

.flic

kr.c

om/p

hoto

s/m

iriam

delir

ium

/

Tooling – what’s left to do?

Dynamic DevelopmentCapacity

Predefined static VM’s LPARs etc

OpenStack Cloud(s)

Docker Cloud

Infrastructure as CodeChef, Puppet, UCD …

OS

Infra On Prem Data Centres

Cloud ProvidersSoftLayer / Amazon etc

Config Containerized Applications

Continuous AvailabilityMesos etc

Deploy

Pipeline

Block Architecture of Hybrid Cloud Dev

SaaS

Primary Audience

GIT / Jenkins / Junit …. Selenium, Jmeter…

✔

✔

✔

✔

✔

Application DIY

Dynamic DevelopmentCapacity

Predefined static VM’s LPARs etc

OpenStack Cloud(s)

Docker Cloud

Infrastructure as CodeChef, Puppet, UCD …

OS

Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc

Config Containerized Applications

Continuous AvailabilityMesos etc

Deploy

Pipeline

Block Architecture of Hybrid Cloud Dev

SaaS

Primary Audience

GIT / Jenkins / Junit …. Selenium, Jmeter…

Other static hosts (‘BYOD’)

DIY

Application DIY

Dynamic DevelopmentCapacity

Predefined static VM’s LPARs etc

OpenStack Cloud(s)

Docker Cloud

Infrastructure as CodeChef, Puppet, UCD …

OS

Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc

Config Containerized Applications

Continuous AvailabilityMesos etc

Deploy

Pipeline

Block Architecture of Hybrid Cloud Dev

SaaS

Primary Audience

GIT / Jenkins / Junit …. Selenium, Jmeter…

Other static hosts (‘BYOD’)

DIY

Compliance / Security ContainersVM Images

Application DIY

Dynamic DevelopmentCapacity

Predefined static VM’s LPARs etc

OpenStack Cloud(s)

Docker Cloud

Infrastructure as CodeChef, Puppet, UCD …

OS

Infra On Prem Data Centres Cloud ProvidersSoftLayer / Amazon etc

Config Containerized Applications

Continuous AvailabilityMesos etc

Deploy

Pipeline

Block Architecture of Hybrid Cloud Dev

SaaS

Primary Audience

GIT / Jenkins / Junit …. Selenium, Jmeter…

Other static hosts (‘BYOD’)

DIY

Compliance / Security ContainersVM Images

Application DIY

You do understand about security and compliance right?

https

://w

ww

.flic

kr.c

om/p

hoto

s/ad

ulau

/

Where’s your data?

Legal restrictions on data locationVary by country even within the EU. Different rules depending on types of data

You already know this?

But now you’re putting the data in the cloud.

DO YOU understand where its goingCan you control / manage / audit the situation?

Now it’s your problem.

Having fun finding tools to help

How’s your security knowledge?• Again – now it’s your problem

– Your code is running in the cloud– You created the services & the containers– Are they secure?– How do you test?– Are you sure?– Are those web services you’re buying secure?

• How much do you know about networking?– Not enough…

• Cyber crime is big business – you will get targeted. https://www.flickr.com/photos/61423903@N06/

“Organized Cybercrime is the most profitable type of crime”

• Cybercrime is estimated to be worth 445 Billion Dollars a Year

• In 2013 the United Nations Office on Drugs and Crime (UNODC) estimated globally the illicit drug trade was worth 435 Billion Dollars

• Guess which one has the least risk to the criminal?• Guess which is growing the fastest?• Guess which one is the hardest to prosecute?

• Guess which one is predicted to reach 2100 Billion Dollars by 2019?

Talk to your Ops team

• They are your best friends.– They know about security and networking. • You need to know too

– They know (some) of the answers• It’s a whole new domain for you

– It’s not a new problem for them• Time to learn

Part 4 - Sharing is Caring

The Results of the Survey Are In…

• Puppet Labs 2015 State of DevOps– Available: puppetlabs.com/2015-devops-report

• Accelerates deployment– High performers 30x more deploys– Code committed to production 200x faster

• Prevents failures and streamlines recovery– High performers 60x fewer failures– Recovery 168x faster

DevOps Topologies (Bad)

• DevOps Anti-Types– Dev and Ops– DevOps Silo– No Ops Needed– Tools Team– Sysadmin– Embedded Ops

http://web.devopstopologies.com/@matthewpskelton

DevOps Topologies (Good)• DevOps Team Topologies

– Dev+Ops– Shared Ops– Ops as IaaS– DevOps-as-a-Service– Teamp DevOps Team– DevOps Evangelists– SRE Team– Container-Driven– DB Capability

http://web.devopstopologies.com/@matthewpskelton

The ’Spine Model’ – The Right Conversations• Effective conversations make for effective

collaboration

• People get stuck in a dilemma where equally plausible options are available

• “Going up the Spine” breaks deadlock

• It’s a TOOL Problem– As a species, we have always been Tool users

and makers. – We use _____ to get our work done http://spinemodel.info/explanation/introduction/

DevOps != Tooling• PRACTICES before Tools

– Decide on the Practices that the tools are there to support– We do _____ to create value

• PRINCIPLES before Practices– Decide on the Principles to measure those Practices against.– We leverage _____ to change the system

• VALUES before Principles– Make as explicit as possible the Values at play in the system.– We optimise for _____

• NEEDS before Values– It all starts at Needs. Why does this system exist in the first

place?– We are here to satisfy _____

http://spinemodel.info/explanation/introduction/

05/03/2023 @danielbryantuk

When Things (Inevitably) Go Wrong

05/03/2023 @danielbryantuk

Optimise for Learning (and Feedback)

www.infoq.com/news/2015/06/too-big-to-fail

Part 5 – Conclusions

Key Messages• Microservices force you to know about distributed systems

– You need to learn how to design and run applications in a new way

• The application/infrastructure ‘platform’ stack is still not fully baked– Essentials are CI/CD, health checks, logging and monitoring

• Think “Safety first”– It’s a wild world out there - security, networking cyber criminals

• Make your ops team your best friend– DevOps is all about the organisation, people and processes

Thanks – Questions?

Daniel Bryant@danielbryantuk

Steve Poole@spoole167

Bonus: Containers Are Not Immutable (By Default)

• Containers can be as susceptible to configuration drift as VMs/bare metal– This surprises many people

• Suggestions– docker run --read-only– docker run --tmpfs /tmp

• Gareth Rushgrove’s CraftConf talk– http://www.ustream.tv/recorded/86186490