How IBM Scaled DevOps: The IBM Marketplace and Continuous Improvement

Ann Marie Fred

Software Engineering Manager and DevOps CoachIBM Marketplace Engineering

Place

Your

Picture

here

February 15, 2016

2

Objectives

By the end of this session, you should be able to:• Describe characteristics of one exemplary team• Identify some agile best practices that work well for IBM• Understand how we safely deploy changes to production many times

per day• Address challenges with scaling DevOps to a large organization• Jump-start a DevOps transformation and continuously improve your

own organization

3

Agenda

• A bit of history: DevOps early adopters at IBM• IBM Marketplace Engineering culture and heritage• Delivering to production, frequently and safely• How to rebuild an organization• Getting through roadblocks

4

Introduction

Why is DevOps important to IBM?• To compete with cloud-native companies, we need to streamline our

engineering and operations.• To foster highly skilled, engaged teams.• To further a culture of innovation and excellence.

5

A bit of history: DevOps early adopters at IBM

Rational/Tivoli DevOps advanced technology team

2011 2012 2013 2014 2015

First DevOps consulting practices

IBM acquires UrbanCode

DevOps practices within legacy software teams

Service Engage

DevOps enablement training

IBM Design Thinking

Whitewater

IBM Marketplace adopts

squad model

6

IBM Marketplace culture and heritageCharacteristics of IBM:• Hierarchical• Federated• Diverse

7

IBM Marketplace culture and heritageWhere we are:• Cloud

•Digital– Marketplace

– Engineering

8

IBM Marketplace

9

My IBM

10

My Products and Services

11

Heritage• Many thanks to…

12

Heritage• Squad model: inspired by Spotify

•More later…• Just culture: inspired by Etsy

•Blameless post-mortems• Microservices: inspired by Netflix

13

Heritage• Lean and Kanban: inspired by Toyota

•Minimum Viable Product (MVP)•Limit work in progress•Pull model

14

Heritage• Scrum: long history of success within IBM

•Helps coordinate work across squads•Weekly sprints•Rational Team Concert

• Design Thinking: IBM-wide initiative•Design Thinking: Build the right thing•DevOps: Build the thing right

15

What is the scale?• IBM Digital (parent) is roughly 500 people, approximately 80 squads• IBM Marketplace Engineering is roughly 100 people, approximately

20 squads

• DevOps adoption & “Squad-ification” varies, even within Marketplace Engineering

16

Making a huge company feel small

Our version of the squad model:• Small and focused• Co-located• Independent• Accountable• Autonomous• Self-managing

17

18

What makes us tick?

Squad Roles• Executive coach• Full-stack developers: develop, test, deploy, operate, support• Tech lead and other committers• Dedicated designers (visual, ux, industrial)• Squad lead / product manager / product owner• Project manager• HR manager / release manager / engineering discipline• Scrum master: rotates

19

The right tools for the job

Experiment, within limits• Free tools and frameworks, no hosting cost• CIO’s office, Whitewater, DevOps Enablement teams• Purchased SaaS tools• Tools hosted by other ops teams• Purchased on-premise tools• Host-your-own

EASY

HARD

20

21

Delivering to production, frequently and safely

22

Characteristics of Continuous Delivery• Tiny, frequent changes• Carefully tested and reviewed• Delivered to production, automatically• During normal business hours• Zero downtime

23

Continuous Delivery reduces risk

• Quick fixes• Quick feedback• Gradual improvement

24

Continuous Delivery without fearThese things work together:• Architecture• Development and Test practices• Deployments• Operations

25

Continuous Delivery: Architecture• Resilient microservices

•Small, easy to maintain•Multiple instances, multiple locations•Fail-over•Graceful degradation

• Bluemix PaaS•We handle the application/service, they handle the infrastructure/operating system

26

Continuous Delivery: Development and Test• Stringent code reviews• 100% unit test code coverage• Automated integration tests• Automated system tests in production• Automated security tests: Rational AppScan

27

Continuous Delivery: Deployments• Automated deployments• Blue/green deployments

28

Continuous Delivery: Operations• Streamlined release management

•Compliance•HA/DR plans•Operations and support plans•At launch, then update periodically

• Monitoring• Alerting• Logging• Feedback (see next slide)

29

Continuous Delivery: Feedback• A/B Testing• Monitoring feedback

30

How to build a DevOps organization

This is awesome, but…• Where did we start?• How did we scale up?

31

Pioneer teams

32

Pioneer teams

• Start from scratch• Strong leaders• Learn, try, adapt, reject, repeat• No cynicism or defeatism• Do the right thing, ask forgiveness later

33

Viral effect• From one squad, one project, to many• Simplify access to best tools• Leaders and SMEs teach others• External community support• Formal training• Cross pollination• Adopt and adapt

34

Learning from failures“Anyone who has never made a mistake has never tried anything new.” – Albert Einstein

Continuous improvement:• Blameless post-mortems• Squad health surveys• Engineering Summit• Retrospectives

35

Getting through roadblocks

Case studies•Too much red tape•This is taking too long!• I want more plans!•We’re fine, thanks.•Two-speed IT

36

Questions?

37

Review of Objectives

Now you can:• Describe characteristics of one exemplary team• Identify some agile best practices that work well for IBM• Understand how we safely deploy changes to production many

times per day• Address challenges with scaling DevOps to a large organization• Jump-start a DevOps transformation and continuously improve

your own organization

38

Contact Info

• amfred@us.ibm.com• @DukeAMO

Thank You!