Lean Agile Transformation - Lean Business Ireland · Openet software enables new business models that innovate how people, machines, and services interact with the world’s largest

Lean Agile Transformation 29th May 2013 Shane O’Flynn – Global Vice President of Engineering

Enterprise Ireland

Driving Competitiveness Dublin Castle

2 © Copyright 2013 Openet – Company Confidential For Use Under Non-Disclosure Only

Who are Openet?

Openet software enables new business models that innovate

how people, machines, and services interact with the world’s

largest networks

• Network Service Provider Focus

• Best in Class Policy and Charging Control

• 80 customers, 28 countries

• Over 10 Billion events per day

• 900+ employees, Offices in Ireland (HQ), US, Brazil, & Malaysia

• Founded 1999

Openet – #1 in Policy Management

2011 & 2012

Selected Customers:


The Engineering Challenge

• 4 Product Lines • 11 Sub-Categories • ~2.5 Million Lines of Code

• ~300 Product Engineers

• 4 Development Centre’s • 2 Outsourced Partners • Work from Home policy in place • Engineering work done in 11 different time zones


The Waterfall Model of Software Development

• Requirements • Design

• Implement • Test

• Maintain

• Doesn’t exist!

• V1.29 Requirements document – that’d be 29 iterations!

• We’ve all been agile for years, we just didn’t know it!


• New Product Add-On developed using Waterfall-ish method

• Kicked Off in March 2011

• Baseline target completion date was October 2011

• Project ran until April 2012 – 100% over budget & 6 months late.

• Main Issues: • High dependency on manual testing

• Lack of automated testing, functional & non-functional

• Attempted to do Agile but requirements written as User Stories with no Acceptance Criteria, and executed in Waterfall framework

• Legacy code issues – technical debt / refactoring was complex & difficult to test. • High risk

• Integration testing executed in the final Phase. • Performance Testing executed in final Phase.

• New Product, new domain & new teams.

Our “Car Crash” Waterfall Project…


The road less travelled… the path to Lean Engineering…

• Product Management & Customer Demand is accelerating

• Product lines are increasing

• Waterfall wasn’t working (for us!)

• Mantra - ‘Add more Engineers’

• Was there another way?


Noble Goals • Get re-Organised

• Use Lean Methods

•Product Quality & Delivery efficiency • Increase backlog agility – JIT decision making.

• Shorten our release cycle times.

• Learn Fast! bka Fail Fast!


• Lean development can be summarized by seven principles, very close in concept to lean manufacturing principles: • Eliminate waste • Amplify learning • Decide as late as possible • Deliver as fast as possible • Empower the team • Build integrity in

• Kanban is a Lean tool.

• Combine Scrum and Kanban ScrumBan

Lean Software Development


Kanban Process / Lean Software • Do not send defective

products to the subsequent process

• The subsequent process comes to withdraw only what is needed

• Produce only the exact quantity withdrawn by the subsequent process

• Level the production

• Kanban is a means to fine tuning

• Stabilize and rationalize the process


Lean Product Release Cycle


3 Major Changes for Openet

• Feature Teams

• ScrumBan & “Done”

• Time boxed Sprints


Feature Teams • Self organising atomic units

• Work assigned to team, not people

• Long lived (12-18 months)

• Generally • 1 Feature Lead to run the team • ~3 developers • ~2 QA (including Performance expertise) • Co-located

• Generalising specialists

• Assigned true specialists on an ad-hoc basis


Scaling Feature teams

Architect(s), Tech Writer(s), Trainer(s), Tech Mentor(s), etc.

PMO

VP, Delivery


PMO

FT

FT FT

Strategic Program #1

FT FT

FT


PMO

FT

FT FT

Strategic Program #2

FT FT

FT

Architect(s), Tech Writer(s), Trainer(s), Tech

Mentor(s)

PMO

FT

FT FT

Strategic Program #n

FT FT

FT FT

FT FT

FT

FT FT

Tactical Program

Trainer(s)

Training Delivery Program


Definition of “Done” – Engineering Release


Using ScrumBan to get to “Done”


Definition of “Done Done” – General Availability


Automated Testing & Test Driven Development

• Automated Test Systems • Functional • System • E2E • Performance • Close-out Sprint

• 120 days down to ~60 days • Target is 20 days (7 elapsed)

• Continuous Build

• Continuous Integration System • Linked to Automated Test Systems • Build cycle ~6 hours (from >24) • Code breaks immediately visible

• Minutes or hours not weeks • Fully virtualized environments


• Absolutely!

Openet Interaction Gateway • Pilot Project

• Delivered on-time, within budget constraints • Engineering Build was actually 2 weeks early!

• 3 “Post-ship” defects discovered • 2 already fixed & patched prior to being reported

• 3 Major changes of direction during lifecycle • Target first customer changed over course of development project • Initial Use Case changed significantly • No impact to final timelines

Success?


Challenges • Coordination & Roll out

• Education & mind-set changes • Resistance to Change • Communications

• User Stories & Acceptance Criteria • Breaking things down to small sizes • Estimation

• Distributed Teams harder

• Project Management Metrics


• Phase 1 Metrics: • Velocity: How many Story Points are delivered per Sprint, per Team & overall, per

Program. Used to calculate future commitments to customers. • Commitment Diligence: Measure of delivered work versus how much work the

team said they would deliver for each Sprint. • > 100% means they’ve under committed originally but taken on more. • < 100% means they’ve over committed originally and didn’t complete everything. • 100% prediction accuracy. • Target is between 80% - 120%.

• Phase 2 Metrics, (to be implemented): • Focus Factor – gives us a measure of disruption to the team. • Adopted Work – additional work taken on during the Sprint from the Backlog. • Found Work – additional work completed during the Sprint for items on the Sprint

Backlog. • Accuracy of Estimation – margin of error on Story Points. Original versus Actual.

Metrics… with Agile


CM Program Metrics

84% 72%

55%

75% 80%

123%

0%

20%

40%

60%

80%

100%

120%

140%

1 2 3 4 5 6Months

CM Commitment Diligence

14 13.5 20

35.5

27

54

0

10

20

30

40

50

60

1 2 3 4 5 6Months

CM Velocity

• Increasing Velocity & Increasing Commitment Diligence showing a very positive trend


IG Program Metrics

90%

114% 119%

91%

150%

0%20%40%60%80%

100%120%140%160%

1 2 3 4 5Months

IG Commitment Diligence

13

24

19 17

13

0

5

10

15

20

25

1 2 3 4 5Months

IG Velocity

2.7 3.3

4.0 3.7

4.6

0

1

2

3

4

5

1 2 3 4 5

Avg.

Vel

ocity

pe

r per

son

Months

IG Velocity - Average Per Person

- Over the last 3 months, IG team sizes have decreased due to natural attrition – hence decreased velocity. Looking at Velocity per person, it is on the rise.


FW Program Metrics

19

38 38 41

05

1015202530354045

1 2 3 4Months

FW Velocity

80% 69%

81%

101%

0%

20%

40%

60%

80%

100%

120%

1 2 3 4Months

FW Commitment Diligence

• Again Increasing Velocity & Increasing Commitment Diligence showing a very positive trend


• Prior to Lean implementation, we didn’t measure well.

• Hard to compare our improvements to our Waterfall world.

• Based on subjective opinion, what we’re doing now is better.

• Entirely new way for Teams to work & deliver working software.

• We now have a measureable “machine”.

• Metrics will now be the basis of decisions.

• Metrics will now back up subjective opinions.

• Metrics will now support the efficient, lean growth of Engineering within Openet

Phase 1 Rollout Complete ~12 months


• Expand & develop our metrics.

• Continuously measure.

• Continuously review.

• Continuously refine.

• Continuously improve.

• Further coaching & training for teams in specific areas.

Phase 2 – Lean Transformation


Immediate Benefits

(Built in)

(Measured)


Questions?

Documents

Lean Agile Transformation - Lean Business Ireland · Openet software enables new business models that innovate how people, machines, and services interact with the world’s largest