Benchmarking – Industry Data and Improvement Tutorial…€¦ · Benchmarking – Industry Data and Improvement Tutorial, OOP, ... Performance Improvement ... Benchmarking Business

V1.1 | 2017-01-30

Christof Ebert, Vector Consulting Services

Benchmarking – Industry Data and ImprovementTutorial, OOP, Munich, 30. Jan. 2017

© 2017. Vector Consulting Services GmbH. All rights reserved. Any distribution or copying is subject to prior written approval by Vector. V1.1 | 2017-01-30

Vector Consulting Services

Welcome

…supports clients worldwide

…improves product development and product strategy, IT and provides interim management

…is as Vector Group globally present with 1700 employees and over 400 Mio. € sales

…offers with the Vector Group a portfolio of tools, software components and services

…is growing and thus continuously hiring

www.vector.com/consultingRailway

IT & Finance

Automotive

Aerospace

DigitalTransformation

Medical

2/47


Christof EBERT

Welcome

[email protected] @christofebertwww.vector.com/consulting

Christof Ebert is managing director at Vector Consulting Services.

He supports clients to improve product strategy and product development and to manage organizational changes.

Prior to that, he held senior management positions for ten years at a global ICT leader, with worldwide responsibility for software technologies.

A trusted advisor for companies around the world, member of industry boards, and professor at the University of Stuttgart and Sorbonne in Paris, Dr. Ebert authored several books.

3/47


Welcome

Performance Improvement

Measurement

Benchmarking

Data Analytics

Case Study: Lean Development

Case Study: Technical Debt

Summary

Agenda

Projects without clear goalswill not achieve their goals clearly.

Tom Gilb's principle of fuzzy targets

5/47


Software and IT Generate Value – But How is it Measured?


A modern car has 50-100 electronic control units

Electrical equipment amounts to 30% of cost

Software delivers 90% of the innovation and value

Increasing complexity and callbacks, liability risks

But not enough measurement to learn and progress

Systematic measurement today is used in all engineering disciplines – except software and IT development

6/47


Software-Driven Value – Innovation and Complexity


“At Apple, we are committed to world-class products. With the launch of our new maps we could not keep this commitment. We will do everything possible to make maps better.”

Tim Cook, CEO

…Complexity

The recall rate of cars is exploding. In 2014 5 times the number of sold cars are recalled in the same timeframe.

50% of the recalls are due to software problems.

Smartphones worldwide annual sales

• 2005: 50 Mio

• 2010: 300 Mio

• 2015: 1,4 Bio.

• 2020: 1,7 Bio.

Unlike the awkward-looking first trials, the incumbent automotive companies, are showing autonomous vehicles that just look like cars.

Innovation…

7/47


Challenge “Complexity Explosion”


High complexity

Long life-time

Mostly maintenance

Source: Vector Benchmarking Suite, Ebert 2016

8/47



Value: Goals

Waste: Fouls

Process: Passes

People: Teamwork

Continuous improvement: Training

Measurements and Benchmarking

Analytics: Goals, speed, corners, …

Improve:Concentration, blood, …


9/47


Measuring the Business Impact: Value and Waste


Department Ping Pong

Added value30%

Overheads40%

Pure waste30%

Source: Vector Benchmarking Suite, Ebert 2016

10/47


Two thirds of performance improvement projects fail

Productivity improvement means a substantial culture change

The change itself is often underestimated and handled ad-hoc

Reasons No management leadership Insufficient communication Lack of internal know-how

Performance Improvement Is Not Easy


Sources: Vector Consulting Services 2016

48%

Internalresistance

Insufficient know-how

Pressurewithoutownership

Other reasons

22%

17%

13%

Performance improvement needs top-down focus and change management.

11/47


Industry Challenges 2017


Innovative Products

Others

Connectivity

Distributed Development

Efficiencyand Cost

Digital Transformation

Governance and Compliance

ComplexityManagement

Securityand Safety

0%

10%

20%

30%

40%

50%

60%

70%

0% 10% 20% 30% 40% 50% 60% 70%

Cost remains biggest short-term challenge across all industries. Connectivity and Digital Transformation evolved as a major challenge.

Mid

-ter

m c

hal

lenges

Short-term challenges

Vector Client Survey 2017. Details: www.vector.com/trends. Horizontal axis shows short-term challenges; vertical axis shows mid-term challenges. Sum > 100% due to 3 answers per question. Strong validity with >4% response rate of 1500 recipients from different industries worldwide.

12/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

You can’t control what you cannot measure.

Tom DeMarco

14/47


Measurement – Motivation

Measurement

What is not measurable, make measurable.

Galileo GalileiA science is as mature as its measurement tools.

Louis Pasteur

If you can‘t measure it, you can‘t manage it.

Peter Drucker

15/47


Bloom (1960-1980) Source code metrics (first publication 1968) Volume, complexity (Halstead, McCabe), Reliability Goal: complexity reduction, release criteria

Boom (1980-2000) Software engineering as a comprehensive discipline Process frameworks like CMMI push for measurement Cost models (Boehm, Putnam) and estimation (FP, COSMIC) Goal orientation (top down definition of metrics) Experiments and validation of data

Doom (2000-2015) Erosion of disciplined software engineering and back to trial and error Agility and time to market more relevant than focused learning Measurement seen as nice to have; enterprise databases disappear

Today… Data analytics and big data suddenly are hype But where are the experts?

A Brief History of Software Measurement

Measurement

16/47


Estimation and Performance Control

Measurement

Relationship of Function Points to duration and effort needs always to be tailored to environmental conditions.

Possible explanations for lower effort compared to predictive model:

Unrealistic estimation

Bad reporting

Better performance

Lower overhead cost

Experience database (same context)Predicted project parametersCurrently planned project parameters

Project effort

Person years

Project size (FPs)

?

17/47


Assess supplier viability, contract reliability, cost and product quality – independent whether it is components or outsourcing

Standard estimation techniques and tools (e.g., COSMIC FP, QSM) for feasibility, health and cost

Risk assessments upfront and during the project

Predictions for progress related to your needs

External audits if you don’t trust process quality

Supplier Management and Project Control

Measurement

Earned Value

100%

Pro-gress

0%

Planned value

Actual cost

0% Calendar time today 100%

Current value: -15% Schedule variance

10% Cost varianceEarned value

Project start Calendar time

100%

Defects

0%

Expected defects Tolerance intervalQuality

Detected defects

Size (FPs)

Effort

50 100 5001

5

10

50 HistoryForecast

Current plan

Feasibility

18/47


Example: Working with Measurements

Measurement

Total cum normalized defects

0

600

S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 D

efects

1 4 7 10 13 16 *Apr‘15

Jul Oct Jan’16

Apr Jul

Total cum effort

0

150

S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9

PM

1 4 7 10 13 16 *Apr'15

Jul Oct Jan’16

Apr Jul

Total defect rate

0

80

S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 D

efects1 4 7 10 13 16 *

Apr'15

Jul Oct Jan’16

Apr Jul

Aggregate staffing rate

01530

S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9 Peop

le

1 4 7 10 13 16 *Apr'15

Jul Oct Jan’16

Apr Jul

Total Cum Normal DefectsTotal Defect RateTotal Cum Effort (PM)Agg. StaffSize (ESLOC(K))PI 12.6 14.6MBI 3.0 3.5

Date 05.09.2016 (17.16 mos)

PlanActual/

ForecastEst. to

Complete613 769 1

1 1123.89 201.93 0.00

0.2137.30 67.30 0.00

Current Plan Actual

Size

04080

S 2 3 4 5 6 7 8 9S 2 4 6 7 8 9

ESLO

C (th

ousan

ds)

1 4 7 10 13 16 *Apr'15

Jul Oct Jan’16

Apr Jul

Interpolated Current Forecast Green Control Bound Yellow Contr. BoundS = Start, 2 = CDR, 3 = FCC, 4 = SIT, 5 = Freeze, 6 = IOC, 7 = Pilot, 8 = 15R, 9 = Release

Actuals, History, Outlookcompared to approved plan

19/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

Poor management can increase software costsmore rapidly than any other factor.

Barry Boehm

21/47


Benchmarking – Motivation

Benchmarking

How much time does the blue car need to its final destination?

It depends on several parameters:

Distance to drive (quantitative: Km, absolute scale)

Average speed (quantitative, ratio scale)

Weather(qualitative: ordinal scale)

Traffic and road(unpredictable: risk to be evaluated)

Car(condition, engine)

Driver(condition, competences)

22/47


Benchmarking is the structured and systematic learning from the best in class.

Benchmarking of IT and Software applies to different dimensions

Products (e.g., quality requirements, product complexity, variance, platforms, technical debt, architecture)

Processes (e.g., roles, methods, tools, value stream, productivity)

Projects (e.g., estimation, supplier management, interfaces)

People (e.g., competences, distributed teams, organization)

What is Benchmarking?

Benchmarking

Prod

uct

ivity

Time

Improvementneeds

SustainableImprovements

Learning, Transitition

23/47


Business Performance and Improvement

Benchmarking

Business Performance

BusinessPain Points

Marketing, Ops, Product Mgmt

(Customer Satis-faction, Prizing, Service, Revenues, Organization, Corporate Culture)

Sales, Marketing, Finance …

IT/Software Impact

Pain Points Performance

Change Program

Continuous Improvement

Vector Benchmarking Suite

Measurements(Gap analysis, Engineering and IT performance, contribution to business, improvement progress)

Industry benchmarks(Product, Defects, Rework, Complexity, Locations, Competences, People management, Processes)

Productivity levers: Value orientation, Waste reduction, People focus

Best practices for change

24/47


Balanced Scorecard

Benchmarking

Financial PerspectiveTargets Performance MeasuresStrength Cash flow

Profit vs. targetsGrowth Sales growthShareholder value Increase of ROCE

Customer PerspectiveTargets Performance MeasuresNew products Share of new products

among all salesFast sales Delivery accuracyPrefered supplier Share of sales to long-

term clients

Innovation and Know how PerspectiveTargets Performance MeasuresTechnologyleadership

Cycle time for new productgeneration

Org. learning inproduction

Lead time until matureproduct

Time to marketaccess

Benchmark of productintroduction with competiton

Internal PerspectiveTargets Performance MeasuresTechnologicalcompetence

Development or productionbenchmark with best of class

Development Development efficiencyProduction know-how

Cycle time, marginal cost,operating profit

25/47


Example: Product Data and Usage

Benchmarking

Always7%

Often13%

Sometimes16%

Rarely19%

Never45%

Source: Ebert 2016; Standish Group Study Reported since 2003 in Chaos Reports.

Goal: Reduce complexity, focus on value Question: How many of delivered features are seen as valuable?Approach: Survey and activation counts with logsAnalytics: Frequency and clusters

26/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

If you cannot measure it, you cannot manage it.

Peter Drucker

28/47


Smart solutionsProducts adapt to the changing environment and use> Connectivity and collaboration

Smart networkingSoftware, mechanics and electronics to create new technologies and business models> Usage data

Smart environmentsCooperating systems to communicate and exchange information> Predictive maintenance

Smart production systemsFlexible production systems enable resource-efficient production and custom products> Flexible integration

Data Analytics – Motivation

Data Analytics


Data analyticsalso: Business Intelligence (BI) or Business Analytics (BA)

Emerging technique that dives into a data set without prior set of hypotheses

The data derive meaningful trends or intriguing findings that were not previously seen or empirically validated.

Data analytics enables quick decisions or help change policies due to trends observed

Approach Accumulation of raw data captured from various sources (i.e. discussion

boards, emails, exam logs, chat logs in e-learning systems) can be used to identify fruitful patterns and relationships

Exploratory visualization – uses exploratory data analytics by capturing relationships that are perhaps unknown or at least less formally formulated

Confirmatory visualization – theory-driven

Data Analytics Approach

Data Analytics


Example: Market Data and Sales

Data Analytics

Usability 0 100%

Performance

Automation

Pricingscheme

Functionality

License model

Service

Installation, administration

Competitors product

New entrant

Goal: Identify New Segments and Buying CriteriaQuestion: Which buying criteria matter for which segments?Approach: Survey and sales figures across similar productsAnalytics: Multivariate analyses

29/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

As far as I can tell “engineering judgement” just means they’re going to make up numbers

Richard Feynman

31/47


Method: Dedicated Techniques from Lean Development


Focus on value

Continuouslyimprove

Empowerpeople

Eliminatewaste

Optimize value streams


Always7%

Often13%

Sometimes16%

Rarely19%

Never45%

Practice: Eliminate Waste


Feature UsageSources:

Vector Benchmarking Suite, Microsoft, Daimler, BITKOM 2012,

Ebert 2016

Our recommendations:

Set a clear goal: Reduce waste

Analyze waste in work flows such as requirements value, requirements changes, duration of open defects, defect detection effectiveness.

Learn from industry best practices and extract concrete prioritized actions

Project: Use quality gates as defined thresholds and to ensure transparent governance.

Product: Reduce technical debt such as insufficient architecture with concrete refactoring actions.

Case Study Focus

33/47


Implement the Change Project


From Classic Development

Client Benefits

25% reduction of lead time and effort before project start

Reduction of hand-offs and ping-pong between project functions

20% reduction of project lead time by concurrent engineering

To Lean Agile

Vector Contribution

Streamlined development processes and interfaces

Value-orientation end-to-end

Product line engineering from requirements onwards

Efficient yet effective safety processes and governance

35/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

As far as I can tell “engineering judgement” just means they’re going to make up numbers

Richard Feynman

36/47


Situation: Market leader in critical infrastructure solutions

Challenge: Complexity, legacy code, volatile requirements

Business needs: Flexible solution and service development, schedule adherence, cycle time reduction

Approach: External benchmarking and internal improvement program to reduce and control technical debt

Goal: Systematically Reduce Technical Debt


1) Make technical debt visible 2) Evaluate trade-offs 3) Control technical debt

Target zone

Effort

Impact

Low

High

High

0%

50%

100%Perfor-mance

Safety

Robust-ness

Port-ability Defects

Maintain-ability

Security

Innovation

Testability

Reliability

02468

1012141618

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Q1-2010

Q2-2010

Q3-2010

Q4-2010

Q1-2011

050

100150200250300350400450

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Q1-2010

Q2-2010

Q3-2010

Q4-2010

Q1-2011

Impact:

0

50

100

150

200

250

300

350

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Q1-2010

Q2-2010

Q3-2010

Q4-2010

Q1-2011

Cost:

0

1

2

3

4

5

6

Q1-2009

Q2-2009

Q3-2009

Q4-2009

Q1-2010

Q2-2010

Q3-2010

Q4-2010

Q1-2011

Deployment:

37/47


Approach: Combination of Different Techniques


Development project

Component design

Component test

Product test

System test

Solu

tions

Develop

men

t

Component development

Commercial start

System approval

End-to-end customer focus: Solutions

Custom

er

SolutionDevelopment

Early defect detection

Incrementaldevelopment

Teamwork and ownership

System specification

38/47


Systematically decreased technical debt – where it mattered most

Efficiency in development and approval, e.g. schedule adherence

Performance Improvement: Control Results


Target zone

Delivery date

Delays

39/47


Welcome


Measurement

Benchmarking

Data Analytics



Summary

Agenda

What gets measured, gets done.

Tom DeMarco

40/47


It is vital in today business climate to understand and manage based on measurements and benchmarks.

Three key principles Systematically measure, analyze and

benchmark Pragmatically evaluate trade-offs and

decide Sustainably control and take action

Don’t exaggerate measurements. It’s one of several tools.Dogmatism will distract potential users.

Measurement: Ambition

Summary

Our own simple law for managing technical debt: RACE

Reduce Accidents and Control Essence

41/47


Analysis:What are the opportunities and threats to my business?Where do I need to be in two years?What do I need to do better to stay competitive?

Assessment:What does that mean to product innovation, development and IT?Which improvements and changes are necessary?

Alignment:Which quantitative targets?Which measurements and analytics?

Your Own Action ListSummary

42/47


Concrete Benchmarks and Measurements

Summary

Software MeasurementEstablish, Extract, Evaluate, Execute

Christof Ebert and Reiner DumkeThird edition, Springer, 2007www.vector.com/books

“Few organizations have really institutionalized measurement of their products and processes. This book is bang up-to-date in both fields and packed with practical advice. For every software engineer."

Charles R. Symons, Inventor of Function Points

44/47


Optimizing Global Projects and Teams

Summary

Global Software and IT

Christof Ebert2. extended Edition, Wiley, 2011 Discount: http://bit.ly/cSjZgD

"This book stands out as the best source of information on distributed software development. Seldom do we see a book with the concepts completely backed by industry experiences and views. Software developers and managers benefit from the broad case studies.“S M Balasubramanian, Vice President, Wipro Technologies

45/47


For more information about Vector Consulting Services please visit

www.vector.com/consulting

Passion. Partner. Value.

Vector Consulting Services

Phone +49 711 80670-0 www.vector.com/consultingFax +49 711 80670-444 [email protected]

Documents

Benchmarking – Industry Data and Improvement Tutorial…€¦ · Benchmarking – Industry Data and Improvement Tutorial, OOP, ... Performance Improvement ... Benchmarking Business