View
217
Download
3
Category
Preview:
Citation preview
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 1
Edwin DandoAssurity, New Zealand
Advancing Expertise in Software Testing
Commercial in Confidence | Assurity Consulting Limited 2013
Software project failureWhat’s killing us and what we can do about it
Edwin DandoConsulting Manager
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 2
Commercial in Confidence | Assurity Consulting Limited 2013
Why study software project failure?
• We all know software projects are often challenged right ?
• But just how challenged are they?• And are NZ projects any more challenged than others?
• And if so why?
Curiosity got the better of me….
Commercial in Confidence | Assurity Consulting Limited 2013
Aim of the study
• Provide up‐to‐date info about failure patterns in software development projects in NZ
• Ali Baigy – my Persian Otago MBA friend• 16 weeks • 28 organizations including
> 18 software companies> 3 universities
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 3
Commercial in Confidence | Assurity Consulting Limited 2013
Why bother?
1. Knowledge of software project failure in NZ is limited, out‐of‐date, incomplete, or irrelevant
2. Our industry is in a crisis• People believe they can demand it therefore it can be done • Under pressure, developers cut quality to support this belief• The results include
˗ Developers and customers resenting the profession˗ Failing products, failing companies, and hateful
work
We are there now
Commercial in Confidence | Assurity Consulting Limited 2013
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 4
Commercial in Confidence | Assurity Consulting Limited 2013
Commercial in Confidence | Assurity Consulting Limited 2013
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 5
Commercial in Confidence | Assurity Consulting Limited 2013
Approach
• Review international studies> What are the common project failure modes from overseas?> Guide questions to see if we are the same
• Approach people> Call first to consider a recent project in their org> Either send questionnaire or do over phone
Commercial in Confidence | Assurity Consulting Limited 2013
The Participants
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 6
Commercial in Confidence | Assurity Consulting Limited 2013
Main Location
0
2
4
6
8
10
12
Auckland Wellington Christchurch Dunedin
Commercial in Confidence | Assurity Consulting Limited 2013
Industries/markets
Banks, insurance and financial services
13%
Government11%
Education14%
Energy7%Industry
17%
Product software companies
5%
Professional services (Accounting,
Warehousing, Health care, etc)
15%
Retail8%
Telecommunications8%
Other2%
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 7
Commercial in Confidence | Assurity Consulting Limited 2013
Number of FTEs
0
1
2
3
4
5
6
7
Less than 6 6 to 20 21 to 100 101 to 500 More than 500
Commercial in Confidence | Assurity Consulting Limited 2013
Revenue
0 1 2 3 4 5 6 7 8
No answer
Less than NZ$ 500K
NZ$ 500K to NZ$ 2 million
NZ$ 2 million to NZ$ 10 million
NZ$ 10 million to NZ$ 100 million
More than NZ$ 100 million
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 8
Commercial in Confidence | Assurity Consulting Limited 2013
Commercial in Confidence | Assurity Consulting Limited 2013
Success rates
0%
10%
20%
30%
40%
50%
60%
Failure In Between Success
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 9
Commercial in Confidence | Assurity Consulting Limited 2013
Types of Development
0%
10%
20%
30%
40%
50%
60%
70%
80%
Custom Software Development General Software Development A mixture of both
Failure In Between Success
The majority of the Custom Software Development (CSD) projects failed
Commercial in Confidence | Assurity Consulting Limited 2013
Failure in their words…
• “A failure because although the delivery was on time, and to budget, the business did not take it up and failed as a consequence.”
• “It earned our company a lot of money, but disrupted our staffing, was unpleasant to work on, was not fully used by the client and was largely irrelevant to the company as a whole.”
• “From a company perspective it was a success; it has generated sales and paved the way for us to enter new markets. From a development standpoint it was a failure; it was significantly late and of lower quality than we expect.”
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 10
Commercial in Confidence | Assurity Consulting Limited 2013
What is success?
• From an IT provider:
“Yes, [the project was a success] because we carried out the clients specs”
• but when we dug deeper:
“The project sponsor didn't like the end result.”
Commercial in Confidence | Assurity Consulting Limited 2013
But this is often normal, right?
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 11
Commercial in Confidence | Assurity Consulting Limited 2013
We carried out the clients specs
But hold on, we already know that• 35% of requirements change throughout project• 33-66% of requirements are seldom or never used
The client doesn’t know what they want.Isn't just following their specs a bit irresponsible?
Commercial in Confidence | Assurity Consulting Limited 2013
So what are people losing sleep over?
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 12
Commercial in Confidence | Assurity Consulting Limited 2013
Major concerns
0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%
Consultant
Developer Team Leader
Development Manager
GM Operations
IS Manager
Managing Director
Project manager
Quality Manager
Support Manager
Cost Quality Scope Time
Senior managers most worried about cost
Support and QA managers most worried about quality & scope
For Consultant time &cost most important
For Dev Lead time much more important than qualityNo surprises here?
Commercial in Confidence | Assurity Consulting Limited 2013
Challenges
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
Failure
In Between
Success
On failed projects, People, Communication and Scope = biggest challenges
On successful projects, Time, Cost, Estimation & Scope = biggest challenges
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 13
Commercial in Confidence | Assurity Consulting Limited 2013
Business case?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
Failure In Between Success
No
Yes
• Failed projects 27%• Successful 57%• In between 67%
Commercial in Confidence | Assurity Consulting Limited 2013
Could the team specify software purpose & use?
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
No Partly Yes No answer
Failure
In Between
Success
• 100% on successful projects• 50% on failed projects
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 14
Commercial in Confidence | Assurity Consulting Limited 2013
Amount of unmanaged scope change
0
1
2
3
4
5
6
7
8
Failure In Between Success
Unmanaged scope change contributes significantly to failure
Num
ber o
f unm
anag
ed s
cope
cha
nges
Commercial in Confidence | Assurity Consulting Limited 2013
Over‐runs
0%
20%
40%
60%
80%
100%
120%
Failure In Between Success
Over cost Over schedule
Failed projects cost twice as much & take 110% longer than planned
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 15
Commercial in Confidence | Assurity Consulting Limited 2013
Planning
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Failure
In Between
Success
• Importance of planning in all categories was moderate to high • A lack of planning was not a failure factor for the projects in this study
Commercial in Confidence | Assurity Consulting Limited 2013
Importance of Leadership
• “There was uncertainty over who was responsible for what, largely due to our manager not being able to give the project as much attention as required.”
• “Leadership was part of the problem. A sponsor with unrealistic expectations and a PM managing a number of projects and really just administrating.”
• “Inexperienced project management and lack of a true project sponsor did not allow the outputs of the project to be worked through with the end users.”
• “The leadership is strong ‐ but allows the individuals to contribute and challenge the leadership and how the project is run. Discussion is encouraged.”
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 16
Commercial in Confidence | Assurity Consulting Limited 2013
Importance of management support
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Failure
In Between
Success
Extremely Low Low Neutral High Extremely high
Increased management support tends to result in higher chance of success
Commercial in Confidence | Assurity Consulting Limited 2013
Development team expertise
0 1 2 3 4 5
Failure
In Between
SuccessAbility to carry out tasks effectively
Ability to understand humanimplications of a new system
Ability to work effectively as a team
Ability to work with top management
Ability to work with uncertainobjectives
Extremely Low Low Neutral High Extremely high
More experienced team tends to equal higher chance of success
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 17
Commercial in Confidence | Assurity Consulting Limited 2013
Learning
0%
10%
20%
30%
40%
50%
60%
70%
No, there was no process Yes, at the end of the project Yes, regularly
Failure
In Between
Success
• < 15% failed projects had regular learning process throughout the project. • Successful projects tended to have a learning process• Lack of a learning process can be considered a contributing factor.
Commercial in Confidence | Assurity Consulting Limited 2013
11 Key Findings
1. Software projects fail in all types of organisations, regardless of size, income, location, number of projects per annum
2. 54% of projects in the study were failures and failed projects cost twice as much & take 110% longer than planned
3. Custom software development is vastly (10 times) more prone to failure than integrating existing software
4. Concern areas> Senior managers – cost> Quality and Support managers – scope and quality> Development Managers – time> Consultants – time and cost
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 18
Commercial in Confidence | Assurity Consulting Limited 2013
11 Key Findings
5. Biggest challenges > Failed projects ‐ People, Communication and Scope > Successful projects ‐ Time, Cost, Estimation and Scope
6. Most failed projects don’t have a business case whereas most successful ones do
7. Scope clarity> On 100% of successful projects, the development team could
specify the purpose & usage of the software > In less than 50% of failed projects the development team could
specify the purpose & usage of the software
Commercial in Confidence | Assurity Consulting Limited 2013
11 Key Findings
8. On failed projects scope changed significantly and not managing this scope change contributed significantly to failure
9. We don't tend to have good processes to learn from our mistakes
10.More planning doesn’t make any difference11. Leadership is critical
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 19
Commercial in Confidence | Assurity Consulting Limited 2013
Commercial in Confidence | Assurity Consulting Limited 2013
The 5 Core Characteristics
1. Majority (54%) fail, cost twice as much, take 110% longer and don’t tend to have a business case
2. Biggest challenges are People, Communication and Scope3. Scope
> Dev team couldn’t specify purpose & usage of the software > Scope changes significantly> Poor processes for managing scope change> Planning more doesn’t help
4. Custom dev = is 10 x more prone to failure than integrating5. Lack leadership and management buy in
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 20
Commercial in Confidence | Assurity Consulting Limited 2013
Lets go deeper on that…
Commercial in Confidence | Assurity Consulting Limited 2013
Sanity Check…
• Cross reference against fresh NZ study> Software Development Practices in New Zealand, The University of Auckland Computer Science Department, Feb 2012 ‐ Diana Kirk and Ewan Tempero
• And their findings?
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 21
Commercial in Confidence | Assurity Consulting Limited 2013
She’ll be right Trev
• “Issues with practices relating to quality”> ineffective or missing design‐and‐code‐checking
practices (e.g. reviews, unit tests)> a lack of independent testing> insufficient separation of environments> major issues with clarity and accessibility of
requirements.
“Represents a possible gap endemic in the NZ approach to developing software”
Commercial in Confidence | Assurity Consulting Limited 2013
Process
• “Organisations do not follow standard models such as Waterfall, Scrum or XP, but rather adapt practices”
• “A majority of respondents claimed to be ‘agile’, but close informal contact with customers was not practiced”
“Individuals do not follow practices in a consistent way. The risk of gaps is clearly large in the New
Zealand context.”
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 22
Commercial in Confidence | Assurity Consulting Limited 2013
So lets make that 7 Core Characteristics…
1. Majority (54%) fail, cost twice as much, take 110% longer and don’t tend to have a business case
2. Biggest challenges are People, Communication and Scope3. Scope
> Dev team couldn’t specify purpose & usage of the software > Scope changes significantly> Poor processes for managing scope change> Planning more doesn’t make any difference
4. Custom dev = is 10 x more prone to failure than integrating5. Lack leadership and management buy in6. Issues with practices relating to quality7. Organisations do not follow standard models but rather adapt practices
Commercial in Confidence | Assurity Consulting Limited 2013
Back to the investigation…
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 23
Commercial in Confidence | Assurity Consulting Limited 2013
#1 ‐Majority (54%) fail, cost twice as much, take 110% longer & don’t have a business case
• Consistent with Product Ownership “Achilles heel”> Why are we doing the project?> What value does each feature add?> Should we continue this project?> How can we trim the tail to get an early working version into customer hands?
• Consistent with my own observations
• One solution – quality Product Ownership> Work with customers + people developing software> Work based on value > Validate assumptions and reduce risk via regular feedback cycles
Commercial in Confidence | Assurity Consulting Limited 2013
# 2 People, Communications & Scope
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 24
Commercial in Confidence | Assurity Consulting Limited 2013
# 2 People, Communication and Scope
• But done properly agile nails people, communication and scope!
• Oh, that’s right: “organisations do not follow standard models but rather adapt practices”
• Epic Fail
Commercial in Confidence | Assurity Consulting Limited 2013
# 3 ‐ Scope
• Agile gives you> Ability to change scope frequently> Stronger people focus through self‐
management and empowerment> Increased communications
All of this for free!!!
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 25
Commercial in Confidence | Assurity Consulting Limited 2013
The Sprint Agreement
Commercial in Confidence | Assurity Consulting Limited 2013
# 4 custom dev = is 10 x more prone to failure than integrating
• But this is where the magic happens!• Innovative products aren’t the result of integration
• Did Steve Jobs integrate existing products? • Did Yikebike integrate existing products?• Did Lars & Jens Rasmussen integrate – no they built!
• We have a system that fosters innovation while managing complexity and controlling risk….
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 26
Commercial in Confidence | Assurity Consulting Limited 2013
# 5 Leadership & Management Buy In
• Servant Leadership – behaviours are values in action• Predictive management versus empirical management• The desire to appear certain vs. the courage to admit software is not predictable and risk is best managed incrementally, with data
• Authority moves down the organization – trust your people!
NZ/AU - Strengths to Opportunities
• Use our isolation, innovation, attitude and independence to lead.
• Harness our pioneering innovation
Commercial in Confidence | Assurity Consulting Limited 2013
The changing role of management
• People don't set out to fail each day ‐ they set out to succeed • Successful managers
> create an environment for success> empower their people > stand aside
• Define the "what" not the "how"• Accept that people don't work for money, they work for AMP
> Autonomy, Mastery, Purpose
• The world is changing and Gen‐Y are helping challenge the status quo
• Manage them as you would a team of volunteers
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 27
Commercial in Confidence | Assurity Consulting Limited 2013
# 6 ‐ Issues with practices relating to quality
• missing design‐and‐code‐checking practices• a lack of independent testing• insufficient separation of environments• major issues with clarity and accessibility of requirements.
• Agile will highlight these insufficiencies quickly and often (like your mother in law coming to stay – you will be reminded of your deficiencies every day….)
“Scrum will highlight every deficiency and impediment that the enterprise has so the enterprise can fix them and change into the best organization in its market”Ken Schwaber, Scrum is Hard & Disruptive, 2006
Commercial in Confidence | Assurity Consulting Limited 2013
• The practices exist for good reason. Don’t skip them until you have mastered the basics
“Scrum is not a methodology that needs enhancing. That is how we got into trouble in the first place, thinking that the problem was not having a perfect methodology. Effort centres on the changes in the enterprise that are needed”‐ Ken Schwaber, Scrum is Hard & Disruptive, 2006
The student follows the teachings precisely. Concentrates on the
task, without worrying about theory.
Basic practices working - student starts to understand the
underlying principles & branch out.
Student isn't learning from other people, rather own
practice. Creates own approaches & adapts
learning to circumstances
# 7 ‐ Organisations don’t follow standard models ‐they adapt practices
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 28
Commercial in Confidence | Assurity Consulting Limited 2013
Finally ‐ Complexity
• The basic problem lies in our comprehension of what complexity is.
• Paradoxically, the deeper we look into complexity the more we see.
• As a result, we now appreciate that prediction of complex systems is (probably) impossible
Newcastle UniversityEngineering Design Centre
Commercial in Confidence | Assurity Consulting Limited 2013
Contrasting Project Failure
Failure, 54%In
Between, 22%
Success, 26%
New Zealand• We succeed less• We fail more• We have less in between
Success
FailureChallenged
Success
Failure
Challenged
ANZTB Test 2013: Advancing Expertise in Software Testing
6 June 2013
Canberra, Australia 29
Commercial in Confidence | Assurity Consulting Limited 2013
SummaryWe are failing for silly reasons and we know better
• Product Ownership ‐ business case, communication, clear view of product purpose & usage, ability to change and manage scope
• People deliver projects – team need a clear view of the product purpose & usage, plus the ability to get involved in this together
• Leadership is critical – you are only a leader if people follow
• Values trump everything – an open “safe to fail” culture wins
• Journey of constant improvement ‐ effort centres on the changes in the enterprise
• Start with the practices and only adapt once you know what you are doing
• Don’t be scared of custom dev – just actively manage the risk using transparency and empirical processes
Commercial in Confidence | Assurity Consulting Limited 2013
The future…
• Failure isn't bad, as long as we learn from it• Our industry is a wonderful industry full of amazing people who want to succeed
• But we must be brave, creative and honest – past management thinking is failing us
• We can use isolation to reduce negative influences to follow and lead in our own way
Recommended