Failure is Becoming the Norm (and what you should do about it ) Part 1 Software industry has a major quality problem Part 2 Enterprise and project tactics

Failure is Becoming the Norm

(and what you should do about it)

Part 1 Software industry has a major quality problem

Part 2 Enterprise and project tactics for solving this problem exist

© 2015 ClearSpecs Enterprises 1

David Gelperin, CTOClearSpecs Enterprises

System Failures and Quality Deficits

“… 43% of companies worldwide have reported breaches in the past year…”

“… the malware that was used [in the Sony breach] would have … probably gotten past 90% of internet defenses that are out there today in private industry …”

“Sale crashes Southwest Airlines website”

Most system failures don’t make the news and are caused by quality, not functional, deficits.

Many software systems have significant quality deficits. For example, some systems can’t be reliably changed after a few years.

These deficits are self-inflicted and becoming systemic as Agile is more widely adopted.


Quality goals are poorly understood


Required quality attributes (quality goals) are difficult to achieve and verify. Most developers (and their managers) have little understanding of either.

Developers understand testing, but not verification. Unfortunately, testing alone is inadequate for the verification of many quality goals. Quality verification may entail analysis, technical review, and measurement, as well as four modes of testing.

You can’t manage risks you don’t understand


Major cause: customers & developers focus on functions


Developers (& their managers) neglect foundations

When a wall cracks (insufficient footings) in your home or your basement floods (insufficient drainage or waterproofing), you understand the need for a solid foundation.

When your system is hacked or crashes under high volume, you understand the need for a solid foundation i.e. set of quality supports.


• Several qualities (e.g. availability, performance, privacy, reliability, safety, security, and usability) are software engineering subfields i.e. have substantial content

• Except for these subfields, quality goal achievement and verification aren’t taught• Lack of a comprehensive (> 50 goals), detailed (> 20 characteristics per goal), and

succinct overview of quality goals• There is little guidance on quality attributes inside software development groups

-- QA & QC groups usually focus on process and test• Project quality lessons are lost

Other causes of poor understanding

Part 2 – Enterprise and project tactics for reducing

quality deficitsa. Establishing a “Quality & Productivity Support Group” (QPSG)

b. Promoting quality awareness

c. Creating rich models of quality goals on each project

d. Sketching achievement and verification strategies for quality goals

e. Recording quality lessons

f. Providing guidance on fundamentals of quality attributes and goals

g. Running study groups on “high-risk” quality goals

h. Holding brown-bag lunch sessions to share quality experiences


a. Primary responsibilities of a QPSG

1. Provide guidance in quality achievement and verification

2. Acquire (and record) quality expertise

3. Identify and develop quality leaders

4. Promote quality-aware and verification-driven development

5. Acquire and evolve rich enterprise quality template(s)

6. Collect and adapt development standards

7. Acquire and record quality support and verification patterns

8. Participate in technical reviews

9. Track product behavior and team productivity

10. Acquire quality and productivity tools and training

11. Provide “safe ears” i.e. what happens in …, stays in …© 2015 ClearSpecs Enterprises 8

a. Objectives of a QPSG

• Develop quality competence in enterprise staff• Develop quality appreciation in all stakeholders to

counterbalance functional bias• Improve productivity


A QPSG is NOT a:• QA or QC group• test group (although it may test)• pool of spare project staff

b. Promoting Quality Awareness

The challenge is to raise stakeholder awareness of quality goals to the same level as their awareness of functional goals.

Quality awareness operates at two levels:

1.System level – identify quality goals, levels, and achievement and verification strategies

2.Component level – implement and verify quality supports


b. System Level Awareness

Implies early and continuing understanding of:• high-priority quality goals and their characteristics

• conflicts between qualities and how conflicts should be resolved

• critical supports for each quality level

• effects of critical supports on each domain function

• how achievement of quality goals will be verified

Implies focus on quality before functionality


b. Quality before functionality


b. Identifying quality goals first

Most quality goals can be accurately identified from knowledge of the software’s operating environments and basic mission e.g. flight control, internet gaming, or stock trading, and use of a rich quality attributes model.

Early identifications may need to be adjusted as understanding deepens, but there is no way to predict when you have enough information to accurately determine a quality goal without waiting until all functional code has been written.

Waiting is an expensive alternative to early identification.


b. Quality-Aware development

Quality-Aware development is NOT a development methodology, but a 3-part supplement to whatever you are doing now or intend to do

It begins with a quality sprint in which you identify quality goals their levels, priorities, challenges, mitigations, supports, achievement strategies and verification strategies. You also acquire and verify a library of crosscutting support components e.g. exception handlers. Using a rich quality attributes model, it should take less than a week to draft a quality goals model.

In each iteration, you reassess and carry out the quality strategies.

Finally, you collect the quality lessons during a project retrospective and record them in the enterprise model of quality attributes and/or in the development standards.


Functional Requirements i.e. behaviors

b. Component-level Awareness


Functional Requirements i.e. behaviors

“Complete” functional components must contain quality support code

Late Identification causes reckless technical debt

Therefore, you should identify quality goals before functions

Domain Function(s)

e.g. delete reservation

• Input verification• Request validation• Exception handling• Logging et. al.

• Safeguards• Security guards• Encryption• Testpoints

c. Skimpy models of quality attributes


ISO 25010 and its peers are skimpy and misleading models of quality attributes

c. 2 skimpy (web) models of quality goals


How is the system A model inconsistent?

c. Developing rich models of quality goals


You should explore the LiteRM quality attributes model

Rich Model of System

Quality Goals

EnterpriseQuality Attributes

Model

RichQuality Attributes

Model

Attribute examples -- 1 of 2


Attribute examples -- 2 of 2


Characteristic examples -- 1 of 3






c. Rich models of quality goals

Rich models of quality goalsare analogous to


On

SystemQuality Goals

d. Q & A about Verification

Isn’t verification just a fancy word for testing?

No

Then, what is it?

A composite checking activity that includes analysis e.g. hazard analysis, technical reviews e.g. code inspections, and measurement e.g. mean time between failures, as well as four modes of testing.

Why should I care?Quality goals must be verified. Testing alone is inadequate e.g. security testing is not enough, also need threat analysis and security guard code inspections.


d. Sketching a Verification Strategy

[See sample in handout]


d. Sketch & Execute Verification Strategies Early


TestThen

Code1986

Verify

Then

Code2015

Prevention-oriented Testing Prevention-oriented Verification

d. Why verify early?

“IT projects that applied NFR (Non-Functional Requirement) verification techniques relatively early in development were more successful on average than IT projects that did not apply verification techniques (or applied them relatively late in development)”

- Eltjo Poort, Nick Martens, Inge van de Weerd and Hans van Vliet “How Architects See Non-Functional Requirements: Beware of Modifiability” REFSQ 2012 [Best Paper]


e. Recording Quality Lessons

Enterprise quality models and development standards are natural places to record information such as stakeholder priorities, achievement and verification strategies, and pointers to support components.

Without recording, lessons are lost or must be relearned.


Wrap up

Quality improvement requires collaboration at the project and enterprise levels

You must bootstrap quality understanding on your project

Resources available at quality-aware.com

Let’s make failure a rare exception, rather than the norm


An Offer

If you have an early stage project, are interested in trying a “quality sprint”, and would like an expenses-only week of guidance, let me know.


Questions or Comments

[email protected]


Documents

Failure is Becoming the Norm (and what you should do about it ) Part 1 Software industry has a major quality problem Part 2 Enterprise and project tactics