Embed Size (px)
Citation preview
Working ourselves out of a job:A passion for improvement
Isabel EvansQuality Manager – Dolphin Computer Access
EuroSTAR2010 Copenhagen
Main points
• Focus on improving the service to the customer, not on improving testing.
• Focus on enabling others to improve, not on increasing the need for our involvement.
• Focus our passion on the big picture while helping others to focus on the details.
A challenge to our comfort fromCapers Jones
Poor software
engineering
Flawed economic
modelsRecession
Cutbacks
Recession
Cutbacks Poor SW engineering
Flawed economic
models
Test community is (partially) culpable…
Recession
Cutbacks
Poor SW engineering
Flawed economic
models
BAD DEVELOPMENT & MAINTENANCE
BAD TEST
BAD MEASUREMENT &
INFORMATION
Our job includes identifying flaws and informing people about risk…
Recession
CutbacksPoor SW
engineering
Flawedeconomic
models
BAD DEVELOPMENT & MAINTENANCE
BAD TEST
FAILURE TO IDENTIFY THE FLAWS
BAD MEASUREMENT &
INFORMATION
FAILURE TO CONVINCE OTHERS ABOUT THE RISK
If we did not identify flaws AND CONVINCE other people…
Poor SW engineering
BAD DEVELOPMENT & MAINTENANCE
BAD TEST
BAD MEASUREMENT &
INFORMATION Flawedeconomic
models
Recession
Cutbacks
Let other pens dwell on guilt & misery… (Jane Austen)
GOOD DEVELOPMENT &
MAINTENANCE
Good SW engineering
Good economic
modelsUp-turn!
Investment!
GOOD TEST
IDENTIFY IMPROVEMENTS
GOOD MEASUREMENT & INFORMATION
IDENTIFY FLAWS
EVIDENCE
CONVINCES
Person, Team or Department
An activity A description or identifier
A thing, object or concept
Test Test are holding a review meeting
I will test the input validation
Is the test environment ready?
Has the test completed successfully?
Tester Who is the tester for the input validation?
The tester mindset
A type of bed…
Testing Have Testing finished all their work?
Has the testing completed?
This is a very testing activity
All the testing is complete
Quality Quality are running the PIR
We’ve done the “quality stuff”
A poor quality product
Quality can be measured
QA QA will check the process …
Use QA to check the process
The QA report has been agreed
QC Jane is joining Quality Control
The team have completed QC
The QC tool set needs installing
QM Quality management will send someone to the meeting
We need to QM this programme
My QM plan includes root cause analysis
Stakeholders
Customers
Managers
Builders
Supporters
Measurers
Society
People who (could) measure qualityCustomers e.g. Req’s Reviews, Task-based testing
Managerse.g. QP and metrics
Builders e.g. Design ReviewsTechnical test
Supporters e.g. Req’s ReviewsTech Test, OAT
Society e.g. Beta, PIR
DO WE NEED A
SPECIALIST ROLE HERE
ANY MORE?
Quality ViewpointsCustomers USER , VALUE, TRANSCENDENT
Managers VALUE,TRANSCENDENT
Builders MANUFACTURING, PRODUCT,TRANSCENDENT
Supporters PRODUCT, USER, MANUFACTURING,TRANSCENDENT
MeasurersUnderstanding quality viewpoints?
Society USER,VALUE,TRANSCENDENT
What roles are required?TESTING is an activity NOT a department or team
TESTING is a role-aspect NOT a job title
• We might find a better and more rewarding job working with the rest of the IT teams to improve what they do.
ACTIVITIESQuality Control
Quality AssuranceQuality
ManagementTesting
AdvocacyAdvisingCoaching
Mentoring
SPECIALIST JOBSQuality EngineerQuality ManagerQuality Facilitator
Excellence AdvocateCustomer Advocate
ROLE-ASPECTSQuality Planning
Acceptance criteria (setter)Review / Inspection Leader
Reviewer / InspectorAudit and process review
Unit testingIntegration testing
System testingAcceptance testing
Test AutomationMetrics and reporting
Focus our passion on the big picture
• Helping others to focus on the details…
• This might include
– coaching and mentoring Business Analysts, Developers and Customers to do better testing.
– involving ourselves in requirements/user story definition and review.
– pair work with Business Analysts , Developers and Customers.
– support for releasing early.
– advising against a release.
– root cause analysis of problems.
Organisational excellence modelse.g. EFQM™
People LEADERSHIP
and vision
Policy & strategy
Partners &
resources
PROCESS
Society & governance measures
Customer measures
People measures
Key FINANCIALS
Continuous improvement feedback
Organizational frameworks e.g.Balanced business scorecard
BBSC
Financial
Customer
Internal
Innovation
Combined scorecard – FOR profit
BBCS
Financial
Customer
People
Society
Internal
Innovation
Combined scorecard – NOT for profit
BBCS
Society
Clients
People
Financial
Internal
Innovation
Combined scorecard – EFQM+Qualities
BBSC
Financial
value
Customer
user
People
user, value, transcendent
Society
transcendent
Internal
manuf. & product
Innovation
user, value, transcendent
Testers measure…
Showing the financial advantagesexample – conversation with PM
• Financial goal for the project: To reduce cost / time • Questions they ask
– Cost and time of last test project:• How much did it cost and how long did it take?• Why was it so expensive last time?• Why did it take so long?
– For the next test project:• Can I have it cheaper and better? • How much effort will it take?• When will you finish?• What resources do you need? • And what will those cost?
• Note possibly conflicting quality goal for the project
Showing the financial advantages (2)
• Metrics the PM would find useful
– Cost of testing: preparation/execution/managing; supporting activities; early preventative activities…
– Cost of failure to test efficiently: e.g. man-days lost waiting for environment…
– Cost of failure to test effectively: e.g. time wasted doing or inappropriate wrong tests
– Cost of failure: cost of defects/failures in live systems
– Money saved by doing testing
– Money that could be saved by improving…
Showing the financial advantages (3)
• Test measures, metrics and indicators:
– Number of significant defects found during previous release testing, cost to repair those defects
– Number of failures in live and cost to repair the damage from those defects
– Time wasted
– Estimated cost of tests that would have found those problems
– Size of previous release and size of next release
– Predicted # defects in next release (see CJ)
– Predicted cost to find those defects (see CJ)
– Predicted cost of not finding those defects (see CJ)
(CJ = Capers Jones: Software Engineering Best Practice)
Showing the financial advantages (4)Metrics from old projects (example only)
# defects release 1
~ %age
# defects release 2
~%age
# defects release 3
~%age
release 4 hunting target
req’s review 0 0 60 15 80 27 25%= 90
design review 0 0 40 10 70 20 20%=72
code review 0 0 20 5 50 13 30%=108
test defects 200 57 200 48 100 27 15%=54
live failures 150 43 100 22 50 13 10%=36
total 350 100 420 100 350 100 360
defects by LOC release 1 release 2 release 3prediction release 4
size 15000 20000 15000 15000
defects 350 420 350 360
defects by LOC 43 appr. 48 appr. 43 appr. 42 appr.
Showing the advantages (5)Language Code size Non-code
defectsCode defects
Predicted total defects Defects/kloc
Example C 125,000 3000 2000 5000 5-25 (Capers Jones)
Our C project 15,000 360 40 (our history)
Defect potential (after Capers Jones)
It looks like the code is about 8 times smallerWe have more defects per KLOC than
the worked Capers Jones example
COSTSCapers Jones
Example Acost per
LOC
Cost of our project with
static test
Capers Jones Example B
cost per LOC
Cost of our project without
static test
inspections 168,750.00 1.35 20,250.00 - - -
static analysis 81,250.00 0.65 9,750.00 - - -dynamic test
stages 150,000.00 1.20 18,000.00 775,000.00 6.20 93,000.00
total pre release 400,000.00 3.20 48,000.00 775,000.00 6.20 93,000.00
total post release 175,000.00 1.40 21,000.00 950,000.00 7.60 114,000.00
total cost 575,000.00 4.60 69,000.00 1,725,000.00 13.80 207,000.00
Showing the advantages 6)
TIMESCALES CJ example AOur project
+ST CJ example BOur project
no STdevelopment schedule
(months) 12 1.44 16 1.92
LIVE FAILURES
Total defects 5000 600 5000 600found by static test 4150 498 0 0
found by dynamic test 500 60 1600 192found post release 350 42 1900 228
Note: this table uses Capers Jones' figures, but history shows our defect per LOC to be about twice as bad as that – need to take that into account when reporting a range for cost, time and defects…
Sample test financial (7)
or get started late… and we can spend 2 months and 93,000 dollars before release for a total cost of 207,000 dollars, with 200-500 defects reported post live
Get started early and…we can spend 1.5 months and 48,000 dollars before release for a total cost of 69,000 dollars, with 40-80 defects reported post live
Financial measures: What is your cost of failure?What does your organisation need?
COF COQ
COF is Cost of Failure - COQ is Cost of Quality
QualitySecurity
Measurement of resultsDevelopment best practices
High costs?
FailureCompromised securityNO measurement of resultsDevelopment worst practicesHigh costsBut fast to market
OKLow costs
Main points
• Focus on improving the service to the customer, not on improving testing.
• Focus on enabling others to improve, not on increasing the need for our involvement.
• Focus our passion on the big picture while helping others to focus on the details.
Thank you for listening.
Isabel Evans [email protected]
Dolphin’s Vision
“It is our vision to deliver independence to people with visionand print impairments all over the World. With over 160 millionvisually impaired people and 600 million dyslexic or learningdisabled people, that's no small challenge. But we're determinedto make a difference and have been working hard on that goalsince Dolphin started in 1986.”
Noel Duffy Managing Director
Dolphin Computer Access
Dolphin’s range of computer software delivers independence to computer users with vision impairments and learning disabilities.
web: www.YourDolphin.com
