View
1.420
Download
4
Category
Tags:
Preview:
DESCRIPTION
Lanette Creamer's presentation on Reducing Test Case Bloat (version 2.1) from QASIG, March 10, 2010
Citation preview
White Master
Replace witha graphic
5.5” Tall & 4.3” Wide
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
ReducingTest Case Bloat
Lanette CreamerAdobe Systems, Inc
QASIG March 2010
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Introduction
Tester and Test Lead at Adobe since 2000
September 2009-First Agile project
Python scripts, collaboration, user focus, testing the whole system, cat photos.
Dysfunctional fiefdoms, clip art, spiders, waiting, retesting.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Agenda
What is a Test Case? What is Test Case Bloat?
Methods to reduce test case bloat in practice.
Barriers to reducing bloat.
How did it go?
Q&A
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section One: Why reduce at all?
“I have no help to send, therefore I must go myself.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
We like our bloat
How is that working for you?
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
What is a Test?
A test is an action which produces discoveries that can be used to evaluate product quality.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Test Case
Is every test a test case?
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
What is Bloat?
Test Case Bloat-More test cases than you can effectively manage* in the time you have. Is all bloat test cases?No.
*Manage may imply some or all of the following: cover, update, execute, report on, allocate, run, automate, respond to.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section Two: Find the bloat
“All we have to decide is what to do with the time that is given to us.”
J.R.R. TolkienThe Fellowship of the Ring 1954
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Is it worth keeping?
Cost versus RiskIf it doesn’t result in a bug is it bloat?If it does result in a bug is it always the most important bug?
Does reduce mean delete?Is it worth keeping all test cases just in case?
Time
Culture
Set up
Trust
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section 3: Ideas used in practice with success
“Their plans were improved with the best advice.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case Bloat
Mathematical Combinations
• Scatterplot
• Pairwise
• Random Selection
• Rotation of Environmental Factors
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case Bloat
Tiered by Importance
• Priority 1, 2, 3
• High, Medium, Low
• Daily, Build, Weekly, Once
Stakeholders Priorities
• Poker Chips
• Backlog
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case
Change Based
• Bug fixes
• Features added
• Components integrated
• New visualization tools
Timing Based
• Covered later elsewhere
• Stop running at ______
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case Bloat
Test Case Find Results
• Areas of breakage
• Date of last failure
• Auto retire?
Model Based
• Interactions with more risk
• Changes to the model
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case Bloat
Code Coverage Based
• Reconcile duplication
• Which code is covered
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Ways to Reduce Test Case Bloat
Customer Workflow Based
• Usage data
• Sales info
• Meeting users
• Studies
• Surveys
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section 4: Letting Go
“The world changes, and all that once was strong now proves unsure.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Does it still matter?
Risk:
High defect density.
Developers are worried about it.
Area of integration.
Area undergoing change.
Provides code coverage.
Data showing it is used.
Bottom line impact.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Does it still matter?
Protecting Your Peeps:
Impacts a customer workflow.
Barrier to adoption.
Top customer issue.
Silent failure is catastrophic.
Stakeholders still care about it.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Letting Go
When I let go of what I am, I
become what I might be.
- Lao Tzu
You do not need a parachute
to skydive. You only need a
parachute to skydive twice.
-Unknown (likely a tester)
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section 5: Human Elements and Change Management
“Valour needs first strength, then a weapon.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Why it may be hard
What if we choose wrong?
Management would rather hear that we ran out of time for the low priority tests rather than that we decided not to test them.
Not everyone agrees on what is important.
Some are ethically opposed to this idea.
No one wants to lose value.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Why it may be hard
Feels safer to just keep them.
We paid good money for these test cases!
We’ve invested too much.
No time to deal with this!
It would be wasteful.
We may need these some day.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Other Ideas
More AutomationOk, but is it maintainable?Will it have value in time?You are making it instead of?Who runs, isolates, owns, maintains it?What about changes to what it relies on?
More TestersOk, but are they trained?Do they save or cost time?Do they come with more tests implied?More or less risk?
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Other Ideas
Do nothing• Status Quo may be ok.• Powerless to change process.• Quality not culturally important.• Bloat is acceptable.• Change not worth the political
cost.• Acceptance may be enough.
Bloat is the norm for the majority of people and now over 50% of all household pets. It is hard to change. Long term success is the most difficult part.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section 6: “Go forth and be spectacular!”-Gretchen Sidener
“Still round the corner there may wait, A new road or a secret gate.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Now What? Now success!
The hard partThere is no after. Only during.Evaluate what worked.Identify what didn’t apply.Test cases usually shouldn’t live forever.Pruning test cases is part of test planning.Know when to start over.Share your own ideas with others!
The good newsThere is no failure, only “another fine learning opportunity”.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Section 7: Disclaimer and Cautionary Note
“...for advice is a dangerous gift, even from the wise to the wise, and all courses may run ill.”
J.R.R. Tolkien
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Choose Wisely
The same level of risk is not appropriate for all situations.
Failure to choose gives you fewer options. This experience report is intended to expand your options, not as a directive.
These ideas have been tested in one context and worked well for my team. Use at your own discretion. No warrantees implied.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Summary
Bloat happens to good people, projects, and pets.
With courage, information, and your own plan you can deal with test case bloat on your own terms.
A whole community of testers is here to help.
Each iteration is a new chance.
®
Copyright 2010 Adobe Systems Incorporated. All rights reserved..
Followup-Technical Details
V A R I AT I O N S O N B L O AT
Jean Hartmann’s work (Microsoft) see http://www.pnsqc.org/conference09/Paper_33_slides.pdf
Reducing Test Case Bloat Technical Paper, Version 1.0 http://www.pnsqc.org/conference09/Paper_25_slides.pdf
Visual Representations of Data http://marlenacompton.com/?p=553
White Master
Replace witha graphic
5.5” Tall & 4.3” Wide
Copyright 2010 Adobe Systems Incorporated. All rights reserved.
Q&A
lanettecream@hotmail.comTwitter @lanettecream Blog at http://blog.testyredhead.com
Recommended