Upload
vananh
View
221
Download
0
Embed Size (px)
Citation preview
Quality assurance & testing
Karel Kravik
Who am I?
� Tartu University� Applied Informatics
� Assistant lecturer “Software Project”
� Webmedia 2000-2007� Programmer
� Analyst/project manager
� Product manager for Changelogic (~4 years)
� Quality manager (~2 years)
� Playtech 2007-...� Integration manager
What I will be talking about?
� Unpredictability of bugs
� Bugs and their effects
� Testing methods & aspects
� Test automation
� Test plans and reports� Test plans and reports
� Bug tracking
� Testing as a career
What one must know about software quality & bugs?software quality & bugs?
What is a software bug?
� Nonconformance to specification� Software doesn’t do something that is required by specification
� Software does something that is prohibited by specification
� Software does something not mentioned in � Software does something not mentioned in specification
� Software does not meet end user needs� Slow, not usable, etc
Nontrivial software has bugs
� You don’t know what nor where the bugs are
� You can’t tell there are no bugs after testing:� You can observe many times when the software behaves right
� You cannot conclude that the software behaves right alwaysright always
� Bug are the software reality� Client: “Please confirm that we don’t make any more bugs!”
According to observations all swans were white. That is, until Australia was discovered.
The bug and it’s effect
� What are the preconditions for a bug to occur?� Only in Vietnamese Windows, when user click ctrl four times in a row
� When it occurs, how bad can it get?� How many users are affected?
� How many transactions are affected?� How many transactions are affected?
� Browsers’ example
� If you could predict how many bugs you have left to discover, you surely couldn’t predict their impact
� Asymmetry between a bug and it’s effect – the less you expect something to happen, the more impact it has
According to my calculations, there are no more P1 bugs in the
software
What to do with bug?
� When having a bug:� You might not be able to reproduce it
� Fix it
� Leave it as is
� Document the current behaviour
It might be a feature☺� It might be a feature☺
� What to do if there are no bugs?� Most probably the software is not tested
Software ships with bugs
� There’s nothing unusual about it
� Known bug is always better than unknown bug� You should know which bugs are left
Economic considerations
� For technical people a bug is a bug and it must be fixed for the sake of perfection
� For business people a bug is another things that wastes developers’ time
� Fixing a bug might introduce new bugs
� Conclusions: � It might be OK to add feature instead of fixing a bug
� It might be OK to go with a hack instead of correct solution
� It might be OK to leave the software as is
� Which leads us to...
Good enough quality
� Software is not written for beauty contest
� Software is written to earn or save money
� Software should earn or save more money than spent on developing it
� Conclusions:� Conclusions:� Software doesn’t have to be perfect (developers are never satisfied with the code, that’s for sure)
� Software must be good enough
� What is good enough?� There’s no simple answer
Practical testing
Difference between testing and QA
� Quality assurance is a broader term, it includes:� Testing
� Test management
� Test process management
Bug tracking� Bug tracking
� Bug prevention
� What is the difference between tester and QA specialist?
Preconditions for testing
� Testing environment� Should be separate from development
� Must be separate from production
� Update procedure
� Installation manual
� Specification or even better, test plan
� Testability features:� Software version – test known state
� Log or debug info
� References to specifications
Types of testing
� Verification� Is the software implemented according to specification?
� But what is the specification is wrong?
� ValidationDid we implement the right thing?� Did we implement the right thing?
� But what if the user cannot express what he needs?
Testing different aspects
� Functionality� Does the application behave like expected?
� Usability� Is it convenient to use?
� Performance & stress� Continuous load & break point
� Security
� Reliability
� ...
Testing methods
� Manual testing� Done mostly in India ☺
� Automated testing� Haven’t seen an actually working and useful solution
I still believe � I still believe they exist
Automated tests rarely find new bugs
When to use automated tests?
� Automated tests are good for:� Testing performance, throughput, response times
� Testing machine interfaces (API, web services)
� Automated tests are not good for:� Functionality
� Usability
� You can automate parts of tests� For example, when testing security, you can use scripts for scanning ports or cracking password
� Test results must be interpreted
Automated testing tools
� Mercury tools (now owned by HP)� Quality Center
� Test Director
� Rational Tools (now owned by IBM)� Functional Tester
� There are apecialized freeware tools
� Tools don’t save you from thinking!
Testing in software lifecycle
Testing in software lifecycle
� (Typical business application example)
� Document inspection – requirements, analysis, design
� Testing prototype/helping developer
� Testing new functionality� Testing new functionality
� Testing changes
� Testing for acceptance
� Handling client feedback
Document inspection
� Completeness� If A, then ..., but what is A is not true?
� Correctness� Human language is vague
� Sometimes, usually, most
Good, effective, stable� Good, effective, stable
� Etc,
� Can it be implemented?
� Can it be tested?
� Is it optimal?
Testing prototype
� Not all projects use prototype, but sometimes developers ask feedback to their very early solution
� Is the concept right?
� Can the concept be implemented?
� Complications
Testing new functionality
� Input validation� Date field accepts YYYY-MM-DD
� Email must contain @ and .
� Business rules� When user has debt, send warning
� Is the data saved correctly? Is it displayed correctly?
� Privileges
� Messages
� Documentation
Testing changes
� What has been changed with this release?
� You need changelog:� Test new features
� Verify fixed bugs
� Are there any dependencies that might be affected?
� Is documentation updated?
� Are machine interfaces changed?
Acceptance testing
� Verify that criteria for production have been met� At first, you have to set these criteria
� Test the main use cases
� Test critical paths
� Test functionality that has been broken before
� Verify that there are no open issues that should have been fixed
Handling client feedback
� Clients should not give feedback directly to programmers, unless you’re doing XP, but then you don’t even have tester’s
� Testers should find out:� If it is a bug at all
If the bug can be reproduced� If the bug can be reproduced
� If there are no duplicates
Tester’s inputs and outputs
Tester’s input and output
� Tester’s input� Application
� Specifications
� Tester’s output� Test plans & test reports
� Bug reports
Test plans & test reports
� Test plan� What should we test?
� How should we test?
� What is the expected outcome?
� Test report� Test plan X was tested
� On version x.y.z
� On dd-mm-yyyy
� By John Smith
� And the result was “test failed”.
� The following bugs were reported: x, y, z
Bug lifecycle
� Reporting
� Bug analysis� Bug is localized, impact and root cause is analyzed
� Fix is scheduled
� Fixing
� Verification� Tester verifies that the bug is really fixed
� And no new bugs are introduced
Reporting bugs
� Bug report should contain:� Steps for reproducing
� Current behaviour
� Expected behaviour
� As much context as possible
As general as possible� As general as possible
� As specific as possible
� No feelings included
Bug analysis (example)� Viga kliendi detailvormil
� Üldistame:� Kas ükski kliendi detailvorm ei tööta? � Kas ükski detailvorm ei tööta?� Kas terve rakendus ei tööta?
� Konkretiseerime:� Osad kliendi detailvormid ei tööta. Mis on vahe � Osad kliendi detailvormid ei tööta. Mis on vahe töötaval ja mittetöötaval?
� Hmm – täpitähed!� Üldistame: Kas ükski täpitähti sisaldav vorm ei tööta?
� Konkretiseerime: “õ”-tähte sisaldavate nimedega klientide detailvormil tekib viga teatega …
Root cause analysis (example cont)
� Kus selline viga veel võib esineda?� Kas õ-tähega asutuse detailvormid on korras?
� Mis on selle vea tegelik põhjus?� Andmebaas kasutab sellist charset’i mis ei saa hakkama õ-tähega?
Kuidas välistada terve vigade klass?� Kuidas välistada terve vigade klass?� Konverteerida andmebaas?
Bug tracking tools
� Bugzilla – freeware, open source� Big user community
� Jira – commercial, site license
� Fogbugz – per user license� Read www.joelonsoftware.com and you might end up buying it
� High-end tools – Rational ClearQuest, etc
Testing as a career
Testing as a career
� It’s a good introduction to software development world
� It’s a good starting point when you want to become analyst or project lead
� How can one be a tester more than 3 years?
What makes a good tester?
� Imagination – a lot of it
� Healthy paranoia – do not assume anything
� Curiosity & exploratory thinking
� Always polite
Why should I want to be a tester?
� Common misunderstandings:� It’s a monkey job – if programming the same app is a monkey job, then testing it probably is too, sorry
� It’s not creative
� It’s not constructive
� It pays less� It pays less
� It’s not a job people usuallydream about, but manyend up doing
Reading
� “Lessons Learned in Software Testing” by Cem Kaner, James Bach, Bret Pettichord
� Read everything that you find useful� Not only about testing, but about software development in generaldevelopment in general
� Be sure to have enough domain knowledge
� And by the way, read Härmel Nestra’s lecture materials from 2005: � http://courses.cs.ut.ee/2005/tvt/Main/Lectures
Anyway, how to test?
Just do it.Just do it.
Just do it!