Agenda
1. Test Case.2. Examples.3. BDD.4. Examples.5. Defects 6. BUG Report7. Examples8. Good BUG9. Additional links
2
About myself
Andrey Oleynik
• About 7 years of work in Quality Assurance area
• QA Automation Engineer
• PhD, Theoretical physics
3
What is a Test Case?
• A set of input values, execution preconditions, expected results and execution postconditions, developed for a particular objective or test condition, such as to exercise a particular program path or to verify compliance with a specific requirement. [IEEE]
• Documentation specifying inputs, predicted results, and a set of execution conditions for a test item. [ANSI]
• A test case is a bunch of steps that lead to consider one "item" as successfully working.
4
Why do we write test cases?
• The basic objective of writing test cases is to validate the testing coverage of the application
• Usually QA engineers follow the test cases standards
• So writing test cases brings some sort of standardization and minimizes the ad-hoc approach in testing
5
What information would the test manager want
out of test case document/s?
• Which features have been tested/ will be tested eventually?
• How many user scenarios/ use cases have been executed?
• How many features are stable?• Which features need more work?• Are sufficient input combinations exercised?• Is software ready to ship? Is testing enough?• What is the quality of the application?
6
Test Case design
• Specification based testing;
• Input Domain testing;
• Risk based testing;
• Scenario testing.
7
Test Case Fields(the most important)
• Test Case id
• Title
• Priority
• Description
• Steps
• Expected result
• Feature/Area8
Test Case fields
• Test case status (Idea/Design/Ready/Update/etc.)
• Product version• Responsible person• Preparation notes• Notes to clean the system• Attachments• Examples• Linked feature/requirement/bug• ...
9
Example
Title: Open file.Description: a file can be opened in the Application.
1. Launch the Application.2. Select "File" menu.
File menu pulls down.3. Choose "Open“.
"Open" dialog box appears.4. Select a file to open.5. Click OK.
Expected result: File should be opened.
13
Important:
1. Template
2. Descriptive and specific
3. Reusable
4. Atomic
5. Positive and negative
6. Refactor
7. Test data
8. Setup and tear down
9. Test Case review
10. Screenshots
14
Test Suite
• A test suite is a set of related tests, usually pertaining to a group of features or software component and usually defined over the same database. Suites are combined into groups. [Beizer]
15
Test management tools
• Gemini• HP ALM (QC)• IBM Rational Quality Manager • Meliora Testlab• Seapine TestTrack• TestLink• Google Docs !• …
16
TDD
Test-driven development is a software development methodology which essentially states that for each unit of software, a software developer must:
• define a test set for the unit first;
• then implement the unit;
• finally verify that the implementation of the unit makes the tests succeed.
19
TDD
• Java (Junit, TestNG);
• C (Check, CUnit);
• C++ (Boost, CppUnit, CppUnitLite, Google Test);
• C# (xUnit, NUnit, MSTest);
• Ruby (Test::Unit, Shoulda, RSpec);
• Python (pyUnit, py.test, doctest);
• …
20
BDD
• Behavior-driven development is a specialized version of test-driven development which focuses on behavioral specification of software units.
• Behavior-driven development specifies that tests of any unit of software should be specified in terms of the desired behavior of the unit.
21
BDD
22
TitleThe story should have a clear, explicit title.
NarrativeA short, introductory section that specifies
•who •which effect the stakeholder wants the story to have •what business value the stakeholder will derive from this effect
Acceptance criteria or scenariosa description of each specific case of the narrative.
•It starts by specifying the initial condition •It then states which event triggers the start of the scenario. •Finally, it states the expected outcome, in one or more clauses.
BDD example
23
Feature: New accountIn order to have new ready to use accountAs a userI want to get an account with balance $0
Scenario: Create new accountGiven I have no accountWhen I create an accountThen I should have an account with balance $0
What is a BUG(defect)?
• A flaw in a component or system that can cause the component or system to fail to perform its required function, e.g. an incorrect statement or data definition. A defect, if encountered during execution, may cause a failure of the component or system.
26
Another BUG definition
• A software bug is an error, flaw, failure, or fault in a computer program or system that produces an incorrect or unexpected result, or causes it to behave in unintended ways.[Wikipedia]
28
BUG names• Mistake• Anomaly• Fault• Failure• Error• Exception• Crash• Bug• Defect• Incident• Side Effect• Change request
29
Types of BUGs• UI • Functional issues• Programming logic• Installation • Memory allocation • System resources• Documentation • Requirements • Localization (L10N) • Internalization (I18N) • Performance • Security• …
30
How to find a bug?
• Understand the whole application in depth
• Prepare good test cases and test data before testing
• Execute tests with different test environment
• Compare actual and expected results
31
What to do next?
• Do monkey testing
• Use test data pattern from previous versions/projects
• Check standard scenarios from other products
• Try to break the application!
32
BUG report
Defect/BUG report: A document reporting on any flaw in a component or system that can cause the component or system to fail to perform its required function. [IEEE]
33
“The point of writing problem report(bug report) is to get bugs fixed” [Cem Kaner].
Defect Fields(the most important)
• Bug ID• Summary• Description/Steps to Reproduce• Expected/Actual result• Status• Severity• Priority• Detected by• Assigned to• Module/Area
34
Defect Fields
• Environment
• Reproducible
• Fixed in Version
• Detected Date
• Modified/Last Updated
• Detected in Version/Build Version
• Comments
• Attachments
• Linked testcases/requirements/defects
35
Defect Resolution
• Fixed
• Won't Fix
• Not Reproducible
• Duplicate
• By Design
• External / 3d party
38
Defect Severity
1 - Urgent• (Critical) • Data loss or corruption. Product is down. Customer operations are
impacted very seriously. There is no any workaround. • For documentation: copyright or offensive content issues,
documentation steps lead to data loss or corruption.
2- High• (Serious) • System / feature is still usable to a high degree but has a serious
issue. Defect has serious customer business impact. • Some data is at risk. No obvious workaround.• For documentation: core functionality is not described.
39
Defect Severity
3 - Medium• Product operational • Application customer data is not at risk. • Obvious/easy workaround available. • Non informative error reporting.• For documentation: Conceptual information is incorrect or
incomplete.
4 - Low• It is more of a cosmetic issue. No serious impedance to
system functionality is noted. Annoyances. • For documentation: typography, formatting.
40
Priority
0. Show-stopper - this needs to be fixed right now, everything else can wait, the build cannot be released with this defect1. Urgent - needs to be fixed before any other high, medium or low defect should be fixed2. High - should be fixed as early as possible3. Medium - should take precedence over low priority defects4. Low - fixing can be deferred until all other priority defects are fixed
41
BUG exampleTitle/Summary: Application crash on clicking the Browse button while file open.
Area: File operations Build Number: x.x.x
Severity: HIGH Priority: HIGH
Status: New/Open/Active
Environment: Windows 2008
Steps To Reproduce:
1) Launch the application
2) Open menu File -> Open.
3) Click on ‘Browse’ button
Actual result: Application crash.
Expected result: On clicking Browse button file browse dialogue should be opened.
Notes: the screenshot and the logs are attached. 44
Good BUG
1. Short and precise title
2. Concise but complete description: Steps to reproduce, consequences of the bug, and suggested behavior
3. Good attachments
4. Complete definition of the categorizing fields
5. Correct Severity & Priority
6. Follow-up and comment
45
Good BUG
7. Include the Results you Expected
8. Include the Results you Actually Observed
9. Explain the Impact on the Customer
10. Avoid speculation
11. Be careful of the tone of your report
12. Avoid duplication - search first (!)
46
Collaboration
• QA engineers and developers are not in fight but are partners
• The main thing is product and its quality
47
BUG Tracking Systems
• Gemini• HP ALM (QC)• IBM Rational Quality Manager • JIRA• Seapine TestTrack• Bugzilla• Google Docs !• …
48
More details
49
• Software testing Help: http://www.softwaretestinghelp.com/
• Про Тестинг: http://www.protesting.ru/• http://software-testing.ru• What is severity and priority in software testing?
http://istqbexamcertification.com/what-is-severity-and-priority-in-software-testing/
• IEEE Std 1044-1993 IEEE Standard Classification for Software Anomalies http://standards.ieee.org/findstds/standard/1044-1993.html
• ISTQB® Glossary http://www.istqb.org/downloads/viewcategory/20.html
More details
50
• Сэм Канер, Джек Фолк Тестирование программного обеспечения
• Роман Савин Тестирование Дот Ком, или Пособие по жестокому обращению с багами в интернет-стартапах
• Луиза Тамре Введение в тестирование программного обеспечения
More details
51
• Introducing BDD by Dan North http://dannorth.net/introducing-bdd/
• Портал об автоматизации тестирования ПО: http://automated-testing.info/
• Конференции sqadays: http://www.sqadays.com
• Software testing Help: http://www.softwaretestinghelp.com/
• Про Тестинг: http://www.protesting.ru/
• http://software-testing.ru
More details
52
• Э.Дастин, Дж. Рэшка, Дж.Пол Автоматизированное тестирование программного обеспечения. Внедрение, управление и эксплуатация
• Matt Wynne and Aslak Hellesoy The Cucumber Book: Behaviour-Driven Development for Testers and Developers
• Криспин, Грегори Гибкое тестирование