View
1.455
Download
1
Category
Tags:
Preview:
Citation preview
Functional System Testing
Written by
Adam Carmi
Black Box Testing2
Outline
• Goal of testing• Test cases, test suites and test data• What is functional system testing?• Coverage• Functional testing techniques:
– Functional analysis– Equivalence partitioning– Boundary value analysis
Black Box Testing3
The goal of software testing
• The process of uncovering evidence of defects in software systems– Does not include efforts associated with tracking down
bugs and fixing them
• No amount of testing will improve the quality of a computer program– The more testing we do of a system, the more
convinced we might be of its correctness– Testing cannot in general prove a system works 100%
correctly
Black Box Testing4
Test cases
• The basic component of testing is a Test Case• In its most general form: (inputs, expected-result)
– inputs include system state, user commands and data values to be processed
– expected result includes visible/audible interface changes or changes in the system state
• Test cases are organized into Test Suites– functionality, security, performance, …
Black Box Testing5
Test case execution
• A running of the software (under test) that provides the inputs specified in the test case and observing the results and comparing them to those specified by the test case– If the actual result varies from the expected
result, then a failure has been detected
Black Box Testing6
Test data
• An effective test strategy requires careful acquisition and preparation of test data prior to testing– Testing can suffer if test data is poor
• Test data concerns:– Depth: quantity and size of data– Breadth: variance of data values and data types– Scope: completeness, relevance and accuracy of data
• Result of a query should be valid for the specific purpose of the query, and not due to a missing or inappropriate value
– Conditions: data should reflect specific “conditions” in the domain• Data that would otherwise arrive after performing specific operations
over time
• Test data and test results are expensive to construct
Black Box Testing7
Example: Test data for TVRS
Name: test1.db
Description: Violation records designed for validating violation lookup
Violation IDOffender’s first nameOffender’s last nameIssuing policeman ID
243567RachelJosef8700342
237812DanLevi6386541
264683DanPorat1346329
255245DinaJosef8245731
000345longFirstNlongLastN8700342
…………
Black Box Testing8
Specification Vs. Implementation
• The basic approaches to testing software are based on its specification and implementation
• White box testing – test cases and data are constructed based on the code that implements the software– Quality and correctness of computations is validated
– Will not be further discussed in this tutorial
• Black box testing – test cases and data are constructed based solely on the software’s specification
Black Box Testing9
Functional System Testing
• Testing of a completed application to determine that it provides all of the behaviors required of it– Testing of completed increments that provide some
degree of end-user functionality
• Search for defects that are variances between the actual operation of the system and the requirements for the system
• System is treated as a black box
Black Box Testing10
How much testing is adequate?
• Completely validating IEEE 754 floating-point division requires 264 test-cases!
float divide(float x, float y)
• From practical and economic perspectives, exhaustive testing is usually not possible– Which software pieces should we test?– Which test cases should we choose?
Black Box Testing11
Coverage
• Coverage is a measure of how completely a test suite exercises the capabilities of a piece of software– “Each line of code should be executed at least once”
– “One test case should be constructed from each specified requirement”
• It is necessary to use testing techniques that narrow down the number of test cases allowing the broadest testing coverage with the least effort
Black Box Testing12
Technique: Functional Analysis
• Analyze the expected behavior of the system according to its functional specification
• Generate a test procedure for each of the possible usage scenarios– Corresponds to use case scenarios
– Analyze how a change in one part of the system affects other parts
– “Grand tour” test cases: the result of one test case produces the data that is the input to the next test case
Black Box Testing13
Example: Functional Analysis I
Use Case: Remove Traffic Violation
1. Supervisor calls for deletion of the chosen Traffic Violation
2. TVRS prompts Supervisor for confirmation
3. Supervisor confirms
4. TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record
5. OffendersDB approves that the Traffic Violation has been deleted
6. TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC
Use Case: Remove Traffic Violation
1. Supervisor calls for deletion of the chosen Traffic Violation
2. TVRS prompts Supervisor for confirmation
3. Supervisor confirms
4. TVRS requests OffendersDB to delete the Traffic Violation from the offender’s record
5. OffendersDB approves that the Traffic Violation has been deleted
6. TVRS allows Supervisor to look up a new Traffic Violation as described in the “Lookup Traffic Violation” UC
Black Box Testing14
Example: Functional Analysis IITest case ID: 134543
Pre-conditions: 1. TVRS initialized with test1.db database
2. Violation 243567 displayed in the “Lookup Violation” dialog
Related use cases: “Lookup Traffic Violation”, “Remove Traffic Violation”
ActionExpected result
Press “Delete” buttonConfirmation dialog is displayed
Press the “Yes” button“Lookup Violation” dialog is displayed
Enter “243567” at “Violation ID” text field and press the “Search” button
A message dialog stating that violation “243567” is not stored in TVRS
Test results
Passed
Failed
Actual results:
Defect diagnosis:
Black Box Testing15
Example: Functional Analysis III
Verify effects of change
Filled when the test case is executed
How do we know that violation 243567 is
stored in the system?
How do we know that violation 243567 is
stored in the system?
Can a tester diagnose the cause of a defect?
Can a tester diagnose the cause of a defect?
In addition, a query could be run on the Offenders database
In addition, a query could be run on the Offenders database
Black Box Testing16
Technique: Equivalence Partitioning
• Identifies ranges of input and initial conditions that are expected to produce the same result
• A group of test cases form an equivalence class if:– They test the same feature/scenario– If one test reveals a fault, the other ones (probably) will
too– If a test does not reveal a fault, the other ones
(probably) will not either• It is adequate to use only a single representative of
the equivalence class
Black Box Testing17
Example: Equivalence Partitioning I
Input value specification for “Lookup Violation” form:Input value specification for “Lookup Violation” form:
Field nameValid values
Violation ID[0-9]{0, 9}
Offender’s first name[a-zA-Z]{0, 10}
Offender’s last name[a-zA-Z]{0, 10}
Black Box Testing18
Example: Equivalence Partitioning II
FieldValid equivalent
classes
Validrepresentative values
Invalid
equivalent
classes
Invalidrepresentative values
Violation IDKnown violation00243567ID < 0 or ID > 999999999
-1, 1234567890
Unknown violation32456720Non numeric ID23ab@
Empty“”
Offender’sfirst name
Unknown violationDavidCharacter# > 10Hasalongname
Single known violation
RachelInvalid characterad0@am
Many known violations
Dan
Empty“”
…………
Black Box Testing19
Example: Equivalence Partitioning III
• The number of test cases to choose from is reduced to(3 + 2) × (4 + 2) × (4 + 2) = 180
• The actual number can be further limited– Single invalid field per test case (3 × 4 × 4 + 6 = 54)– Importance of use case– Resources available– Most frequent input– Life-critical software– Infeasible test cases– Randomly– ...
Black Box Testing20
Technique: Boundary Value Analysis
• Based on experience / heuristics– Testing boundary conditions of equivalence classes is
more effective
• Choose input boundary values as equivalence classes representatives
• Choose inputs that invoke output boundary values• Examples:
– (0, 10] ⇒ validate using 0, 1, 2, 9, 10, 11
– Read up to 5 elements ⇒ validate reading 0, 1, 4, 5, 6 elements
Black Box Testing21
BVA as an equivalence partitioning extension
• Choose one (or more) arbitrary value(s) in each equivalence class
• Choose valid values exactly on lower and upper boundaries of equivalence class
• Choose invalid values immediately below and above each boundary (if applicable)
Black Box Testing22
General purpose test-suite construction technique
• May be used to obtain reasonable coverage with little effort– Use cautiously!– Unsuitable when values of different fields are related
1. While test cases can be added• Each new test case should include as many un-included valid non-boundary
equivalence class representatives as possible2. While test cases can be added
• Each new test case should include as many un-included valid boundary equivalence class representatives as possible
3. While test cases can be added• Each new test case should include a single invalid equivalence class
representative that has not been included before4. Manually replace/remove redundant or infeasible test-cases
1. While test cases can be added• Each new test case should include as many un-included valid non-boundary
equivalence class representatives as possible2. While test cases can be added
• Each new test case should include as many un-included valid boundary equivalence class representatives as possible
3. While test cases can be added• Each new test case should include a single invalid equivalence class
representative that has not been included before4. Manually replace/remove redundant or infeasible test-cases
Black Box Testing23
Example: Country Club I
DaySunday - ThursdayFriday - Saturday
Guest
status
VisitorMemberStudentVisitorMemberStudent
Age (years)
Admission fee
[0, 16)251020351030
[16, 60)502545702565
[60, 120]351530501545
SpecificationSpecification
Black Box Testing24
Example: Country Club II
FieldValid equivalent
classes
Validrepresentative values
Invalid
equivalent
classes
Invalidrepresentative values
DaySun - ThuMon, Sun, Thu
Fri - SatFri, Sat
Guest status
VisitorVisitor
MemberMember
StudentStudent
Age[0, 16)2, 0, 15Non-numeric value
14@a
[16, 60)34, 16, 59Age < 0 or
Age > 120
-1, 121
[60, 120]100, 60, 120
A combo box is used for choosing the day
and guest status
A combo box is used for choosing the day
and guest status
Black Box Testing25
Example: Country Club IIITest case IDDayGuest statusAgeResult
1MonVisitor225
2FriMember3425
3MonStudent10030
4SunVisitor025
5SatMember1610
6ThuStudent6030
7SunMember1510
8SatStudent12045
9ThuVisitor5950
10MonMember14@aInvalid age
11FriStudent-1Invalid age
12FriVisitor121Invalid age
valid
valid)boundary(
invalid
Recommended