Upload
rachel-stearman
View
215
Download
1
Tags:
Embed Size (px)
Citation preview
Requirement-based Testing
A Presentation by
Harry Sneedfor the
John von Neumann Society Budapest, 27 May 2009
The many Dimensions of Software Testing
Test Objects• Units (Modules/Classes)• Components• Subsystems• SystemsTest Types• Functional Testing• Performance Testing• Load Testing• Usability Testing
Test Approaches• Random Testing• Creative Testing• Systematic TestingTest Bases• Code-based testing• Interface-based testing• Reference-based testing• Model-based testing• Require.-based testing
Software Units
ComponentsSystems
Functional Specification
(Oracle)
In 1978 Bill Howden introduced the notion of a test oracle in connection with Functional Testing. It means testing the System against the Requirement Specification (functional & non-functional Requirements)
A Test is always a Test against Something
Oracle
CodeTest inputsTest results &
Actual paths
SpecificationExpected results Expected paths
Potential Paths
SpecifiedPaths
compare Results & Paths
Corresponds to a Unit Test (White-Box)
Code-based Testing = Testing the Program against itself
CodeTest inputs Test results
Specification Expected results
compareData
Representative Values
Interfaces/Files/Tables
Specified Results
Interfaces/Databases
Corresponds to an Integration Test (Grey-Box)
Data-based Testing = Testing the Program against the Data
Interfaces/Files/Tables
CodeTest inputs Test results
Specification Expected results
compare
Files/Tables/Panels
Specified Results
Corresponds to a System Test (Black-Box)
Results
Files/Tables/Panels
Specified Inputs
Requirement-based Testing = Testing the Program against the Specs
StaticAnalysis
ExecutionTest
Verf. Cond.Generator
SyntaxAnalysis
SyntaxErrors Semantic Errors Exec Errors Struct Errors
Graph Checking+
Call Checking+
Units Consistency+
Mode Checking+
Asserted/Actual Use
+Set/Use Checking
Test Results+
Execution Errors
+Coverage
Report+
Assertion Exceptions
Annotated Listing
+VC'S
+Symbolic
Execution of Expressions
+Simplified
VC'S
Indented Program Listing
+Diagnostics
Correct CorrectCorrectCorrect
SourceProgram
Programwith Assert
"Correct"Source Prog.
Assertions taken from Specs(RSL)
The Mother of all Test Tools = RXVPRequirement Evaluation & Verification Package
FORTRAN
Dr. Edward Miller from General Research Corporation
The idea was to put dirty software into a washing maschine and get out cleaned software .
P1
P2
P3
P4
Entry
Exit
Test Object = Procedure or Module
Test Case = Path thru Test Object
1. Path
2. Path
3. Path
5. Path
Test Cases were extracted from the Code
4. Path
PreconditionsPostconditions
BeforeState After
State
The Budapester Testlabor
• The Budapester Test Laboratory was set up by Harry Sneed and Dr. Ed Miller from Software Research Inc in cooperation with the SZKI and Szamok in May, 1978. This was the first known attempt to outsource the testing activity.
• The customer was Siemens, Munich, which was developing a large scale, distributed software system for the German Railroad – the Integrated Transport System (ITS).
Test Process in the Test Labor
• Compilable Modules were delivered by Siemens every week on a magnetic tape.
• Sources were statically analyzed to produce flow and structure diagrams.
• Paths were marked thru the flow diagrams with magic markers.
• Conditional variables were assigned values to confirm or negate every decision.
• Values were assigned at test time to the test driver before each test case execution.
• Results were logged to be checked later against the expected results.
Experience with the Test Labor
• In the first half year 128 modules with 60881 SPL statements passed through the test labor.
• 4378 code branches were tested with a coverage of 89.7 % by 1544 test cases.
• 192 program errors were discovered via dynamic analysis.
• 1396 design and code deficiencies were discovered via static analysis.
• The costs of the quality assurance remained below 4 DM pro statement. This was less than 10% of the development costs.
Summary of first 6 Months
Comp Modules Stmts Testcases
Branches Coverage Design &Code Defis
ProgramErrors
A 6 4029 138 5
B 37 7588 232 603 91% 130 22
K 71 40735 1064 2843 87.5% 868 143
N 6 2847 101 140 94% 110 14
S 8 5682 147 792 95% 150 6
Total 128 60881 1544 4378 91.9% 1396 190
Resume on Module Testing
• In one year 281 Modules with 134 K stmts were tested with 3864 test cases at a cost of DM 75,- per test case = DM 289.800
• The minimum branch coverage was 85%• 403 Errors were reported at a cost of DM 150,- per error =
DM 60.450• Later in the system test by the customer, another 386
errors were uncovered.• The intensive module test had uncovered only 51% of the
total errors found.• This demonstrates the fact that module testing of the code
against itself is not sufficient. The software as a whole must be tested against the requirements.
StoreMeasure
R-Net
ReceiveSignals
ASSERT
ASSERT
ReportEngine ok
ASSERT
CheckMeasure
ReportSignalError
ASSERT
sendAlarm
ASSERT
Invalid Measure Valid Measure
Validation Points
MonitoringEngine
Safe Range Out of safe Range
RXVP progressed to test the systems against Requirements Nets
ReceiveSignal
Measuringdevice
&
StoreMeasure
Message Log
CheckMeasure
Report ok AlarmNurse on Duty
AlarmDisplay
ReportMeasure Error
SystemDisplay
3. test case
2. test case
1. test case
R-Net-Entry
invalid MeasureValid Measureand
or
or Engine problemEngine ok
Engine Monitoring System
Requirement Specification
with RSL
Extracting Test Cases from Requirement Nets
Action DescriptionIf Account_Balance minus
the withdrawal is morethan the credit limitallow the withdrawal
In case the withdrawalcauses the Account_Balanceto fall below the creditlimit refuse the
withdrawal.
Account_Balance minus the withdrawalis more than the credit limitWithdrawal causes the Account_Balanceto fall below the credit limitAllow the withdrawal
Refuse the withdrawal
Test Case 1 2 3 4
Y Y N N
Y N Y N
- X - -
X - X -
Service Function Test Case Generation
Conditional keywords arescanned.
Each Condition is treatedseparately and combinedwith the other conditionsin the same action to givethe number of testcases = nr_conditions2
Extracting Test Cases from Business Rules
One Use Case can have one or more Test case
Normally, there are at least 2 Test Cases
Positive outcome
Negative outcome
CashWithdrawal
Not enoughcash
Accountoverdrawn
Cashdelivered
WrongPIN Number
InvalidCard
Use Case =
Test Cases = Use Case Paths
Extracting Test Cases from Use Case Descriptions
Test Cases
Functional
Non-Functional
Objects
Actions
States
Conditions
Requirements Text
Extracting Test Cases from Requirement Documents
Actions upon objects:The customer account should be deleted.The invoice should be sent within 30 days.
Object states:The customer account is overdrawn.The customer is an employee.
Rules:If the account is overdrawn a notice should be sent.Whenever someone fails to pay within 30 days, a reminder should be sent.Customer accounts should be deleted, if they are overdrawn more than 3 months.In case of a system crash, the recovery routine should be started.
Grammar types indicating Declarations to be tested
[RF 41] Notification can be divided into action and non-action (=information) notification.
[RF 42] In case of an “action needed” notification, the system provides a link to the place where the action is needed.
[RF 43] Notifications are added to a notification interface, which contains information aboutThe reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification
If action needed: A link to the place where the action is needed.
[RF 44] The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”. If the action is finished, the user name is stored by the notification.
[RF 45] If at least one provider changed his feed data e.g. changing starting time, a notification is added to the notification table (see [RF 43]) so that each user can see the notification. The notification is only added to the notification table if the data is unequal to the current event data (see also [RF 30]).
Sample Requirements Text
TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0163 Require RF_43 perform action Notifications are added to a notification interface, which contains information about:
The reason of the notificationThe kind of the notificationThe type of the notificationThe date/time of the notification
DFI0164 Require RF_43 checks rule If action is needed a link to the place where the action is needed.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0165 Require RF_44 perform action The user is able “take” a notification (locked for others), do the action and mark the notification as “finished”.DFI0166 Require RF_44 checks rule If the action is finished, the user name is stored in the notification.--------------------------------------------------------------------------------------------------------------------------------------------------------TestCase-Id BaseType Requirement Relation TestType TestCase Description--------------------------------------------------------------------------------------------------------------------------------------------------------DFI0167 Require RF_45 checks rule If at least one provider changed his feed data changing starting
time, a notification is added to the notification table (see [RF_43]) so that each user can see the notification.
DFI0168 Require RF_45 checks rule The notification is only added to the notification table if the data is unequal to the current event data (see also [RF_30]).DFI0169 Require RF_45 checks rule The user choose Provider A as primary source, so the event time is the time of Provider A.DFI0170 Require RF_45 checks rule If Provider B updates his time to the time of the event, no notification is added.DFI0171 Require RF_45 checks rule If Provider B sends a time different to the event time, a notification is sent.
Test Cases extracted from the Requirement Text
The order entry system must be able to handle at least 1000 customer web pages at one time. A customer order must be processed within 2 seconds. The article database may contain up to 10.000 articles. The system should be able to produce as many as 25.000 bills in one billing run. A billing run should not take more than 2 hours. If the system crashes it should be back up and in operation within 15 minutes. Database backups should take place every 4 hours. Customers should be checked both by number and by password. Passwords should be rejected after 3 tries, customer number should be rejected after 2 tries.
Sample Non Functional Requirements
1. 999 customer orders at one time
2. >1000 customer orders at one time
3. response time per order < 2 sec.
4. response time per order >2 sec.
5. 9.999 article records
6. > 10.000 article records
7. 24.999 bills in one billing
8. >25.000 bills in one billing
9. run time of billing < 2 hours
10. run time of billing >2 hours
11. If system crash, recovery < 15 minutes
12. If > 4 hours, database secured
13. Submit 3 passwords where 3rd password is correct
14. Submit 3 passwords where all 3 are incorrect
15. Submit 2 customer number where 2nd number is correct
16. Submit 2 customer number where both are incorrect
Typical Non-Functional Test Cases
A Test Case:
has a unique identifier serves a purpose refers to a Testobject refers to a requirement or use case has a precondition state has a postcondition state has a set of required input parameters has a set of expected results has a predecessor test case has a given environment has an owner has a status has a date of execution
Properties of a Test Case
An international standard is required for the minimumProperties of test cases!!!
TestCase ID: IC-Order-7Test Case – Purpose: to test a fulfilled orderTest-Object: Customer – OrderPrecondition: Ordered Article is on stock.
Article. Amount is > Order.Position.Amount(Article. Amount - Order. Position.Amount ) > minimum – Amount
Postcondition: Article.Amount = a Pre.Article.Amount – Order.Position.AmountNo supply order
Inputs: Order.Cust-Nr = 200100Customer.Credit = ’A’Order.Position-Nr = 2Order.Position.Art – Nr = 4711Order.Position.Amount = 3Article.Amount = 10Minimum – Amount = 7
Results: Text-Msg = „Article order fulfilled”Environment: MS – Internet, Windows – XPUsecase tested: Customer.Order.ProcessingStatus: testedDate: 29. 8. 2005.
Functional Test Case Specification
TextAnalysis
TestCases
RepositoryLoader
Test CaseDB
RepositoryUnloader
TestScriptsTest
Testerenhances & refinesTest Cases
TesterexecutesTest Cases
Text AnalyzergeneratesTest Cases
Test Script Generatorgenerates Test Scriptsfor Test Automation
From Requirements via Test Cases to Test Scripts
Text
Processor
TxtAnalTextPre
Processor
TxtScan
Object
List
ObjectModel
TestCases
RuleViolations
SpecMetrics
RepositoryEntries
Objects- Use Cases - Actions - States - Rules
TextExcelXML
Format RulesRupp Rules
SizeQuantityComplexity
Sections- Use Cases - Objects - Actions - States - Rules
TextAnal = Tool for extracting Test Cases from English and German Language Documents
Requirements Analysis Demo
• Bet&Win Requirement Document• Setting Key Words• Scanning for Objects• Analyzing the Document• Extracting Test Cases• Collecting Requirement Metrics• Documenting Cross References• Generating a test design
Egovernment Web Site with three applications and circa 33 K Java Stmts
1844 Test cases were specified of which 1272 were generated
1495 Test cases were executed = 81% Test Case coverage
352 Test cases discovered at least one error = 24% Hit rate
450 Errors were reported by testers = 13 per KDSI
56 Errors were reported by users after release = 11% of all errors
System test discovered 89% of all reported errors = test efficiency
System test lasted 4 months from January to May 2005
392 Test days were booked against the project
= 3.8 Test cases per Testertag
= 1.1 Errors reported per tester day
Customer was satisfied with the test result.
Egovernment Web Site with three applications and circa 33 K Java Stmts
1844 Test cases were specified of which 1272 were generated
1495 Test cases were executed = 81% Test Case coverage
352 Test cases discovered at least one error = 24% Hit rate
450 Errors were reported by testers = 13 per KDSI
56 Errors were reported by users after release = 11% of all errors
System test discovered 89% of all reported errors = test efficiency
System test lasted 4 months from January to May 2005
392 Test days were booked against the project
= 3.8 Test cases per Testertag
= 1.1 Errors reported per tester day
Customer was satisfied with the test result.
Experience with Requirement-based Testingof the Public Internet Portal for the state of Saxony
Conclusions
• Test methods have not really improved since the 1970s.
• We are still using the basic approaches introduced back then.
• Test tools have however improved significantly.• Today there are more and better testing tools.• The future lies in test automation.• Only through automation can we really reduce test
costs and increase the test ROI.• Test ROI = (TestBenefits – TestCosts) / TestCosts