Upload
lorraine-cobb
View
214
Download
1
Tags:
Embed Size (px)
Citation preview
15 Nov 2005 Jones - IASTED/SEA 1
Automated Automated Support for Support for Test-Driven Test-Driven
SpecificationSpecificationEdward L. JonesEdward L. Jones
IASTED SEA ConferenceIASTED SEA Conference
Phoenix, ArizonaPhoenix, Arizona
October 14-16, 2005October 14-16, 2005
15 Nov 2005 2Jones - IASTED/SEA
OutlineOutline
Test-Driven Specification ParadigmTest-Driven Specification Paradigm Background Background An ExampleAn Example Automation SchemeAutomation Scheme Future WorkFuture Work
15 Nov 2005 3Jones - IASTED/SEA
The Paradigm: Test Driven The Paradigm: Test Driven SpecificationSpecification
Test Cases
+
Assess Test Cases
SpecificationAssess
Specification
Software Development
Software Testing
Time
15 Nov 2005 4Jones - IASTED/SEA
BackgroundBackground
Early Attention to Testing is GoodEarly Attention to Testing is Good V-Model of Software Testing (planning)V-Model of Software Testing (planning) W-Model of Software Testing (verification)W-Model of Software Testing (verification) Test-Driven Development (coding phase)Test-Driven Development (coding phase)
Requirements/Specification remain the hard part of software Requirements/Specification remain the hard part of software engineeringengineering
Verification options – prototyping, testing, model checking, Verification options – prototyping, testing, model checking, automated test case generationautomated test case generation
15 Nov 2005 5Jones - IASTED/SEA
Specifics of Our WorkSpecifics of Our Work
Black-box: focus is software functionBlack-box: focus is software function Specification Language = Decision TableSpecification Language = Decision Table Decision table-based test adequacy Decision table-based test adequacy
criteriacriteria Automated conversion of decision table Automated conversion of decision table
to analyzer programto analyzer program Data-driven verification to reveal model Data-driven verification to reveal model
anomalies and test data inadequaciesanomalies and test data inadequacies
15 Nov 2005 6Jones - IASTED/SEA
Decision TablesDecision Tables
Logic based, cause-effect specificationLogic based, cause-effect specification Within family of state-based specificationsWithin family of state-based specifications Old technology from 1950s-70sOld technology from 1950s-70s
Automated programming systemsAutomated programming systems Basis for software testingBasis for software testing
Specification language for rule based Specification language for rule based systems systems Vanthienen’ Prologa tool, other decision Vanthienen’ Prologa tool, other decision
management tools for decision sciencemanagement tools for decision science
15 Nov 2005 7Jones - IASTED/SEA
An Example SpecificationAn Example Specification
Calculate employee pay, including overtime paid at 1.5 times the hourly rate of hourly employees for qualifying hours. The normal work week is 40 hours; the maximum number of paid hours is 80. Salaried employees earn over $30 per hour, and are paid overtime only for hours in excess of 60 hours. Salaried employees working less than 40 hours are paid for the normal work week. Hourly employees earn less than $30 per hour, and are paid for all hours worked up to the maximum; they are paid overtime for hours in excess of 40.
15 Nov 2005 8Jones - IASTED/SEA
Stimulus-Response View of Stimulus-Response View of Specification Specification
Calculate Pay
hours
rate
regular_pay
overtime_pay
15 Nov 2005 9Jones - IASTED/SEA
Decision Table for Decision Table for Specification (Incomplete)Specification (Incomplete)
---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ----------
ConditionsDecision
Rules
ActionsAction Rules
Computation
Rule
15 Nov 2005 10Jones - IASTED/SEA
Decision Table based Decision Table based Adequacy CriterionAdequacy Criterion
Possible adequacy criteria w.r.t. Possible adequacy criteria w.r.t. conditions, actions, rulesconditions, actions, rules
Functional CoverageFunctional Coverage Measures the thoroughness of testing Measures the thoroughness of testing
w.r.t. decision table rules w.r.t. decision table rules
FC = FC = # computation rules satisfied # computation rules satisfied
# computation rules in DT # computation rules in DT
15 Nov 2005 11Jones - IASTED/SEA
Automated CheckingAutomated Checking
Specification anomaliesSpecification anomalies Ambiguous rulesAmbiguous rules - - test tuple satisfies test tuple satisfies
multiple rulesmultiple rules Missing ruleMissing rule -- -- test tuple satisfies no ruletest tuple satisfies no rule
Test data set adequacyTest data set adequacy Superfluous test data – rule satisfied by Superfluous test data – rule satisfied by
multiple test tuplesmultiple test tuples Incomplete test data – rule not satisfied Incomplete test data – rule not satisfied
by any test tupleby any test tuple Computation rule defines test oracle Computation rule defines test oracle
15 Nov 2005 12Jones - IASTED/SEA
AutomationAutomation
Build Analyzer
Generate Analyzer
Decision TableRun
Analyzer Test Data Set
TestCoverage
Report
SpecCoverage
Report
15 Nov 2005 13Jones - IASTED/SEA
DT Version #1DT Version #1
---------------------------- ---------- hours > 40 | N N rate >= 30 | Y N ----------------------------- --------- overtime_pay = 0; | X X regular_pay = hours * rate; | X regular_pay = 40 * rate; | X ---------------------------- ----------
Missing Rules
15 Nov 2005 14Jones - IASTED/SEA
Analysis Reports - Analysis Reports - Version #1Version #1
TEST SET
hours rate 39 9 41 29 40 30 45 40
SPEC COVERAGE REPORT
VALID 35 9 315 0MISSING 41 29 MISSING MISSINGVALID 40 30 1200 0MISSING 45 40 MISSING MISSING
TEST COVERAGE REPORT
Rules -->Test -- --Case 1 2==== -- -- 1 X 2 3 X 4RULES NOT SATISFIED:FUNCTIONAL COVERAGE = 2/2
15 Nov 2005 15Jones - IASTED/SEA
DT Version #2DT Version #2------------ ------------------------------------------N N Y Y - - | hours > 40- - - - Y - | hours > 60- - - - - Y | hours > 80N Y N Y Y Y | rate >= 30------------ ------------------------------------------X - - - - - | regular_pay = hours * rate;- X X X X X | regular_pay = 40 * rate;X X - - - - | overtime_pay = 0;- - X - - - | overtime_pay = 1.5 * rate * (hours - 40);- - - X X X | overtime_pay = 1.5 * rate * (hours - 60);------------ ------------------------------------------
TEST SET hours: 35 41 40 45 70 90 rate: 9 29 30 40 40 50
15 Nov 2005 16Jones - IASTED/SEA
Analysis Reports - Analysis Reports - Version #2Version #2
TEST COVERAGE REPORT
Rules -->Test -- -- -- -- -- --Case 1 2 3 4 5 6==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X X
RULES NOT SATISFIED:FUNCTIONAL_COVERAGE = 6 / 6
SPEC COVERAGE REPORT
VALID 35 9 315 0VALID 41 29 1160 43.5VALID 40 30 1200 0VALID 45 40 1600 -900AMBIGUOUS 70 40 1600 600AMBIGUOUS 70 40 1600 600AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250
15 Nov 2005 17Jones - IASTED/SEA
DT Version #3DT Version #3------------ ------------------------------------------N N Y - - - | hours > 40- - - Y Y - | hours > 60- - - - N Y | hours > 80N Y N Y Y Y | rate >= 30------------ ------------------------------------------ X - - - - - | regular_pay = hours * rate;- X X X X X | regular_pay = 40 * rate;X X - - - - | overtime_pay = 0;- - X - - - | overtime_pay = 1.5 * rate * (hours - 40);- - - X X X | overtime_pay = 1.5 * rate * (hours - 60); ------------ ------------------------------------------
TEST SET hours: 39 41 40 45 70 90 rate: 9 29 30 40 40 50
No similar rule for salaried employee.
15 Nov 2005 18Jones - IASTED/SEA
Analysis Reports - Analysis Reports - Version #3Version #3
TEST COVERAGE REPORT
Rules -->Test -- -- -- -- -- --Case 1 2 3 4 5 6==== -- -- -- -- -- -- 1 X 2 X 3 X 4 X 5 X X 6 X X
RULES NOT SATISFIED:FUNCTIONAL_COVERAGE = 6 / 6
SPEC COVERAGE REPORT
VALID 35 9 315 0VALID 41 29 1160 43.5VALID 40 30 1200 0MISSING 45 40 MISSING MISSINGAMBIGUOUS 70 40 1600 600AMBIGUOUS 70 40 1600 600AMBIGUOUS 90 50 2000 2250AMBIGUOUS 90 50 2000 2250
15 Nov 2005 19Jones - IASTED/SEA
ConclusionConclusion Have presented a black-box approachHave presented a black-box approach
Test-driven specification as a paradigm Test-driven specification as a paradigm
for interaction between developer and for interaction between developer and
testertester
Demonstrated benefits of automationDemonstrated benefits of automation
Claim that approach is applicable to Claim that approach is applicable to
existing decision based specification existing decision based specification
methodsmethods
15 Nov 2005 20Jones - IASTED/SEA
Future WorkFuture Work
Extend to State-Based SpecificationsExtend to State-Based Specifications Mill’s State-Box modelMill’s State-Box model SCR state tablesSCR state tables Pre-/Post-condition specificationsPre-/Post-condition specifications
Consider random generation of test Consider random generation of test data used to verify specificationdata used to verify specification
Use tool in software engineering Use tool in software engineering coursescourses
15 Nov 2005 22Jones - IASTED/SEA
Motivation / ReferencesMotivation / References
Hoffman & Strooper’s 2001 paperHoffman & Strooper’s 2001 paper
“Prose + Test Cases = Specification”
Pamela Zave’s IEEE-TSE 1991 paper, a post-mortem on Pamela Zave’s IEEE-TSE 1991 paper, a post-mortem on her 1980’s work on the Paisley operational specifications her 1980’s work on the Paisley operational specifications system system
“ “An Insider’s View of Paisley”An Insider’s View of Paisley”