Upload
abigail-mcbride
View
218
Download
0
Embed Size (px)
Citation preview
Balancing Practices: Inspections, Testing, and Others
JAXA scenario (formal method)
Masa KatahiraJapanese Space Agency
Strategy to select methods
• Methods Compliment– Inspection/Review– Testing– Formal Method
• Theorem Proving • Auto Model Checking
• Inspections/Reviews– Hard to cover all aspects
• Testing– Not complete, too late in some case
• Formal Method– TP: Complex for practical use– MC: State explosion possible
• We realize that – the correct use of particular methods, – the combination of several methods
are very important.• But how?
– Quality Goals
– Budget Limitation
– System Characteristics
– Data availability, Development Phase
Selection and Scalability of Methodologies(sample)
Completeness/Consistency
Selection (Depth)Selection (Depth)Light
Full set Phases
Modeling/Model CheckingInspection/Review
(Check List)
Simulation
Interface Validation
Design Coverage & Timing
Verification Coverage
Auto Test Case Generation &
Robustness EvaluationTest Case & Test
Test Result Review
Compliance/Traceability
Risk Analysis (Robustness)
Process & Quality
Static analysis(Problem Reports)
In line Process Monitor (SMIP)
Manual Check(Tools Support)
Auto Equivalency Check
Hazard Analysis/SFMEA
REA
Ass
essm
ent A
ttri
bute
s A
sses
smen
t Att
ribu
tes
(Sam
ple)
(Sam
ple)
0% 20% 40% 60% 80% 100%
A
B
C
D
E
Review Model Checking Review with Checklist
Fig.1 Each methods’ effectiveness among all significant issues
A
B
C
D
E
A
B
C
D
E
Fig.2 Each methods’ effectiveness among all Editorial Errors
Fig.3 Each methods’ effectiveness among all Significant issues and Editorial errors
ConsiderationConsideration
For projects A,B,C, there is not enough time to perform model checking. Review with check list instead. For Project D, a checklist is useful for the data correctness and consistency of data handling system.For Project E, the effective of model checking is confirmed due to having enough time.
Review also shows important role in erroneous description in the specificationFor design phase such as C,E, model checking and a checklist help finding errors which can not be found by review.
For projects A,B,C, there is not enough time, and the review with check list shows efficiency.For Project D, Review as well as review with checklist shows the efficiency for data handling system.
For such as Project E case, when enough time is assigned, the model checking shows good results.
Lessons Learned Summaryin JAXA case study
Method Advantage Disadvantage
Formal Model/Model Checking
●It is useful to find the problem concerning the complicated state/mode transition and processing timing issues which is hard to be found by manual.●Erroneous description in the spec. at modeling which is more effective than normal review.
●A certain mount of time are necessary to modeling and model checking.●Need to know modeling and model checking knowledge.●Low cost effectiveness for software which does not have complicated logic such as data handing, or transformation
Review withChecklist (Inspection)
●High cost effectiveness even if there is very short time to access it.●Erroneous descriptions are covered by check items in the list.●It does not depend on the skill of evaluators than model methods.
●There are limited based on the items in the checklist. ●It is hard to check the detailed behaviors in the complex system and to cover the possible combination.
Boundary of Formal Method Application
Available man-month
Safe
ty C
riti
cal
Syst
em C
hara
cter
istics
Compl
exity
Important Border
Needs and remaining issues of formal method
[Problem Statement]• Need to assure the high reliability of spacecraft• Facing to the difficulty to prove the goodness only by test and inspection
because of system complexity and safety requirements such “must not work”
• Large number of defects are introduced mostly at the Requirement Phase or the Early Design Phase. Unintended or Unexpected system/software behavior is difficult to be found at the inspection/review by manual.
[Challenge]• Knowledge base inspection/review is still very important, but model
checking gives a chance to detect important findings which are easy to miss by the reviewer.
• Modeling task itself gives a chance to think enough deeply what the specification really says as if the reviewers build the software by themselves.
[Remaining Issues]• Quality of Model and model checking task
– Large amount of time is spent to correct the erroneous model• Abstraction and partitioning techniques
– To avoid state explosion, and missing the scenarios• Better Productivity
– Hard to find real problems from thousands of auto checking results• Personnel skill
Modeling and model checking in JAXA
Requirement Model
(SpecTRM*1, Uppaal)
DesignModel
(SpecTRM, Uppaal)
Req. Spec.ICDHazard Report
Design Spec.ICDHazard Report
Modeling
Natural Language Input Tools
Natural Language Input Tools
UppaalUppaal
Flow DiagramToolFlow DiagramTool
Model Checking(Static Analysis)
CompletenessConsistencyReachability
CompletenessConsistencyReachability
EquivalencyEquivalency
SMV, SPINSMV, SPIN
Executable Code
Simulation(Dynamic Analysis)
RobustnessRobustness
Behavior AnalysisBehavior Analysis
FindingsFindings
Findings
Test Case Proposal
Operation Model
ProcedureOpe. Scenario
ConsistencyTask Model ToolTask Model Tool
*1:SpecTRM: Specification Tools and Requirements Methodology
Direct Modeling
Productivity improvement
0
2
4
6
8
10
12
14
DirectModeling
NaturalLanguage
Model
Flow ChartModeling
Modeling Task Cost
Real issues from the results of the modeling and model checking
0 0.2 0.4 0.6 0.8 1
Proj A/ Req.
Proj A/ Preli Design
Proj A/ Detail Design
Proj B/ Req.
Proj B/ Preli Design
Proj B/ Detail Design
Proj C Design
Modeling Task
Consistency Analysis
Completeness Analysis
SPIN
Identified issues in the specification
• Modeling – Can organize information and execute the modeling in the brain – Identify lots of basic problems in the specification (ambiguous
descriptions, inconsistency of the contents, unclear data definition) as to make the accurate model of the specification in the formal language
• Automated Consistency analysis– Effective to identify the inconsistency in the requirement specification– Identify the inconsistency among the procedures in case that multiple
tasks executions are allowed simultaneously in the design level. • Automated Completeness analysis
– Check whether the nodes after the transition at the branch in the flowchart meet the number of the transition conditions and its contents, and whether all error handling and exceptional procedure are covered in the design level.
• Formal Validation of the functional behavior using SPIN – Effective to validate whether the procedures are executed without
stagnation and those behavior meet the requirements for the procedure flow in the detailed design
– Effective to verify whether hazard control function/failure recovery functions are working without unintended stop in the real time
Lessons Learned from industry use of modeling and model checking
Questions? (Formal Method)A) What is the role of formal method (Theorem Proving, Model
checking etc.) in many quality practices?
B) When is a Formal Method necessary or efficient?
C) What is a Formal Method useful for? Specific Aspects?
D) What are most important research issues to deploy the method into real projects? Industry Needs?
E) What empirical data gathered at the industry will be useful to future research?
F) What is an expected benefit from use of formal method?
Findings from each methods(Spacecraft’s Projects)
Significant: Signification Issues to be modified such as incorrect or missing functions/logic/data
Editorial: Editorial Errors in the specification
No Issue: Non real problem (misunderstanding/modeling mistake)
Type of system
Findings type and number from each methods
Review Formal Modeling and Model Checking
Review with Checklist
Significant Editorial No Issue Sum
System A(Controller) /Req.
0 1 0 1 1 1 0 2 2 2 1 5
System B(Controller) /Req.
8 4 4 16 4 2 1 7 9 2 1 12
System C(Controller)/Design
0 0 2 2 1 0 1 2 1 5 4 10
System D(DataHandling), Design
0 24 34 58 2 0 3 5 4 0 5 9
System E(Controller)/Design
2 0 27 29 3 13 8 24 2 0 32 34
Significant Editorial No Issue Sum Significant Editorial No Issue Sum
SpecTRM (Model) Based Robustness Test Environment (SpecRobusT)
Outline:• By using specification models, the important test cases are generated for full software simulation during
development contractor’s test phase automatically and comparing results.• Especially, all inputs are verified in the model to generate the test cases.• Auto tests are performed at 10,000 – 100,000 cases / sec.Results of Project application:• # of Test Case : 550,870,000,000 • Benefits:
– Verification at very early phase – Introduction to automated test environment – Introduce “Test Before Development” paradigm into development process
Implementation Procedure