Upload
others
View
7
Download
0
Embed Size (px)
Citation preview
Offline Model-based Testing and
Runtime Monitoring of the Sensor Voting Module
Paolo Arcaini Angelo Gargantini Elvinia Riccobene
Università of Bergamo- Italy
Università di Milano - ItalyTolouse, ABZ 2014
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
Motivation
• For safety critical components, formal verification and validation of models must be combined with the validation of the implementation. • confidence that the system has been implemented as specified, i.e., its
implementation conforms to its requirements.
• regardless the correctness of the model (guaranteed by formal verification, simulation and so on), the implemented system must be validated itself.
• Even when formal refinement and/or program formal verification are applied
“Who would want to fly in an airplane with software proved correct, but not tested?”
Ed Brinksma in his 2009 keynote at the Dutch Testing Day and Testcom/FATES
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
General scenario
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
Model-basedtesting
Runtime Monitoring
requirements
Formal specification
Implementation
Models used for test data generation and verdicts computation (oracle)In off-line, test suites are derived from the model and saved.
During runtime, checking whether a runof a system satisfies or violates a givencorrectness property
Conforms to ?
Proposed validation process
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
ASM model
Java
Req.
1.modelling
5. Offline testing6. Runtime monitoring
3. Implementing
4. Link Implementation and specification
2 Model validation
Applied to the sensor voting module
1. ASM model for SVM
• Using AsmetaL• http://asmeta.sourceforge.net/
• function channel represents the signals coming from the three channelsdynamic monitored channel: Channel −> Boolean
• function validCh keeps track if each channel is still validdynamic controlled validCh: Channel −> Boolean
• the output is sensor and its validity is simply defined as a derived function validdynamic controlled sensor: Boolean
derived valid: Boolean
• RULES to set the value and the validity
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
2. Model validation
1. Simulation• User guided
• Scenario guided• “all the sensors stay valid, the value is what expected”
• “sensor1 becomes invalid and after some steps also sensor2 fails”
2. Formal Property Verification • By model checking
• Once the sensor becomes invalid, then it will always remain invalid:• CTLSPEC AG( not(valid) implies AG (not(valid)))
• There exists a path in which the sensor eventually becomes invalid:• CTLSPEC EF (not(valid))
• There exists a path in which the sensor always remains valid:• CTLSPEC EG (valid)
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
We found at least one errorin our initial specification
3. Write the Java implementation
• The ASM and the Java implementation have been developed independently• once we have agreed upon the interface
• A method to pass the current values of the channels
• A way to get the sensor value and validity
• More freedom (thanks to Java)• Use of generics
• System.out / log4j
• The Java class can contain further methods not interested bythe spec
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
4. Linking a Java class with its ASM model
• We reuse the approach CoMA
Conformance Monitoring by ASMs• Paolo Arcaini and Angelo Gargantini and Elvinia Riccobene CoMA:
Conformance Monitoring of Java programs by Abstract State Machinesin 2nd International Conference on RUNTIME VERIFICATION RV 2011
• Based on the use of Java annotation (@)
• The implementation itself contains the definition of the links to the specification
• The link should be maintained during software evolution
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
4. Linking a Java class with its ASM model
• Given a Java class C and its ASM model 𝐴𝑠𝑚𝐶
• 1. Link the class its specification
@Asm("SensorVotingModule.asm")
class Sensor {
….
}
2. Link the state
Java state Abstract State
3. Link the execution
Java methods Rules
Java ClassC
AsmSpecification
𝐴𝑠𝑚𝐶
link
@Asm
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
4b. Java observed state to ASM signature• Java observed state
OS(C) consists of all public fields, and pure non void public methods of the class C the user wants to observe
𝑙𝑖𝑛𝑘: 𝑂𝑆 𝐶 → 𝑓𝑢𝑛𝑐𝑡𝑖𝑜𝑛𝑠(𝐴𝑠𝑚𝐶)
For the voting module:class Sensor {…@MethodToFunction(“sensor")public boolean getValue(){ return value;}
@MethodToFunction("valid")public boolean isValid(){ return sensorValid;}
Java ClassC
AsmSpecification
𝐴𝑠𝑚𝐶
@FieldToFunction@MethodToFunction
Signature
OBSERVED STATE
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
class Sensor{….@StartMonitoringpublic Sensor() { …}@RunSteppublic void computeSensorValue() {…}
}
4c Java methods to ASM execution
• the execution of the Java code must be linked with an execution(i.e., a run) of the ASM.
• Identify Java changing methodsCM(C)
is the set of methods of C whose execution is responsible for changing values of OS(C) and that the user wants to observe
Java ClassC
AsmSpecification
𝐴𝑠𝑚𝐶
@StartMonitoring@RunStep
Rules
Changing Methods
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
5. Offline testing
• Test generation
Using model checking (Spin) (ATGT) and several coverage criteria to generate “abstract” tests
• Test concretization
The test are “adapted” to concrete test execution
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
ASM model
Java
TESTS
Test generation
Test concretization
5a. Test generation
• ATGT exploits the counter example generation of the model checker SPIN
• ATGT has derived 38 test predicates • 20 for the BRC and 18 for the URC
• Each test predicate represents a test goal or test requirement
• For example
valid = false : the sensor becomes invalid
• ATGT produces • 38 test sequences
• 11 test sequences if tests reduction is applied
• They are enough to cover all the test goals
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
−−−− state 0 −−−−−−− controlled −−valid = true−− monitored −−channel(ONE) = falsechannel(TWO) = falsechannel(THREE) = true−−−− state 1 −−−−−−− controlled −−sensor = falsevalid = true
Example of test sequence
5b. Test concretization
• Test are translated to JUnit code
• Deriving a concrete Java test• A sequence of method calls
• with suitable checks (i.e., asserts)
• novel technique
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
The test concretization leverages the linking between the Java class and the ASM and the definitions of state conformance and step conformance of point 3
ATS - ABSTRACT TEST SEQUENCE Junit @Test
Test translation to JUnit
• @StartMonitoring: create a new Object
• For each state of the ATS, call the method annotated with @RunStep (+ parameters)
• Build the oracle by exploiting the annotations @FieldToFunction and @MethodToFunction.
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
@Test
public void test() {
// state 0
Sensor sut = new Sensor ();
assertEquals(true, sut.isValid());
sut.computeSensorValue(false, false, true);
// state 1
assertEquals(false, sut.getValue());
assertEquals(true, sut.isValid());
}
−−−− state 0 −−−−−−− controlled −−valid = true−− monitored −−channel(ONE) = falsechannel(TWO) = falsechannel(THREE) = true−−−− state 1 −−−−−−− controlled −−sensor = falsevalid = true
6. Runtime Monitoring (CoMa)
1. During the execution of the Java object of class C
2. an observer evaluates when the observed state is changed• Using AspectJ
3. makes the abstract ASM to perform a machine step• Using AsmetaL simulator
4. An analyzer evaluates the step conformance between the Java execution and the ASM simulation
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
ASM model
𝑂𝐶 - Java OBSERVER
SIMULATOR
ANALYZER
1 23 4
OK/FAIL
Runtime conformance
Runtime conformance 𝑂𝐶 is runtime conformant to its 𝐴𝑠𝑚𝐶 if:
1. no specification invariant of 𝐴𝑠𝑚𝐶 is ever violated.
2. the initial state of the computation of 𝑂𝐶 conforms to the initial state of the computation of 𝐴𝑠𝑚𝐶
3. every observed change step conforms with the step of 𝐴𝑠𝑚𝐶
𝐴𝑠𝑚𝐶
𝑂𝐶
Non changing methods steps are not checked
@StartMonitoring @RunStep
conf conf
@RunStep
computation (states)
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
Experimental result – Sensor Voting
• MBT: 38 Junit tests
• CoMA: we have simulated the environment by instantiating 10 times a new sensor and computing 10 times the sensor value passing three random values as inputs for the three channels.
• Evaluation• Code coverage by EclEmma: 100% in both cases
• Mutation score by PIT 57 killed mutants over 74 in both cases
• Both techniques are equivalent regarding detecting faults inserted by the standard PIT mutation operators.
• CoMA has a great runtime overhead (ASM simulator must run in parallel)
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
When runtime monitoring is better?
• we have simulated a delayed short circuit fault that causes isValid to return true after 5 times it is called.
int nvCount = 0;
boolean isValid() { return valid | nvCount++ > 5;}
• The tests produced from the specification do not detect this fault
• the rule coverage of the specification does not imply the coverage of this faulty behavior in the implementation.
• However, monitoring the code with CoMA exposes the failure by any run in which valid becomes false and isValid is called at least 5 times.
• In general, unspecified anomalous behaviors of the implementation are better detected by runtime monitoring than by MBT
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
Conclusions
• A complete ASM-based methodology for the validation of code
• Guidelines/methodology for:• Linking the code and its specification
• Deriving test cases from models
• Performing runtime monitoring
• Lesson learned• Case study very useful for students/practitioners
• Importance of a linking (for both MBT and runtime, but also future use)
• This can limit the use of Java (skeleton generation could be useful)
• Future work: compare w.r.t other approaches like Junit tests written by hand/ JML - Design by Contract/ test generation
ABZ 2014 Angelo Gargantini - MBT and runtime monitoring SVM
-