Upload
others
View
0
Download
0
Embed Size (px)
Citation preview
Model-‐Based Tes,ng: An Introduc,on Mohammad Mousavi
Halmstad University, Sweden
Halmstad Summer School on Tes,ng, June 2013
Based on joint work with: H.R. Asaadi, R. Khosravi, N. Noroozi and T.A.C. Willemse
Outline
• Overview • Mo,va,on: Tes,ng a Funds Transfer Switch • Theory • Open Problems
2
Program: June 3
3
09:15-09:30 Welcome, opening remarks (Walid Taha, Halmstad) 09:30-10:30 Introduction to Model-Based Testing (Mohammad Mousavi, Halmstad) 10:30-11:00 Coffee Break 11:00-12:30 Industrial-Strength Model-Based Testing and its Methodology, Part I (Jan Peleska, Verified and University of Bremen) 12:30-14:00 Lunch Break 14:00-15:30 Industrial-Strength Model-Based Testing and its Methodology, Part II (Wen-Ling Huang, University of Bremen) 15:30-16:00 Coffee Break 16:00-17:00 Closing the V - by going from V to DEL (Tony Larsson, Halmstad)
Program: June 4
4
09:00-10:30 Testing Concurrent Java Programs, Part I (Corky Cartwright, Rice) 10:30-11:00 Coffee Break 11:00-12:30 Testing Concurrent Java Programs, Part II (Corky Cartwright, Rice) 12:30-14:00 Lunch Break 14:00-15:30 Testing and Verifying Software Properties with ACL2 and ProofPad, Part I (Rex Page, Oklahoma) 15:30-16:00 Coffee Break 16:00-17:30 Testing and Verifying Software Properties with ACL2 and ProofPad, Part II (Rex Page, Oklahoma)
Program: June 5
5
09:00-10:30 Accurate Programming Using ScalaCheck (Walid Taha, Halmstad) 10:30-11:00 Coffee Break 11:00-12:30 Hands-on ScalaCheck (Rickard Nilsson, Lund) 12:30-14:00 Lunch Break 14:00-15:30 Property-based testing with QuickCheck, Part I (John Hughes, QuviQ and Chalmers) 15:30-16:00 Coffee Break 16:00-17:30 Property-based testing with QuickCheck, Part II (John Hughes, QuviQ and Chalmers)
System development: A planned experiment
Activity and Tools
Reasoning Techniques
Hypotheses (Models)
Unit Programming
Unit Integration
System Integration
Programming languages
Frameworks, Middleware
Protocols,
Deductive Reasoning Assertions
Contracts
Behavioral models
ConcJUnit, Inductive Reasoning ScalaCheck,
Qcheck
ProofPad
RT-‐Tester
Outline
• Overview • Mo,va,on: Tes,ng a Funds Transfer Switch • Theory • Open Problems
7
Tes,ng a Funds Transfer Switch
8
Complex Architectures
9
Thread Pool Integra,on Services
Session Management
Object Cache
Logging
Database Domain Logic
Security
Persistence Frameworks
Transac,on Management
10
Database
Thread Pool Integra,on Services
Session Management
Object Cache
Logging
Domain Logic
Security
Persistence Frameworks
Transac,on Management
Thread Pool Integra,on Services
Session Management
Object Cache
Logging
Domain Logic
Security
Persistence Frameworks
Transac,on Management
Thread Pool Integra,on Services
Session Management
Object Cache
Logging
Domain Logic
Security
Persistence Frameworks
Transac,on Management
11
The System Under Test
• An Opera,onal EFT Switch • Java Applica,on (∼100 KLOC) • Extensive use of Java frameworks
• Already tested, but not with a disciplined view on concurrency
12
Netw
ork Manger
Message M
anager
Flow M
anager
Service Layer
Com
ponent Layer
ProtocolToIFxHandler
Message Binder
Authorization
Message Processor
IFXToProtocolHandler
Exception Handler
Transaction Service
Terminal Service
Cell Charge Service
Settlement Service
DAO Service
Financial Entity Service
Reversal Handler
Settlement
Electronic Funds Transfer (EFT)
EFT Switch
Interbank Network
EFT Switch
Core Banking System
13
Example Scenarios
EFT Switch Core Banking System
POS Terminal
purchase request
purchase request
purchase response
purchase response
14
Example Scenarios
EFT Switch Core Banking System
POS Terminal
purchase request
purchase request
purchase response purchase resp.
reversal response reversal request
15
ISO 8583 Standard
Message Flows
16
ISO 8583 Standard
Transac:on Flows
17
ISO 8583 Standard
Business Rules
18
Concurrent Transac,ons
Purchase Transac,on #1
POS #1 Reversal Transac,on #2
Purchase Transac,on #2
POS #2
One LTS instance per transac:on in the Switch 19
Our Method
• Using Model-‐Based Tes,ng
• Using a set of integrated tools – Genera,ng and running test cases – Logging and priori,zing test cases – Measuring test coverage – Tes,ng business rules
20
Model-‐Based Tes,ng • Modeling the desired behavior (system) / possible interac,ons (environment)
• Deriving test cases from the model rcv_pos_req?
snd_core_req! snd_pos_resp!
rcv_core_resp? ,me_out?
21
Model-‐Based Tes,ng
• Abstrac,ons from reality • Separa,ng different concerns • Approxima,ng system behavior and / or its environment
22
– Restric,ng environment interac,ons – Simpler than actual system – Easier to verify
Equivalence By Observa,on
rcv_pos_req?
snd_core_req! snd_pos_resp!
rcv_core_req? ,me_out?
snd_core_req! snd_pos_resp!
rcv_core_req? ,me_out?
rcv_pos_req?
23
Outline
• Overview • Mo,va,on: Tes,ng a Funds Transfer Switch • Theory: What is Observa,on? • Open Problems
24
a
b Next
25
a
a
b Next
26
b
a
b Next
27
a
b Next
28
Completed Trace Equivalence
I ≈ctr S
traces(I) = traces(S)
c_traces(I) = c_traces(S)
29
a
c
a
b c b
a
30
a
c
a
b 0 1
a b c Next
31
a
a
c
a
b 0 1
a b c Next
32
a
a
c
a
b 0 1
a b c Next
33
a
c
a
b 0 1
a b c Next
34
a
c
a
b c b
a
35
a a
c
a
Environment System
b b
What we did using switches
36
Tes,ng Equivalence
I ≈te S
for every environment E:
traces(E || I) = traces(E || S)
c_traces(E || I) = c_traces(E || S) 37
a
c
a
b 0 1
a b c idle
Next
38
a
a
c
a
b 0 1
a b c idle
Next
39
a
a
c
a
b 0 1
a b c idle
Next
40
a
c
a
b 0 1
a b c idle
Next
41
a
c
a
b 0 1
a b c idle
Next
42
c
a
c
a
b 0 1
a b c idle
Next
43
a
c
a
b 0 1
a b c idle
Next
44
a
c
a
b 0 1
a b c idle
Next
45
a
c
a
b 0 1
a b c idle
Next
46
a
c
a
b 0 1
a b c idle
Next
Now, we know that we have completed a trace.
47
a
a
c
a
Env
S1
b
b
θ
c c b
a
S2 occurs if no other transition can
48
a
Env
S1
b
a
c
a
b
θ
c c b
a
S2
c_traces(Env ⎤⎪S1) = {ab, aθc}
c_traces(Env ⎤⎪S2) = {ab} 49
Refusal Equivalence
I ≈rf S
for every environment E:
traces(E ⎤⎪ I) = traces(E ⎤⎪ S)
c_traces(E ⎤⎪ I) = c_traces(E ⎤⎪ S) 50
And there is more!
51
You are here!
The Linear Time – Branching Time Spectrum Rob van Glabbeek
Defining specifica,ons at a higher-‐level
c b
a
Specifica,on
a 0 1
a b c idle
Next
a 0 1
a b c idle
Next
Impl. 1
Impl. 2
52
Tes,ng Pre-‐order
I te S
for every environment E:
traces(E || I) ⊆ traces(E || S)
c_traces(E || I) ⊆ c_traces(E || S) 53
Restric,on to Specifica,on
Feature #1 Par,al Specifica,on
a 0 1
a b c idle
Next Full Implementa,on
c b
a a
b
a
c
a
b
Feature #2 Par,al Specifica,on
Feature #3 Par,al Specifica,on 54
Restric,on to Specifica,on
I conf S
for every environment E:
traces(E || I) ∩ traces(S) ⊆ traces(E || S)
c_traces(E || I) ∩ traces(S) ⊆ c_traces(E || S)
55
I/O Transi,on Systems
Dis,nguishing between input and output ac,ons
!error !response
?request
?response
!request
56
Pre-‐orders on I/O transi,on systems
• The same no,ons apply here – I/O test pre-‐order – I/O refusal pre-‐order
ior for every environment E:
traces(E ⎤⎪ I) ⊆ traces(E ⎤⎪ S)
c_traces(E ⎤⎪ I) ⊆ c_traces(E ⎤⎪ S)
I S
57
I/O Conformance
Informally, I/O Conformance = I/O Refusal restricted to specifica,on traces
Feature #1 Par,al Specifica,on
a
0
1
a b c
idle
Next
Full Implementa,on
c b
a
a
b
a
c
a
b
Feature #2 Par,al Specifica,on
Feature #3 Par,al Specifica,on
I ioco S
58
Black-‐box tes,ng for ioco
c b
a
Specifica,on
a 0 1
a b c idle
Next
Implementa,on
a
b
Test Case
59
Example
?b
!a
?b
?b
?b
!a
?b
τ
!b
fail pass pass
θ
θ θ
?a
?a ?a
60
ioco Test Cases
• I/O transi,on systems • The only terminal states: and • Reversed I/O ac,ons • Special ac,on θ • Finite and determinis,c !b
fail pass pass
θ
θ θ
?a
?a ?a
pass fail
61
Automa,c Test Case Genera,on
• Init: – Generate an ini,al state
• Recursion: – At each point in the recursion choose non-‐determinis,cally between: 1. Stopping the recursion 2. Supplying an input 3. Observing an output (one transi,on per output ac,on)
62
Outline
• Overview • Mo,va,on: Tes,ng a Funds Transfer Switch • Theory • Open Problems
63
Back to the real world!
64
Switch Specifica,on
• Modeling in UPPAAL • ≈ 25 automata
Switch behavior in purchase scenario 65
Purchase Use Case
Specifica,on Structure
POS Model Switch Model Core Model P-‐S
Connector S-‐C
Connector
Four uses cases :ll now
66
Tes,ng Ecosystem
c b
a
Switch Spec.
ioco Test-‐Case Generator
(UPPAAL TRON) a 0
1 a b c
idle
Next
Switch Impl.
Adaptor
Test Result: pass or fail
(+ counterexample)
Coverage Metrics (Cobertura)
67
Measuring Coverage Using Cobertura code coverage tool What about model-‐based coverage? 68
Tes,ng Ecosystem
c b
a
Switch Spec.
ioco Test-‐Case Generator
(UPPAAL TRON) a 0
1 a b c
idle
Next
Switch Impl.
Adaptor
Test Result: pass or fail
(+ counterexample)
Coverage Metrics (Cobertura)
Offline Test Suite
69
Logging Offline Test Suite
• The adaptor logs the messages from TRON • Priori,za,on based on coverage • Used for regression tes,ng
ioco Test-‐Case Generator
(UPPAAL TRON)
Adaptor
Offline Test Suite
70
Tes,ng Ecosystem
c b
a
Switch Spec.
ioco Test-‐Case Generator
(UPPAAL TRON) a 0
1 a b c
idle
Next
Switch Impl.
Adaptor
Test Result: pass or fail
(+ counterexample)
Coverage Metrics (Cobertura)
Offline Test Suite
Test DB
71
Tes,ng Business Rules
72
Adaptor TRON Switch
purchase req(amnt, …)
purchase req(amnt, …)
purchase resp(amnt, …)
remember req. amount
purchase resp(amnt, …)
validate resp. amount
Results
• Have found a number of bugs in the opera,onal system – One traced back to null-‐pointer dereferencing – Poor excep,on handling
• Code coverage of about 40%
73
Observa,ons
• Modeling language and tool limita,ons – Using asynchronous models?
• Scalability Issues – Running mul,ple TRON instances – Reducing buffer lengths, pool sizes, etc.
74
Outline
• Overview • Mo,va,on: Tes,ng a Funds Transfer Switch • Theory: • Open Problems
75
Future Work
• Characterizing of the conformance induced by internal choice test cases
• Formula,ng syntac,c condi,ons to sa,sfy seman,c constraints for asynchronous tes,ng
• Experimental results: – Size of the quo,ent models / test-‐cases – Applying decomposi,onal tes,ng to actual code
Neda Noroozi PAGE 76 8/5/13
Future Work
• Applying decomposi,onal tes,ng to product lines
• Addressing data selec,on issues in actual implementa,ons
Neda Noroozi PAGE 77 8/5/13