The Yogi ProjectSoftware property checking via static analysis
and testing
Aditya V. Nori, Sriram K. Rajamani, Sai Deep Tetali, Aditya V. Thakur
Microsoft Research India
Synergizing Testingand Static Analysis
Idea: Combine testing and static analysis
through a “synergy” of both
New algorithms (Synergy, Dash, SMASH) to do scalable proofs using testing
Industrial strength tool (3 person years)
Synergy: Gulavani, Henzinger, Kannan, N., Rajamani, FSE 2006 Dash: Beckman, N., Rajamani, Simmons, ISSTA 2008SMASH: Godefroid, N., Rajamani, Tetali, MSR-TR-2009
The problemQuestionIs error() unreachable for all possible inputs?
Input
Testing: finds bugs, but can’t prove their absenceStatic analysis: can prove the absence of bugs, but can result in false errors
a↦true b↦falselimit↦2×
Testing×
a↦true b↦falselimit↦2i↦0lock↦1x=0y=0
a↦true b↦falselimit↦2×
Testing×
×
×
×
×
×
×
×
×
××
×
×
a↦true b↦falselimit↦2×
Testing×
×
×
×
×
×
×
×
×
××
×
×
test
Abstraction1
2:ρ×
×
s1s2
Abstraction 0
23 4
5
1
9
6 78
11 10
Proof
2:ρ
0
2: ¬ρ3:ρ 4: ¬ρ
5: ¬ρ
1:ρ
9:ρ
6:ρ 7: ¬ρ8: ¬ρ
11 10
9: ¬ρ
1: ¬ρ
4:ρ3: ¬ρ5:ρ
7:ρ6::¬ρ8:ρ
ρ = (lock != 1)
Key Ideas
Frontier: Boundary between tested and untested regions
WPα: New refinement operation that does not depend on whole program alias information
01234
789
×
××
××
××
× ×
56××
××
10
frontier
Key IdeasStep 1: Try to generate a
test that crosses the frontier– Perform symbolic
simulation on the path until the frontier and generate a constraint 1
– Conjoin with the condition 2 needed to cross frontier
– Is 12 satisfiable?
01234
789
×
××
××
××
× ×
56××
××
10
1
2
frontier
Key Ideas01234
789
×
××
××
××
× ×
56××
××
10
frontier
Step 1: Try to generate a test that crosses the frontier– Perform symbolic
simulation on the path until the frontier and generate a constraint 1
– Conjoin with the condition needed to cross frontier 2
– Is 12 satisfiable? [YES]
Step 2: run the test and extend the frontier
×
×
×
Key IdeasStep 1: Try to generate a
test that crosses the frontier– Perform symbolic
simulation on the path until the frontier and generate a constraint 1
– Conjoin with the condition needed to cross frontier 2
– Is 12 satisfiable? [NO]
01234
789
×
××
××
××
× ×
56××
××
10
1
2
frontier
Key IdeasStep 1: Try to generate a
test that crosses the frontier– Perform symbolic
simulation on the path until the frontier and generate a constraint 1
– Conjoin with the condition needed to cross frontier 2
– Is 12 satisfiable? [NO]
Step 2: use WPα to refine so that the frontier moves back!
0123
4:¬ρ789
×
××
××
××
× ×
56××
××
10
frontier
4:ρ
DASH: Proofs from Tests– Algorithm uses only test case generation
operations– Maintains two data structures:
• A forest of reachable concrete states (tests)– Under-approximates executions of the
program• A region graph (an abstraction)
– Over-approximates all executions of the program
– Our goal: bug finding and proving• If a test reaches an error, we have found
bug• If we refine the abstraction so that there is
*no* path from the initial region to error region, we have a proof
– Handles the richness of C• New operator WPα uses only aliases α that
are present along concrete tests that are executed
• Algorithm uses recursive invocations to handle inter-procedural analysis
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
no
Proof! yes
no
Input:Program P
Property ψ
The DASH Algorithm
Example
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
no
Proof! yes
no
Input:Program P
Property ψ
τ=(0,1,2,3,4,7,8,9)
Example
Symbolic execution +
Theorem proving
frontier
Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
no
Proof! yes
no
Input:Program P
Property ψ
01234
56
789
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
y = 1
Refinement 01234
56
789
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
8:¬ρ 8:ρ9
89
ρ= (lock.state != L)
× ×refine
Refinement
8:¬ρ 8:ρ9
89
ρ= (lock.state != L)
× ×
01234
56
78 :¬ρ
9
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
8:ρ
refine
Example
τ=(0,1,2,3,4,7,<8,p>,9)
01234
56
78 :¬ρ
9
×
× ×
× ×
× ×
× ×
×
×
× ×
×
10×
8:ρ Can extend test beyond frontier?
Refine abstraction
Construct initial abstractionConstruct random tests
Test succeeded? Bug!
Abstractionsucceeded?
τ = error path in abstraction f = frontier of error path
yes
no
yes
no
Proof! yes
no
Input:Program P
Property ψ
frontier
Correct, the program is0123
4⋀¬s5⋀¬s6⋀¬r
9
×
× ×
× ×
× ×
××
×
×
7⋀¬q×
8⋀¬p×
4⋀s5⋀s6⋀r7⋀q
8⋀p×
10
Handling procedure calls
Key ideaPerform a recursive Dash query on the called procedure and usethe result to either generate a test or compute WPα
Sk-1
Sk
×
Sk-2 T×
CALL(foo(i, j)) frontier
Interprocedural analysis
Sk-1
Sk
×
Sk-2 T×
CALL(foo(i, j)) Dash[assume(φ1), foo(i, j), assert(¬φ2)]- pass: perform refinement- fail: generate test
12
Correctness• Theorem. If Dash terminates on (P, φ), then
either of the following is true:– If Dash returns (“pass”, Σ≃), then Σ≃ is a proof that P
cannot reach ¬φ– If Dash returns (“fail”, t), then t certifies that P reaches ¬φ
• Theorem. Every Dash iteration entails exactly one theorem prover call
• However … Dash need not terminate! Thus, usefulness is based on empirical evidence
Locked
do { //get the write lock
KeAcquireSpinLock(&devExt->writeListLock);nPacketsOld = nPackets; request = devExt->WriteListHeadVa;
if(request && request->status){devExt->WriteListHeadVa = request->Next;
KeReleaseSpinLock(&devExt->writeListLock);
irp = request->irp;if(request->status > 0){
irp->IoStatus.Status = STATUS_SUCCESS;irp->IoStatus.Information = request->Status;}
else{irp->IoStatus.Status = STATUS_UNSUCCESSFUL;irp->IoStatus.Information = request->Status;
}SmartDevFreeBlock(request);IoCompleteRequest(irp, IO_NO_INCREMENT);nPackets++;
}} while (nPackets != nPacketsOld);
KeReleaseSpinLock(&devExt->writeListLock);
LL
Windows Device Drivers
Unlocked
Error
UU
Source Code
TestingDevelopment
API Usage Rules(SLIC)
Software Model Checking
Read forunderstanding
New API rules
Drive testingtools
Defects
100% pathcoverage
Rules
Static Driver Verifier
Source Code
TestingDevelopment
API Usage Rules(SLIC)
Software Model Checking
Read forunderstanding
New API rules
Drive testingtools
Defects
100% pathcoverage
Rules
Static Driver Verifier
Architecture of Yogi
Yogi IR (yir)
C Program slamcl
Slam.li database
sliccSLIC property
Instrumented Slam.li database
li2yir
Test case that exposes a
bug
Proof that program satisfies property
Alias and mod-ref information
Z3 theorem prover
YAbsManYSim
polymorphic
region graphs
initialfunctions
summaries
Implementation
• ~6K lines of F#• Z3 theorem prover• Integrated with SDV for Windows• Engineering effort: 3 person years
Static Driver Verifier Runs• #drivers = 69, #properties = 85• largest driver = ~30 KLOCs• #KLOCs = ~350 KLOCS• Yogi works on 129 runs where SLAM “gives up”• Yogi takes ~12 hours whereas SLAM takes ~42 hours
Thanks to …
• Nels Beckman (CMU)• Bhargav Gulavani (IIT
Bombay)• Thomas Henzinger
(EPFL)• Yamini Kannan
(Berkeley)• Robert Simmons
(CMU)
• Leonardo de Moura (MSR Redmond)
• Nikolaj Bjorner (MSR Redmond)
http://research.microsoft.com/yogi