Strauss: A Specification Miner Glenn Ammons Department of Computer Sciences University of Wisconsin-Madison

Strauss:A Specification

Miner

Glenn AmmonsDepartment of Computer SciencesUniversity of Wisconsin-Madison

2

How should programs behave?

• Strauss mines temporal specifications• for example, always call free after malloc

• Why?• To debug

• To verify

• To test

• To modify

• To understand

3

How should programs behave?

• Strauss mines temporal specifications• for example, always call free after malloc

• Why?• To debug

• To verify

• To test

• To modify

• To understand

• Specs constrain programs• like structured programming, types, etc.

4

Strauss output

Spec. says what programs should do:•What order of calls? •What values in calls?

X := socket()

Y := accept(sock = X)

close(sock = Y)

read(sock = Y)

write(sock = Y)

start

end

For all calls Y := accept(sock = X):

5

Strauss inputs

...socket(domain = 2, type = 1, proto = 0, return = 7))...

Traces


...7 := socket(domain = 2, type = 1, proto = 0)...

StraussStrauss

LibraryClient programs

Interface (e.g., sockets)

6

Strauss inputs


Traces



StraussStrauss



Scenario criterion

accept

State transition model (STM)

def socket()close(def use sock)def accept(use sock)read(use sock)write(use sock)

7

My contributions• A dynamic approach to mining

• analyze communication, not source code

• A tool for mining specifications• practical: scalable and mostly automatic• temporal specs may refer to multiple data

values• requires little info about code

• New method for debugging specifications• Birds of a feather flock together (concept

analysis)

• Found specifications and bugs

8

Summary of results

• Mined 17 specs. for X11• example: XInternAtom should only be

called during initialization, not in the event loop.

• Found 199 bugs• In widely distributed programs

• wish (Tk), xpdf, xpilot, ...

• Included races, leaks, logic errors, and performance bugs

• By verifying dynamically (on traces)

9

Related work: obtaining specs.

• By hand• From programmers

• Types• Specification languages and annotations

• From analysts, testers• Abstraction tools (Bandera [Dwyer et al.])

• With specification mining

10

Comparing spec. miners• Strauss

• Specs: temporal, multiple objects, arbitrary NFAs

• Mining: from traces, local, no code necessary

• Goal: specs. for interfaces

• Other work• Concurrent work

• FSMs for Java objects [Whaley, Martin, and Lam]• Simple templates [Metacompilation]• Loop invariants [ESC/Java]

• Earlier work• Arithmetic properties [Daikon]• Cliches [Programmer’s Apprentice]• Communicating processes [Verisoft]

11

Overview of Strauss

Strauss

Programs and interface

Instrumentedprograms

Training traces

STM and scenariocriterion

Unvalidatedspecification

Preprocess

Mine

Debug

Developer’sjudgement

Specification

Scenarios

Tracer

Run ontest inputs

Scenarioextractor

Learner

Cable

12

Trace + STM = Semantic trace

1 7 := socket(domain = 2, type = 1, proto = 0)2 0x100 := malloc(size = 23)3 8 := accept(sock = 7, addr = 0x40, addr_len = 0x50)4 23 := write(sock = 8, buf = 0x100, len = 23)5 0 := close(sock = 8)...



Semantic trace

Vars: v7, v8

1 v7 := socket()2 skip3 v8 := accept(sock = v7)4 write(sock = v8)5 v8 := close(sock = v8)...

13




Semantic trace

Vars: v7, v8


1 7 := socket(domain = 2,domain = 2, type = 1,type = 1, proto = 0proto = 0)2 0x100 := malloc(size = 23)0x100 := malloc(size = 23)3 8 := accept(sock = 7,, addr = 0x40,addr = 0x40, addr_len = 0x50addr_len = 0x50)4 23 := write(sock = 8,, buf = 0x100,buf = 0x100, len = 23len = 23)5 0 :=0 := close(sock = 8)...

14




Semantic trace

Vars: v7, v8


1 7 := socket(domain = 2,domain = 2, type = 1,type = 1, proto = 0proto = 0)2 0x100 := malloc(size = 23)0x100 := malloc(size = 23)3 8 := accept(sock = 7,, addr = 0x40,addr = 0x40, addr_len = 0x50addr_len = 0x50)4 23 := write(sock = 8,, buf = 0x100,buf = 0x100, len = 23len = 23)5 0 :=0 := close(sock = 8)...

15

A different STMState transition model (STM)

def malloc()free(def p)read(use buf)write(use buf)

Semantic trace

Var: v0x100

1 skip2 v0x100 := malloc()3 skip4 write(buf = v0x100)5 skip...

1 7 := socket(domain = 2, type = 1, proto = 0)2 0x100 := malloc(size = 23)3 8 := accept(sock = 7, addr = 0x40, addr_len = 0x50)4 23 := write(sock = 8, buf = 0x100, len = 23)5 0 := close(sock = 8)...

16

A different STMState transition model (STM)

def malloc()free(def p)read(use buf)write(use buf)

Semantic trace

Var: v0x100

1 skip2 v0x100 := malloc()3 skip4 write(buf = v0x100)5 skip...

1 7 := socket(domain = 2,7 := socket(domain = 2, type = 1,type = 1, proto = 0)proto = 0)2 0x100 := malloc(size = 23size = 23)3 8 := accept(sock = 7,8 := accept(sock = 7, addr = 0x40,addr = 0x40, addr_len = 0x50addr_len = 0x50)4 23 :=23 := write(sock = 8,sock = 8, buf = 0x100,, len = 23len = 23)5 0 := close(sock = 8)0 := close(sock = 8)...

17

From dependences to scenarios

Pick a seedSlice forwardsSlice backwardsChop each pair

18



19



20



21



22

Extracting scenarios: example

Vars: v7, v8

1 v7 := socket

2 v8 := accept(sock = v7)

3 write(sock = v8)

4 v8 := close(sock = v8)


23


Pick a seedVars: v7, v8

1 v7 := socket


3 write(sock = v8)



24


Pick a seedSlice forwards

Vars: v7, v8

1 v7 := socket


3 write(sock = v8)



25


Pick a seedSlice forwardsSlice backwards

Vars: v7, v8

1 v7 := socket


3 write(sock = v8)



26



Vars: v7, v8

1 v7 := socket


3 write(sock = v8)



27

Overview of Strauss

Strauss



Training traces



Preprocess

Mine

Debug


Specification

Scenarios

Tracer

Run ontest inputs

Scenarioextractor

Learner

Cable

28

Overview of learning

standardizeScenarios(dep. graphs)

ACEGB

ACEGB

ACEGB

Strings

NFA Learn NFA

Scenario standardize

NFA

Scenario acceptor (SA)

String

Yes or no

Specification = (SA, STM, scenario criterion)

29

Standardizing scenarios

v7 := socket()

v8 := accept(sock = v7)

write(sock = v8)

read(sock = v8)

v8 := close(sock = v8)


v2 := socket()



read(sock = v3)

write(sock = v3)


30


v7 := socket()



read(sock = v8)

write(sock = v8)


v2 := socket()



read(sock = v3)

write(sock = v3)


31


X := socket()


X := close(sock = X)

read(sock = Y)

write(sock = Y)

Y := close(sock = Y)

X := socket()


X:= close(sock = X)

read(sock = Y)

write(sock = Y)


32

Standard scenario = string

X := socket()


X := close(sock = X)

read(sock = Y)

write(sock = Y)


X := socket()


X:= close(sock = X)

read(sock = Y)

write(sock = Y)


A

B

C

D

E

F

33

Overview of learning

standardizeScenarios(dep. graphs)

ACEGB

ACEGB

ACEGB

Strings

NFA Learn NFA

Scenario standardize

NFA

Scenario acceptor (SA)

String

Yes or no

Specification = SA + STM + scenario criterion

34

Raman’s PFSA algorithm1. Build a weighted retrieval tree2. Merge similar states

35


socket

accept

read write

write

write

read

read

write

write

write

100

5

5

95

70

7025

25

25

5

end70

525

36


socket

accept

read write

write

write

read

read

write

write

write

100

5

5

95

70

7025

25

25

5

end70

525

37


socket

accept

read write

writeread

read

write

write

100

5

5

95

70

7025

25

5

end100

25

38


socket

accept

readwrite

read

readwrite

100

595

7025

25

end100

25

5

75

39


socket

accept

readwrite

readwrite

100

595

70

25

25

end100

25

5

75

40

Overview of Strauss

Strauss



Training traces



Preprocess

Mine

Debug


Specification

Scenarios

Tracer

Run ontest inputs

Scenarioextractor

Learner

Cable

41

Overview of Debugging

Cable

Debugged specification

SpecificationACEGB

ACEGB

ACEGB

Classifiedscenarios

Learner

Scenarios

42

Classifying scenarios

b0b2

b1

G0G0

G0

G0G0

g0

g1

43


b0b2

b1

G0G0

G0

G0G0

b0b2

b1 G0G0

G0

G0G0

g0

g1

g0

g1

44


b0b2

b1

g0

g1

G0G0

G0

G0G0

b0b2

b1 g0

g1

G0G0

G0

G0G0

g0

g1

G0G0

G0

G0G0

45

Concept analysis: mammals

4-legged

hairy smart marine thumbed

cats yes yes

dogs yes yes

dolphins

yes yes

gibbons yes yes yes

humans yes yes

whales yes yes

46

Concept analysis: mammals

4-legged

hairy smart marine thumbed

cats yes yes

dogs yes yes

dolphins

yes yes

gibbons yes yes yes

humans yes yes

whales yes yes

47

Cable method: good pets

Good pets:

Bad pets:

48


catsgibbonsdogsdolphinshumanswhales

{}

Good pets:

Bad pets:

49


hairy

catsgibbonsdogs

Good pets:

Bad pets:

50


4-leggedhairy

cats dogs

Good pets:

Bad pets:

51


4-leggedhairy

cats dogs

Good pets:cats, dogs

Bad pets:

52



Bad pets:

gibbons dolphinshumanswhales

smart

53



Bad pets:gibbons dolphins humanswhales

gibbons dolphinshumanswhales

smart

54

Concept analysis: scenarios

Takestransition 0

Takestransition 1

Takes transition 2

Takes transition 3

Takes transition 4

Scen. 0 yes yes

Scen. 1 yes yes

Scen. 2 yes yes

Scen. 3 yes yes yes

Scen. 4 yes yes

Scen. 5 yes yes

55

Cable method: good scenarios

Good scenarios:

Bad scenarios:

56


scen. 0scen. 1scen. 2scen. 3scen. 4scen. 5

{}

Good scenarios:

Bad scenarios:

57


scen. 0scen. 1scen. 2

Takes transition 1

Good scenarios:

Bad scenarios:

58


scen. 0 scen. 2

Takes transition 0Takes transition 1

Good scenarios:

Bad scenarios:

59


scen. 0 scen. 2

Takes transition 0Takes transition 1

Good scenarios:0, 2

Bad scenarios:

60


Takes transition 2Takes transition 3Takes transition 4

Good scenarios:0, 2

Bad scenarios:

scen. 1 scen. 3scen. 4scen. 5

61


Takes transition 2Takes transition 3Takes transition 4

Good scenarios:0, 2

Bad scenarios:1, 3, 4, 5

scen. 1 scen. 3scen. 4scen. 5

62

Experimental results

• Where to find bugs?• in programs (static verification)?

• or in traces (dynamic verification)?

• Specifications and bugs

63

How Strauss verifies a spec

extract scenarios

Scenario acceptor




socket(...)

accept(...)

read(...) write(...)

close(...)

socket(...)

accept(...)


close(...)

socket(...)

accept(...)


close(...)

Traces Scenarios(dep. graphs)

Positive specifications:Accept trace iff every seed hasa scenario that the SA accepts.

Negative specifications:Accept trace iff every seed hasno scenario that the SA accepts.

Yes or no

64

Experimental results

•90 traces•72 programs

•7 “book” rules•10 “graph” rules

Rule STM States Edges Bugs

PrsAccelTbl 9 3 10 1PrsTransTbl 5 3 4 0Quarks 4 63 93 0RegionsAlloc 4 8 14 16RegionsBig 33 370 604 12RmvTimeOut 3 3 3 0XFreeGC 3 10 13 44XGContextFromGC 4 22 36 44XGetSelOwner 3 5 6 9XInternAtom 17 3 11 42XPutImage 9 3 7 2+2XSaveContext 8 989 1519 33XSetFont 4 20 37 44+1XSetSelOwner 16 3 8 7XtFree 4 95 171 45XtOwnSel 16 4 23 1XtRealizeProc 4 22 36 0

65

1. X11 selection protocol specs.:• Found 17 bugs total.• English spec is buggy!

2. Performance: XInternAtom should not be called in the event loop

• 42 buggy programs (out of 72); degree varied

3. XPutImage spec. (more on this later)• 2 different bugs! But, also 2 false positives.

4. Other bugs:• 138 total (resource errors)

Bugs found

66

Cable was usefulCost = concepts visited+ concepts labeled

SpecificationBase Optimal Expert

PrsAccelTbl 18 4 8PrsTransTbl 6 2 2Quarks 16 2 2RegionsAlloc 20 8 16RegionsBig 540 X 149RmvTimeOut 4 2 2XFreeGC 20 6 12XGContextFromGC 50 14 24XGetSelOwner 4 4 5XInternAtom 20 4 5XPutImage 12 6 10XSaveContext 184 X 42XSetFont 56 X 53XSetSelOwner 12 4 5XtFree 224 X 28XtOwnSel 14 4 5XtRealizeProc 76 12 13

Cost of labeling

67

XSetSelectionOwner

English: Pass XSetSelectionOwner the timestampfrom the last event.

For all calls XSetSelectionOwner(time = X)

XSetSelectionOwner(time = X)

(event.time = X) := XNextEvent()XFilterEvent(event.time = X)XtDispatchEvent(event.time = X)(event.time = X) := XCheckWindowEvent()cb_XtActionProc(event.time = X)

68

XInternAtom

English: Don’t call XInternAtom in the event loop.

For no calls XInternAtom(display = X)

cb_XtEventHandler(event.display = X)cb_XtActionProc(event.display = X)(event.display = X) := XtAppNextEvent()(event.display = X) := XNextEvent()XFilterEvent(event.display = X)XtCallActionProc(event.display = X)(event.display = X) := XWindowEventXtDispatchEvent(event.display = X)(event.display = X) := XCheckWindowEvent()

XInternAtom(display = X)

69

XPutImage

English: The image and GC passed to XPutImagemust have been created on the same display.

For all calls XPutImage(display = X, image = Y, gc = Z)

Z := XCreateGC(display = X)

Y := XCreateImage(display = X)Y := XShmCreateImage(display = X)

XPutImage(display = X, image = Y, gc = Z)

70

Conclusions• Strauss is

• local• scalable• dynamic

• Strauss finds• real specs.• real bugs

• Does Strauss• generalize well?• work for others?• do too much or too little?

71

Suggestions for future work

• Increasing automation• Exploiting patterns• Exploiting labels

• Beyond mining from traces• Static; hybrids (profile-based)

• More expressive models• Structured heaps; arithmetic properties

• Other domains• Program understanding; testing

72

My publications• On Strauss.

• Debugging temporal specifications with concept analysis. With Rastislav Bodik, James R. Larus, and David Mandelin. PLDI03.

• Mining specifications. With Rastislav Bodik and James R. Larus. POPL02.

• Path profiling• Improving data-flow analysis with path profiles.

With James R. Larus. PLDI98. Selected for 20 Years of PLDI (1979-1999).

• Exploiting hardware performance counters with flow and context sensitive profiling. With Tom Ball and James R. Larus. PLDI97.

73

End of talk

74


b0b2

b1

g0

g1

G0G0

G0

G0G0

b0b2

b1 g0

g1

G0G0

G0

G0G0

g0

g1

G0G0

G0

G0G0

75

Overview of StraussTraces and STM

apply STM extract scenarios

socket(...)

accept(...)


close(...)

Trace programs(with PDGs)

socket(...)

accept(...)


close(...)

socket(...)

accept(...)


close(...)

Seed pattern

Scenarios

Classify and learn

Specification

X := socket()

Y := accept(so = X)

close(so = Y)close(so = X)

read(so = Y)

write(so = Y)

start

end

..

76

Overview of StraussTraces and STM

apply STM extract scenarios

socket(...)

accept(...)


close(...)

Trace programs(with PDGs)

socket(...)

accept(...)


close(...)

socket(...)

accept(...)


close(...)

Seed pattern

Scenarios

Classify and learn

Specification

X := socket()

Y := accept(so = X)

close(so = Y)close(so = X)

read(so = Y)

write(so = Y)

start

end

..

77

Strauss inputs


Traces





StraussStrauss



Documents

Strauss: A Specification Miner Glenn Ammons Department of Computer Sciences University of Wisconsin-Madison