Deriving State-Based Test Deriving State-Based Test Oracles for Conformance Oracles for Conformance

TestingTesting

Jamie AndrewsJamie AndrewsAssociate ProfessorAssociate Professor

Department of Computer ScienceDepartment of Computer ScienceUniversity of Western OntarioUniversity of Western Ontario

London, OntarioLondon, Ontario

Plan for TalkPlan for Talk

Testing backgroundTesting background Log file analysis (LFA)Log file analysis (LFA) Process for developing artifactsProcess for developing artifacts Refinement of processRefinement of process

Testing Background

• 3 main tasks to testing:– Selecting test inputs– Running tests– Checking test outputs

• Checking test output not always trivial

SoftwareUnder Test

Test Input

Test Output

Checking Test Output

• May be too complex to check visually

• May be legitimately different from output of previous version

SoftwareUnder Test

Test Input

Test Output

Test Oracles

• Programs that check the output of other programs

SoftwareUnder Test

Test Input

Test Output Test Oracle

Pass/Fail

Test Oracles

• Input and output may be difficult to capture

• Oracle may have to parse complex I/O

SoftwareUnder Test

Test Input

Test Output Test Oracle

Pass/Fail

Log File Analysis (LFA)

• Log file: simple text output

• LFA: dynamic analysis for conformance checking

SoftwareUnder Test

Test Input

Test OutputLog FileAnalyzer

Pass/Fail

Log File

LFA ChallengesLFA Challenges

Make sure logging doesn’t slow down Make sure logging doesn’t slow down system too muchsystem too much– Can send logging data to another machineCan send logging data to another machine

Decide on logging policy for softwareDecide on logging policy for software Write log file analyzer programWrite log file analyzer program

– Special-purpose state machine-based language Special-purpose state machine-based language to help with thisto help with this

Log File Analysis Language Log File Analysis Language (LFAL)(LFAL)

Analyzer = collection of state machinesAnalyzer = collection of state machines Each machine notices some log file lines, Each machine notices some log file lines,

ignores othersignores others Log file lines trigger transitionsLog file lines trigger transitions Machine reports error if it:Machine reports error if it:

– Notices a lineNotices a line– Does not have a legal transition on itDoes not have a legal transition on it

ProcessesProcesses

Need a process for gettingNeed a process for getting– From requirementsFrom requirements– To logging instrumentation, analyzerTo logging instrumentation, analyzer

““Big-step” process:Big-step” process:– Used and taught to studentsUsed and taught to students

““Small-step” process:Small-step” process:– Suggested refinementSuggested refinement

Example: Elevator SystemExample: Elevator System

Requirement to check:Requirement to check:

““The doors are never The doors are never open when the open when the elevator is in motion.”elevator is in motion.”

Big-Step ProcessBig-Step Process

Requirements

SPFEs

Log File AnalyzerProgram

Logging Policy

Situations withPermitted andForbiddenEvents

Example: SPFEsExample: SPFEs

SPFE1:SPFE1:– Situation: Elevator door is openSituation: Elevator door is open– Permitted event: Door closesPermitted event: Door closes– Forbidden event: Elevator starts movingForbidden event: Elevator starts moving

SPFE2:SPFE2:– Situation: Elevator is movingSituation: Elevator is moving– Permitted event: Elevator stops movingPermitted event: Elevator stops moving– Forbidden event: Door opensForbidden event: Door opens

RelationshipsRelationships

SPFEs should re-express requirements to SPFEs should re-express requirements to be checkedbe checked

Logging policy should specify that we log all Logging policy should specify that we log all events that allow us to determine:events that allow us to determine:– Whether we are in each SituationWhether we are in each Situation– Whether each Permitted/Forbidden Event has Whether each Permitted/Forbidden Event has

happenedhappened

Example: Logging PolicyExample: Logging Policy

Log all door open / close events in the form Log all door open / close events in the form door_opendoor_open, , door_closedoor_close

Log all elevator move / stop events in the Log all elevator move / stop events in the form form start_movestart_move, , stopstop

Can instrument code based on thisCan instrument code based on this

From SPFEs to AnalyzerFrom SPFEs to Analyzer

Situations correspond to statesSituations correspond to states Permitted events correspond to transitionsPermitted events correspond to transitions Forbidden events should not have any Forbidden events should not have any

corresponding transitioncorresponding transition

Example: LFAL AnalyzerExample: LFAL Analyzer

machine door_safety;

initial_state closed_stopped;

from closed_stopped, on start_move, to moving;

from moving, on stop, to closed_stopped;

from closed_stopped, on door_open, to open;

from open, on door_close, to closed_stopped;

final_state Any.

closed_stopped

open

moving

stop

start_move

door_close

door_open

Some Past ProjectsSome Past Projects

Steam boiler simulator and analyzer (X. An)Steam boiler simulator and analyzer (X. An) WAP client development and testing (V. Liu)WAP client development and testing (V. Liu)

– 5 KLOC; 3 protocol layers verified5 KLOC; 3 protocol layers verified

Test case generation from oracles (R. Fu)Test case generation from oracles (R. Fu)

Problems with Big-Step ProcessProblems with Big-Step Process

Not always explicit:Not always explicit:– Which requirements are to be checkedWhich requirements are to be checked– Under what assumptions/conditionsUnder what assumptions/conditions

Sometimes “abstract” events mentioned in Sometimes “abstract” events mentioned in SPFEs cannot be directly loggedSPFEs cannot be directly logged– e.g. “door open” event may actually correspond e.g. “door open” event may actually correspond

to “send release command to door lock to “send release command to door lock actuator”actuator”

– Need more concrete, loggable eventsNeed more concrete, loggable events

Small-Step ProcessSmall-Step Process

CheckingAssumptions

Abstract Events

Log File AnalyzerProgram

Logging Policy

Concrete EventsSPFEs

Requirementsto be Checked

Future WorkFuture Work

Case studies of small-step processCase studies of small-step process Teaching small-step processTeaching small-step process ExperimentsExperiments

Potential Benefits and ProblemsPotential Benefits and Problems

Benefits: improved reliability, flexibility, Benefits: improved reliability, flexibility, scalability, traceabilityscalability, traceability

Problems:Problems:– False negatives and positivesFalse negatives and positives– Instrumentation maintenanceInstrumentation maintenance– Process weightProcess weight

SummarySummary

LFA is a method for test result checkingLFA is a method for test result checking We have used and taught a process for We have used and taught a process for

applying itapplying it We propose a refined process for future We propose a refined process for future

workwork

Frequently Asked Questions

• Do we have to discard regression testing?– No; can use as a complement

• How do we know what to put in log file?– Recommend identify reqs to check with LFA first– Develop logging policy, log file analyzer from those

• Efficiency? Scalability?– A few years ago processed 1000 lines/sec– Biggest analyzer 1200 NLOC, from 10-page spec– Recommend starting small, seeing if works for you