Department of Computer Engineering
PROPANEPROPANEAn Environment for Examining theAn Environment for Examining thePropagation of Errors in SoftwarePropagation of Errors in Software
Martin HillerMartin Hiller, Arshad Jhumka, Neeraj Suri, Arshad Jhumka, Neeraj Suri
Chalmers University of TechnologyChalmers University of Technology
Göteborg, SwedenGöteborg, Sweden
{{hillerhiller, arshad, suri}, arshad, suri}@[email protected]
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
2
Overall ObjectivesOverall Objectives
A
D
C
E
B
F
Software reliability can be provided/increased Software reliability can be provided/increased by adding mechanisms that detect and correct by adding mechanisms that detect and correct data errors data errors wrappers, assertions, etc. wrappers, assertions, etc. (especially for black-box software)(especially for black-box software)
Given a software system and limited resources one would like to Given a software system and limited resources one would like to concentrate work on the most vulnerable/exposed parts of the concentrate work on the most vulnerable/exposed parts of the software, i.e. …software, i.e. …
Where do upcoming errors propagate?Where do upcoming errors propagate?
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
3
OutlineOutline
• Error Propagation & Basic ApproachError Propagation & Basic Approach• Examples of results obtained by using PROPANEExamples of results obtained by using PROPANE
– Aircraft arrestment systemAircraft arrestment system• Overview of PROPANEOverview of PROPANE
– Tool suiteTool suite– Requirements & limitationsRequirements & limitations
• Conclusions & some future directionsConclusions & some future directions
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
4
Error PropagationError Propagation
B
F
ErrorError = A system state which is = A system state which is different from the state in a ”correct” different from the state in a ”correct” execution of the system (i.e, execution of the system (i.e, notnot mutations or software defects)mutations or software defects)
Modules in a software system have different levels of exposure and Modules in a software system have different levels of exposure and different ”ability” to break error propagation (i.e., different levels of different ”ability” to break error propagation (i.e., different levels of error containment) – examining error propagation gives the developer error containment) – examining error propagation gives the developer a picture/profile of these levels.a picture/profile of these levels.
D
C
E
A
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
5
Basic Approach Basic Approach Error Injection Error Injection
A
D
C
E
B
F
1.1. Generate Generate Golden RunGolden Run, i.e., an error free , i.e., an error free reference runreference run
2.2. Generate Generate Injection RunInjection Run, i.e., a run in which , i.e., a run in which an error (i.e. erroneous system state) is an error (i.e. erroneous system state) is injectedinjected
3.3. Compare Compare Injection Run Injection Run with with Golden Run Golden Run to see which parts of the system were to see which parts of the system were corrupted by the injected errorcorrupted by the injected error
PROPANE PROPANE – Propagation Analysis Environment– Propagation Analysis Environment By instrumenting the target software, PROPANE can, during execution, log By instrumenting the target software, PROPANE can, during execution, log individual variables and events and inject errors into individual variables.individual variables and events and inject errors into individual variables.
= probe= probe= injection = injection locationlocation
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
6
Example: Aircraft Arrestment SystemExample: Aircraft Arrestment System
CLOCK
CALC
DIST_S
PRES_AV_REGPRES_S
ms_slot_nbr i
mscntpulscnt
slow_speedstopped
IsValue
OutValue TOC2ADC
TCNTTIC1PACNT
SetValueRotationsensor
Pressuresensor
Pressurevalve
Computer
Pressurevalve
Tape drum(original)
Tape drum(mirror)
Cable
Target system overviewTarget system overview Target software overviewTarget software overview
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
7
Results Generated by PROPANEResults Generated by PROPANE
• From low level to high level informationFrom low level to high level information– Signal/Variable PlotsSignal/Variable Plots– Propagation SignaturesPropagation Signatures– Propagation GraphsPropagation Graphs– Propagation SummariesPropagation Summaries
• Data compilations that can be used for further Data compilations that can be used for further analysisanalysis
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
8
Signal/Variable PlotsSignal/Variable Plots
PACNT
0
50
100
150
200
250
300
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000t (ms)
va
lue
Golden RunInjection Run
SetValue
0
500
1000
1500
2000
2500
3000
0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000t (ms)
va
lue
Golden RunInjection Run
One-cycle single-bit upset at t = 1500 msOne-cycle single-bit upset at t = 1500 ms SetValue erroneous at t = 1539 msSetValue erroneous at t = 1539 ms
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
9
Propagation SignaturesPropagation Signatures
• Each injected error generates a Propagation Each injected error generates a Propagation SignatureSignature showing when and where the error showing when and where the error propagatedpropagated
1490 1500 1510 1520 1530 1540 1550 1690 1700t (ms)
PACNTpu
lscnt
i SetVal
ueO
utVal
ueTOC2ADC
IsVal
ue
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
10
Propagation GraphsPropagation Graphs
The Propagation Graph is generated in three different formatsThe Propagation Graph is generated in three different formats– PROPANE native formatPROPANE native format– GML (Graph Markup Language)GML (Graph Markup Language)– dot dot (part of the (part of the graphvizgraphviz tool suite from AT&T Research) tool suite from AT&T Research)
ProbedProbedlocationlocation
Incoming errors from locations earlier in the Incoming errors from locations earlier in the propagation pathpropagation path
Outgoing errors to locations later in the Outgoing errors to locations later in the propagation pathpropagation path
error counterror countttmin min / t/ tavg avg / t/ tmaxmax
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
11
Propagation Graph for PACNTPropagation Graph for PACNTStarting point
Propagation pathEach arc carries information regarding number of propagated Each arc carries information regarding number of propagated errors and propagation timeerrors and propagation time
Variables along the propagation trajectory
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
12
Propagation Summary for PACNTPropagation Summary for PACNT
SignalSignal Error Error countcount
Error Error raterate ttminmin ttavgavg ttmaxmax
PACNTPACNT 18401840 1.0001.000 00 00 00
pulscntpulscnt 18401840 1.0001.000 00 00 2020
ii 18111811 .984.984 00 44 2020
SetValueSetValue 12751275 .693.693 11 613613 41594159
OutValueOutValue 12751275 .693.693 11 613613 41594159
TOC2TOC2 12751275 .693.693 33 615615 41614161
ADCADC 12651265 .688.688 1010 629629 41684168
IsValueIsValue 12021202 .653.653 155155 682682 34673467
slow_speedslow_speed 769769 .418.418 00 20042004 58905890
mscntmscnt 11841184 .643.643 476476 29822982 62016201
ms_slot_nbrms_slot_nbr 11841184 .643.643 476476 29822982 62016201
TCNTTCNT 11841184 .643.643 476476 29822982 62016201
TIC1TIC1 11841184 .643.643 476476 29822982 62016201
Probed locationProbed location(variable)(variable)
For all incoming arcs to one node:For all incoming arcs to one node:
• Total error count and error rateTotal error count and error rate
• Combined tCombined tmin min / t/ tavg avg / t/ tmaxmax
Propagation Graphs and Propagation Summaries indicate which parts of the Propagation Graphs and Propagation Summaries indicate which parts of the observed system state that are observed system state that are most vulnerable most vulnerable to the injected errorsto the injected errors perhaps a perhaps a good location for error detection/recovery mechanismsgood location for error detection/recovery mechanisms??
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
13
Error Propagation from PACNTError Propagation from PACNT
Highest Error RateHighest Error Rate
Lowest Error RateLowest Error Rate
ms_slot_nbr i
mscnt
pulscnt
slow_speed
stopped
IsValue
OutValueTOC2ADC
TCNT
TIC1
PACNT
SetValue
CLOCK
PRES_S V_REG PRES_A
CALC
DIST_S
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
14
OverviewOverview
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
15
PLPLPROPANE LibraryPROPANE Library
• Static C Library providing the injection/logging APIStatic C Library providing the injection/logging API– propane_inject() propane_inject() /* Injects an error *//* Injects an error */– propane_log_var() propane_log_var() /* Variable probe *//* Variable probe */– propane_log_event() propane_log_event() /* Event probe *//* Event probe */
• Target must be instrumented with these functions and linked Target must be instrumented with these functions and linked with the librarywith the library
• Experiments are then set up using description filesExperiments are then set up using description files– Error types Error types (what errors to inject, e.g., bit-flips)(what errors to inject, e.g., bit-flips)– Error triggers Error triggers (when to inject errors, e.g., once/periodically)(when to inject errors, e.g., once/periodically)– Active probes Active probes (perhaps not all instrumented variables/events have to be (perhaps not all instrumented variables/events have to be
logged)logged)
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
16
Instrumentation ExampleInstrumentation Example
int spherical_volume( double radius ){ double volume;
volume = 4.0 * (PI * pow(radius, 3.0)) / 3.0;
return volume;}
int spherical_volume( double radius ){ double volume;
/* Injection location for radius */ propane_inject( IL_SPHERE_VOL, &radius, PROPANE_DOUBLE );
/* Probe the value of radius */ propane_log_var( P_RADIUS, &radius );
volume = 4.0 * (PI * pow(radius, 3.0)) / 3.0;
/* Probe the value of volume */ propane_log_var( P_VOLUME, &volume );
return volume;}
Original codeOriginal code Instrumented codeInstrumented code
At this point, instrumentation is At this point, instrumentation is unfortunately still a manual task. unfortunately still a manual task. However, automation is the next However, automation is the next step in the development of step in the development of PROPANE. PROPANE.
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
17
PSCPSCPROPANE Setup CreatorPROPANE Setup Creator
• Once the target is instrumented, experiments can Once the target is instrumented, experiments can be set upbe set up
• Given information on injection locations, probes, Given information on injection locations, probes, error types and occurrences, PSC will createerror types and occurrences, PSC will create– description files for running experimentsdescription files for running experiments– description files for data extraction and analysisdescription files for data extraction and analysis
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
18
PCDPCDPROPANE Campaign DriverPROPANE Campaign Driver
• After instrumentation and setup, the PCD runs the After instrumentation and setup, the PCD runs the experimentsexperiments– Invokes instrumented target executable containing the PL and all links to Invokes instrumented target executable containing the PL and all links to
external modules, e.g., environment simulatorexternal modules, e.g., environment simulator
• Provides continuous information on experiment status and Provides continuous information on experiment status and remaining workremaining work
• User controlUser control– PausePause– AbortAbort– SkipSkip
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
19
PDEPDEPROPANE Data ExtractorPROPANE Data Extractor
• Extracts and analyses data from the raw readouts produced Extracts and analyses data from the raw readouts produced during experimentsduring experiments– Perform Perform Golden Run ComparisonsGolden Run Comparisons, i.e., compare an injection run with a , i.e., compare an injection run with a
”clean” reference run to trace errors, generating a ”clean” reference run to trace errors, generating a Propagation SignaturePropagation Signature
– Collapse multiple propagation signatures into Collapse multiple propagation signatures into Propagation Graphs Propagation Graphs and and Propagation SummariesPropagation Summaries
– Prepare/compile data for further external analysis (using e.g. MatLab)Prepare/compile data for further external analysis (using e.g. MatLab)
1490 1500 1510 1520 1530 1540 1550 1690 1700t (ms)
PACNTpu
lscnt
i SetVal
ueO
utVal
ueTOC2ADC
IsVal
ue
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
20
Requirements & LimitationsRequirements & Limitations
• Programming languageProgramming language– The library itself is written in C The library itself is written in C the language in the target system must the language in the target system must
be able to interface with C-librariesbe able to interface with C-libraries• Environment simulatorEnvironment simulator
– Stimuli to the target system must be provided by an environment simulatorStimuli to the target system must be provided by an environment simulator– PROPANE provides basic interfacing functionality such that control of the PROPANE provides basic interfacing functionality such that control of the
EnvSim can be made part of the experiment setup (~plug-in’s)EnvSim can be made part of the experiment setup (~plug-in’s)• Target instrumentationTarget instrumentation
– Difficulties with real-time applications that cannot be made to run in Difficulties with real-time applications that cannot be made to run in simulated timesimulated time
• Current implementation focused towards single-node SWCurrent implementation focused towards single-node SW– Distributed SW not considered at this pointDistributed SW not considered at this point
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
21
ConclusionsConclusions
• PROPANE enables experimental evaluation of error PROPANE enables experimental evaluation of error propagation in SWpropagation in SW– Propagation profiles indicate which parts of system state that Propagation profiles indicate which parts of system state that
are vulnerable are vulnerable aid for equipping SW with error detection aid for equipping SW with error detection and recovery mechanismsand recovery mechanisms
• We envision that PROPANE can be used as a We envision that PROPANE can be used as a design stage aid for analysis of single node SWdesign stage aid for analysis of single node SW
• Limitations apply for some apsects of real-time and Limitations apply for some apsects of real-time and distributed SWdistributed SW
Department of Computer Engineering
July 22, 2002Martin Hiller
PROPANE: An Environment for Examining the Propagation of Errors in SoftwareISSTA 2002
22
Some Future DirectionsSome Future Directions
• Increased automationIncreased automation– E.g. automatic instrumentation of target softwareE.g. automatic instrumentation of target software
• Extended analysis capabilitiesExtended analysis capabilities– E.g. provide your own analysis plug-insE.g. provide your own analysis plug-ins
• Open sourceOpen source– Possibly in the near futurePossibly in the near future