Reasoning About Precisely Defined Processes Leon J. Osterweil ([email protected]) Lab. For Advanced SE Research (LASER) University

Reasoning About Precisely Defined Processes

Leon J Osterweil (ljocsumassedu)

Lab For Advanced SE Research (LASER)

httplasercsumassedu

University of Massachusetts

Amherst MA 01003

Institute for Software Research

University of California Irvine

25 April 2014

Thanks to Collaborators

Faculty and Staffbull Lori A Clarkebull George Avruninbull Barbara Lernerbull Sandy Wise

Studentsbull Bobby Simidchievabull MS Raunakbull Stefan Christovbull Huong Phanbull Heather Conboybull Xiang Zhoubull Seung Yeob Shinbull Huong Phan

ndash And othershellip

A Focus on Human-Intensive Systems

bull Integrate contributions ofndash Software systemsndash Hardware devicesndash Human participants

bull They control much of the worldrsquos workndash So it is important that they be defect-free secure

bull They are increasingly complexndash Concurrent distributed complex exception richndash Making it hard to be sure of them

Some Examples

bull Electionsbull Medical Procedures

ndash Blood transfusionndash Chemotherapy administration

bull Software Developmentbull Management processesbull Manufacturing Processesbull Emergency planning and support

Some Examples

Our Approach

bull Human-intensive systems are collections of processes

bull Model thembull Analyze thembull Continuously improve them

An Example Health Care Process Engineering

bull ~100000 people die in US hospitals each year due to preventable medical errorsndash 1999 IOM report estimatendash Doesnrsquot count serious injury pain-and-suffering needless

bull Errors likendash Transfusing the wrong type of bloodndash Delivering incorrect medicationndash Amputating the wrong legndash Removing the healthy lung (leaving the cancerous one in)

bull Recent NY Times article estimates it is probably more like 440000 deaths per yearndash Third leading cause of death in the US

~100000 people each year in US hospitals due to preventable errors

One fully loaded 747 per day

Another Example Elections in the US

bull Elections entail far more than casting and tabulating votesbull Need to consider the entire process

ndash Voting machines play a partndash Humans are also key participantsndash Databases too

bull The election process is large and complex and in the US varies from jurisdiction to another

bull Election processes vary over time as well

Goalbull To identify potential defects threats to security in election

processes and evaluate approaches to correcting them

Our Approach Continuous Process Improvement

bull Create a precise accurate model of a real-world process

bull Use formal analysis methods to automatically identify potential problems in the modelndash Eg single points of failure (SPFs)

bull Modify process model to address the problemsndash Verify that the modification makes things better

bull Deploy improvements in real-world process

Approach Consider a process to be a kind of software Apply software engineering technologies

Programming Human-Intensive Processes

bull Process programming language requirements

ndash Capture complexity of systems clearly cleanly in detail

ndash Rich semantics (eg functionality concurrency resource utilization exceptions human participation)

ndash Precisely defined semantics to support static analysis simulations and executions

ndash Understandable to the domain experts (facilitate validation that the definition models actual process)

Process definition

PropertiesModel Checker

(FLAVERS)

Discrete event simulator

Failure mode and effects analyzer

Fault tree generator

Hazards

Failure modes

Scenario specifications

Satisfied properties violated properties +

counterexamples

Fault trees minimal cut sets

Effects of failure modes

Discrete event simulation runs

Little-JIL narrator

Property elicitor (PROPEL)

Process editor(Little-JIL editor)

Textual representation of process definition

Process Improvement Environment Architecture

Requirements Derivation

Derived RequirementsDevice model

Process definition + requirements

AnalysisAnalysis Feedback

Improvements new family members

The Little-JIL Process Definition Languagebull Blends proactive and reactive controlbull Coordinates human and automated agentsbull Emphasizes exception specification managementbull Facilities for abstraction scoping hierarchybull Supports artifact flowbull Concurrency synchronization with message-passingbull Articulate specification of resourcesbull Steps have agents that can be humans software hardwarebull Semantics for aborting stepsbull Prepost condition constructsbull Facilities for human choicebull Rigorously defined using finite state machine semanticsbull Visual language

ldquoSteprdquo is the central Little-JIL abstraction

TheStepName

Interface Badge(parameters resources agent)

Prerequisite Badge Postrequisite Badge

Substep sequencingHandlers

Artifactflows

Exception type

continuation

Define an election process

bull Use the Little-JIL process definition languagendash Consists of coordination diagram and other

specifications (eg agents artifacts resources)ndash Especially appropriate for modeling concurrency

and complex exception handling that arise in elections

ndash Visual representation facilitates communication and validation

pass authentication and vote

present ID

Perform pre-vote authentication

Check off voter as voted Issue ballot

Record voter preference

Top-Level simplified election process

Hierarchy Scoping and Abstraction in Little-JIL

bull Definition is a hierarchical decompositionbull Think of steps as procedure invocations

ndash They define scopesndash Copy and restore argument semantics

bull Encourages use of abstractionndash Eg system fragment reuse

present ID

Record voter preference=

Adding some elaborations

Confirm voter ID matches voter

Confirm voter ID matches voting roll

Confirm voter has not voted

Exception Handling A Special Focus of Little-JIL

bull Steps may have one or more exception handlersbull Handlers are steps themselves

ndash With parameter flow

bull React to exceptions thrown in descendent stepsndash By Pre- or Post-requisitesndash Or by Agents

bull Four different continuations

present ID

Record voter preferenceLet voter voter with provisional ballot

Fill out provisional ballot

Submit provisional

ballot

And some exception managementMissing ID ExceptionInadmissible ID ExceptionID Mismatch Exception

Voter Already Checked Off Exception

exceptionsID Mismatch

ExceptionsMissing IDInadmissable ID

exceptionsVoter Already Checked Off

Properties needed to support Finite-State Verification (Model-Checking)

bull Refine the requirements for an election processndash High-level requirementsndash Low-level requirementsndash Precise properties or event sequences

bull Identify event alphabetbull Annotate graph with events used to define

propertiesbull Verify the process adheres to the properties

ndash Run formal analysis using finite-state verification

Decompose high-level requirements

bull Example refinement of high-level requirement into a collection of low-level requirements

each unique voter is allowed at most one vote

voter must receive ballot before choosing to vote

voter must leave voting booth after choosing to vote

voter must be authenticated before entering voting booth voter must be checked off before entering voting booth voter must enter voting booth before choosing to vote

Formally define the propertiesUse the PROPEL property elicitation tool to

formally define a property corresponding to the low-level requirement ldquovoter must be authenticated before entering voting boothrdquo

Example propertyVoter must be authenticated before entering voting boothbull Disciplined English view

ndash VoterEntersVotingBooth cannot occur until after VoterIsAuthenticated has occurred VoterIsAuthenticated is not required to occur however

ndash VoterIsAuthenticated can occur multiple times before the first subsequent VoterEntersVotingBooth occurs

ndash After VoterIsAuthenticated occurs other events can occur before the first subsequent VoterEntersVotingBooth occurs

ndash After VoterEntersVotingBooth occurs neither VoterIsAuthenticated nor VoterEntersVotingBooth can occur again

bull FSA view

FLAVERS finite-state verifier

Binding property events to process steps

Property FSA specified in PROPEL Little-JIL process definition

Bindings between property events and process steps

Yes the process satisfies the property

No the property could be violated Here is a counter-example

Finite-state verification with FLAVERSbull The FLAVERS FSV verifier has been extended to automatically

construct finite models of the Little-JIL process definitionsbull Finite model represents all possible event sequences for the events

in a property that could occur for all the possible traces through the process definition

bull Apply dataflow analysis algorithm to determine if the model is consistent with the property

bull If the process is inconsistent with the property a counter-example trace is produced

bull FLAVERS determines whether the election process as defined in Little-JIL adheres to the property ldquovoter must be authenticated before entering voting boothrdquo

(Voter Already Checked Off Exception)

(Voter Enters Voting Booth Event)

(Voter Votes Or Does Not Vote Event)

(Voter Leaves Voting Booth Event)

[pass authentication and vote]

[present ID]

[perform pre-vote authentication]

[let voter vote with provisional ballot]

[fill out provisional ballot]

[submit provisional ballot]

Violation detectedbull An unauthenticated voter can vote with provisional ballot

ndash Counter-example produced by FLAVERS to demonstrate how the property ldquovoter must be authenticated before entering voting boothrdquo could be violated

Violation explanationbull The parallel step creates a race condition

ndash The pre-vote authentication step is executed in parallel with two others

ndash Exceptions can occur in any orderndash Exceptions may appear to be independent but they are notndash If confirm voter has not voted wins that creates problems

bull Forcing sequential execution can correct this situationbull After correcting the process definition the FLAVERS

verifier can verify that the new process definition satisfies the ldquovoter must be authenticated before entering voting boothrdquo property as well as the other properties

Is this a ldquorealrdquo problem

bull Humans would probably never let this happenndash They will be watching and using their judgment

bull But suppose this process were automatedndash Steps executed by hardwaresoftware wherever

possiblendash This scenario could actually happenndash Would manifest itself as a ldquobugrdquo

bull Prior diagnostic analysis prevents this

In Medical Domain

bull Have found race conditions deadlocksbull Unsafe sequences

ndash Administering medication with checking dosage permission etc

ndash Not being sure to weight patients upon arrivalndash Letting patients into emergency department

without wristbands

Other kinds of problemsbull Finite state verificationmodel checking looks

for event sequence defectsbull But assumes that all steps are performed

correctlybull Humans may make errors

ndash Software toobull Looking for consequences of incorrect

performance done using Fault Tree Analysis

Fault Tree Analysis (FTA)bull A well accepted and widely practiced safety

analysis technique that identifies all possible combinations of events that could lead to a given hazardndash Hazard A condition in which loss of life or

serious loss of property becomes possible

bull Approachndash Specify a hazard that is of concern

ndash Create a fault tree for that hazard

ndash Derive Minimal Cut Sets (MCSs)--minimal event combinations that can cause the hazard

Process definition

(FLAVERS)

Hazards

Failure modes

counterexamples

Little-JIL narrator

Process Improvement Environment

Fault Tree Analysis (FTA)

bull FTA is a deductive top-down analysis to find out which events in a system could lead to a given hazard

bull A fault tree is a graphical model of various combinations of events that could produce the hazard

BACKGROUND

hazard

primary event

Minimal Cut Set (MCS)

bull A minimal cut set (MCS) is a minimal set of primary events all of whose occurrence ensures that the hazard event occurs

bull MCS can be computed automatically from a Fault Tree using Boolean Algebra

bull A MCS indicates a system vulnerability that an adversary may be able to exploit to create the hazardndash Eg A singleton MCS called a single point of

failure (SPF) is a particularly worrisome vulnerability

BACKGROUND

Our Approach Generate the Fault Tree from the Process Definition

bull Specify a hazardndash Consider hazards created by the delivery of an

incorrect artifact to a process step

ndash Generation based on templates for the semantics of the language

bull Use Fault Tree Analysis to develop all Minimal Cut Setsndash Automatically calculated from the fault tree using

Boolean algebra

Small example part of a real generated fault tree

Details of our Approachbull Use our rigorously defined model of the process

ndash Derived from and validated by domain expertsbull Obtain election hazards from domain expertsbull Apply fault tree analysis

ndash To detect vulnerabilities bull Using hazard analysis

ndash To define attacks that can exploit the vulnerabilitiesbull In ongoing work we are also

ndash Composing attacking and defending processesndash Evaluating the defenderrsquos resistance to such attacks

bull Using model checking

FTA for Medical Processes

bull Use to identify critical steps that should be double-checked

Finding Vulnerabilities in The Simple Blood Transfusion Process

A Derived Fault Tree

Calculating Minimal Cut Sets

Each gate corresponds to an equation 1 E1 = E2 2 E2 = E3 + E4 3 E3 = E5 + E6 4 E5 = E7 E8 5 E6 = E9 E13 6 E7 = E11 + E12 7 E9 = E11 + E10

=gt E1 = ( E4 ) + ( E11 ) + ( E12 E8 ) + ( E10 E13 )

Single points of failure

An Actual Generated Fault Tree

Process definition

(FLAVERS)

Hazards

Failure modes

counterexamples

Little-JIL narrator

Dynamic Analysis too by generatingdiscrete event simulations

Analysis Feedback

Analysis

Driving Simulations to Optimize Resource Allocations

bull Define processes in Little-JILbull Hypothesize resource behaviorsbull Define resource allocation strategiesbull Run simulationsbull Tabulate

An Example part of an ED process

An Example Resource Type specificationltresource type=medical doctorgtltattribute name=location value= gtltattribute name=shift value= gtltcapacity assignment_available=1reservation_available=1gtltcapability name=MDgtltreservation guard=location==artifact(patientlocation)ampamp timegt=start(shift) ampamp timeltend(shift)contention_policy=SickestFirst ProblemSpecificselection_policy=LeastUtilizedFirst ProblemSpecificeffort_needed=0 gtltassignment guard=location==artifact(patientlocation)ampamp timegt=start(shift) ampamp timeltend(shift)contention_policy=SickestFirst ProblemSpecificselection_policy=LeastUtilizedFirst ProblemSpecificeffort_needed=1 gt

Some Resource Instance Specificationsltinstantiate type=medical doctor number=10 gtltinstance type=medical doctor id=1set_attribute=location value=main1048576track gtltinstance type=medical doctor id=1set_attribute=shift value=7AM104857610485763PM gtltinstance type=medical doctor id=2set_attribute=location value=main1048576track gtltinstance type=medical doctor id=2set_attribute=shift value=3PM1048576104857611PM gtltinstance type=medical doctor id=3set_attribute=location value=main1048576track gtltinstance type=medical doctor id=3set_attribute=shift value=11PM104857610485767AM gt

Priority-based scheduling policy greatly reduces Length of Stay for patients across all acuity levels

Waiting times by acuity level using Sickest-first scheduling policy

Waiting times by acuity level using Priority-Based scheduling policy

The number of handoffs decreases when doctors and nurses stop accepting new patients 1 hour

before their shifts end

Triage Nurse cancannot place patient in bed

Elapsed time (in simulation time units)

Summary of Results

bull Found some defects in some healthcare processesbull Effected a 70 reduction in the number of ldquoerrors

reaching the patientrdquo at Baystate Oncology Divisionbull Identified defects and vulnerabilities in election

processesbull Automating some code refactoring processesbull While alsomdash

ndash Improving our process languagendash Creating automated Fault Tree generationanalysis toolset

Another Example Domain

Software Engineering

bull Analysis of Scrum processesndash Precise definition(s)ndash Simulations of task assignment strategies

bull Understanding Rework in software developmentndash What is rework ndash Application to refactoring

ScrumActivity Skeleton

Development Iteration

Sprint Planning Meeting Sprint Sprint Review Sprint Retrospective

product Product

sprint backlog channel Backlog Channel

sprint backlog sprint backlog channel

product product

agent ScrumMaster

owner ProductOwner

deadline Hours = 4

product Product

product product

agent team

Now Elaborate on the Sprint Step

product Product

product product

agent ScrumMaster

owner ProductOwner

deadline Hours = 4

product Product

product product

agent team

SprintActivity Skeleton

Sprint

Daily Sprint

Daily Scrum

Checked Work

Revise Sprint Backlog

Sprint Step DetailsSprint

Daily Sprint

Daily Scrum

sprint backlog sprint backlog channel sprint backlog sprint backlog channel

agent ScrumMaster

team Team

sprint burndown BurndownTool

editor BacklogTool

deadline Minutes = 15

sprint backlog Backlog

product product

product Product

product Productdeadline Days = 1

agent Teamproduct Product

agent Team

editor BacklogTool

Now elaborate onldquoChecked Workrdquo

Checked Work

Checked Work ElaborationSprint

Daily Sprint

Daily Scrum

agent ScrumMaster

team Team

editor BacklogTool

product product

product Product

agent Team

editor BacklogTool

Checked Work

Checked Work Subprocess

Checked Work

Rework

Integrate

Checked Work

Integrate

Report Build Failedproduct Product

product ProductBuild Failed

report Build Fail Report

product product

product Productreport Build Failed = report U Build Fail Report

Check Build

product product

agent Team

agent Builder

agent Team

product Product

product product

agent ScrumMaster

owner ProductOwner

deadline Hours = 4

Product Product

product product

agent team

ldquoBegin Sprintrdquo event ldquoEnd Sprintrdquo event

SprintSprint

Daily Sprint

Daily Scrum

agent ScrumMaster

team Team

editor BacklogTool

product product

product Product

agent Team

editor BacklogTool

SprintSprint

Daily Sprint

Daily Scrum

agent ScrumMaster

team Team

editor BacklogTool

product product

product Product

agent Team

editor BacklogTool

4sprint backlogchange

SprintSprint

Daily Sprint

Daily Scrum

agent ScrumMaster

team Team

editor BacklogTool

product product

product Product

agent Team

editor BacklogTool

4sprint backlogchange This is benign because the step is performed by Team

Simulation of Different Task Assignment Strategies

bull Hypothesizedndash Distribution of task sizes and difficultiesndash Distribution of worker coding and testing skill levelsndash Different team makeupsndash Different strategies for task assignment

bull Fault injection to simulate coding bugs and inadequate testing

bull Iterate until no more bugs found

Different strategies for task assignment

bull Strategiesndash Random assignment of tasks to workersndash Greedy Hardest tasks to strongest workersndash Prev Tasks not completed reassigned to previously

assigned workersndash Greedy Prev Combination of Greedy and Prev

bull Different task assignment strategies produced sharp differences inndash Lines of code producedndash Number of residual bugs

What is ldquoreworkrdquo

in software development

In other intellectual work

Traditional Software Development Process

Requirements

Develop Rqmt Element

Declare and Define Rqmt

Define Rqmt ElementDeclare Rqmt Element

~ Rqmt OK

Inter-requirementConsistency Check

Rqmt OK

Rework in aRequirementsSpecificationSub-Process

Rework in a Design Sub-Process

Requirements Rework May Be TriggeredDuring Design

Requirements Rework Process

Contains a Previously Executed Step

That We Saw Previously Here

Requirements Rework

Invocation of step originally defined as substep of Requirements

Requirements Rework

Same exceptionthrown

Requirements Rework

Different invocationcontext -gt differentresponse

High-Level Design

Declare and Define HLDesign Elements

Declare HLDesign Element

Requirements

~ A Rqmt OK

HLDesign OK

Define HLDesign Elements

High-Level Design

~ HLD OKDeclare HLDesign Elements

~ Rqmts OK9

AnotherRework-centered

Design Process

Coding

Develop Code Modules

Define Module Interfaces

Code All Modules

Define A Module Interface

X~Rqmts OK

~HLD OK

Low-Level Design

Requirements

High-Level Design

Coding

hellip

InterfaceOK

CodeOK

~LLD OK

~Code OK

~ A Rqmt OK

Coding

Final Observations

bull Many kinds of analysis applicable to processesndash FSVModel Checkingndash Fault Tree Analysisndash Reachability Analysisndash Discrete Event Simulation

bull Requires rigorous definitions of processes and propertieshazards

bull Broadly applicable to many diverse domains

What we have learned about system and process software could show us how to do better application software

bull Resource managementndash Which supports integration of humans ldquoinside the

boxrdquobull Rework

ndash Which entails effective use of retrospection inspection

Resource Management

bull A resource is an entity that is characterized byndash Ability to provide one or more ldquocapabilitiesrdquo

bull Capability The ability to support doing some taskactivitywork

ndash A set of descriptive attributesbull Attribute a (name value) pair

bull Capability set changes with context circumstancesndash Attribute values do too

bull A resource is a set ofndash Guarded capabilitiesndash Guarded attributes

Resource Management

bull Seems overlooked in application softwarendash Old view Software = algorithms + data ndash New view Software = activities +artifacts+resources

bull Resources rediscovered in Service-orientated software development

bull A big issue with lots of hard questions

Rework

bull Pervasive in virtually all creative activitiesbull The essence is in the data (and resources)

rather than the activitiesbull Rework is a kind of recursionbull State reification and inspection helpsbull So does historical retrospection

We should focus more on similarities than differences

bull Strong Temptations to do the oppositendash And good rewards too

bull Everything is different from everything elsebull But there are often important similarities too

ndash And we should try to learn from similaritiesndash And we should find strong rewards in doing so

What we do is more fundamental than we may think

bull Can we carry to other domains our clear insights intondash Abstractionndash Concurrencyndash AnalysisReasoningndash Phased development and evolution

bull And learn from related disciplines about ndash Reworkndash ldquohumans in the boxrdquondash Agentsresources

bull And fashion an understanding of something still far deeper and more fundamental

Thank you

Questions

present ID

Submit provisional

ballot

Recall the previous processMissing ID ExceptionInadmissible ID ExceptionID Mismatch Exception

Confirm voter has not votedexceptions

ID Mismatch

Issue regular ballot

Issue provisional ballot

Now elaborate the issue ballot step

present ID

Submit provisional

ballot

Now elaborate the issue ballot stepMissing ID ExceptionInadmissible ID ExceptionID Mismatch Exception

ID Mismatch

Detecting Vulnerabilities in This Process

bull Process has numerous checks and double-checks but are they enough

bull What combinations of incorrect performances could cause a hazard

bull Can the wrong artifact reachndash The wrong stepndash The wrong agent

bull How to find these situations

An Example Election Process Hazard An unqualified voter gets to vote with a regular ballot

Is modeled based upon our Little-JIL process definition asThe ldquorecord voter preferencerdquo step receives an incorrect ldquoballotrdquo artifact

Artifact flowbull Primarily along parent-child edges

ndash As procedure invocation parametersndash Passed to exception handlers too ndash Often omitted from coordination diagrams to reduce

visual clutterbull Parent-child data flow is inadequate

ndash Artifacts also need to flow laterallyndash And subtasks need to communicate with each other

bull Little-JIL also incorporates channelsndash For concurrency synchronization preemptionndash And for passing data laterally

present ID

Submit provisional

ballot

voterName gtgt

gtgt voterNamevoterRegistered gtgt

voterQualified gtgt

gtgt voterQualified

voterQualified==true

voterRegistered==true

voterQualified==falsevoterQualified==true

gtgt voterQualified

ballot gtgt

ballot gtgt ballot gtgt gtgt voterQualified

gtgt voterQualified

gtgt ballot

Add artifact flow (and adjust exception management)

Missing ID ExceptionInadmissible ID ExceptionID Mismatch Exception

The Fault Tree automatically derived from the Little-JIL this process definition

PRELIMINEARY RESULTS

Hazard an unqualified voter gets to vote with a regular ballot

Hazard specification using the FTA tool artifact ldquoballotrdquo input into the step ldquorecord voter preferencerdquo is wrong

The Resulting MCSs

bull There are 11 MCSs in the fault treebull Example

1 Step get voter name produces wrong voterName2 Step verify voter has not voted does not throw VoterUnregistered

exception (while checking prerequisite)3 Step check off voter as voted does not throw VoterUnqualified

exception (while checking prerequisite)4 Step issue regular ballot does not throw VoterUnqualified

exception (while checking prerequisite)

One informal interpretation An imposter provides an incorrectname but three different checks fail to pick this up

There are other interpretations too

1 Step get voter name produces wrong voterName2 Step verify voter has not voted does not throw

VoterUnregistered exception (while checking prerequisite)3 Step check off voter as voted does not throw

VoterUnqualified exception (while checking prerequisite)4 Step issue regular ballot does not throw VoterUnqualified

One Interpretation Imposter provides name of qualified voter who has not

yet voted

There is really only one incorrectly performed step namely the first oneThe others are ldquoincorrectrdquo only because they get incorrect inputmdashbut they

are correct given their inputs

present ID

Submit provisional

ballot

voterName gtgt

voterQualified gtgt

gtgt voterQualified

ballot gtgt

gtgt voterQualified

gtgt ballot

An impostor has the name of a registered voter who has not voted

Alternative Interpretation Imposter provides name of qualified voter who has voted but official does not notice

present ID

Submit provisional

ballot

voterName gtgt

voterQualified gtgt

gtgt voterQualified

ballot gtgt

gtgt voterQualified

gtgt ballot

Alternative interpretation An impostor has the name of a registered voter who has voted but the election official does not notice

Some Examples

Some Examples (2)

Our Approach

~100000 people each year in US hospitals due to preventable er

The Little-JIL Process Definition Language

And some exception management

Properties needed to support Finite-State Verification (Model-

Formally define the properties

Example property

Finite-state verification with FLAVERS

Violation detected

Violation detected (2)

Violation explanation

In Medical Domain

Documents

Reasoning About Precisely Defined Processes Leon J. Osterweil ([email protected]) Lab. For Advanced SE Research (LASER) University