Cleanroom Software Cleanroom Software EngineeringEngineeringBy Derek B. LarsonBy Derek B. Larson

Cleanroom Software Cleanroom Software EngineeringEngineering

►What is Cleanroom Software What is Cleanroom Software Engineering?Engineering?

►Cleanroom ProcessCleanroom Process►Waterfall Model into a CleanroomWaterfall Model into a Cleanroom►Case StudiesCase Studies

IntroductionIntroduction►Developed IBMDeveloped IBM►The stated goal of Cleanroom is

software that exhibit a zero defect rate. mathematical and statistical methods

►IBM developed a device controller product using Cleanroom Software Engineering that had zero failures in three years used at 300+ locations.

[1]

Intro Cont.Intro Cont.Hardware cleanrooms keep problems out bykeeping potential contaminating factors from reaching the product.

Why Cleanroom SEWhy Cleanroom SE►The reason to use Cleanroom Software

Engineering is simple: quality►Quality improvements of 10 to 20 times

have been reported when the Cleanroom process was demonstrated in industry

►If defects can cause loss of life or critical financial loss

►Increases in productivity

Why Cont.Why Cont.►Increases in productivity?

Cleanroom is a uniquely defined process that decreases required testing, error correction, and maintenance.

These savings offset any additional overhead needed for the quality control

“Errorless Software Development”. If no errors enter development, no errors need to be tested or debugged. Elimination of testing and debugging means faster product development.

Reqs for Cleanroom SEReqs for Cleanroom SE► Most spend a lot of time and money Most spend a lot of time and money

at the start of the project for at the start of the project for preventing defects.preventing defects.

► Most use statistical methods to Most use statistical methods to ensure quality.ensure quality.

► Most formally state and “prove” Most formally state and “prove” requirement needs. (YEAH ‘Z’)requirement needs. (YEAH ‘Z’)

[2]

Cleanroom FunctionsCleanroom Functions►Cleanroom uses three types of

functions►All code that is developed will follow

the basic structure of these functions. ►Because the functions are sound, the

code will likewise be sound.►These functions are called “Boxes”

[1]

The Three BoxesThe Three Boxes►Black BoxBlack Box►State Box State Box ►Clear Box Clear Box

Black BoxBlack Box►Black box is a view of an object that Black box is a view of an object that

hides the implementation process hides the implementation process and data .and data .

►By modeling code as a series of black boxes, we can ensure its quality verses our specification by ensuring that the actual black box performs according to the black box definition.

[1]

Black Box Cont.Black Box Cont.►It will describe how that system It will describe how that system

will respond to stimuli; this is done will respond to stimuli; this is done usually in a formal specification usually in a formal specification language. language. Z (Zed)Z (Zed)

Picture from http://uebb.cs.tu-berlin.de/zeta/zetadistdoc/Z-img13.gif

State BoxState Box►State box is where the view of an State box is where the view of an

object shows the data implementation, object shows the data implementation, but does not show the implementation but does not show the implementation process process

► It describes how state information is It describes how state information is being transformed being transformed

►In essence, the “history” of the black box is replaced by an existing “state”.

[1]

State Box Cont.State Box Cont.►This is an abstraction of the history

that allows us to take a higher-level view of the system

►Must ensure that there is no “history” case that is unaccounted for.

[1]

Clear BoxClear Box►Clear box shows both the data and Clear box shows both the data and

process implementationprocess implementation►The goal is to stepwise refine functions The goal is to stepwise refine functions

and prove them as being correct. and prove them as being correct. ►Clear boxes show what is actually

necessary to go between the old state and the new state.

[1]

Clear Box Cont.Clear Box Cont.►Sometimes there are multiple paths or

multiple states that can result from a state box – the clear box lets us examine and design these transitions with flow control.

►In the clear box, the procedures required to implement the state box transition function are defined explicitly.

[1]

Cleanroom Approach Cleanroom Approach ►Requirements Analysis Requirements Analysis

Producing and reviewing “informal Producing and reviewing “informal specifications.” specifications.”

►High-level Design High-level Design Converting the requirements into state Converting the requirements into state

machines and functions machines and functions ►Detailed Design Detailed Design

Further refinement of functions Further refinement of functions [3]

Cleanroom Approach Cont.Cleanroom Approach Cont.► Coding by incrementCoding by increment

Developing code and verifying it using formal Developing code and verifying it using formal methods. methods.

Compiling code or unit testing is prohibited Compiling code or unit testing is prohibited ► Pretest by increment Pretest by increment

Generation of test cases Generation of test cases ► Statistical Testing by IncrementStatistical Testing by Increment

The code is compiled, linked, and tested. Then The code is compiled, linked, and tested. Then the results are validated. the results are validated.

[3]

Cleanroom Approach Cont.Cleanroom Approach Cont.►Cleanroom Software Engineering Cleanroom Software Engineering

prohibits unit testing prohibits unit testing ► In a Cleanroom development, In a Cleanroom development,

correctness verification replaces unit correctness verification replaces unit testing and debugging testing and debugging

►After coding is complete, the software After coding is complete, the software immediately enters system test with immediately enters system test with no debugging. no debugging.

[3]

Cleanroom Approach Cont.Cleanroom Approach Cont.► All test errors are accounted for from the first All test errors are accounted for from the first

execution of the program with no private execution of the program with no private testing allowedtesting allowed

► The role of system testing in the Cleanroom The role of system testing in the Cleanroom process is to certify the quality of the process is to certify the quality of the software with the systems specifications in software with the systems specifications in mind. mind.

► Not doing unit testing can only be done if the Not doing unit testing can only be done if the above requirements are followed, that way above requirements are followed, that way many of the defects are already found and many of the defects are already found and fixed, so when the system is done coding it fixed, so when the system is done coding it should be close to no defects. should be close to no defects.

[3]

Correctness Verification►Once a piece of code is developed it goes Once a piece of code is developed it goes

through the Correctness Verification process.through the Correctness Verification process.► correctness verification phase takes the

developed code and compares it against the specification to see if it really does what is outlined in the specification.

►The specifications define the conditions that code must meet in order to fulfill the function for which it was developed.

[1]

Correctness Verification Cont.

►Correctness verification uses function-theoretic static code analysis to do just that.

►The term ‘function theoretic’ implies that there is a one-to-one mapping between the code and the specifications.

Correctness Verification Example

function isNumeric(char c){

if ((c >= ‘0’) and (c <= ‘9’))return true;

elsereturn false;

}

Example ExplanationExample Explanation► If the character passed in is in the set

[0,1,2,3,4,5,6,7,8,9] we expect the function to return true.

►Based on what we know about character sets and the language used to develop the code if the character is in the set then the logic ((c >= ‘0’) and (c <= ‘9’)) will return true.

►When the character is not in the set then the logic will return false.

►Both cases result in the expected behavior for the entire method, and so the code passes the correctness verification. [1]

Statistical Usage Testing► In conjunction with the box structure

specifications in the pre-development phase, usage specifications are created for the statistical testing phase.

►Usage specifications are simply descriptions of how the system will be eventually used.

►Usage models need to be defined for all possible scenarios for a given piece of code along with the probabilities of each scenario occurring. [1]

Statistical Usage Testing Cont.

►Use Markov Chains►Markov chains are essentially directed

graphs with nodes as states of use and arcs as the stimuli that cause state transitions.

►This is the most efficient way to test software, since the most destructive problems will be eliminated first, and money will not be spent on potentially harmless problems if it is not available. [1]

Waterfall Model into a Waterfall Model into a CleanroomCleanroom

►Waterfall model…we all know what Waterfall model…we all know what that isthat is

►We want to take the Cleanroom model We want to take the Cleanroom model and add some milestones to the and add some milestones to the model. model.

Waterfall Model into a Waterfall Model into a Cleanroom MilestonesCleanroom Milestones  

►Software Specification Review Software Specification Review    (Historical) Addresses requirements and (Historical) Addresses requirements and

external interfaces external interfaces (Cleanroom) Remains the same with increment (Cleanroom) Remains the same with increment

plan-mapping requirements to increment. plan-mapping requirements to increment. ►Preliminary Design Review Preliminary Design Review

(H) Top-level architecture (H) Top-level architecture (Cr) Top-level architecture updated as needed (Cr) Top-level architecture updated as needed

in later increments in later increments

Waterfall Model into a Waterfall Model into a Cleanroom MilestonesCleanroom Milestones  Cont.Cont.

►Critical Design ReviewCritical Design Review (H) Detailed Design(H) Detailed Design (Cr) Detailed Design of functionality for (Cr) Detailed Design of functionality for

particular increment particular increment ►Qualification Test Qualification Test

(H) Verify requirements (H) Verify requirements (Cr) Verify requirements. Performed on final (Cr) Verify requirements. Performed on final

increment. Earlier increments have informal increment. Earlier increments have informal QT QT

Case StudiesCase Studies►STARS STARS ►Tank-automotive and Armaments Tank-automotive and Armaments

Command Command

STARSSTARS►Developed by the Department of Developed by the Department of

DefenseDefense►STARS receives radar data and flight STARS receives radar data and flight

plan information and presents the plan information and presents the information to air traffic controllers on information to air traffic controllers on high resolution, 20" x 20" color high resolution, 20" x 20" color displays allowing the controller to displays allowing the controller to monitor, control, and accept hand-off monitor, control, and accept hand-off of air traffic of air traffic [4]

STARS Cont.STARS Cont.STARS is capable of tracking up to 1350 airborne aircraft simultaneously within a terminal area.  The system interfaces with multiple radars (up to 16 short and long range), 128 controller positions, 20 remote towers, and a 400 by 400 mile area of coverage. [4]

STARS Cont.STARS Cont.► The “STARS” program emphasized on three The “STARS” program emphasized on three

main points main points Process-driven Process-driven Re-use basedRe-use based Integrated software engineering environmentIntegrated software engineering environment

► STARS evaluated current "traditional" STARS evaluated current "traditional" processes. Then determined that a quality processes. Then determined that a quality management philosophy (putting decision management philosophy (putting decision making in the hands of workers, focusing on making in the hands of workers, focusing on processes, quantitative measurements) is processes, quantitative measurements) is critical and that Cleanroom Software critical and that Cleanroom Software Engineering follows this philosophy Engineering follows this philosophy [4]

STARS Cont.STARS Cont.►Cleanroom was combined with the Cleanroom was combined with the

TRW (spiral) Ada Process Model TRW (spiral) Ada Process Model ►Produced software at a rate of $30-40 Produced software at a rate of $30-40

per line of code versus industry per line of code versus industry averages of $130 per line for software averages of $130 per line for software of similar complexity and development of similar complexity and development timeframe (the size of the application timeframe (the size of the application is greater than 300 KLOC) is greater than 300 KLOC) [4]

STARS SavingsSTARS Savings►$130 - $30 = $100 per line of code$130 - $30 = $100 per line of code►The project is around 300K lines of codeThe project is around 300K lines of code►So, $100 * (around)300K = (around) So, $100 * (around)300K = (around)

$30,000,000$30,000,000►Could buy 30, million dollarCould buy 30, million dollar

houseshouses►Or about one month of Alex Or about one month of Alex

Rodriguez playing baseballRodriguez playing baseball[4]

Tank-automotive and Tank-automotive and Armaments CommandArmaments Command

►TACOM generates, provides, and sustains TACOM generates, provides, and sustains mobilitymobility, , lethalitylethality, and , and survivabilitysurvivability for for soldiers, other U.S. Armed Services, and soldiers, other U.S. Armed Services, and our allies - all to ensure Army readiness our allies - all to ensure Army readiness todaytoday, , tomorrowtomorrow, and , and beyondbeyond

Tank-automotive and Tank-automotive and Armaments Command Cont.Armaments Command Cont.

►After seven project increments After seven project increments (approximately 90K lines of code)(approximately 90K lines of code)

►4.2:1 productivity increase 4.2:1 productivity increase ►20:1 return on investment has been 20:1 return on investment has been

documented documented ►Projects experienced an overall testing Projects experienced an overall testing

error rate (represents all errors found error rate (represents all errors found in all testing) of 0.5 errors/KLOC in all testing) of 0.5 errors/KLOC [4]

SummarySummary►Cleanroom software development may be a

wonderful advance in the process of software development or may just be a downright weird approach, most likely a little of both.

► Looking at Cleanroom from a theorists’ point of view Cleanroom provides a theoretical foundation to software development in its use of mathematically based software development and statistical quality control.

SummarySummary►By not introducing errors into the

development phase there should be no testing requirement in the process.

►Statistical testing provides the benchmark as to performance and failure rate and helps verify the inputs to the development process by checking its outputs.

On the other HandOn the other Hand►Many people sees Cleanroom as too radical a

departure from conventional software development.

►The level of experience and training required to have a functional team of Cleanroom developers may not cost effective

►Developers develop code, and Cleanroom is more about specifications and statistical models than coding.

In the EndIn the End►One has to understand the appropriate use of

Cleanroom software development►Cleanroom is suitable for very particular types

of software where the human and financial risks of having errors are too great to be left to chance

►This generally does not fit the mold of mainstream software development, in which the concentration is often on getting the best price in the best time period.

► From what has been shown, Cleanroom is anything but a mainstream development process.

References References 1. Cleanroom Software Engineering.

http://www.criticaljunction.com/werbicki/SENG623/Group/SENG623W03_Cleanroom.pdf

2. Uta.edu, Cleanroom Software Engineering. Found at: www.uta.edu/cse/levine/fall99/cse5324/cr/clean/page.html

3. DACS, The Data and Analysis Center for Software. Found at: www.dacs.dtic.mil/databases/url/key.php?keycode=64

4.4. Carnegie Mellon, Software Engineering Institute. Carnegie Mellon, Software Engineering Institute. Found at: Found at: www.sei.cmu.edu/str/descriptions/cleanroom.htmlwww.sei.cmu.edu/str/descriptions/cleanroom.html