Cleanroom Software

Engineering

Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 25

Required Reading and Additional

Reference

• Required Reading:

– Linger, Cleanroom Software Engineering for Zero-Defect Software, Proceedings, 15th Int. Conf. on Soft. Eng. (1993), IEEE Computer Society Press, pp. 2-13.

• Additional relevant reference:

– Linger, Trammell, Cleanroom Software Engineering Reference Model. CMU/SEI-96-TR-022, Software Engineering Institute, 1996.

Cleanroom SE Philosophy

• Cleanroom Software Engineering is a software development philosophy.

• First introduced in the ‘80s within IBM by Harlan Mills.

• Based on the notion that defects in software should be avoided rather than detected and repaired.

• Software development should not be viewed as a trial-and-error undertaking.

(cont’d)

Cleanroom SE Philosophy

• Cleanroom Software Engineering is a software development philosophy.

• First introduced in the ‘80s within IBM by Harlan Mills.

• Based on the notion that defects in software should be avoided rather than detected and repaired.

• Software development should not be viewed as a trial-and-error undertaking.

(cont’d)

Cleanroom SE Philosophy

• Cleanroom Software Engineering is a software development philosophy.

• First introduced in the ‘80s within IBM by Harlan Mills.

• Based on the notion that defects in software should be avoided rather than detected and repaired.

• Software development should not be viewed as a trial-and-error undertaking.

(cont’d)

Cleanroom SE Philosophy

• Cleanroom Software Engineering is a software development philosophy.

• First introduced in the ‘80s within IBM by Harlan Mills.

• Based on the notion that defects in software should be avoided rather than detected and repaired.

• Software development should not be viewed as a trial-and-error undertaking.

(cont’d)

Cleanroom SE Philosophy (cont’d)

“In traditional software development, errors were regarded as inevitable. Programmers were urged to get software into execution quickly, and techniques for error removal were widely encouraged. The sooner the software could be written, the sooner debugging could begin.”

(cont’d)

Cleanroom SE Philosophy (cont’d)

“Today, debugging is understood to be the most error-prone process in software development, leading to ‘right in the small, wrong in the large’ programs...”

Characteristics

• Team-oriented…

“The functional specification is created by the development team, or by a separate specification team for large projects, and the usage specification is created by the certification team.”

• Object-based box structure specification and design

• Stepwise refinement

(cont’d)

Characteristics

• Team-oriented…

“The functional specification is created by the development team, or by a separate specification team for large projects, and the usage specification is created by the certification team.”

• Object-based box structure specification and design

• Stepwise refinement

(cont’d)

Characteristics

• Team-oriented…

“The functional specification is created by the development team, or by a separate specification team for large projects, and the usage specification is created by the certification team.”

• Object-based box structure specification and design

• Stepwise refinement

(cont’d)

Characteristics

• Team-oriented…

“The functional specification is created by the development team, or by a separate specification team for large projects, and the usage specification is created by the certification team.”

• Object-based box structure specification and design

• Stepwise refinement

(cont’d)

Characteristics (cont’d)

• Uses function-theoretic correctness verification – components are not executed or developer-tested!

“Team correctness verification takes the place of unit testing and debugging, and software enters system testing directly, with no execution by the development team...no private debugging (is) permitted.”

(cont’d)

Characteristics (cont’d)

• Uses function-theoretic correctness verification – components are not executed or developer-tested!

“Team correctness verification takes the place of unit testing and debugging, and software enters system testing directly, with no execution by the development team...no private debugging (is) permitted.”

(cont’d)

Characteristics (cont’d)

• Statistical usage testing (of integrated increments) is undertaken for quality certification (‘‘statistical quality control’’).

“The certification (test) team is responsible for...certifying the quality of software with respect to its specification. Certification is carried out by statistical usage testing that produces objective assessments of product quality.”

(cont’d)

Characteristics (cont’d)

• Statistical usage testing (of integrated increments) is undertaken for quality certification (‘‘statistical quality control’’).

“The certification (test) team is responsible for...certifying the quality of software with respect to its specification. Certification is carried out by statistical usage testing that produces objective assessments of product quality.”

(cont’d)

Characteristics (cont’d)

• Incremental development…

“Management planning and control...is based on developing and certifying a pipeline of software increments that accumulate to the final product.”

• Structured programming

Characteristics (cont’d)

• Incremental development…

“Management planning and control...is based on developing and certifying a pipeline of software increments that accumulate to the final product.”

• Structured programming

Characteristics (cont’d)

• Incremental development…

“Management planning and control...is based on developing and certifying a pipeline of software increments that accumulate to the final product.”

• Structured programming

Impact on Development Cycle

“Experienced...teams...can achieve

substantially reduced product development cycles. The precision of Cleanroom development eliminates rework and results in dramatically reduced time for certification testing compared to traditional methods. And Cleanroom teams are not hostage to error correction following product release.”

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Stimulus

Stimulus History

Response

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Stimulus

Stimulus History

Response

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Old State

New State

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Box Structure Specification and

Design

• Incorporates black box (external behavior), state box (retained data), and clear box(processing) forms.

• “Transition Functions:”

– Black box: (S, SH -> R)

– State box: (S, OS) -> (R, NS)

– Clear box: (S, OS) -> (R, NS) by procedure (intended function)

• Intended functions are refined into control structures (programs)

Verification

• Development teams employ mental proofs of correctness in team reviews…

“Every correctness condition of every control structure is verified – every team member must agree that each condition is correct.”

Verification

• Development teams employ mental proofs of correctness in team reviews…

“Every correctness condition of every control structure is verified – every team member must agree that each condition is correct.”

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification

• Based on statistical quality control in manufacturing

• Process (statistical usage testing):

– sample population of user executions based on expected frequency (stratified random sampling): operational profile

– measure quality by determining if executions are correct

– extrapolate to the population of possible executions (statistical inference)

– if quality is inadequate, identify and correct flaws in development process

(cont’d)

Quality Certification (cont’d)

• Alternate distributions can be defined for low-probability, high-consequence functions.

• Errors tend to be found in failure-rate order on average (coverage testing is not biased to find errors in any particular rate order).

Quality Certification (cont’d)

• Alternate distributions can be defined for low-probability, high-consequence functions.

• Errors tend to be found in failure-rate order on average (coverage testing is not biased to find errors in any particular rate order).

Cleanroom Software

Engineering

Prepared by

Stephen M. Thebaut, Ph.D.

University of Florida

Software Testing and Verification

Lecture 25