2
Ediior’s Comer: Robert L. Glass Some Thoughts on Software Errors* This is a set of personal thoughts, gleaned from my nearly 35 years of experience in the industrial and academic worlds of software engaging. In these thoughts, I: attempt to collect and focus some insight on the software engineering process and the software engineering product, using software errors as the magni- fying glass through which the collecting and focusing OCCUTS. Thought l-Not all software errors can be found by known error-removal techniques. This thought is about process. The known processes for removing software errors can be categorized as a) review, b) test, and c) proof. We know a lot about the worth of these processes. Review-all the levels of review, from requirements to code-has been shown to be the most cost-effective of the three. Testing-all the levels of testing, from unit to system-has been shown to be an absolutely essential supplement to review. Proof has yet to prove itself as an industrial-strength error removal technique, but it offers a third and d~ati~~ly different alternative to the other two. But what is really the important part of this thought is that no one of these processes is enough. Extrapolating that thought a little further, it is fair to conclude that . . . there is not now a process answer to the problem of software errors. Thought 2-Not all software errors are found EVER. This thought is about product. Many people believe that the software product is the most complex thing ever constructed by human beings. Certainly, when we look at the number of logic segments in a program, and then try to consider the laby~n~ian paths made up of those segments, we encounter a combinatorial explosion and a near infinity of different ways the executing program can proceed Small wonder that error-free software has been an elusive target. The thought that we cannot remove all *This material was originally presented as a talk at the Third International Conference on Fault-Tolerant Computing Systems. errors from software is an important one. It does not excuse us from trying as hard as we can to remove all the errors-that is not the importance of this thought. What is important is that . . . software in critical systems must take precautions beyond error removal for defense against software errors. That, in fact, is the reason we have the emerging and important discipline of fault tolerant software. Fault tolerance is the fallback solution when the inevitable error occurs. It must be a~~~ that this is a con~ove~i~ ~ougllt . Harlan Mills and others from IBM claim to be able to produce error-free software using properly psychologi- cally conditioned software engineers and disciplined use of certain processes. Perhaps they, unlike others who have made similar claims in the past, will be able to do what they say. But it is safe to say that both the state of the art and the state of the practice in 1988 demand that the possibility of latent errors in critical software must be dealt with. Thought 3-Not all software error-finders are equal. This thought is about the people who use the process to build the product. We have seen again and again that there are enormous differences between software devel- opers, ranging in magnitude up to 30-l. Glenford Myers found that some software professionals found seven times as many software errors in a software product as others. Nancy Leveson found that only nine of 24 participants in a software study found any errors at all in a software product with 60 known errors in it. The cover of Barry Boehm’s Software Engineering Economics shows us very dramati~ly that the quality of software people has a far more profound impact on software productivity than any known software process. What this thought tells us is this . . . the most important element in software error removal is not the nature of the product or the nature of the process, but the choice of people. The JoumatOfSystermand Softwa~ 9.243-244 (1989) @ 1989 Elsevia Science ~blishing Co., Inc., 1989 243

Editor's corner: Some thoughts on software errors

Embed Size (px)

Citation preview

Page 1: Editor's corner: Some thoughts on software errors

Ediior’s Comer: Robert L. Glass

Some Thoughts on Software Errors*

This is a set of personal thoughts, gleaned from my nearly 35 years of experience in the industrial and academic worlds of software engaging. In these thoughts, I: attempt to collect and focus some insight on the software engineering process and the software engineering product, using software errors as the magni- fying glass through which the collecting and focusing OCCUTS.

Thought l-Not all software errors can be found by known error-removal techniques.

This thought is about process. The known processes for removing software errors can be categorized as a) review, b) test, and c) proof. We know a lot about the worth of these processes. Review-all the levels of review, from requirements to code-has been shown to be the most cost-effective of the three. Testing-all the levels of testing, from unit to system-has been shown to be an absolutely essential supplement to review. Proof has yet to prove itself as an industrial-strength error removal technique, but it offers a third and d~ati~~ly different alternative to the other two.

But what is really the important part of this thought is that no one of these processes is enough. Extrapolating that thought a little further, it is fair to conclude that . . .

there is not now a process answer to the problem of software errors.

Thought 2-Not all software errors are found EVER.

This thought is about product. Many people believe that the software product is the most complex thing ever constructed by human beings. Certainly, when we look at the number of logic segments in a program, and then try to consider the laby~n~ian paths made up of those segments, we encounter a combinatorial explosion and a near infinity of different ways the executing program can proceed Small wonder that error-free software has been an elusive target. The thought that we cannot remove all

*This material was originally presented as a talk at the Third International Conference on Fault-Tolerant Computing Systems.

errors from software is an important one. It does not excuse us from trying as hard as we can to remove all the errors-that is not the importance of this thought. What is important is that . . .

software in critical systems must take precautions beyond error removal for defense against software errors.

That, in fact, is the reason we have the emerging and important discipline of fault tolerant software. Fault tolerance is the fallback solution when the inevitable error occurs.

It must be a~~~ that this is a con~ove~i~ ~ougllt . Harlan Mills and others from IBM claim to be able to produce error-free software using properly psychologi- cally conditioned software engineers and disciplined use of certain processes. Perhaps they, unlike others who have made similar claims in the past, will be able to do what they say. But it is safe to say that both the state of the art and the state of the practice in 1988 demand that the possibility of latent errors in critical software must be dealt with.

Thought 3-Not all software error-finders are equal.

This thought is about the people who use the process to build the product. We have seen again and again that there are enormous differences between software devel- opers, ranging in magnitude up to 30-l. Glenford Myers found that some software professionals found seven times as many software errors in a software product as others. Nancy Leveson found that only nine of 24 participants in a software study found any errors at all in a software product with 60 known errors in it. The cover of Barry Boehm’s Software Engineering Economics shows us very dramati~ly that the quality of software people has a far more profound impact on software productivity than any known software process.

What this thought tells us is this . . .

the most important element in software error removal is not the nature of the product or the nature of the process, but the choice of people.

The Joumat Of Systerm and Softwa~ 9.243-244 (1989) @ 1989 Elsevia Science ~blishing Co., Inc., 1989

243

Page 2: Editor's corner: Some thoughts on software errors

Editor’s Corner

Unfortunately, this thought does not tell us how to identify those people. We must still use subjective and ad hoc techniques to try to optimize our people choices.

Thought 4-Not all software errors are equal.

This thought is about focusing the preceeding thoughts. First of all, not all software errors are bad. Sometimes the benefit of removing an error is less than the cost of removing it. Furthermore, researchers such as Elliot Soloway who study what good software engineers do have found that good people tend to make more false starts than “bad” people-that is, at least early in the life cycle, the path to good software must inevitably lead through “errors” as ideas are tried and rejected. Here, we see clearly that some errors are not as bad as others.

More important, some software errors are much worse than others. Those are a) the errors that persist through all the error-removal processes into the produc- tion use of the software, and then b) cause the system of which the software is a part to do unsafe or expensive things. I showed in a 1981 study that these persistent errors tend to be those where the software is less complex than the problem the software is to solve-for example, the software tests two conditions in an IF statement but neglects a third, or a logic branch fails to reset a variable it has used back to the value expected in the next segment encountered. Nancy Leveson has linked software errors and system safety, and has shown further that independent software developers tend to make common errors. Timm Gramms calls these “bi- ased errors” and says that they come from “thinking traps.”

The importance of this thought is that . . .

both error removal and fault tolerance must concentrate on the worst of the errors-the ones that can result in unsafe systems.

SUMMARIZING THOUGHTS Through the magnifying glass of software errors, we can see and focus clearly on several implant conclusions about software itself:

1. There is not now a process answer to the problem of software errors. It is not that we lack for process. We have l review, test, proof processes l requirements and structural testing l statistical and safety testing . unit, integration and system tests l a plethora of tools and techniaues The problem appears, simply, to be larger than any processes we know.

2. Software in critical system must take precautions beyond error removal for defense against software errors. The product, for all of our efforts and all of our wishes, must be treated as if it still has errors in it.

3. The most important element in software error removal (and, for that matter, in all software work) is the choice of people to do the work. Good people will tend to find good solutions. “Bad” people will tend to find much worse ones.

4. Both error removal and fault tolerance must concentrate on the worst of the errors. Some errors appear to be easy to make, some errors appear to be hard to find, and some errors result in unsafe systems. More often than not, the easy to make and hard to find errors, in fact, lead to the unsafe system.

The news in 1989, then, is not as good as we wish it were. Software error removal and fault tolerance must employ

l carefully chosen people l a variety of processes of known effectiveness l understanding of the nature of errors and safety.

Anything less than that is simply not good enough.