Introduction

On October 5, 1960, the American BallisticMissile Early-Warning System station at Thule,Greenland, indicated a large contingent of Sovietmissiles headed towards the United States .l For-tunately, common sense prevailed at the informalthreat-assessment conference that was immediatelyconvened : international tensions weren't particu-larly high at the time . The system had only recent-ly been installed . Kruschev was in New York, and allin ;all a massive Soviet attack seemed very unlikely .As a result no devastating counter-attack waslaunched . What was the problem? The moon hadrisen, and was reflecting radar signals back toearth . Needless to say, this lunar reflectionhadn't been predicted by the system's designers .

Over the last ten years, the Defense Depart-ment has spent many millions of dollars on a newcomputer technology called "program verification" -a branch of computer science whose business, in itsown terms, is to "prove programs correct" . Pro-gram verification has been studied in theoreticalcomputer science departments since a few seminalpapers in the 1960s , but it has only recentlystarted to gain in public visibility, and to beapplied to real world problems . General Electric,to consider just one example, has initiatedverification projects in their own laboratories :they would like to prove that the programs used intheir latest computer-controlled washing machineswon't have any "bugs" (even one serious one candestroy their profit margin) . 3 Although it usedto be that only the simplest programs could be"proven correct" - programs to put simple listsinto order, to compute simple arithmetic functions -slow but steady progress has been made in extendingthe range of verification techniques . Recent papershave reported correctness proofs for somewhat morecomplex programs, including small operatingsystems, compilers, and other material of modernsystemsystems, compilers,

aPrepared for the Symposium on UnintentionalNuclear War, Fifth Congress of the InternationalPhysicians for the Prevention of Nuclear War,Budapest, Hungary, June 28 - July 1, 1985 .

Copyright (c) 1985 Brian Cantwell Smith

THE LIMITS OF CORRECTNESS'

Brian Cantwell Smith*

18

What, we do well to ask, does this newtechnology mean? How good are we at it? Forexample, if the 1960 warning system had beenproven correct (which it was not), could we haveavoided the problem with the moon? If it werepossible to prove that the programs being writtento control automatic launch-on-warning systemswere correct, would that mean there could not bea catastrophic accident? In systems now beingproposed computers will make launching decisionsin a matter of seconds, with no time for any humanintervention (let alone for musings aboutKruschev's being in New York) . Do the techniquesof program verification hold enough promise sothat, if these new systems could all be provencorrect, we could all sleep more easily at night?These are the questions I want to look at today .Arid my answer, to give away the punch-line , is no .For fundamental reasons - reasons that anyone canunderstand - there are inherent limitations towhat can be proven about computers and computerprograms . Although program verification is animportant new technology, useful, like so manyother things, in its particular time and place,it should definitely not be called verification.Just because a program is "proven correct", inother words, you cannot be sure that it will dowhat you intend .

First some background .

2 . General Issues in Program Verification

Computation is by now the most importantenabling technology of nuclear weapons systems :it underlies virtually every aspect of the defensesystem, from the early warning systems, battlemanagement and simulation systems, and systemsfor communication and control, to the intricateguidance systems that direct the missiles to theirtargets . It is difficult, in assessing the chancesof an accidental nuclear war, to imagine a moreimportant question to ask than whether these per-

*Xerox Corporation : Intelligent Systems Laboratory,Palo Alto Research Center, 3333 Coyote Hill Road,Palo Alto, California 94304 USA ; Stanford Univer-sity : Department of Philosophy, and Center forthe Study of Language and Information, VenturaHall, Stanford, California 94305 USA ; and Compu-ter Professionals for Social Responsibility :P .O . Box 717, Palo Alto, California 94301 USA .

vasive computer systems will or do work correctly .

Because the subject is so large, however, Iwant to focus on just one aspect of computersrelevant to their correctness : the use of modelsin the construction, use, and analysis of computersystems . I have chosen to look at modellingbecause I think it exerts the most profound and,in the end, most important influence on the systemswe build . But it is only one of an enormous numberof important questions . First, therefore,- in orderto unsettle you a little - let me just hint at someof the equally important issues I will not address :

1 . Complexity: At the current state of the art,only very simple programs can be proven correct .Although it is terribly misleading to assumethat either the complexity or power of a com-puter program is a linear function of length,some rough numbers are illustrative . Thesimplest possible arithmetic programs are mea-sured in tens of lines ; the current state ofthe verification art extends only to programsof up to several hundred . It is estimated thatthe systems proposed in the Strategic DefenseInitiative (Star Wars), in contrast, will re-quire at least 10,000,000 lines of code .5 Byanalogy, compare the difference between re-solving a two-person dispute and settling thepolitical problems of the Middle East . There'sno a priori reason to believe that strategiessuccessful at one level will scale to the other .

2 . Harman interaction : Not much can be "proven",let alone specified formally, about actualhuman behaviour . The sorts of programs thathave so far been proven correct, therefore, donot include much substantial human interaction .On the other hand, as the moon-rise exampleindicates, it is often crucial to allow enoughhuman intervention to enable people to over-ride system mistakes . System designers,therefore, are faced with a very real dilemma :should they rule out substantive human inter-vention, in order to develop more confidencein how their systems will perform, or shouldthey include it, so that costly errors can beavoided or at least repaired? The Three-Mile Island incident is a trenchant exampleof just how serious this trade-off can get :the system desiqn provided for considerablehuman intervention, but then the operatorsfailed to act "appropriately" . Which strategyleads to the more important kind of correct-ness?

A standard way out of this dilemma is tospecify the behaviour of the system relativeto the actions of its operators .

But this, aswe will see below, pressures the designers tospecify the system totally in terms of inter-nal actions, not external effects . So you endup proving only that the system will behave inthe way that it will behave (i .e ., it will

raise this line level 3 volts), not do whatyou want it to do (i .e ., launch a missile onlyif the attack is real) . Unfortunately, thelatter is clearly what is important . Systemscomprising computers and people must functionproperly as integrated systems ; nothing isgained by showing that one cog in a misshapenwheel is a very nice cog indeed .

Furthermore, large computer systems aredynamic, constantly changing, embedded in com-plex social settings . Another famous "mis-take" in the American defense system happenedwhen a human operator mistakenly mounted atraining tape, containing a simulation of afull-scale Soviet attack onto a computer that,just by chance, was automatically pulled intoservice when the primary machine ran into a pro-blem . For some tense moments the simulationdata were taken to be the real thing .6 Whatdoes it mean to install a "correct" module in-to a complex social flux?

3 .

Levels of ballure :

Complex computer systemsmust work at many different levels . It fol-lows that they can fail at many differentlevels too . By analogy, consider the manydifferent ways a hospital could fail . First,the beams used to frame it might collapse . Orthey might perform flawlessly, but the operatingroom door might be too small to let in ahospital bed (in which case you would blame thearchitects, not the lumber or steel company.)Or the operating room might be fine, but thehospital might be located in the middle of thewoods, where no one could get to it (in whichcase you would blame the planners) . Or, to takea different example, consider how a lettercould fail . It might be so torn or soiledthat it could not be read . Or it might lookbeautiful, but be full of spelling mistakes .Or it might have perfect grammar, but disastrouscontents .

Computer systems are the same : they canbe "correct" at one level - say, in terms ofhardware - but fail at another (i .e ., the systemsbuilt on top of the hardware can do the wrongthing even if the chips are fine) . Sometimes,when people talk about computers failing, theyseem to think only the hardware needs to work .And hardware does from time to time fail,causing the machine to come to a halt, oryielding errant behaviour (as for example whena faulty chip in another American early warningsystem sputtered random digits' into a signal ofhow many Soviet missiles had been sighted,again' causing a false alert7 ) .

And the connec-tions between the computers and the world canbreak :' when the moon-rise problem was firstrecognized, an attempt to override it failedbecause an iceberg had acgidentally cut anundersea telephone cable .

But the more impor-tant point is that, in order to be reliable, asystem has' to be correct at every relevant level :the hardware is just the starting place (andby far the easiest, at that) . Unfortunately,however, we don't even know what all the rele-vant levels are . So-called "fault-tolerant"computers, for example, are particularly goodat coping with hardware failures, but the soft-

ware that runs on them is not thereby iproved .9

4 . Correctness and Intention : What does correctmean, anyway? Suppose the people want peace,and the President thinks that means havinga strong defense, and the Defense departmentthinks that means having nuclear weaponssystems, and the weapons designers requestcontrol systems to monitor radar signals, andthe computer companies are asked to respondto six particular kinds of radar pattern, andthe engineers are told to build signalamplifiers with certain circuit character-istics, and the technician is told to writea program to respond to the difference betweena two-volt and a four-volt signal on a parti-cular incoming wire . If being correct meansdoing what was intended, whose intent matters?The technician's? Or what, with twenty yearsof historical detachment, we would say shouldhave been intended?

With a little thought any of you could extend thislist yourself . And none of these issues eventouch on the intricate technical problems thatarise in actually building the mathematical modelsof software and systems used in the so-called"correctness" proofs . But, as I said, I want tofocus on what I take to be the most importantissue underlying all of these concerns : thepervasive use of models . Models are ubiquitousnot only in computer science but also in humanthinking and language ; their very familiaritymakes them hard to appreciate . So we'll startsimply, looking at modelling on its own, and comeback to correctness in a moment .

3 .

The Permeating Use of Models

When you design and build a computer system,you first formulate a model of the problem youwant it to solve, and then construct the computerprogram in its terms . For example, if you wereto design a medical system to administer drugtherapy, you would need to model a variety ofthings : the patient, the drug, the absorptionrate, the desired balance between therapy andtoxicity, and so on and so forth . The absorp-tion rate might be modelled as a number propor-tional to the patient's weight, or proportionalto body surface area, or as some more complexfunction of weight, age, and sex .

Similarly, computers that control trafficlights are based on some model of traffic -of how long it takes to drive across the inter-section, of how much metal cars contain (thesignal change mechanisms are triggered by metal-detectors buried under each street) . Bicyclists,as it happens, often have problems with automatictraffic lights, because bicycles don't exactlyfit the model : they don't contain enough iron totrigger the metal-detectors . I also once saw atractor get into trouble because it couldn't moveas fast as the system "thought" it would :

thecross-light went green when the tractor was onlyhalf-way through the intersection .

To build a model is to conceive of the worldin a certain delimited way. To some extent you

must build models before building any artifact atall, including televisions and toasters, but com-puters have a special dependence on these models :you write an explicit description of the modeldown inside the computer, in the form of a setof rules or what are called representations -essentially linguistic formulae encoding, inthe terms of the model, the facts and datathought to be relevant to the system's behaviour .It is with respect to these representations thatcomputer systems work .

In fact that's really whatcomputers are (and how they differ from othermachines) : they run by manipulating representa-tions, and representations are always formulatedin terms of models . This can all be summarizedin a slogan : no computation without represen-tation .

The models, on which the representations arebased, come in all shapes and sizes . Balsa modelsof cars and airplanes, for example, are used tostudy air friction and lift . Blueprints can beviewed as models of buildings : musical scores asmodels of a symphony . But models can also beabstract . Mathematical models, in particular, areso widely used that it is hard to think of any-thing that they haven't been used for : from wholesocial and economic systems, to personality traitsin teen-agers, to genetic structures, to the massand charge of sub-atomic particles . These models,furthermore, permeate all discussion and communi-cation . Every expression of language can beviewed as resting implicitly on some model of theworld .

What is important, for our purposes, is thatevery model deals with its subject matter at someparticular level of abstraction, paying attentionto certain details, throwing away others, groupingtogether similar aspects into common categories,and so forth . So the drug model mentioned abovewould probably pay attention to the patients'weight, but ignore their tastes in music . Mathe-matical models of traffic typically ignore thetemperments of taxi drivers . Sometimes what isignored is at too "low" a level ; sometimes too"high" : it depends on the purposes for which themodel is being used . So a hospital blueprint: wouldpay attention to the structure and connection ofits beams, but not to the arrangements.of proteinsin the wood the beams are made of, nor to theefficacy of the resulting operating room .

Models have to ignore things exactly becausethey view the world at a level of abstraction('abstraction' is from the Latin abstrahere, 'topull or draw away') . And it is good that they do :otherwise they would drown in the infinite rich-ness of the embedding world . Though this isn'tthe place for metaphysics, it would not be toomuch to say that every act of conceptualization,analysis, categorization, does a certain amountof violence to its subject matter, in order toget at the underlying regularities that groupthings together . If you don't commit that act ofviolence - don't ignore some of what's going on -you would become so hypersensitive and so over-come with complexity that you would be unable toact.

To capture all this in a word, we will say

that models are inherently partial .

All thinking,and all computation, are similarly partial . Fur-thermore - and this is the important point -thinking and computation have to be partial : that'show they are able to work .

4 . Pu.ZZ-blooded Action

Something that is not partial, however, isaction . When you reach out your hand and grasp aplow, it is the real field you are digging up, notyour model of it . Models, in other words, maybe abstract, and thinking may be abstract, andsome aspects of computation may be abstract, butaction is not . To actually build a hospital, toclench the steering wheel and drive through theintersection, or to inject a drug into a person'sbody, is to act in the full-blooded world, not ina partial or distilled model of it .

This difference between action and modellingis extraordinarily important . Even if your everythought is formulated in the terms OT some model,to act is to take leave of the model and partici-pate in the whole, rich, infinitely variegatedworld . For this reason, among others, action playsa crucial role, especially in the human case, ingrounding the more abstract processes of modellingor conceptualization . One form that grounding cantake, which computer systems can already takeadvantage of, is to provide feedback on how wellthe modelling is going . For example, if an indus-trial robot develops an internal three-dimensionalrepresentation of a wheel assembly passing by ona conveyor belt, and then guides its arm towardsthat object and tries to pick it up, it can usevideo systems or force sensors to see how well themodel corresponded to what was actually the case .The world doesn't care about the model : the clawswill settle on the wheel just in case the actuali-ties mesh .

Feedback is a special case of a very generalphenomenon : you often learn, when you do act, justhow good or bad your conceptual model was . Youlearn, that is, if you have adequate sensory ap-paratus, the capacity to assess the sensed exper-ience, the inner resources to revise and recon-ceptualize, and the luxury of recovering fromminor mistakes and failures .

5 .

Computers and Models

What does all this have to do with computersand with correctness? The point is that computers,like us, participate in the real world :

they takereal actions . One of the most important factsabout computers, to put this another way, is thatwe plug them in . They are not, as some theoreti-cians seem to suppose, pure mathematical ab-stractions, living in a pure detached heaven . Theyland real planes at real airports ; administer realdrugs ; and - as those of you here today know alltoo well - control real radars, missiles and com-mand systems . Like us, in other words, althoughthey base their actions on models, they have con-sequence in a world that inevitably transcends thepartiality of those enabling models . Like us, inother words, and unlike the objects of mathematics,they are challenged by the inexorable conflict bet-ween the partial but tractable model, and the

actual but infinite world .

And, to make the only too obvious point : wein general have no guarantee that the models areright - indeed we have no guarantee about much ofanything about the relationship between model andworld . As we will see, current notions of"correctness" don't even address this fundamentalquestion .

In philosophy and logic, as it happens, thereis a very precise mathematical theory called"model theory" . You might think that it would bea theory about what models are, what they are goodfor, how they correspond to the worlds they aremodels of, and so forth . You might even hope thiswas true, for the following reason : a great dealof theoretical computer science, and all of thework in program verification and correctness,historically derives from this model-theoretictradition, and depends on its techniques . Unfor-tunately, however, model theory doesn't addressthe model-world relationship at all . Rather,what model theory does is to tell you how yourdescriptions, representations, and programscorrespond to your model .

The situation, in other words, is roughly asdepicted in Figure 1, below . You are to imaginea description, program, computer system (or evena thought - they are all similar in this regard)in the left hand box, and the very real world inthe right . Mediating between the two is the in-evitable model, serving as an idealized or pre-conceptualized simulacrum of the world, in termsof which the description or program or whatevercan be understood . One way to understand themodel is as the glasses through which` the programor computer looks at the world :

it is` the world,that is, as the system sees it (though not, ofcourse, as it necessarily is) .

The technical subject of "model theory", asI have already said, is a study of the relation-ship on the left . What about the relationship onthe right? The answer, and one of the main pointsI hope you will take away from this discussion', isthat, at this point in intellectual history, we'have no theory of this right-hand side relation-ship .

COMPUTER

M0DE

Figure 1 : Computers, Models, and the EmbeddingWorld

There are lots of reasons for this, some verycomplex . For one thing, most of our currentlyaccepted formal techniques were developed, duringthe first half of this century, to deal with mathe-matics and physics . Mathematics is unique, withrespect to models, because (at least to a firstlevel of approximation) its subject matter is theworld of models and abstract structures, and there-fore the model-world relationship is relativelyunproblematic . The situation in physics is morecomplex, of course, as is the relationshipbetween mathematics and physics . How apparentlypure mathematical structures could be used tomodel the material substrate of the universe is aquestion that has exercised physical scientistsfor centuries . But the point is that, .whether ornot one believes that the best physical models domore justice and therefore less violence to theworld than do models in so-called "higher-level"disciplines like sociology or economics, formaltechniques don't themselves address the questionof adequacy .

Another reason we don't have a theory of theright-hand side is that there is very little agree-ment on what such a theory would look like . Infact all kinds of questions arise, when one studiesthe model-world relationship explicitly, aboutwhether it can be treated formally at all, aboutwhether it can be treated rigorously, even if notformally (and what the relationship is betweenthose two), about whether any theory will be morethan usually infected with prejudices and precon-deptions of the theorist, and so forth . The inves-tigation quickly leads to foundational questionsin mathematics, philosophy, and language, as wellas computer science . But none of what one learnsin any way lessens its ultimate importance .

In theend, any adequate theory of action, and, conse-quently, any adequate theory of correctness,will have to take the model-world relationshipinto account .

6 .

Correctness and Relative Consistency

Let's get back, then, to computers, and tocorrectness . As I mentioned earlier, the word'correct' is already problematic, especially asit relates to underlying intention . Is a programcorrect when it does what we have instructed itto do? or what we wanted it to do? or what historywould dispassionately say it should have done?Analysing what correctness should mean is toocomplex a topic to take up directly . What I wantto do, in the time remaining, is to describe whatsorts of correctness we are presently capable ofanalysing .

In order to understand this, we need tounderstand one more thing about building computersystems . I have already said, when you design acomputer system, that you first develop a model ofthe world, as indicated in the diagram . But youdon't, in general, ever get to hold the model inyour hand : computer systems, in general, are basedon models that are purely abstract . Rather, if youare interested in proving your program "correct",you develop two concrete things, structured in termsof the abstract underlying model (although these arelisted here in logical order, the program is veryoften written first) :

1 . A specification : a formal description in somestandard formal language, specified in termsof the model, in which the desired behaviour isdescribed ; and

2 . The program : a set of instructions and repre-sentations, also formulated in the terms of themodel, which the computer uses as the basis forits actions .

How do these two differ? In various ways, ofwhich one is particularly important . The programhas to say how the behaviour is to be achieved,typically in a step by step fashion (and often inexcruciating detail) . The specification, however,is less constrained : all it has to do is tospecify what proper behaviour would be, independentof how it is accomplished . For example, a specifi-cation for a milk-delivery system might simply be :"Make one milk delivery at each store, driving theshortest possible distance in total ." That's justa description of what has to happen . The program,on the other hand, would have the much more diffi-cult job of saying how this was to be accomplished .It might be phrased as follows : "drive four blocksnorth, turn right, stop at Gregory's Grocery Storeon the corner, drop off the milk, then drive 17blocks north-east, . . ." . Specifications, to useuse some of the jargon of the field, are essentiallydeclarative ; they are like indicative sentencesor claims . Programs, on the other hand, areprocedural : they must contain instructions thatlead to a determinate sequence of actions .

What, then, is a proof of correctness? Itis a proof that any system that obeys the programwill satisfy the specification .

There are, as is probably quite evident, twokinds of problems here . The first, often acknow-ledged, is that the correctness proof is in realityonly a proof that two characterizations of some-thing are compatible . When the two differ - i .e .,when you try to prove correctness and fail - thereis no more reason to believe that the first (thespecification) is any more correct than the second(the program) . As a matter of technical practice,specifications tend to be extraordinarily complexformal descriptions, just as subject to bugs anddesign errors and so forth as programs . In factthey are very much like programs, as this intro-duction should suggest . So what almost alwayshappens, when you write a specification and a pro-gram, and try to show that they are compatible, isthat you have to adjust both of them in order toget them to converge .

For example, suppose you write a program tofactor a number C, producing two answers A andB . Your specification might be :

Given a number C, produce numbers A andB such that A x B = C .

This is a specification, not a program, because itdoesn't tell you how to come up with A and B . Allit tells you is what properties A and B should have .In particular, suppose I say : ok, C is 5,332,114 ;what are A and B? Staring at the specificationjust given won't help you to come up with theanswer . Suppose, on the other hand, given this

specification, that you then write a program - say,by successively trying pairs of numbers until youfind two that work . Suppose further that you thenset out to prove that your program meets your spe-cifications . And, finally, suppose that thisproof can be constructed (I won't go into detailshere ; I hope you can imagine that such a proofcould be constructed) . With all three things inhand - program, specification, and proof - youmight think you were done .

In fact, however, things are rarely that sim-ple, as even this simple example can show . In par-ticular, suppose, after doing all this work, thatyou try your program out, confident that it mustwork because you have a proof of its correctness .You randomly give it 14 as an input, expecting 2and 7 . But in fact it gives you back the answersA = 1 and B = 14 . In fact, you realize upon fur-ther examination, it will always give back A = land B = C . It does this, even though you have aproof of its being correct, because you didn't makeyour specification meet your intentions . You wantedboth A and B to be different from C (and also differ-ent from 1), but you forgot to say that . In thiscase you have to modify both the program and thespecification . A plausible new version of thelatter would be :

Given a number C, produce A and B suchthat A J 1 and B J 1 and A x B = C .

And so one and so forth : the point, I take it, isobvious . If the next version of the programgiven 14, produces A = -1 and B - -14, you wouldsimilarly have met your new specification, butstill failed to meet your intention . Writing "good"specifications - which is to say, writing specifi-cations that capture your intention - is hard .

It should be apparent, nonetheless, thatdeveloping even straightforward proofs of "correct-ness" is nonetheless very useful . It typically forcesyou to delineate, very explicitly and completely, themodel on which both program and specification arebased . A great many of the simple bugs that occurin programs, of which the problem of producing 1and 14 was an example, arise from sloppiness andunclarity about the model . Such bugs are notidentified by the proof, but they are often un-earthed in the attempt to prove . And of course thereis nothing wrong with this practice ; anything thathelps to erradicate errors and increase confidenceis to be applauded . The point, rather, is to showexactly what these proofs consist in .

In particular, as the discussion has shown,when you show that a program meets its specifica-tions, all you have done is to show that two for-mal descriptions, slightly different in character,are compatible . This is why I think it is some-where between misleading and immoral for computerscientists to call this "correctness" . What iscalled a proof of correctness is really a proof ofthe compatibility or consistency between two formalobjects of an extremely similar sort : program andspecification .

As a community, we computer scien-tists should call this relative consistency, anddrop the word 'correctness' completely .

What proofs of relative consistency ignore is

the second problem intimidated earlier . Nothingin the so-called program verification process perse deals with the right-hand side relationship :the relationship between the model and the world .But, as is clear, it is over inadequacies on theright hand side - inadequacies, that is, in themodels in terms of which the programs and speci-fications are written - that systems so commonlyfail .

The problem with the moon-rise, for example,was a problem of this second sort . The difficultywas not that the program failed, in terms of themodel . The problem rather, was that the modelwas overly simplistic ; it didn't correspond towhat was the case in -the world . Or, to put itmore carefully, since all models fail to corres-pond to the world in indefinitely many ways, aswe have already said, it didn't correspond towhat was the case in a crucial and relevant way .In other words, to answer one of our originalquestions, even if a formal specification hadbeen written for the 1960 warning system, and aproof of correctness generated, there is no reasonto believe that potential difficulties with themoon would have emerged .

You might think that the designers weresloppy ; that they would have thought of the moonif they had been more careful . But it turns outto be extremely difficult to develop realisticmodels of any but the most artificial situations,and to assess how adequate these models are . Tosee just how hard it can be, think back on thecase of General Electric, and imagine writingappliance specifications, this time for a refri-gerator . To give the example some force, imaginethat you are contracting the refrigerator out tobe built by an independent supplier, and thatyou want to put a specification into the contractthat is sufficently precise to guarantee that youwill be happy with anything that the supplier de-livers that meets the contract .

Your first version might be quite simple -say, that it should maintain an internal tempera-ture of between 3 and 6 degrees Centigrade ; notuse more than 200 Watts of electricity ; cost lessthat $100 to manufacture ; have an internal volumeof half a cubic meter ; and so on and so forth .But of course there are hundreds of other pro-perties that you implicitly rely`'on :

it should,presumably, be structurally sound :

you wouldn'tbe happy with a deliciously cool plastic bag .It shouldn't weigh more than a ton, or emit loudnoises . And it shouldn't fling projectiles outat high speed when the`door' is' opened .

In general,it is impossible, when writing specifications, toinclude everything that you want :

legal contracts,and other humanly' interpretable specifications,are always stated within a background ofcommonsense, to cover the myriad unstated and unstatableassumptions assumed - to' hold in force .

(Currentcomputer programs, alas, have no common sense, asthe cartoonists know so well .)

So it is hard to make sure that everythingthat meets your specification will'really be arefrigerator ; it is also hard to make sure thatyour requirements don't rule out perfectly goodrefrigerators . Suppose for example a customer

plugs a toaster in, puts it inside the refrigeratorand complains that the object he received doesn'tmeet the temperature specification and must there-fore not be a refrigerator . Or suppose he triesto run it upside down . Or complains that itdoesn't work in outer space, even though you didn'texplicitly specify that it would only work withinthe earth's atmosphere . Or spins it a 10,000 rpm .Or even just unplugs it . In each case you wouldsay that the problem lies not with the refrigeratorbut with the use . But how is use to be specified?The point is that, as well as modelling theartifact itself, you have to model the relevantpart of the world in which it will be embedded .It follows that the model of a refrigerator as adevice that always maintains an internal temperatureof between 3 and 6 degrees is too"strict to coverall possible situations . One could try to modelwhat appropriate use would be, though specificationsdon't ordinarily, even try to identify all the rele-vant circumstantial factors . As well as therebeing a background set of constraints with respectto which a model is formulated, in other words, thereis also a background set of assumptions on which aspecification is allowed at any point to rely .

? . The Limits of Correctness

It's time to summarize what we've said so far .The first challenge to developing a perfectly"correct" computer system stems from the sheercomplexity of real-world tasks . We mentioned at theoutset various factors that contribute to thiscomplexity : human interaction, unpredictable fac-tors of setting, hardware problems, difficultiesin identifying salient levels of abstraction, etc .Nor is this complexity of only theoretical concern .A December 1984 report of the American DefenseScience Board Task Force on "Military Applicationsof New-Generation Computing Technologies" identifiesthe following gap between current laboratory demon-strations and what will be required for successfulmilitary applications - applications they call"Real World ; Life or Death" . In their estimationthe military now needs (and, so far as one cantell, expects to produce) an increase in the powerof computer systems of nine orders of magnitude,accounting for both speed and amount of informationto: be processed . That is a 1,000,000 .000-foldincrease over current research systems, equivalentto the difference between a full century of theentire New York metropolitan area, compared to oneday in the life of a hamlet of one hundred people .And remember that even current systems arealready several orders of magnitude more complexthat those for which we can currently developproofs of relative consistency .

But sheer complexity has not been our pri-mary subject matter . The second challenge tocomputational correctness, more serious, comes fromthe problem of formulating or specifying an appro-priate model . Except in the most highly artificialor constrained domains, modelling the embeddingsituation is an approximate, not a complete, endea-vour . It has the best hopes of even partial suc-cess in what Winograd has called "systematicdomains" : areas where the relevant stock of objects,properties, and relationships are most clearly andregularly pre-defined . Thus bacteremia, or ware-house inventories, or even flight paths of air-

planes coming into airports, are relativelysystematic domains, at least compared to conflictnegotiations, any situations involving intentionalhuman agency, learning and instruction, and soforth . The systems that land airplanes are hybrids- combinations of computers and people - exactlybecause the unforeseeable happens, and because whathappens is in part the result of human action,requiring human interpretation . Although it isimpressive how well the phone companies canmodel telephone connections, lines, and evendevelop statistical models of telephone use, at acertain level of abstraction, it would neverthe-less be impossible to model the content of thetelephone conversations themselves .

Third, and finally, is the question of whatone does about these first two facts . It isbecause of the answer to this last question thatI have talked, so far, somewhat interchangeablyabout people and computers . With respect to theultimate limits of models and conceptualization,both people and computers are restrained by thesame truths . If the world is infinitely rich andvariegated, no prior conceptualization of it, norany abstraction, will ever do it full justice .That's ok - or at least we might as well say thatit's ok, since that's the world we've got . Whatmatters is that we not forget about that richness- that we not think, with misplaced optimism, thatmachines might magically have access to a kind of"correctness" to which people cannot even aspire .

It is time, to put this another way, thatwe change the traditional terms of the debate .The question is not whether machines can do things,as if, in the background, lies the implicit assump-tion that the object of comparison is people . Plansto build automated systems capable of making a"decision", in a matter of seconds, to annihilateEurope, say, should make you uneasy ; requiring aperson to make the same decision in a matter ofthe same few seconds should make you uneasy too,and for very similar reasons . The problem isthat there is simply no way that reasoning of anysort can do justice to the inevitable complexityof the situation, because of what reasoning is .Reasoning is based on partial models . Which meansit cannot be guaranteed to be correct . Which means,to suggest just one possible strategy for action,that we might try, in our treaty negotiations,to find mechanisms to slow our weapons systemsdown .

It is striking to realise, once the comparisonbetween machines and people is raised explicitly,that we don't typically expect "correctness" forpeople in anything like the form that we pre-sume it for computers . In fact quite the opposite,and in a revealing way . Imagine, in some by-goneera, sending a soldier off to war, and giving him(it would surely have been a "him") final instruc-tions .

"Obey your commander, help your fellow-soldier", you might say, "and above all do yourcountry honour" . What is striking about this isthat it is considered not just a weakness, but apunishable weakness - a breach of morality - toobey instructions blindly (in fact, and for rele-vant reasons, you generally can't follow instruc-tions blindly ; they have to be interpreted to thesituation at hand) . You are subject to court-

martial, for example, if you violate fundamentalmoral principles, such as murdering women andchildren, even if following strict orders .

In the human case, in other words, our socialand moral systems seem to have built in an accep-tance of the uncertainties and limitations inherentin the model-world relationship . We know that theassumptions and preconceptions built into instruc-tions will sometimes fail, and we know that instruc-tions are always incomplete ; we exactly rely onjudgment, responsibility, consciousness, and soforth, to carry someone through those situations -all situations, in fact - where model and world partcompany . In fact we never talk about people, interms of their overall personality, being correct ;we talk about people being reliable, a much moresubstantive term . It is individual actions, fullysituated in a particular setting, that are corrector incorrect, not people in general, or systems .What leads to the highest number of correct humanactions is a person's being reliable, experienced,capable of good judgment, etc .

There are two possible morals here, forcomputers . The first has to do with the notionof experience . In point of fact, program verification is not the only, or even the most common,method of obtaining assurance that a computersystem will do the right thing . Programs areusually judged acceptable, and are typically accep-ted into use, not because we prove them "correct",but because they have shown themselves relativelyreliable in their destined situations for somesubstantial period of time . And, as part of thisexperience, we expect them to fail : there alwayshas to be room for failure . Certainly no onewould ever accept a program without this in situtesting : a proof of correctness is at best addedinsurance, not a replacement, for real life expe-rience . Unfortunately, for the ten million linesof code that is supposed to control and coordinatethe Star Wars Defense System, there will never,God willing, be an in situ test .

One answer, of course, if genuine testing isimpossible, is to run a simulation of the realsituation . But simulation, as our diagram shouldmake clear, -tests onZy the Zeft-hand side reZation-ship .

Simulations are defined in terms of models ;they don't test the relationship between the modeland world . That is exactly why simulations andtests can never replace embedding a program inthe real world . All the war-games we hear aboutand hypothetical military scenarios, and electronicbattlefield simulators, and so forth, are allbased on exactly the kinds of models we have beentalking about all along . In fact the subject ofsimulation, worthy of a whole analysis on its own,is really just our whole subject welling up allover again.

I said earlier that there were two morals tobe drawn, for the computer, from the fact that weask people to be reliable, not correct . The secondmoral-is for those who, when confronted with thefact that genuine or adequate experience cannot behad, would say "oh, well, let's build responsibilityand morality into the computers - if people canhave it, there's no reason why machines can't haveit too ." Now I will not argue that this is inhe-

rently impossible, in a metaphysical or ultimatephilosophical sense, but a few short comments arein order . First, from the fact that humans some-times are responsible, it does not follow that weknow what responsibility is : from tacit skills noexplicit model is necessarily forthcoming . We sim-ply do not know what aspects of the human conditionunderlie the modest levels of responsibility towhich we sometimes rise . And second, with respectto the goal of building computers with even humanlevels of full reliability and responsibility, Ican state with surety that the present state ofartificial intelligence is about as far from thisas mosquitos are from flying to the moon .

But there are deeper morals even than these .The point is that even if we could make computersreliable, they still wouldn't necessarily alwaysdo the correct thing . People aren't provably "cor-rect", either : that's why we hope they are respon-sible, and it is surely one of the major ethicalfacts is that correctness and responsibility don'tcoincide . Even if, in another 1,000 years, someonewere to devise a genuinely responsible computersystem, there is no reason to suppose that it wouldachieve "perfect correctness" either, in the senseof never doing anything wrong . This isn't a failurein the sense of a performance limitation ; it stemsfrom the deeper fact that models must abstract, inorder to be useful . The lesson to be learned fromthe violence inherent in the model-world relation-ship, in other words, is that there is an inherentconflict between the power of analysis and concep-tualization, on the one hand, and sensitivity tothe infinite richness, on the other .

But perhaps this is an overly abstract way toput it . Perhaps, instead, we should just rememberthat there will always be another moon-rise .

Notes

1 . Edmund Berkeley, The Computer Revolution,Doubleday, 1972, pp . 175-177, citing news-paper stories in the Manchester GuardianWeekly of Dec . 1, 1960, a UPI dispatch pub-lished in the Boston Traveller of Dec . 13,1960, and an AP dispatch published in theNew York Times on Dec . 23, 1960 .

2 . McCarthy, John, "A Basis for a MathematicalTheory of Computation", 1963, in P .Braffortand D . Hirschbert, eds ., Computer Programmingand Formal Systems, Amsterdam : North-Holland,1967, pp . 33-70 . Floyd, Robert, "AssigningMeaning to Programs", Proceedings of Symposiain Applied Mathematics 19, 1967 (also in F .T .Schwartz, ed, Mathematical Aspects ofComputer Science, Providence : American Math-ematical Society, 1967) . Naur, P ., "Proof ofAlgor.ithms by General Snapshots", BIT Vol . 6No . 4, pp . 310-316, 1966 .

3 . Al Stevens, BBN Inc ., personal communication .

4 . See for example R .S . Boyer, and Moore, J .S .,eds ., The Correctness Problem in ComputerScience, London : Academic Press, 1981 .

5 . Fletcher, James, study chairman, and McMillan,Brockway, panel chairman, Report of the Study

on Eliminating the Threat Posed by NuclearBallistic Missiles (U), Vol . 5, BattleManagement, Communications, and Data Pro-cessing (U), U .S . Department of Defense,February 1984 .

6 . See, for example, the Hart-Goldwater reportto the Committee on Armed Services of theU .S . Senate : "Recent False Alerts from theNation's Missile Attack Warning System"(Washington, D .C . : U .S . Government PrintingOffice, Oct . 9, 1980) ; Physicians forSocial Responsibility, Newsletter,"Accidental Nuclear War", (Winter 1982),P . 1 .

7 . Ibid .

8 . Berkeley, op . cit . See also Daniel Ford'stwo part article "The Button", New Yorker,April 1, 1985, p . 43, and April 8, 1985,p . 49, excerpted from Ford, Daniel, TheButton, New York : Simon and Schuster,1985 .

9 . In point of fact, developing software forfault-tolerant systems is an extremely trickybusiness .