Conscious Machines - Minsky

7/29/2019 Conscious Machines - Minsky

1/11

http://kuoi.com/~kamikaze/doc/minsky.html

Conscious Machines

Marvin Minsky

M.I.T.

Published in "Machinery of Consciousness", Proceedings, National Research

Council of Canada, 75th Anniversary Symposium on Science in Society, June1991. I don't have the final publication date.

Many people today insist that no machine could really think. "Yes," they say,

"machines can do many clever things. But all of that is based on tricks, just

programs written by people to make those machines obey preconceived rules.The results are useful enough -- but nowhere in those cold machines is there

any feeling, meaning, or consciousness . Those computers simply have nosense that anything is happening."

They used to say the same about automata vis-a-vis animals. "Yes, thoserobots are ingenious, but they lack the essential spark of life." Biology then,

and psychology now: each was seen to need some essence not mechanical.

The world of science still is filled with mysteries. We're still not sure of how

the Sun produces all its heat. We do not know precisely where our early

ancestors evolved. We can't yet say to what extent observing violence leads to

crime. But questions like those do not evoke assertions of futility. We can try

harder to detect more neutrinos, find more fossils, or perform throrough

surveys. However, in certain areas of thought, more people take a differentstance about the nature of our ignorance. They proceed to work hard, but not

toward finding answers, but toward trying to show that there are none. ThusRoger Penrose's book [1] tries to show, in chapter after chapter, that human

thought cannot be based on any known scientific principle.

I have already written a book [2] that discusses various attempts to show that

men are not machines, but mainly works to demonstrate how the contrary

might well be so. You might object that no one has time to read all such

books, so why can't I just summarize? And that's what this essay is all about:that certain things are too complex to summarize! This includes the

mechanisms of highly evolved organisms and, especially, the workings oftheir nervous systems. It also includes the highly evolved systems that we call

cultures and societies. And especially, it includes what we call consciousness.

In particular, consider the problem of describing the brain in detail -- in view

of the fact that it is the product of tens of thousands of different genes. We cancertainly see the attractiveness of proposing to get around all that stuff, simply
http://kuoi.com/~kamikaze/doc/minsky.htmlhttp://kuoi.com/~kamikaze/doc/minsky.html


2/11

by postulating some novel "basic" principle by which our minds are animated

by some vital force or essence we call Mind, or Consciousness, or Soul.

That tendency is not confined to religion and philosophy. The same approach

pervades our everyday psychology. We speak of making decisions by

exercising 'freedom of will'; or by finding what something 'means', or of

discovering truths by means of 'intuition'. But none of those terms explains

very much; each only serves to name another set of mysteries.

The situation is different in Physics. Consider the whirlpools that form whenwater flows down drains. When a scientist says that this can be explained by

the Conservation of Momentum, that's very different from attributing it tosome convenient Whirlpool God -- because precisely the very same

mathematical rule can be used to explain and predict a vast range of other

phenomena, with a precision and lack of exception found in no other realm ofideas. That principle apparently applies to everything in our universe and,

because of its singularly good performance, we regard this sort of

"fundamental" or "unified" principle as an ideal prototype of how to accountfor mysterious phenomena. But one can carry that quest too far by only

seeking new basic principles instead of attacking the real detail. This is what Isee in Penrose's quest for a new basic principle of physics that will account for

consciousness.

The trouble is that this approach does not work well for systems whose

behavior has evolved through the accretion of many different mechanisms,over the course of countless years. For example, in physiology, the excretion

of excess potassium in the urine occurs because our ancestors evolvedelaborates system of receptors and transport mechanisms, along with intricate

machinery for controlling them. This is understood so well today, that no onefeels that there's any need to postulate a separate, special principle for the

Conservation of Potassium. Progress in this area is no longer news for biology

because we have seen two hundred years of great success accrued from

working out details. Since Harvey, Darwin and Pasteur, the idea of a Vital

Force has nearly vanished from biology. Why is it still so much a part ofpresent-day psychology?

I'll argue that vitalism still persists because we're only starting to find a way to

understand the brain. (I see this as the irony of Penrose's book, because thepath toward understanding lies in that flood of new ideas that began to grow

around the time, half a century ago, along with the emergence of computers in

the 1950s -- include the work of Turing in 1936, McCulloch and Pitts in 1943,

and the hundreds of thinkers who joined them afterward. Yet Penrose takes

the other side, and argues that the abilities of human mathematicians to

discover new mathematical truths cannot be explained on the basis ofanything a machine could do. He argues in [1], p110, that this kind of thinking


3/11

must be based on "insights that cannot be systematized -- and, indeed, must lie

outside any algorithmic action!" He bases this on the assumption that anythinking machine we build for attempting to discovering knowledge about

mathematics must itself be based on some absolutely consistent logical

foundation -- that is, one that cannot possibly produce any type of logicalcontradiction or inconsistency. This is the same assumption used in Godel's

celebrated 'incompleteness theorem'. Penrose's application of this idea to

psychology is due, as Penrose notes, to J.R.Lucas, in Philosophy, 36, pp120-4, 1961.

Consistency

It seems to me that all of this stands upon a single and simple mistake. It

overlooks the possibility, as my colleague Drew McDermott once remarked,of including systems "that are mistaken about mathematics to some degree, orsystems that can change their minds." By inadvertently ruling such machines

out, you've simply begged the question whether human mathematicians can be

kinds of machines -- because people do indeed change their minds, and can

indeed be mistaken about some parts of mathematics. An entire generation of

logical philosophers has thus wrongly tried to force their theories of mind to

fit the rigid frames of formal logic. In doing that, they cut themselves off fromthe powerful new discoveries of computer science. Yes, it is true that we can

describe the operation of a computer's hardware in terms of simple logical

expressions. But no, we cannot use the same expressions to describe themeanings of that computer's output -- because that would require us to

formalize those descriptions inside the same logical system. And this, I claim,

is something we cannot do without violating that assumption of consistency.

If you are not a logician, then you might wonder what's all the fuss about.

"What could possibly be wrong with logical consistency. Who wants thosecontradictions, anyway?" The trouble with this is that the problem is worse

than it looks: paradoxes start to turn up as soon as you permit your machine touse ordinary common-sense reasoning. For example, troubles appear as soon

as you try to speak about your own sentences, as in "this sentence is false" or

"this statement has no proof" or in "this barber shaves all persons who don't

shave themselves." The trouble is that when you permit "self reference" you

can quickly produce absurdities. Now you might say, "Well then, why don't

we redesign the system so that it cannot refer to itself?" The answer is that thelogicians have never found a way to do this without either getting into worse

problems, or else producing a system too constrained to be useful.

Then what do ordinary people do? So far as we know they scarcely use any

logic at all. The studies made by the great child psychologist Jean Piagetsuggest that the abilities required for to manipulating formal expressions are


4/11

not reliably available to children until their second decade, if ever. And even

as a mathematician, I cannot recognize the psychology Penrose describes.When doing mathematics, my mind is filled with many things non-logical. I

imagine examples based on gears and levers, I imagine conversations that

might reveal to me what Andrew Gleason or Dana Scott might do in the samesituation, or I imagine explaining my solution to a student and discovering

something wrong with it There's little sign of consistency in any of that

experience. Nor is that famous 'intuition' really a privileged route to the truth,because although the answer seems to come with a feeling of certainly, later

it's likely to turn out to be wrong.

Perhaps the most important aspect of how humans work are the ways in which

we ask ourselves (not necessarily by using words) what problems have we

seen before that most closely resemble the present case, and how did we

manage to deal with them. For those were where we made our mistakes andthen sometimes managed to learn from them. And notice that in doing so, we

somehow must employ some capabilities for retrieving and then manipulatingsome descriptions of some of our earlier mental activities. Now, notice how

self-referent this is. Often when you work on a problem you consider doingsome certain thing -- but before you actually carry that out, you often inquire

about yourself, about whether you actually be able to carry it through. Solving

problems isn't merely applying rules of inference to axioms. It involves

making heuristic assessments about which aspects of the problem areessential, and which of one's own abilities might be adequate to dealing with

them. Then, whatever happens next arouses various feelings and memoriesabout of situations that seem similar and methods that might be appropriate. Is

this done by some kind of non-physical magic or it is accomplished, as I

maintain, by the huge and complex collection of knowledge-base

representations and pattern-matching processes that we all regard as 'common

sense'?

Now it happens that when we do such things, we often find that we talk to

ourselves about what we're doing. And when we thus "refer to ourselves" we

sometimes speak of being conscious or aware. I think it no coincidence thatPenrose feels that this, too is something present-day science cannot explain.

Indeed, he Could this result from just that fear of inconsistency and self-reference? Indeed, Penrose sometimes speaks of a "reflection [principle" with

something resembling awe: "The type of 'seeing' that is involved in a

reflection principle requires a mathematical insight that is not the result of the

purely algorithmic operations that could be coded into some formalmathematical system (p110)." In my opinion this is just a mistake! He appears

to assume that when this is applied to humans, the word "consistent" can befreely inserted between 'some' and 'mathematical' -- as though people possess

some marvelous gift whereby they can tell which assertions are true. But inview of the many mistakes we all make, I see no compelling evidence that


5/11

anyone has any direct such access to truth. All we can depend upon (including

the power of formal proof) is based on our experience. I think. And in anycase there really is no problem at all in programming a computer to perform

that sort of reflective operation. Indeed John McCarthy has pointed out that

forming a Godel sentence from a proof predicate expression (which is thebasis of the Lucas-Penrose argument) requires no more than a one line LISP

program. So in my view Penrose and many other philosophers have put the

problem upside down: the difficulty is not with making algorithms that can doreflection -- which is easy for machines, but with consistency -- which is hard

for people. In summary, there is no basis for assuming that humans are

consistent -- not is there any basic obstacle to making machines use

inconsistent forms of reasoning.

Consciousness.Even the most technically, sophisticated people maintain that whatever

consciousness might be, it has a quality that categorically places it outside the

realm of science, namely, a subjective character that is makes it utterly private

and unobservable. Why do so many people feel that consciousness cannot be

explained in terms of anything science can presently do?

Instead of arguing about that issue, let's try to understand the source of thatskeptical attitude. I have found that many people maintain that even if a

machine were programmed to behave in a manner indistinguishable from aperson, it still could not have any subjective experience. Now isn't that a

strange belief -- considering that unless you were a machine yourself, how

could you possibly know such a thing? As for 'subjectivity,' consider

that talking about consciousness is a common, objective form of behavior.

Therefore, any machine that suitably simulated a human brain would have to

produce that behavior. Then, wouldn't it be curious for our artificial entity tofalsely claim to have consciousness? For if it had no such experience, then

how could it possibly know what to say? Of course a classic question inphilosophy is asking for proof that our friends have minds; perhaps they are

merely unfeeling machines. But then one must ask how they'd know how to

lie.

In any case, we have much the same problem with ourselves; try asking a

friend to describe what having consciousness is like. Good luck! Most likely

you hear only the usual patter about knowing oneself and being aware, ofsensing one's place in the universe, and so on. Why is explaining

consciousness so dreadfully hard? I'll argue that this is something of anillusion, because consciousness is actually easier to describe than most other

aspects of mind; indeed, our problem is a far more general one, because ourculture has not developed suitable tools for discussing and describing thinking


6/11

in general. This leads to what I see as a kind of irony; it is widely agreed that

there are "deep philosophical questions" about subjectivity, consciousness,meaning, etc. But people have even less to say about questions they'd consider

more simple:

How do you know how to move your arm?

How do you choose which words to say?

How do you recognize what you see?

How do you locate your memories?

Why does Seeing feel different from Hearing?

Why does Red look so different from Green?

Why are emotions so hard to describe?

What does "meaning" mean?

How does reasoning work?

How do we make generalizations?

How do we get (make) new ideas?

How does Commonsense reasoning work?

Why do we like pleasure more than pain?

What are pain and pleasure, anyway?

We never discuss these in everyday life, or bring them up in our children's

schools. An alien observer might even conclude that those Earth-people seemto have a strong taboo against thinking about thinking. It seems to me that this

is because our traditional views of psychology were so mechanistically

primitive that we simply had no useful ways to even begin to discuss such

things. This is why I find such irony in the arguments of those who reject the

new mechanistic concepts of psychology -- the new ideas about computational

processes that promise at last to supply us with adequate descriptions of these

complex processes.

The science of Psychology, as we know it today, is scarcely one hundredyears old. Why did humanity wait so long before the emergence of thinkers

like Freud, Piaget, and Tinbergen? I think the answer lies in the fact that the

brain is not merely a kind of machine, but one that is far more complex than

anything ever imagined before. The pivotal notion provided by those three

pioneers was that the mind has many parts. A person doesn't simply See by

"looking out" through the eyes. Instead, vision involves many differentprocesses, cooperating, competing, being promoted and inhibited by other

processes, being managed and regulated by yet others. You can not simply'recognize' a telephone, because that is scarcely at all a matter of vision;

instead, you have to "re-cognize" it -- that is, the input has to somehow

activate some memory representations of a device with a certain kind of

structure (handset and dial, say) coupled with a certain functional disposition

(to hold to the mouth and ear for communication purposes). This is nothing

like the sorts of unitary concepts found both in commonsense and philosophy,e.g., of a platonic ideal of a telephone, or some sort of model inside the head.

In recent years we've learned much more about the complexity of the brain. It

now appears that perhaps fully half of our entire genetic endowment is

involved in constructing our nervous systems. This would suggest that the

brain is nothing like a single large-scale neural net; instead, it would have


7/11

even more parts than the skeletomuscular system -- which can be seen to have

hundreds of functional parts. If you examine the index of a book onneuroanatomy, you will find the names of several hundred different organs of

the brain. A good fraction of those are already known to have psychologically

distinct functions. To pursue the analogy a little further, note that the skeletalanatomies of animals have been known for millennia, but only in rather recent

years have scientists understand the mechanics of locomotion and its various

gaits; that had to wait until scientists learned more about the mechanics offorces and materials. Similarly, mechanistic theories of psychology may have

to wait even longer for adequate conceptual tools because the 'mechanics' of

heuristic computation could turn out to be more complex than those of

physics. Before these new ideas emerged, with the era of complex

information-processing computer models, such models were not considered

convincing -- perhaps because there were no feasible experiments. I don't

mean to say that there was no progress at all before computers, only that therewas precious little. Freud himself was one of the first to conceive of "neural-net-like" machines -- only no one would listen to him except Fliess. Later

came the astounding insights of Post, Godel, and Turing, followed by those of

Rashevky's group, McCulloch and Pitts, and Grey Walter's simple yet

somewhat life-like mini-robots. But significant progress began only in the

1950s when more serious models could be conceived, tested, and discarded in

days or weeks instead of years. Soon the researchers in Artificial Intelligencediscovered a wide variety of ways to make machine do pattern recognition,

learning, problem solving, theorem proving, game-playing, induction andgeneralization, and language manipulation, to mention only a few. To be sure,

no one of those programs seemed much like a mind, because each one was so

specialized. But now we're beginning to understand that there may be no need

to seek either any single magical "unified theory" or and single and hitherto

unknown "fundamental principle"-- because thinking may instead be the

product of many different mechanisms, competing as much as cooperating,and generally unperceived and unsuspected in the ordinary course of our

everyday thought.

What has all this to do with consciousness? Well, consider what happened inbiology. Before the 19th century there seemed to be no alternative to concept

of "vitality" -- that is, the existence of some sort of life-force. There simply

seemed no other way to explain all the things that animals do. But then, as

scientists did their work, they gradually came to see no need for a "unified

theory" of life. Each living thing performed many functions, but is slowlybecame clear that each of them had a reasonably separate explanation! For the

most part each separate function was served by a different and specializedorgan! Thus the lungs oxygenate the blood, while the heart pumps it to other

organs. The nucleus reproduces all the organs' structural information, while

the ribosomes translates those codes into proteins which then self-configurethemselves. For some time that subsequent appeared to entail a mystery. It


8/11

seemed natural to assume that those configurations were based on a uniform

energy-minimizing mechanism -- but simulations did not bear this out. Thisappears to not be so; instead, each protein has had to evolve this property on

its own. (A random string of peptides cannot usually manage it.) Conclusion:

There is no central principle, no basic secret of life. Instead, what we have arehuge organizations, painfully evolved, that manage to do what must be done

by hook or crook, by whatever has been found to work.

Why not assume the same for the mind? (I could have said the brain, instead --but in my view minds are simply what brains do.) Why else would our brains

contain so many hundreds of organs? Of course there are many old argumentsagainst localization of brain-functions because it seemed that often a mind still

works when some of its brain has been lost. One answer to that is to argue that

many functions are accomplished in multiple ways, not only to provide

resistance to some injuries, but perhaps more important, because no particularway is likely to be always reliable. To be sure, there still seem to be some

mental phenomena that have not yet been shown to "organ-ized". So there isstill some room for theories about mechanisms that are not so localized. But

now, I maintain, it is time for "insulationism" to take its place along with, andin complementary opposition to, connectionism.

Then what might be the functions and the organs of what we call

consciousness? To discuss this, we'll have to agree on what we're talking

about -- so I'll use the word consciousness to mean the organization of

different ways we have for knowing what is happening inside your mind, yourbody, and in the world outside. Here is my thesis; some people may find it tooradical:

We humans do not possess much consciousness. That is, we have very littlenatural ability to sense what happens within and outside ourselves.

In short, much of what is commonly attributed to consciousness is mythical --and this may in part be what has led people to think that the problem of

consciousness is so very hard My view is quite the opposite: that some

machines are already potentially more conscious than are people, and that

further enhancements would be relatively easy to make. However, this does

not imply that those machines would thereby, automatically, become much

more intelligent. This is because it is one thing to have access to data, butanother thing to know how to make good use of it. Knowing how your

pancreas works does not make you better at digesting your food. So consider

now, to what extents are you aware? How much do you know about how you

walk? It is interesting to tell someone about the basic form of biped

locomotion: you move in such a way as to start falling, and then you extend

your leg to stop that fall: most people are surprised at this, and seem to havewhich muscles are involved; indeed, but few people even know which


9/11

muscles they possess. In short, we are not much aware of what our bodies do.

We're even less aware of what goes on inside our brains.

Similarly we can ask the extents to which we're aware of the words we speak.

At first one thinks, "yes, I certainly can remember that I just pronounced "the

words we speak." But to what extent are we aware of the process that

produced those particular words? Why, barely at all! We have to employ

linguists for lifetimes of research even to discover the simplest aspects of the

language production process.

Finally, I can ask you questions like, "Can you tell me what you are thinking

about." The answers to such questions are hard to interpret. The listener mightlist the names of some subjects or concerns that were recently in mind, and

sometimes can describe a bit of the trains of thought that led to them. These

kinds of answers clearly feed upon memories of recent brain-activities. Butevery such answer seems incomplete, as though the act of probing into any

one of those memories interferes with subsequently reaching any other ones.

In any case, I cannot think of any aspect of consciousness that could operatewithout making use of short-term memories, and this suggests that the term

'consciousness' is usually used in connection with whatever processes brainsuse for accessing memories of their recent states.

This raises the question the extent to which such memories might really existinside our brains. Clearly there is a problem: if the same neural network has

been used recently for only a single purpose, then it may still containsubstantial information about what it recently did. But if it was used for

several things, then most of those traces will have been overwritten -- unlesssome special hardware has been evolved for maintaining such records. For a

modern computer, there is much less of a problem with this because we canwrite programs to store such records inside the machine's 'general purpose

memory". Of course, there will be ultimate limits on the size of such records,

but not on the nature of their contents. For example, most LISP language

systems allow the user to specify that all the activations of an arbitrary set of

program-components will have traces stored recursively. If you specifyenough of this before you run your program, then subsequently you'll be ableto find out everything it did -- and even to simulate running it backwards.

However, as we've already said, having such access does not by itself enablethe machine to make a good interpretation of those records. Certainly a certain

degree of consciousness -- in the sense of access to such records -- is

necessary for a person (or machine) to be intelligent. But even a large degree

of such 'consciousness' would not by itself yield intelligence.

So this finally leads us to some really important questions about what are the

uses of consciousness. It seems entirely clear to me that consciousness hasusefulness. It can't be what some philosophers claim: some sort of useless


10/11

metaphysical accessory. On the contrary, there are important ways to exploit

short term memories. For example, one has to keep out of loops -- that is,repeating an unsuccessful action many times -- which requires knowing what

already has been done. Also, after one has successfully solved a difficult

problem, one wants to "assign credit" to those actions that actually helped.This may involve a good deal of analysis -- in effect, thinking about what

you've recently done -- which clearly requires good records. Furthermore,

such evaluations must be done on various scales; did you waste the last fewmoments, and why; or did you waste an entire year? Why do we use the term

consciousness only about the shorter term memories?) On each such scale,

you'd better have an adequate array of memories. Otherwise you cannot

intelligently revise your plans, adjust your strategies, take stock of your

resources, and in many other ways maintain some control over your future. On

how many such time-scales do we work, and how many different mechanisms

are involved with each? Because we're living in the early times of psychology,no one can yet answer such questions. Clearly it is time to begin to seekconstructive ways to study them. To do this we should prepare ourselves for

coping with complexity, because it seems unlikely that so many different

functions can emerge from a single, completely new principle.

Then what is the alternative. We'll simply have to face the facts that our

many-hundred-organ-ed brain is not a useless luxury. By the time of your

birth the brain contains hundreds of specialized agencies, and by the time thatyou're an adult, most of those systems have probably grown through dozens of

stages of development. Now at various times in those first few yours, some ofthose systems create the most supremely useful of all fictions, namely, that the

unwritten novel that constitutes your life is centered on a principle protagonist

-- that you conceive of as your consciousness, like an actual person inside

your head! Some sections of [2] describe in more detail why this illusion is so

useful in life; indeed, in effect, it makes itself true. But the point of all this is

to emphasize that none of those old simplistic concepts from the past -- thosespirits, souls, and essences -- can help us with that modern task, of

understanding how all those different resources, are constructed, operated and

managed. Surely they work to a large extent as a partially cooperative parallelsystem -- but also, surely, those are largely controlled (much as Dennett

suggests in [3]) by one or several sequentially controlled systems, which in

turn are assembled from smaller parts. The first sentence in my book [2],

attributed to Einstein, is "Everything should be made as simple as possible,

but not simpler." The first step to take toward doing that is to exorcise thoseSpirits from Psychology.

[1] The Emperor's New Mind, Roger Penrose

[2] The Society of Mind, Marvin Minsky


11/11

Documents

Conscious Machines - Minsky