¥

SOME CHARACTERISTICS OF HUMANINFORMATION PROCESSING

Earl Hunt and Walter Makous

Department of PsychologyUniversity of Washington " Seattle

i *

SOME CHARACTERISTICS OF HUMAN

INFORMATION PROCESSING

Earl Hunt and Walter Makous

This research was partially supported by the NationalScience Foundation, Grant No. NSF 87-1438R, to theUniversity of Washington, and partially by the NationalInstitute of Neurological Deseases and Blindness, GrantNo. MH 15564-01, to the University of Washington.

Department of PsychologyUniversity of Washington — Seattle

Technical Report No. 68-1-19August 23, 1968

I

ADDENDUM

The pages It^tA taicv; »ho?jlc' -be -corrected co read as shown.

Cevsr, XXX. Insci eete of Nwreiogical Diseases and Blindness, GrantPg-X L.22: wane. We trust learn how best to connect computers t0...Pg.6, L.Bi Tf.fi perfonsancs of the 83'stem as a vhole leads co the same

f.'.;-;:xcluotorj.

PgX, hPi: deuce ca.c be yeeru lis fact, Walraven' and others at the...P^-.tO, L,3: Had Xopp_ bits, but here again the InformationPg.ii, Xl6: c<jacl«sio« Is clearly Intolerable for computet science,...'Pg.l3, XX eenfc to other areas of Che brain. Analogies to general...ft. .15,. X 27-; Even without a detailed ' knowledge of Its parts, we have

learned touch about the behavioral characteristics of the brain.Pg.J.6, L.Z4: 'th* dlSflaj had gone off you received a signal (for instance...

(175?g.30, 1, .11: ready for the© 7 . . Other evidence however, indicates that_ tiis phenomenon ■ "

Pg,4l, t,5: tha more recent Ideas presented by Quillian and Reltman ' .Pg..'s6, l.Plk: optiwßitt' values for the visual variables characterising visual dls-?g, 34 .. 1.... IP- ; ..... , especially Martian^lp-.54 s 7.23: I3SP?SIX on? humans—from.Fr . 61 , L . 7 : „ .. ,-«» (t ~t, ) /tau ,„MD * P I^e 1 dt ,

.*1 C

Reference 57: W. 1,, Makous and t. D» Brown, Systematic...TABLE VTI, title: SHORTCOMINGS OF EXISTING INTERACTIVE CQNSOLES^^-i'IGUBE CAPTIONS, L. 1: Figure 11. Necker cube. Perception...

FIGURE CAPTIONS, L. 3: Figure 1. Example of the effect...Figure I: Figure lls upside down. It must be turned around for effect.Footnotes: Add to last page of Footnotes the footnote on Page 52, "This

example is a modification of one offered by Quillian(73) .3

>**...

-, - *~ '

*-...T "■"'"' ''"'"- " ' ' -' -" V.■aiak..'.: -A, t . ... , , .-. „... „..(/. ' ,itjz».^,.jA...-,51:.,.... Aimffltfn*n7 - -'"■-7"^^Pff7* -"<>■■ - ' a' "i7u.. "■:... ,"* .* - . .AA*"1?-;-', .' - nJ.'j* .*.' : .-*..-,A "" "

TABLE OF CONTENTS

1. INTRODUCTORY REMARKS. 1

2. COMMUNICATING WITH THE ENVIRONMENT: THE 1-0 SYSTEM. 4

2.1 The Interface Problem 4

2.2 Information Analysis of Sensory Function 5

2.3 Information Analysis of Human Output 8

2.4 Is Information a Good Measure of Human Performance? 9

3. THE BRAIN: PHYSICAL CONSIDERATIONS. H

1 13 . 1 Introduction

3.2 Anatomy and General Functioning H

133 . 3 Memory

4. A FUNCTIONAL ANALYSIS OF HUMAN DATA HANDLING. 16

4.1 Buffering 16

4.2 Code Analysis in the Buffers 18

4.3 Code Conversion 22

4.4 Progressive Code Changes Over Time 25

4.5 Storage Management

4.6 Real Time Problem Solving 33

4.7 Multi-Programming

4.8 Long-Term Memory

.5. MAN IN AN INFORMATION SYSTEM. A5455.1 Introduction

5.2 Computer Output; Human Input

5.3 Human Output; Computer Input hl

405.4 The Place of Man

5.5 Why the Difference Between Man and Machine 53

Table of Contents Cont'd.

6. ROBOTS. 54

6 . 1 Some General Remarks 54

6.2 The Problem of Perception 57

6.3 Conclusion 67

M

Some Characteristics of Human

Information Processing

Earl Hunt and Walter Makous

The University of Washington

I. INTRODUCTRY REMARKS

Our technology is as concerned with the movement of signals as with the

movement of objects. This observation immediately leads to thoughts of com-

puters, but computers are not the only components of an information proces-

sing system. One must consider the performance of all system components;

telephone lines, and teletype terminals as well as central processors.

Perhaps most important, one must know the operating characteristics of

man.

We process information In order to report the state of the world

to man. Therefore this report must be in a form which a man finds conven-

ient to use, a point that producers of thousands of pages of computer

output would do well to remember. Man may often be an internal system

component as well. The feasibility of telephone networks is at least

partially determined by the short-term memory of the operator. Similarly,

in sonar and radar systems, man's unparalleled capabilities for visual pat-

tern detection make him an essential part of a scanning system. We could

continue with as many examples as anyone could wish. The point should

be clear. Man is as important as any other component of an information

system. How should he enter into the system designer's equations?

In order to give the question complete consideration, we would have

to add still another to the plethora of general psychology textbooks.

2

This is neither desirable nor practical. (For those interested, however >we do recommend a recent textbook(30> which approaches psychologyfrom the information processing viewpoint.) What we shall do is to dis-cuss a few characteristics of man which appear to us to be importantin determining his ability to handle information, if he is motivated todo so. Throughout, we shall assume that this motivation exists, ignoringthe question of why this should be so.

Like any computing system, man must be able to receive informationfrom his environment, process it, perhaps referring to previously storeddata, and communicate the results. Man's ability to do this depends uponhis senses, and his motor responders, which, in computing terms, are theinput and output devices interfacing between the world and the brain.We shall first discuss some of the transmission characteristics of theseinterfaces. Abstractly, this gives us a clue as to the kind of computerman is. More practically, it leads to several specific suggestions forthe design of man-computer communication. This topic is becoming more andmore important with the increased use of real time computer control systems,in which man may often be an integral part.

To say the least, an evaluation of a computer would be incomplete

without a discussion of the central processor and the memory units. Simi-larly, we cannot talk about man without discussing the brain. But howshould we describe the brain? As a physical mechanism? Hardly, for two

reasons. There are major gaps in our knowledge of how the brain worksat a physical level. Even if we did know all about the physical mechanismsof the brain, it is likely that reciting their details would obscure theoverall picture of mental functioning. Here the computer analogy is

3

particularly apt. Most descriptions of how a computer works do not concern

themselves with the logic of floating point circuitry. We shall take the

same approach one does in describing a computer. After surveying very

briefly some of the physical features of the brain, we shall concentrate

on functional characteristics. Our approach will be to describe performance

on a few key tasks, themselves unimportant, which (hopefully) are direct

tests of the brain's ability to perform elementary operations. Again

there is a computing analogy: in evaluating a computer one often writes

special test programs which have no purpose other than to exercise a parti-

cular machine capacity. Unfortunately things are not quite so neat in

evaluating man. One can go to any desired level in analyzing the micro-

structure of a computer, and hence can be reasonably sure that appropriate

test programs have been written. We cannot do this with the brain, so our

"test programs", in fact, psychological experiments, represent attempts

to test operations which we think are basic to the brain. We may be wrong.

It seems to us, however, that this manner of proceeding is inevitable

in the study of information processing in a natural science, no matter

how inappropriate it would be in the study of information processing in

an engineering science. Hopefully, we have written this paper to indicate

where we are guessing with confidence, and where we are speculating.

Let the reader beware!

Following the discussion of the computing characteristics of man,

we consider briefly how these characteristics can best be blended with

the computing characteristics of machinery. In particular, we consider

the desiderata of man-computer interfaces, and comment on the sorts of

problems which, in our present state of knowledge, ought to be reserved

4

for men.

In our final section, we examine the robot oroblem. We know that

computers can supplement human problem solving, but can they replace it?

If machines are going to substitute for man, should they be designed to

mimic him, or should they achieve his capabilities in some quite different

manner? In spite of the intense interest in robots on the part of many

social commentators., and a few scientists, we shall conclude that man

has some capabilities which we have very little chance of duplicating with-

out further advances in our scientific knowledge.

2. COMMUNICATING WITT! THE ENVIRONMENT : THE 1-0 SYSTE*

2.1 The Interface Problam.—General Considerations

A few years ago, one of us was asked about the possibility of develop-

ing microelectrode techniques Dermittin? a computer to tap directly into

the brain. Somewh.it nore facetiously, a third narty to the conversation

suggested developing a telepathic link between man and computer. Aside

from technical, theoretical, or ethical questions, it is not clear that

we could imnrove on what exists. The processes of evolution have already

developed for man uniquely satisfactory interfaces betxreen his brain and

his environment. Ttan's senses do very well in the task of getting informa-

tion into his brain in the correct format; similarly his hands, and the

tools they manipulate, are outstanding devices for manipulating the environ

ment. T7e must learn how best to correct computers to our senses and our

hands, not how to substitute computers for them.

The foregoing remarks do not mean that man is uniquely suited to wort;

with any sort of computing equipment. All we x*ish to stress is that in

designing and improving man-commiter teams we must identify the process

5

5

that limits rate of performance. There is little point, for example, in

working to increase the rate of information output from the computer if

that rate already exceeds the rate at which humans can receive the infor-

mation, or if some other link in the process is what limits progress. With

this in mind, let us look at some data for human information reception.

2.2 Information Analysis of Sensory Function

Theoretically, the senses are capable of extremely high rates of

information transmission, with little information being lost on its way

to the brain. Take, as an example, the visual system.

The most primitive aspect of vision is the simple detection of the

presence of light in a particular spatial location. A bit of information

from the environment, carried by a photon, is encoded by the photoiso-

metrization of a pigment molecule within the eye. There are approximately

3 x 107 such molecules in each receptor within a mammalian eye. More

specifically, the human eye contains two different kinds of receptors,

the rod and the cones, that are parts of functionally distinct systems which

assume control of the channel to the brain, the optic nerve, under different

conditions. The rod receptors, 1.25 x 108 in number,(66) can replace used

(79)pigment at a maximum rate of one complete replacement every 15 mm.

The rod system of the human eye is capable of encoding information at a

steady rate of 3(107) (1.25) (10 8)/(15) (60) - 4(10 12) bits per sec. Although

the cones are 25 times less numerous than the rods, they can replace

. (6,80,81) _.pigment molecules about 7 3/2 times as fast as the rods. Thus, the

greater rate of photopigment regeneration in cones partially compensates for

their smaller number, so that the cone system is capable of encoding infor--12

mation at about 1/3 the rate of the rod system, or about 1.25(10 ) bits per

6

The membrane of an optic nerve fiber at a particular point in time

and space can be categorized as either active or inactive. It can cycle3through these t^o states 10 times per second. Each optic nerve contains

about 10 such fibers p conseouentlv, each optic nerve is capable of

carrying no more than 10' bits per sec. Hence, the receptors can receive

and encode information about 1000 times as fast as it can be transmitted to

the brain via the optic nerve.

The performance of the svsten as a whole leads to the conclusion. There

is a maximum number o^ photons in a brief flash of light that the visual

system is capable of signalling. Increases in the quantity of photons above

that number cause no change in the output of the system: i. c. s all flashes

look the same, so long as the number of photons they contain are eaual to

or greater than a critical number. The information carried by the additional

photons cannot be transmitted. Although the immediate visual response

elicited by two flashes of light consisting of different numbers of photons

may not be discrininable i* both flashes contain a sufficient number of

photons, the retina may vet encode and store soTne information on the dif-

ference between the flashes under these conditions, for one may be able to

detect a difference in the after-images produced by the two flashes. 12^

This may be true of flashes containing as many as I^o to 1000 times as many

photons as that which evokes a maximum response fron the visual system.

(7 89 NSince after-images appear to be caused by bleached pigment, ' ' the

difference in appearance in the after-mages must be caused by differences,caused by the differing number of absorbed ouanta, in the amount or nature of

the pigment photoisonerized by the two flashes.

Loss of this information in the eve seems wasteful and inconsistent

with the very high efficiency of the eye in so many other respects. It

7

i

also seems inconsistent with the data showing that absorption of between

two^ ' and ten photons in sufficient spatial and temporal coinci-

dence can be seen. In fact, Walravem and others at the Institute for

Perception at Soesterberg, the Netherlands, have experienced a certain

degree of success in treating various aspects of visual function as though

(10 95)the eye were an ideal detector. ' That is, if the retina is subjected

to stable conditions of illumination, the number of photons absorbed

by a particular receptor within successive time periods follows a poisson

function. These poissonian fluctuations in photon absorptions, analogous

to the clicks of a Geiger counter, constitute a noise from which the

visual system must extract the signal. In many respects it appears that

the eye approaches the ideal in the task of extracting statistical infor-

mation from this noisy signal. How, then is this high efficiency to be

reconciled with the conclusion reached in the two preceding paragraphs

that substantial amounts of information are lost within the eye?

The answer is to be found in the changes that occur within the visual

system as the level of illumination increases. When illumination is well

above threshold, and the information inherent in the light is more plenti-

ful, the eye goes beyond detection to extract information on the spectral,

spatial, and temporal distribution of the light. This is done by a suc-

cession of fixations, each lasting about a quarter of a second. During

these quarter-second fixations, the eye performs an integration, with respect

to time, of the light falling on each receptor. Naturally, one effect

of the integration is the loss of information about the arrival of indi-

vidual photons. Averaging, however, retains the redundant information

which (usually) defines the visual picture. The loss of some information

concerning the number of photons, then, is a consequence of extracting

other kinds of information from the noise.

8

When illumination is still greater, and the response of the eye approaches

its upper limits, rejecting some redundant information can actually increase

the amount of relevant information transmitted. During a quarter-second

eye-fixation, an optic nerve fiber can conduct, at most, 250 impulses. If

each impulse signalled the absorption of a single photon, for example, a

stimulus 250 times greater than threshold for the unit, i. c., 2.4 log units

above threshold, would nlicit from the fiber the maximum response, saturating

it. As the visual system is reauired to function over a range of illum-

ination spanning more than 12 lor units, this could be incapacitating. By

transmitting only part of the (redundant) information from each photon

absorption, the eye can function over a much extended range.

Some information, then, is lost within the eye, but the amount lost

there is not great unless the amount transmitted is already great.

Jakobson has estimated that the visual system can encode and transmit

data at the rate of 4.3(10 ) bits per second. Although some of his assump-

tions are questionable, this estimate is not likely to err by more than two

or three orders of magnitude. In fact, the existence of individuals with

eidetic imagery (photographic memory, in common parlance), and the perfor-. (^6)mance of the remarkable Individual described in Luria s recent book,

The Mind of a Mnemonist, suggest that Jakobson"s estimate may be very nearly

correct, for it is not abstracted Information in the stimulus that these

individuals store, like most of us, but the entire imase. Thus, the

transmission rate of the visual system is in the thousands of bits per

second, at the least. Similar consideration of other senses lead to the

same conclusion.

2.3 Information Analysis of Human Output

The high rate at which information can enter the brain contrasts

starkly with the Iot? rata at which it issues fron tha brain. To our

9

knowledge, the highest rate of output reported a human is 35 bits/sec

for oral reading. What process limits the rate at which the vast amounts

of information entering the brain can be read out? At least part of the

limitation is inherent in the process of speaking the words, for silent(71) (23)

reading may progress at the rate of 44v or even 52.5 bits/sec. The

serial nature of the output may be one constraint on rate of output, for

piano playing, which permits some parallel output, may progress at the rate

of 22 bits/sec, as against 15 bits/sec for the serially-constrained typing.

Similarly, stenotyping, which also permits parallel output is faster than

ordinary typing. The act of forming the words, however, cannot constitute

the ultimate limit on the rate of output, for speaking spontaneously slows a

subject down from his oral reading speed to 18 bits/sec, perhaps with bursts

up to 26 bits/sec, without necessarily chanrrincr the words spoken. In other

words s a subject can read a transcript of his spontaneous speech much more

rapidly than he can emit it in the first place.

2.4 Is Information a Good Measure of Human Performance?

Before "oing further, we should point out that information, or entropy,

is not truly a fit measure of intellectual output. Even in the simplest case,

where the product is fairly represented by a string of symbols, the rate of

information transmission Is affected at the outset by the choice of unit, a

choice that is always artificial and usually arbitrary. For example, the

entropy in a given sentence differs according to whether words, syllables,

phonetns 5 or characters are the basis for the computation. Estimates of the

entropy of a typical word of text vary from about 10 to 12 bits, <55 >but the information transmitted in a particular situation varies further,

with the characteristics of the receiver (number of words he can recognize,

number of words he can make a good guess at, history of word frequencies,

previous experience, and mutual inter-dependencies), the nature of the text.

10

.

the immediate context of the word, and so forth. At the other extreme of(23)

size of ur.its, the entropy of a character is estimated at between 1.5( 55 )

and 2.0 5 bits, but here again the information transmitted is highly

specific to the details of the situation in which the character occurs.

At the input side, quantifying in terns of entropy, as \ie have done/no _ 01 7 \

above, is at least as unsatisfactory. To nuote Cherry, 'p '

"...an observer looking down a microscone, or reading instruments, is not

to be equated with a listener on a telephone receiving spoken messages.

Mother Nature does not communicate to us with si<?ns or language. A

c.ommunication channel should be distinguished from a channel of observation

and, without wishing to seem too assertive, the writer would suggest that in

true communication problems the concept of entropy need not be evoked

af. allX

We do not mean to imply that information theory is not relevant for

neural or for psychological processes, on.lv that the units and concepts of

information theory are net natural for the nervous system or for behavior.

In distinguishing between natural units and those that are not. natural, we

follow Hempel in su^p-cstino: that some concepts and units of measure

are more likely than others to result in simple expressions of natural laws.

T?or example, although, the rate of information transmitted through the

nervous system is limited, these limits are not fundamentally determined

in terms of entropy. The reading rate of most individuals is essentially

United by the rate at which syllables and phonemes can be emitted, and the

rate of information thereby transmitted is limited only secondarily, by the

entropy of these units of speech. Nevertheless, there are situations in

which it may be not only useful but necessary to treat such situations in

terms of information theory. These situations include both practical

situations where the fundamental question concerns the transmission of

11

information, and the human is only one link in the communication channel;

and it includes also some cases where information theory can tell something

(37)fundamental about physiological mechanisms. *"or example, Hagins used

information theory, along with other theoretical and experimental consider-

ations, to arrive at a set of constraints defining certain necessary

properties of the processes in the squid retina that must communicate the

information carried by a photon absorption through to the development of an

experimentally measurable potential. From, these constraints it is possible(25)

to show that the early receptor potential cannot take part in communica-

tion of this signal.

3. THE BRAIN: PHYSICAL CONSIDERATIONS

3.1 Introduction

Tten's sensory and perceptual organs—his input mechanisms if you

w j_ll are quite impressive. If the rest of man had similar information

handling capacities we would hardly need computers at all. Fince this

conclusion ls clearly Intolerable for computer science, we seek a defect.

We shall find it in two places, human ability to store information and to

perform internal computations.

First we shall consider some facts and recent speculations about the

physical basis of memory. Host of the facts on which our remarks are based

have been gathered from physiological stud Les of memory in animals, and

probably apply to vertebrates in general. While we are more interested in

function than anatomy, a brief discussion of physical structures is necessary

3.2 Anatomy and General functioning

The brain is an extension of the spinal cord which folded over the end

of the cord in a complicated manner. In primitive vertebrates (e.g. the

shark or dogfish) it is little more than a swelling on the cord. In mammals,

and especially in primates, the brain folds back over the spinal cord. In

the higher animals there is an extensive, mushroom-like growth of "new

brain," or cortical, structures. This is particularly marked in man. At one

time it was thought that all "higher mental functions" "resided" in the

cortex, but it is now clear that this is a gross oversimplification. It is

hard to imagine a function in a normal animal that is not influenced by

activity in the cortex, and it is impossible for a function to involve the

cortex without also involving other brain structures, for there are no

direct connections between receptors or effectors and the cortex. If a dis-

tinct function is ever established for the cortex, it is unlikely that it

will correspond to any "function," as the word is applied to behavior and

psychology. It Is at least as probable that the cortex has evolved in

parallel with other brain structures serving the same function, but that the

cortex permits some refinement in the performance of that function.

In addition to the top-bottom distinction between cortical and sub-

cortical structures, the brain displays a rough bilateral symmetry. This

is particularly visible in the cortex, which forms the external covering

of the right and left hemispheres. In general, a hemisphere receives most

of its input from, and exercises most of its control over, the contralateral

side of the body. The bifurcation is not complete. Speech, a uniquely

human capacity, is in most individuals a strictly unilateral function

dependent upon an area more often in the left hemisphere than in the right.

Other functions, such as vision and memory for complex situations, involve

both hemispheres.

When messages about the environment are transmitted to the brain, they

take the form of coded electrical impulses which travel along two pathways.

One is the "direct" route to specific sensory projection areas of the. cortex

12

13

A second, more diffuse, pathway transmits information through the brain

stem to various subcortical centers, and from there it projects signals widely

throughout the entire brain. This second pathway often seems to serve an

alerting function, somewhat like a priority interrupt flag in a computing

system. A signal, then, can have two effects. Via a specific pathway,

determined by the physical nature of the signal, a recognition procedure may

be initiated in the cortex. At the same time an "alerting" signal may be

snet to other areas of the brain. Analogies to general alerting signals

in computer hardware, or even to social systems, such as the "General

Quarters" alarm on a ship, come readily to mind, but they are too superficial

to be of much value. A non-specific alerting signal can partially interrupt

ongoing activity, and increase reactivity. How it does this, and the specific

functions this serves, are not clear.

In addition to the "alerting" track in the brain stem, there is a

descending pathway which is potentially quite important in determining human

reactions. There is evidence in all senses except taste, that the brain can

lower the sensitivity of a peripheral Input channel which is not concernedft

with the primary task being processed. The result is that to an animal

the world is not a jumble. It is a selected stream of input, wh.^re the

activity at this moment determines the input at the next. The animal imposes

selectivity and priority queing on its input in very much the same way as a

time-sharing computer queues its users.

3 . 3 Memory

What happens when a piece of information reaches the brain. Now is it

stored? Here we are pushing the limits of our knowledge, so much so that

much of what we have to say must be speculation. On the other hand, recent

studies in physiological psychology have led to the formulation of a rather

different picture of memory than that presented in introductory psychology

texts of only a few years ago (See John for a review cast in the con-

text of one of the newer viewpoints)

Memory is a multi-stage process in which the coding of information is

changed both in terms of its information content and its physical nature.

Initially, as we have said, information is transmitted in the form of a

train of electrical pulses. Insofar as information bearing is concerned,

this train can be regarded a digital signal. It is clear that memory cannot

be stored in any such digital form, however, because at each synapse the

digital train of impulses arriving on one particular neuron is transformed

into a continuously graded depolarization, the magnitude of which is affected

not only by the activity of other neurons but also by the recent history

of activity in the particular neuron in question. Permanent memory is a

permanent change in the dispositional properties of a set of neurons. That

is, some change in the neurons alters the probability that they will or will

not fire in a particular way in response to the same input. Evidently the

change is due to alteration of the physical and chemical properties of

individual neurons. The precise nature of the alteration is not yet clear,

although a number of hotly contested theories have been advanced. Further,

the locus of the set of neurons involved in any particular class of learning

is not known, although some recent progress has been made on this question

using such techniques as split brain preparations, single unit recording(52) (22)

during learning, and the isolation of simple learning systems.

This picture of memory accentuates some of the classic contrasts between(94)

people and computers (e.g., Yon Neuman ). Even if only a small fraction

of the 10 neurons in the brain do participate in memory, the increase in

the number of internal discriminable states of the brain due to some neurons

14

15

having assumed one of several possible states will he enormous. Man's

potential memory may be so large that he does not live long enough to fill

it. Compare this to the allocation of disk space in a reasonably active time

sharing system!

If memory is so large, and the capacity of our input devices so great,

why do we ever forp,et anything? One bottleneck is in the transformation

from intermediate to permanent memory. Information transmission from the

sense organs to the sensory projection regions of the brain is quite rapid.

From the sensory projection areas information apparently goes to a working

memory area, although where this is is not at all clear. It may not

even be in the cortex.* T Te shall argue in the next section that functionally

there may be several short term memory areas, although it does not follow

that they are physically distant. Two functions must be performed in short

term memory. Pictures of the currently present environment must he compared

to previously stored codes and the current environment must be itself

recorded in permanent storage. Note that what will be stored is the signal

which results from the interaction of an "outside" signal with the signals

received from permanent memory during the recognition process. It may take

an appreciable time to complete this analysis. Similarly, ouite a consider-

able time may be required to complete the permanent storage process. Here

we are very handicapped by species differences. It is difficult to obtain

the necessary data from humans. Animal research can give us some idea of

the process, but gives us little information about the absolute times

involved, since strong species differences have been noted between animals

closely related as the rat and the mouse, and even between different strains

of inbred rats and mice.

Let us turn now from physical considerations to functional ones. Even

4. A FUNCTIONAL ANALYSIS OF HUMAN DATA HANDLING

4.1 Buffering.

Discussion of the sensory system is apt to leave one with the impres-

sion that the brain is a huge telephone network, with cables flowing in

and out of some magical central unit where conscious thought takes place.

This is far from the case. The input system is a highly buffered one.

As you read this page you experience a continuous flow of data to your

brain. In fact, however, your eye is jumping across the page, fixing one

spot for about 200 msec, then taking 40 msec to jump to the next fixation

point. Somehow discrete blocks of data are buffered to produce a subjec-

tively continuous input.

The peripheral visual system contains a relatively large capacity

buffer which can hold information for about one second. This has been

shown in a series of experiments on the recall of very rapidly presented

material '. Suppose you are asked to observe a very brief (50 msec)

display of letters in the following form:

XRQ X T S

JVK L B N

XAW V Z E

If you are typical, you will be able to report four or five letters,

probably those beginning on the left of the first line. One's first

thought is that only the first line can be read during the time allowed,

but further experiments show that this is not the case. Suppose that after

the dispaly had gone off you received a signal (for instance, a high

16

pitched tone) to report the third line of the display. You would still

be able to report only four or five letters, but this time they would be

from the third line. What does this mean? First, you must have read the

entire display into a sensory buffer. Next you must have transferred

data from the buffer into a more permanent, conscious memory. This process

appears to be a slower one than the first, for as it occurs, information

fades from the sensory buffer. Thus, if you process in the normal read-

ing order, beginning at the upper left-hand corner, the buffer contents

will, have faded by the time the top line has been read into the central

memory. What the signal does then, is to alter the order in which infor-

mation is read, not from the page into the eye, but from the sensory buffer

*into central memory.

The buffer appears capable of holding information for about one second,

under the conditions of these experiments. This follows from the observa-

tion that recall cannot be altered if the selective attention signal is

delayed for longer than a second. Is this delay a property of the visual

system or is highly buffered input a more general characteristic? Cer-

tainly the particular buffer used here is affected by subsequent visual

events, for if the stimulus is followed by a bright flash (sooner than

any data can be read from the buffer), recall drops to nearly zero.

On the other hand, data from a variety of studies indicate that buffering

is a phenomenon of other sensory systems.

Two observations are of particular interest because they provide

evidence for buffering in a very different experimental situation. In

the last few years many studies have been conducted which show that material

17

which would normally be remembered very easily can be forgotten in a few

seconds if the stimulus presentation is followed immediately by interfering

activity If a person is shown three consonants- (e.g. , DYQ) ,and then immediately asked to begin counting backwards by threes from an

arbitrary number, the probability of correct recall of the letters will

be less than 0.2 after only fifteen seconds. Of course, if there is no

counting, recall is perfect. The results can be duplicated with spoken

or visually presented stimuli. This is very important, since it shows

that the interference effects are not due to competition for specific

*sensory pathways. In fact, the interference is produced by a central

activity (counting backwards) , which does not produce input over any sensory

channel. In computing terms, what counting backwards must do is occupy

a central memory area which is required for transfer from the sensory buffer

to a more permanent store. This would decrease the rate at which informa-

tion could be read out of the buffer, without decreasing the rate at which

the information held in the buffer was decaying. As a result, less infor-

mation would reach the more permanent storage area.

4.2 Code Analysis in the Buffers.

During the passage from peripheral to central storage buffers, meaning

ful information is subjected to a number of categorizations, so that

the information which is eventually processed by the brain is a compactly

coded representation of the information in the original stimulus.

Some of the coding is implicit in physical characteristics of the

nervous system. The frog: ( 34, 59 \ cat(43,44) . rabbit (1 » 66) , and the

(45)monkey have in their visual systems special "feature detecting" neurons

18

which fire when particular patterns impinge on the retina. For example,

the cat has horizontal and vertical line detectors, and the frog even

has a unit which, without too much of a stretch of the imagination, can

be thought of as a detector for schematized bugs. Thus the message which

is received centrally is not an ensemble of bits, it is a set of characters.

If the character set reflects the meaningful units of environment, then

the brain will receive the basic information It needs to enable the animal

to adjust to the world. If the character set does not provide a good coding

for the environment, then the animal is in trouble.

Physical feature detection is only a small part of the coding which

occurs in buffers. Very early in the perceptual process stimuli are also

recognised as familiar units. We have argued that in each of the buffers

the stimulus is, in effect, a fading message. The problem the nervous

system must solve is how to recognize and code this message before it fades

completely. Instead of seeing the message as an ensemble of on-off signals,

it is more useful to think of it as a multi-dimensional signal, where each

dimension transmits information. We need to know the capacity for coding

a single dimension, and the ways in which information from several dimen-

sions may be combined to obtain a code for the entire stimulus. Signal

recognition studies (sometimes called absolute judgement experiments)

attack these questions.

In signal recognition a set of possible signals is established, and

on each trial one of these is shown. The person's task is to make a re-

sponse identifying the signal. The resulting data can be analyzed using

the same techniques one uses to analyze the transmission characteristics

19

of an electronic channel. The basic statistic is the probability that res-

sponse lis made, given that stimulus J_ was sent. This is formally analogous

to the probability that signal i will be received given that signal j_( 3.2)was sent . .

( 61 )Miller , after reviewing a number of experiments, noted that

the human ability to transmit information is remarkably constant across

the senses. Table I summarizes the results of several studies. What

PUT TABLE I ABOUT HERE.

this table says is that if a stimulus varies along one dimension only,

we are able to identify about seven different positions on that dimension.

Now, obviously, we are able to identify more than seven things. This is

because most of the stimuli which we face differ along more than one dimen-

sion. A number of studies have shown human channel transmission capacity

can be substantially increased by the use of multi-dimensional stimuli,

but that the information gain across channels is not additive. This is

shown in Table U. Clearly there is incomplete addition of information

PUT TABLE H ABOUT HERE.

from the different dimensions. A question that arises is whether this is

because simultaneous but less accurate judgements are made along each dimen-

sion, or because people react to global impression as a single stimulus.

The available data indicates that the former is the case; separate

20

(32 )uni-^dimensional judgements are made, but each with decreased accuracy

At first glance, these figures defy our intuition. We know that

we recognize a myriad of objects; faces, birds, trees and the like, and

we do so quickly. We know we can remember a very large number of impressions

Yet when we look closely, we find that we have a very limited capacity to

identify points along any one stimulus dimension. How, then, are the com-

plex tasks accomplished? Hie answer apparently lies in our ability to

code stimuli in terms of familiar, well learned objects. What is stored

and what is remembered is a record of how we interpreted our experience,

in terms of our previous knowledge. We suggest that the nervous system's

capacity for making distinctions between stimuli based on their physical

variations is only relevant in analyzing the peripheral buffers of bur

memory and storage system. Very early in this recognition process, the

information in the multi-dimensional stimulus is receded as a set of labels

referring to previously learned categories. This Is what we remember.

If the above analysis is correct, the amount of information a person

can process will be determined as much by the number of labels required

to describe the state of the world as by the amount of information to be

transferred. In the same article in which he pointed out the stability

of human absolute judgement, Miller pointed out that memory is limited in

quite a different way. In immediate memory experiments we can recall seven

to ten characters, regardless of whether they are digits or letters, al-

though the information in a sequence of letters is obviously greater

than the information in an equally long sequence of digits. Even more

dramatically, a person who is really familiar with binary to octal

21

conversion, can "remember" a string of 36 binary digits by recod ing them

as 12 octal digitsl (To do this, you must become very, very, very familiar

with binary to octal coding.)

Recoding is more than a mnemonic trick, for recoding not only gives

one a more efficient memory code, it also provides protection against

interference. Recall those experiments in which three letter trigrams

were forgotten within seconds in the presence of an interfering activity.

It has been found that a crucial variable is the number of labels to be

remembered . If nonsense trigr.ims are presented , e.g., TCA, then three

code words, the letters, must be held in memory. If the letters are easily

recoded, a single label, as in CAT, then the recall of this single code

unit should be more like the recall of a single letter. This is indeed

the case. Recalling three familiar words in the face of interfering( 62 )

activity is like recalling three letters

4.3 Code Conversion

Buffering permits the recoding of environmental information into

progressively more economic forms. Such a process requires that at each

level the current message be scanned and matched against possible codings.

Are all components of a buffer analyzed simultaneously, or is attention

focussed on parts of the message in sequence? The data presented earlier

on tachistoscopic presentation S 87 » 5 ' indicated that sequential atten-

tion was the case. Given that a particular chunk of the buffer contents

has been selected for recoding, the search through memory could proceed

in several fashions. One method would be to send a "broadcast" message

throughout memory, describing the input message to see what matches existed

22

for it in memory. It also might be the case that a single copy of the input

was compared successively against our memory for each possible signal

until one was found that matched. This would be, in effect, a simple

table look-up. Finally, the input could be broken into features, and the

features tested successively to establish progressively narrower sets of

possible memory matches.

Neisser *64 * has presented a strong case for different processes

at different levels. In the analysis of sensory buffers the first thing

that must be determined is what the stimulus units are. This analysis,

which takes about 100 msec, seems to be parallel for the entire stimulus.

The purpose of the analysis is to determine what segments of the stimuli

probably correspond to code words in memory. For example, at this state,

dark letters would be distinguished from light backgrounds, even though

the letters themselves were not yet recognized. The stimulus chunks would

then be taken, one by one, and subjected to a more careful analysis of their

features

The feature analysis itself appears to be a sequential decision-making

process. Three lines of evidence for this can be cited. One is the data

from reaction time experiments, which study the time required to make

an identifying response, with varying numbers of possible stimuli. For

equally likely stimuli, this time is a logarithmically increasing function( 49)

of the number of alternatives X Now suppose that the stimuli are not

equally likely. Reaction time will then be a function of the information

in the stimulus display . Even more interesting, the reaction times

to an individual stimulus will be inversely proportional to its probability

23

(29)occurrence . Thes . observations support the following picture of

stimulus identification. When a stimulus is presented, its sensory repre-

sentation is read into a buffer. A parallel process then divides the buffer

into regions, each of x/nich will be r^codei in sequential process. In

recod:Xg one of these regions, a sequential search is conducted for fea-

tures. The search is evidentLy a flexible one, which can be effected by(29)

learning . . Tests are mada which progressively narrow the set of

possibly correct codes. A cods will be assigned to a stimulus complex

as soon as the odds are high eaough that that code is correct. For this

reason stimuli with greater a priori likelihood will be recognized more

rapidly than unlikely ones.

There are two more experimental observations which provide some inherently

interesting data which fit rather nicely into the model proposed here.

Neisser * , asked how people search a list for an item. He pre-

sented up to fifty lines of letters. An abbreviated example of his material

is

EH V P

SW I Q

VF C J

The task was to search the list for a line containing a critical letter

(e.g., look for a line with aX in it). Not surprisingly, the time it

takes do this is a function of the position of the correct line in the

list of lines. This says, in effect, that it takes a constant amount of

time to scan each line. More surprising, a person can be asked to scan

for a line containing any of several critical items (e.g., scan for a line

24

with a Zor aX) without increasing the search time. Notice that it should

not, if the sequential branching process was used. The reason is that

the feature analysis would proceed the same way in the case of a search

for multible patterns as it would for single patterns. A search would

be conducted for features that either identified the input as a target pat-

tern or that prove out the pattern. The number of features to be detected

*need not increase linearly with the number of patterns sought.

Our final application of the model is to the well known phenomena

of perceptual set. It is common knowledge that we see and hear what we

expect to see and hear. Undoubtedly there are motivational components

to this behavior, but it can also be produced as a pure decision process.

If a person is presented with the stimulus E his perception of it as

a 3 or a / > will depend on whether or not he has been lead to expect

letters or numerals * . Similarly, it takes longer to recog-.

nize a picture of a violin held by a man dressed in a track suit than a

picture of a discus in the same setting . This is exactly what we expect

from a sequential decision process. The number of cues which must be

detected before one feels sufficiently confident of a categorization to

make a response will be a function of the cues detected previously.

4.4 Progressive Code Changes Over Time

If stimulus information is progressively encoded as it passes from

buffer to buffer, certain aspects of complex mental processing shoild be

predictable. Consider the problem of recognizing that a stimulus falls

into a particular category. The time this takes should depend upon the

nature of the category; higher order recognitions which require more

25

categorization, should take longer. Performance, en complex tanks requiring

several categorizations should be predictable by a model in which a com-

plicated description is broken down into sequence of recoding steps.

Finally, when a stimulus is complex, it should take tine for an appropriate

coding to develop.

Fosner and Mitchell measured the time needed to make a classifi-

cation based on the physical attributes of a stimulus and the time required

to m&Ue a classification of tha same stimulus basad upon its membership in

an abstract class. In a typical experiment a subject would be shown a pair

of letters, and then asked whether they were "the same" or "different".

The basis for "sameness" varied under different conditions. In the simplest

case only physically identifiable stimuli (e.g., AA) were to be called

the same. At a more complex level, two forms of the same letter might be

called the same (e.g., Aa) . This was called "name identity." In a more

complex "rule identity" two stimuli were to be called the same if they

were members of the same class e.g., vowels (Ac) . In the more complex cases,

"sameness" should be easier to detect than differences, since if two

stimuli are the same at one level (e.g., physical identity) they must

be the same at higher levels. Table 111 summarizes some of the reaction

time Posner and Mitchell obtained. These were the times required to give

correct judgements based upon rule identity when in fact identity could

be detected at varying levels. Note that the more primitive the possible

detection of sameness, the faster the reaction time.

Table 111 should be placed about here.

26

Posner and Mitchell found that requiring a high level analysis has

very little effect upon the time required for peripheral analysis. What

effect there was could be reduced by practice. In a subsidiary experiment

they found that the time to decide whether or not two digits are both odd

or both even is virtually identical when the digits were presented through

ear phones, visually, or one digit In each mode. In each case a time of

about 800 msec was required. This was apparently composed of a 600 msec

period in which the sensory information x*as coded and its name determined,

and a 200 msec period in which the inclusion of each stimulus in the

set of odd or even digits was determined. These results are consistent

with the serial analysis proposed here, although, as Posner and Mitchell

were careful to point out, one can develope an alternate parallel processing

model to account for the data.

Now let us consider the problem of analyzing performance on a complex

task using a model which assumes sequential discrimination. In nonsense

syllable learning subjects are required to learn arbitrary associations

between meaningless letter combinations—e.g., JAX—GYR. Simon and Feigen-

baum 84 proposed a model of the required learning. (Interestingly,

their model was physically presented as a computer program since the

logic of the model was so complex that analytic predictions were difficult

to derive.) The Simon and Feigenbaum model states that when a stimulus

is exposed, it is first seen as collections of lines and angles. These

are passed through a "tree" of sequential tests (a discrimination net

in Simon andFeigenbaum 's terms) which results in recognition of letters.

The ensemble of letters is then passed through a similar discrimination

27

net whose output is a recognized syllable. The syllables, in their turn,

are passed through a net which detects parts of syllable pairs, and the

identity of the pair. Thus given only the stimulus term in a stimulus

response pair (JAX-GYR) in the example above, and if equipped with the

proper discrimination nets, Simon and Feigenbaum' s EPAM (Elementary Per-

ceiver and Memorizer) program was able to generate the response syllable.

In the EPAM model, learning is equated with the development of dis-

crimination nets adequate for the task at hand. Learning at one level

is not possible until an adequate discrimination net has been established

to handle the coding required at prior levels. This, of course, is consis-

tent with our analysis of recognition in memory. Simon and Feigenbaum

were able to simulate a number of experiments on nonsense syllable learning.

In particular, they were able to mimic the effect of varying degrees of

similarity between stimulus items during learning. Obviously this would

be of crucial importance in developing an appropriate discrimination net.

Another implication of our argument is that the progressive recodings

of a stimulus can, in a sense, be elaborations upon the original image

as it is assigned a meaningful code. Neisser takes this argument

somewhat further, taking the position that recognition occurs when the

nervous system is able to synthesize a meaningful percept out of the

fragmentary parts provided by the environment. If so, then when a meaning-

ful stimulus is presented under bad viewing conditions, it should become

sharper over time, as it is elaborated by reference to memory. This effect,

which is a sort of "inverse forgetting" in which more can be recalled the

greater the elapsed time between presentation and recall, has been demonstrated

28

29

experimentally^27 A Lines of 24 letters each, equally spaced, were pre-

sented visually for less than a second. From one to ten seconds later,

observors were asked to reproduce them. In fact, the displays were either

randomly arranged letters, eight three-letter words which did not form

a sentence, or grammatical sentences consisting of the eight three-letter

words. An example of the latter is

HISB I GN E'W DOGBIT H E R 0 L DC AT.

Crawford, et al. found that the accuracy of recall increased over time.

The effect was most marked for the sentences (which could be subjected

to the highest level of recoding), and hardly present for randomly

arranged letters.

Finally, one can ask what the eventual coding form is. Do we store

things as pictures, spoken words, or in some ineffable central nervous

system coding? For normal adult humans there is evidence that the auditory

coding of stimulus information plays a crucial role, even when the infor-

mation Is presented visually. If letters are presented to the eye and the

observor then asked to recall them, the commonest confusions are between

those letter pairs which sound alike, rather than those which look alike

(26 ) . Th« presumption is that this confusion is due to: the fact

that the stored image of the visually presented letter is identical to

the auditory coding for the letter

4.5 Storage Management

Earlier we referred to the ability of the central nervous system

to literally turn off the peripheral processor, as in the case of a cat

suppressing auditory responses while watching a mouse. A similar selective

*

attention phenomena has been demonstrated for humans, but it appears to

have a quite different basis. Suppose a parson is presented with t;;o dif-

ferent messages, simultaneously, one to each ear. To be specific, imagine

an arrangement in which it is possible to speak "1, 2, 3," into one ear

and "7, 8, 9" into the other. If the listener is then asked to repeat

what he has heard, in the order in which he has hea^d it, he will reply first

with the entire message going into one ear, then with the entire message

to the other ear * ' . Originally this was interpreted as evidence

for a peripheral phenomenon, almost as if each ear had a buffer in which

It held messages for about one second, until the central mechanism was

ready for them . Other evidence however, indicates that the phenomena

is a central one Involving the organization of messages from different

sources. This was shown by altering the simultaneous message technique

slightly . As before, the messages consisted of pairs of items pre-

sented simultaneously, one in each ear. Instead of always using the same

type of item (digits) , different types of items (digits or words) were

used. Thus at one presentation a subject might hear a digit in one ear

and a word in the other. Table IV shows the typical sequence. The listener

was then asked to report what he had heard either by types (all digits,

then all words), by ear (all items in the right ear, then all in the

left) or by order of arrival (first pair, then second pair, then third).

By far the easiest report to make was by class of item, even if (as in

the example of Table IV) the items of one type were received by different

ears.

How are we to explain this? By the time the coded stimulus arrives

30

TABLE IV ABOUT HERE

in central memory, it will have been assigned a good many identifying

tags, such as a digit or a word, or markers. Yntema and Trask hypothesized

that in central short-term memory items are initially sorted into lists

of information with similar tags. At recall, these lists are selected

and read out one at a time. The same explanation can be applied to Broad-

bent's earlier results by noting that when the stimuli are all of the same

type the only differential tagging possible is by sensory channel over

*which they arrive.

The importance of coding impressions by type in short-term memory

is illustrated in studies of human capability to keep track of the current

state of several variables ' . Suppose an observer receives a serieSuppose an observer receives a series

of messages of the form

THE DIRECTION OF A IS NORTH

THE SPEED OF B IS SLOW

THE COLOR OF C IS RED

THE COLOR OF A IS BLUE

Aperiodically he is asked to state the current value of one of the

variables, that is, to respond to questions such as "What is the direction

of A?" In this task the variables can be considered to be attributes

(color, size, direction) of objects (A, B, C) . It Is much easier to keep

track of n attributes on one object than one attribute of n objects.

When there is only one attribute, the states are all the same, so that

31

32

the messages look like this

THE DIRECTION OF A IS NORTH

THE DIRECTION OF C IS SOUTH

THE DIRECTION OF B IS NORTH

THE DIRECTION OF A IS WEST(47 )and they are easily confused. Hunt simulated "keeping track" per-

formance by a computer program which assumed that a person had a very

limited capacity for remembering whole messages, but that he could recall

fragments of recently heard messages. If the type of a fragment identifies

the variable to which it belongs (as it would in the problem of keeping

track of color, size and direction of one object), a good guess can be

made about the last message sent. All the observer has to do is to find

the most recent item on the appropriately tagged list. If the task is

one of keeping track of the same attributes of different objects, then

the list to which a message fragment is assigned no longer identifies

the variable, so message reconstruction is far more error prone.

We have now reached the point at which information has arrived in

central short-term memory. We have found that prior to the arrival the

message passes through several buffers, at each of which it is coded.

The final contents of short-term memory probably consists of auditory

codes, sorted according to various tags. Tags will have been established

by consulting long-term memory via a sequential search technique which

is itself much influenced by the context in which past searches have been

conducted. Our attention has been focussed on how man learns what is pre-

sent in his short-term memory. In the next sections we will ask how he

uses this information to solve problems which may extend over time.

4.6 Real Time Problem Solving

Most recent models of real time problem solving assume three sorts

of memory; a sensory buffer which is used to code input, a memory buffer,

and a long-term store, in which information resides for the duration of

the problem-solving session. The force of our previous argument has been

that the sensory buffer is really a series of buffers and that they are

probably not that closely tied to sensory systems. In the next few pages,

when we refer to memory buffers we will be referring to a holding area

in which a meaningful semantic unit is held in short-term memory. We

shall for the most part assume that no further recoding is being done.

Rather, processing is now being devoted to establishing relationships

between messages that have arrived in memory at different times.

The time periods we are thinking of are of the order of seconds and

minutes. We will speak of a memory buffer as haLng capable of holding mes-

sages for several seconds, perhaps even a minute. Auditory rehearsal is

usually suggested as a mechanism for doing this. Long-term store will

refer to a storage process capable of holding information for several moments

Clearly there are shorter memory buffers and longer long-term storage

processes.

The short-term memory buffer has two Interesting characteristics

which set it apart from sensory processing. It is evidently quite small.

If we define the immediate memory span as the number of items which can

be repeated back just subsequent to their presentation, the usual estimate

of short-term memory is from six to ten items. As we have seen, this figure

is independent of the amount of information per item. If a person is

33

required to do something with his memory, however, the immediate memory

buffer becomes still shorter. This has been investigated in a recent

series of experiments in which people were required to keep track of the

changing states of several variables, but in a situation somewhat different(2 3 )

from the Yntema and Meuser studies ' . Short-term memory capacity

in this situation varied from between two and five items. Further analysis

indicated an even more unusual phenomena—the subjects had considerable

control over what went into their short-term memories. To illustrate(2)

hoxj this happened a brief explanation of the Atkinson, et al. " procedure

is necessary. A sequence of letter-number pairs were presented one at a

time. Periodically a letter alone would be presented, and the observer

had to repeat the number which had last been paired with it. Atkinson,

et al. analyzed their experiments by assuming a mathematical model in which

the information at any one presentation either might or might not enter

short-term memory. If it did it would force the removal of a previously

entered item. Finally, while it was in short-term memory, Atkinson, et al.

assumed that some information about the item could be passed into longer

*term less specific memory. It was found that the best model of how people

use this buffered memory arrangement varied depending upon the conditions

of the experiment. For example, if on every presentation of a letter it

was re-paired with a new number, then each presentation should be treated

independently. The data indicated that a newly presented item would

be entered into short-term store with probability 0.39. This is a way of

saying that the observer could ignore some presentations while he concen-

trated on storing information about those items already in the buffer.

34

If the experimental conditions were changed slightly, the model for using

memory also had to be changed. When the letter-number pairs were repeated,

that is leaving an answer unchanged, in order to get the model to fit the

data in this situation, it became necessary to assume that the observer

first searched his memory to see if he could correctly answer the question.

Only if he could not, did he consider entering the item into short-term

memory. This would be an efficient strategy because it would not take up

valuable short-term memory space to store information about items which

were already correctly placed in memory.

Several other changes in the experimental conditions could force other

changes in the model. The exact details are unimportant, the principle

is. Memory can be viewed as having two components; structural detail and

the control processes. Buffer and long-term stores are, presumably, fixed

structural aspects of memory. How these resources are assigned in executing

a particular task will depend on the control process ("strategy" in a more

cognitive psychology) which is adopted.

Let us consider control processes for tasks which require short-term

memory. Obviously, we cannot list an engineering handbook description

of human control processes. This would be like listing the programs that

could be written for a computer. Instead we shall illustrate with two quite

different uses of short-term memory in problem solving.

Mental arithnetic is our first example. The evidence is entirely anec-

dotal, based on accounts of professional mental calculators in persons

who are capable of doing what appear to be prodigious feats of calculations

without the aid of pencil or paper( 31) .* The stage calculator appears

35

to have two tricks. First, he uses strategies which reduce the number of

numbers which must be remembered at any one time, although it may increase

the number of steps required to perform a

vihg the problem "236 x 47" in the usual

you (1) compute 236 x 7, (2) remember the

(4) multiply the answer by 10, (5) recall

calculation. For example, sol-

school fashion requires that

answer, (3) compute 236 x 4,

the answer stored at step two, then

(6) add the two numbers obtained to get the final answer. The alternative

method, used by the stage calculators, is to perform the vector multipli-

cation

(40, 7) 20030

6 ,where one needs only to remember a running sum while performing x;iell-practiced

simple multiplications. The process is greatly simplified, by a second

trick, memorizing larger multiplication tables than the familiar 10 by

*10 one.

A task that has been better analyzed than mental calculation, but is

much less interesting, is called the concept learning task. Here the learner

observes a sequence of stimuli which vary along well defined dimensions.

Examples are: large gray square, little red circle—the objects here

varying in size, color and shape. Each object will have been assigned

into a class by some predefined, and usually simple rule. An example is,

"All big red objects are +'s, all others are -'s." The learner's task is

to discover the classifying rule. In computer science terminology, this

can be thought of as a pattern recognition task, with the peculiarity

that the objects are given in sequence, so that the classification rule

36

must be continuously updated. There is a very large literature on concept

( 11, 46) _. .learning, which we shall not attempt to summarize here . This

( 48 ,literature, and especially some of the more recent observations

36, 76, 91)^ seems cons istent with the following picture. In concept

learning the learner stores two things, his current guess about the correct

answer and a record of the last n objects presented. The object storage

is analogous to the short-term memory buffer in the keeping track studies.

Probably less than five records are stored, and these are probably not com-

plete pictures of the objects. That is, if the last object shown was

a "large green star over a red bar," the subject might only remember

"large green star." As new information is presented, an attempt is made

to alter the working hypothesis to fit the data. Alteration strategies(21) (91 )

have been described by Bruner, et aIX , and Trabasso and Bower

In some cases alteration will be Impossible, so the hypothesis must be

abandoned for a new one. The new hypothesis will be developed by analyzing

the information in the object storage memory at the time. Note that if

the object storage were of size 0, then the new hypothesis would have to

be selected at random. In fact, this assumption almost, but not quite,

fits the data from many concept learning experiments.

4.7 Multi-Programming

The current style in computing is to have several tasks active within

a large computer at the same time. To what extent can people do similar

work? Our answer is that the limits on human multi-processing are determined

by the availibility of appropriate memory buffers. If two tasks do not

compete for the same level buffer, they can coexist. Similarly, if two

37

tasks can pass information to each other via a particular buffer, they

may follow each other in rapid succession, or be interleaved. If they are

at the same level but do not communicate, then time must be allowed to

clear the buffers before beginning the second task.

The point concerning multi-level tasks is fairly easy to prove.

You can talk while driving. You can even carry on a conversation at a

cocktail party. We suspect this is possible because well practiced tasks

are handled by more peripheral buffer systems. Now suppose that at a

cocktail party someone who is not party to your conversation speaks your

name. Your attention will be diverted. The example is commonplace,

but look what had to happen. A fairly complex series of pattern recog-

nition programs had to be initiated in peripheral buffers, first to segment

sounds into phonemes, then phonemes into morphemes, and finally to recognize

a specific word. The retrieved image of this word must have carried xjith

it something like a priority interrupt signal indicating that higher level

processing was required—hence the intervention of conscious attention

buffers. The interesting point is that the same series of analyses must

have been carried on—up to the point at which it was clear that a priority

interrupt xras not required—for all those sounds reaching your ear to which

you did not respond.

To illustrate the problem of task complexity where two tasks compete

for the same buffer, or share a buffer, we will turn to some data on free

recall. If a person listens to a list of unconnected items, and then

is asked to recall them, he will do best on those items at the front of

the list (primacy) and at its end (recency) . This is what would be expected

38

from the Atkinson and Schiffrin buffer model of memory. Initially the

short-terra memory buffer would not be filled, so those items at the front

of the list would all be placed in it, and information about them trans-

ferred to long-term store. As the short-term buffer filled there would

be a greater probability that a new item would not enter the buffer or

would be there for only a short time, thus limiting the amount of informa-

tion that could be transferred to long-term memory. This accounts for the

primacy effect. If recall is tested immediately after a list is presented,

the last items would still be in the buffer, so they would have a high

probability of recall. This accounts for recency. But what if the memory

buffer should be refilled immed.-.'.ately after the last presentation of an

item, but before recall xv-as required' According to the model, primacy

effects should remain, and recency effects should be destroyed. This

was demonstrated by asking people to do simple arithmetic after the list

of items to be remembered has been presented, but before they are recalled.

Only primacy was observed

Compatible and incompatible buffer uses at the same level have been

illustrated in some recent studies of decision making and memory in our

own laboratories . The memory task was to keep track of several vari-

ables, all of which had numerical values. The decision was to predict

the value of an additional variable, which was in fact an unknown linear

function of the n variables the subject had to remember. As the number

of to-be-remembered variables increased, the subject was faced with the

difficult task of keeping track of several variables x*ith the same states.

As would be expected, performance deteriorated rapidly. The decision task,

however, receives information from but does not compete with, the memory

39

task. It was found that any deterioration in decision making could be

completely accounted for by the deterioration in the memory task. Another

way of saying this is that the person who had to make a decision based upon

remembered information did as well as he could with his faulty records.

Similarly, accuracy of memory was the same whether or not a decision was

required. People could switch from the memory to the decision tasks and

back again without any difficulty, because the two made complimentary,

rather than competing, use of memory.

4.8 Long-Term Memory

Finally, we approach the most interesting question. Hox* do people

use their long-term memory in decision making and problem solving? Un-

fortunately, we must be anticlimactic. In spite of the importance of this

question, there is much less knoxm about long-term than about short-term

memory. This is true if we consider either possible physical mechanisms

for memory or if we consider functional models to describe memory. We

will be unable to cite as many experimental findings, partly because of

the difficulty involved in making controlled observations of humans. over

periods of months and years. The best we will be able to do is to cite

a few studies, recall some facts everyone knows, but seldom thinlcs of,

and then put forward a very tentative model for information storage and

use over long periods.

First, the commonplace facts. Long-term memory is "output limited"

—you often cannot produce all the information you know you have. (Can

you state the names of all the people you know?) Items of information

are clustered together in recall, so that if one item in a group of items

40

is given, the remaining items are more likely to follow. If you name one

member of your bowling team, the remaining members of the team will follow.

Finally, an item may appear in several clusters of items. It may be

possible to recall a particular name either by recalling the names of

bowling team members or recalling the names of people living on your block.

Observations such as these have led to suggestions that experiences

are stored as a network of association. Although the idea is an old one

it goes back at least to Locke... our presentation xd.ll draw heavily from

(73 ) (75=)the more recent ideas presented by Quillian and Reitman

As in computer based systems, let us consider records, the items stored,

and the relations between records, the pointers which serve to organize

the storage system. We shall hypothesize two types of records, records

of the relationship between the abstract categories and records describing

specific events.

We see our general knowledge of the world as being determined by a

network relating abstract categories to each other. The effect is that(73 )

of an elaborately cross-referenced dictionary. Quillian offers the

example of our memory of the word plant (either written or spoken), which

would be tied to three nodes, referring it to the living thing called plant,

a manufacturing plant, and plant as a verb. These definitions would, in

turn, be implemented by describing their relation to other symbols in the

general knowledge network. Thus the auditory stimulus plant would be tied

to plant 1, plant 2, and plant 3 by the relation external stimulus-internal

code. Plant 1 would be tied to living system by the relation member of

set, and to the internal symbol tomato by the relation tomato exemplifies

41

plant 1. Similarly, plant 2 (manufacturing plant) would be a subset of

buildings, and tied to the concept production by the relation use. An

adequate model of the knowledge held by any human being would have to be

a very large network indeed, with multiple connections possible between

nodes .Now consider codes for specific events. To do this we introduce two

new ideas, the idea of a unitary event and of an event graph. A unitary

event is defined as a set of references to the labels of our general know-

ledge memory. For example, consider the recollection of a "big black dog

chasing a cat." We envisage both animals as being stored as sets of references

to the appropriate labels, dog, big, etc.; while the experience xrould be

stored as a graph (dog, big, black) chasln| (cat, little, white). The

major difference between event and general knowledge storage is that

in the event storage temperal sequences are permitted as possible linkages

between nodes in the graph.

Event storage depends heavily upon linkages to general concept memory

We do not believe that we store experiences, rather we store references

to previous ideas. The possibility exists that there is also a parallel

storage of uncoded sensory experiences themselves. If so, this memory

is usually not accessible to most of us. The evidence for its existence

at all rests largely upon reports of unusual individuals or special situa-

tions (see footnote on page 29 ) .Retrieval of information from the long-term network is assumed to

be a search and screening process. Assume that several nodes are given

as guidance for search of long-term store. Items connected to these

42

nodes will be selected, drawn into short-term memory buffers, and examined

for relevance. For simplicity, assume that one of three decisions must

be made; the new item is irrelevant, the new item is the answer desired,

or the new item is not the answer, but a further search should be conducted

of nodes connected to It. If bad decisions are made at this stage, memory

buffers may become jammed, or one may simply run out of leads even though

the necessary information is in memory. If good search decisions are made,

then memory may be screened quite effectively.

These proposals have more of the status of a philosophy for thinking

about thinking than they do of a formal model of memory. Let us see, briefly,

how they can be used to account for some selected phenomena concerning

human memory and recall. First, consider the übiquitous effect of context(72 73)

on human recall and reasoning. Quillian ' has suggested that we

think of connections in a memory network as being more or less open, de-

pending upon how recently they have been activated. If this were so,

it would account for the clustering phenomena in free recall, and also

for our limited ability to recall names. If relations are symmetrical,

and we have been recalling names based on a "teammate" relation, there

will be a tendency to cycle. Strict symmetry is not necessary, the exis-

tence of short loops would do as well. A similar explanation can be offered

for an apparently dissimilar task, performance on verbal analogies questions.

Consider the two items:

UP IS TO DOWN AS RED IS TO [PINK, BLUE, GREEN]

COMMUNIST IS TO SOCIALIST AS RED IS TO [PINK, BLUE, GREEN]

The network memory interpretation is that the first item establishes

43

a bias toward responding along the relationship opposite, while in the

second case there is a call for a subtler search via a representative

color relation.

In our final example, we point out that if this model is correct

we would expect recall of specific experiences to be facilitated if, at

the time of storage, time were allowed to construct event graphs with

many links to permanent memory, and to make several connections other than

temperal contiguity between the links within the event graph. Bower

(personal communications) has reported a series of experiments which can

be interpreted in this light. His subjects were required to memorize

arbitrary lists of words. In one condition the task was presented as one

of rote memorization, in another the subjects were urged to make up stories

in which the words appeared in order. In terms of the quantitative infor-

mation involved, remembering a story would add to the amount of information

the subjects had to recall, while in terms of the memory network, it

increases the number of links between the nodes. Not surprisingly, the

subjects who made up stories were far better able to recall the lists of

words then those who had treated the task as one of memorization.

Bower's results, and the other data which has been cited for long-

term memory, may or may not be due to the accuracy of our memory network

model. In any case, they do illustrate a general principle. Human memory

is large, but storing information into it is a slow process. Immediately

after information is presented to a person, it enters a fragile, labile

storage stage. If later recall is desired, it is most important that people

be allowed to fix the information from the labile stage into long-term

44

45

'

memory. The more complex the information to be stored is, the longer

the fixation process takes. This is a point xrtiich should be kept in

mind by anyone who uses man as an information storage device, be he

the designer of an educational or a command and control system.

5. MAN IN AN INFORMATION SYSTEM.

5.1 Introduction

Man is an essential, Indispensable part of many information

processing systems. In all cases, the ultimate goal of the system

is to present information to a human. In many systems man also func-

tions internally, as one of the system components. We have discussed

some of the operating characteristics of man "in" isolation. Here

we take up the problem of getting information from the computer into

the human and vice versa. No happy combination of variables nor any

schema for presenting information is likely to increase the basic

rate at which ordinary humans can receive information. Rather, the

emphasis here is on making communication between man and computer

as easy as is possible, so that nothing interferes with the funda-

mental tasks occupying the man's mind, or actually impedes his

performance. Thus, the emphasis is on preventing hindrance rather

than on special assistance.

5.2 Computer Output; Human Input

Since the WWII studies on the design of dials for instruments,

much work has been done on the physical arrangement of display, but

relatively little of great significance or generality has resulted.

Oneexception is Yntema's summary(98 }of the ways in which information

46

should be coded for an operator x*ho must deal with a changing situation

(a) Each variable of which he must keep track should

have its own exclusive set of possible states. To take a

simple example, suppose that the operator is to keep track

of the altitude and speed of an airplane, and suppose that

the values of each variable are categorized into three dif-

ferent levels. Then, if altitude is categorized as high,

medium, or low, speed should be categorized as fast, inter-

mediate, and slow.

(b) There should be few variables with many possible

states, not many variables with few states.

(c) The frequency with which a variable changes state

should be kept to a minimum.

Yntema further concludes:

(a) Capacity for random information is low. People

make mistakes when keeping track of two or three things at

once .(b) Performance is not much improved when each variable

goes through a regular, predictable sequence of states.

(c) Performance is greatly improved by correlation be-

txjeen the present states of different variables, at least when

the correlation is extreme.

Recently, Gould and Gould and Makous have summarized the

optimum variables for the visual variables characterizing visual dis-

plays consisting of cathode ray tubes and laser-powered displays,

respectively. Their conclusions are summarized in Table V.

TABLE V ABOUT HERE

i

i

I

47TABLE VI ABOUT HERE

5.3 Human Output; Computer Input

Some displays are part of a system that not only presents com-

puter output for human consumption, but also permits input to the

computer from the human, via the display, a light pen and a typewriter

Table Vlshows a list of desirable features for such a display, all(55 )

of which appear in Licklider r s recent book . We have given with

each item Licklider' s subjective estimate of its importance, on a

10 point scale of increasing importance. Table VII gives a list of(55)

shortcomings of presently existing consoles. "" ' ' Licklider- also makes

the following prediction:

"The oscilloseope-and-lisht-pe'i schema of the next decade

should have a hard, tough surface upon which both the user

and the computer can print, write, and draw, and through

which the user's markings will be communicated to the com-

puter. Even when this surface is flush with the top of a

desk, no "electron gun" sticks down through the desk and

bumps the user's knees. The marks appear on the surface,

of course, and not on a lower subsurface: there is no

explosion screen and no parallax. .ideally, the user and

the computer should make their marks in precisely the

same coordinate frame, so that it will not be necessary

to compensate for poor registration. It is easy and

natural to designate part of an observed pattern by

pointing to it or touching it directly with fingertip

or stylus."

TABLE VII ABOUT HERE

48

While on this topic, we should remember some things any programmer

knows: (1) Present techniques for manipulating a computer are very

clumsy, like writing with an x-y recorder by twittling two potentio-

meters, instead of simply grasping d pen and applying it directly

to paper; (2) Much of the feeling of clumsiness disappears as one

practices and becomes more and more skillful at the initially awkward

manipulation. The second principle can be used to minimize the first.

That is, awkwardness may be minimized by using the most practiced

psychomotor skills we have, as a means of getting information into( 55)

the computer. Licklider points out that an exhaustive list of

skills that are both complex and xd-despread in our population contains,

at most, five items: (i) getting about in three dimensional space;

(ii) speaking and understanding natural language; (iii) writing;

(iv) playing musical instruments; and (v) typing—and he has misgivings

about the last three.'

Although typing is at the end of the list, it is by far the most

common means of entering information into a computer. Its borderline

quality is evident in the fact that so few people who can type, even

rapidly, can also compose text on a typewriter; and even among those

who can, many eke out a more stilted prose on a typewriter than by

paper and pencil. If all problems were put aside, speech would be

the preferred method of communication with a computer, in both direc-

tions. Not only does it have the advantage of being highly practiced

and widespread, but there may even be a "wired in" propensity for

vocal communication. Recall the data we presented on verbal encoding

i

I

I

ii

49

of visually presented information.

A Utopian dream we hold is that writing might be done in the fol-

lowing way: The writer speaks to a computer, which analyzes his words

and presents them on a CRT almost as he says them, punctuating as he

speaks according to grammatical rules, perhaps assisted by breaks

in the tempo of speech. To change a word, the writer points to it

on the CRT, speaks the desired substitution, and the computer instantly

makes it, adjusting spacing if necessary. Similarly, one sentence

would be inserted between two others in the middle of the text by

pointing to the location and saying, "insert ..." and then give the

sentence. Long text would be searched by pushing the text up or down

with the light pen, as though moving a scroll, or by a "moving window"

(55 ) Text is deleted or erased by light pen either with a button,

on the pen itself, or by accompanying vocal instruction.

Now we could do all this if the display, instead of being controlled

by a computer were controlled by another man! Only a rudimentary

start has been made on the task of getting the computer to handle the

sort of information on which human social interchange depends. It

appears that the human has formidable powers after all. He does, and

they are concentrated in the fields of pattern recognition and language

analysis.

5.4 The Place of Man

Given these human characteristics, what task assignments should

be made in a man-computer team? It is easy to make vague statements

such as "people should be relieved of tedious tasks, to handle flexible

i

50

decision making." At the other extreme, one can list specific tasks

and indicate which are done best by man and which by present day machines

Even easier, one can list and deplore human frailties. We have taken

a fourth alternative. We will discuss two tasks which people do better

than machines, in spite of considerable effort to automate them.

The reasons for human superiority in these tasks seem to us to be

reasons which will often dictate where a human should be used.

The first task is visual pattern recognition in a familiar world .It is well known that there are many pattern recognition tasks, such

as the recognition of faces, which are very easy for humans but quite

beyond the most advanced automated systems. A man can recognize his

wife in a new dress and hairdo, while a computer struggles to tell

that the chair is behind the table. On examination the discrepancy

is not surprising, since mammals are well designed for pattern

recognition. The information transmitting capability of the visual

system is far in excess of that of computer graphic inputs. A picture

which a human would refer to as grainy television xtfill fill a computer

input channel. In the visual tract itself, we have seen that verte-

brates have a number of "wired in" decoders, such as the line or

spot detectors of the frog, which automatically extract and code in-

formation which is usually useful to the animal. Human pattern

recognition is geared to a situation in x*hich certain codings, such

as "straight line counts," should always be applied to the environ-

ment, the amount of information required to describe the environment

is very large, and, finally, an approximate answer is required rapidly.

i

51

In studies in which these conditions are not met, computer pattern

recognition programs have exceeded human performance. Examples are

studies of the classification of patterns constructed from random

collections of black and white dots, where the normally useful codes(92 )

of the visual system do not apply , and studies of pattern recog

nition where the correct classification rule is based upon detailed

logical analysis of the pattern to be classified

Our second example is a task which has been the bane of machine

oriented linguists. Humans are very good at recognizing Items in

context, especially in linguistic problems. When "time flies like an

arrow," man does not look for a species of insect. Extralinguistic

cues can also alter human interpretation of words. The Virginian

could distinguish between the meaning of a noun phrase uttered by

friendly or unfriendly acquaintances in the time it took to draw a

.44. It is extremely difficult to get computers to perform similar

feats of disambiguation. The standard approach is to try to recover

all possible meanings of a sentence, with the result that a syntactic

analyzer may spew forth literally hundreds of meanings to a sentence

(53 )which humans do not find at all puzzling

We conjecture that people handle context well because they are

the sort of special computer designed for this task. Again, short-

term memory plays a key role. Unlike time sharing computers, humans

do not clear their primary memories when they shift from one task

to another. At any one time the state of the working store will depend

on the current stimulus and the traces from both the immediate past

I

I

I

52

stimulus and any memories which it recalled. Looking ahead, the

future state of working memory will be determined by the next external

stimulus and any retrieval from permanent store which the current

state produces. With this in mind, it is hardly surprising that people

have no trouble with the two sentences, "The umpire called a strike,"

and "The union called a strike." The early words in the sentence

establish the semantic context within which the possibly ambiguous

term "strike" will be interpreted. Human ability to disambiguate

does not depend upon word order, provided that the sentence is not

too long. Short-term memory can be used to collect a list of sounds

(or sights) which cannot be interpreted until the last one arrives.

An example would be a sentence in literary German, where the verb

would come at the end. Of course, this could not be done if the sen-

tence were so long that short-term memory was filled before the words

in it could be interpreted. German avoids this error by making the

individual words sufficiently complex that they can be parsed rather

like the phrases of English.

Our conjecture, then, is that the context sensitivity of humans,

and presumably other vertebrates, is intimately tied to two biological

features of their information processing mechanism. Recognition in

context depends on a slow paced short-term memory and a relatively

rapid, parallel search and retrieval mechanism for recognizing the

closest approximation to the current contents of short-term memory

from many records in long-term store.

1

53

5.5 Why the Difference Between Man and Machine?

In Information Sciences there is a persistant tendency to speak

as if there were certain inviolate rules of information management.

At one level of generality, of course, there are. Shannon's theorems

of information transmission apply to nerve axons and coaxial cables.

At times, however, this universalist view may be misleading. We think

it is here. We have argued that man can be regarded as a computer

with very great input capabilities, a tremendous and efficient perma-

nent secondary memory, and a very slow, inefficient internal computing

unit and primary memory. This is exactly the opposite of the

characteristics of a digital computer system. Why the difference?

Look at the environments in which both systems exist. The computer

system was developed to handle discrete jobs which arrived frequently.

Any one job has to be handled quickly, with a high degree of accuracy.

Fortunately each job is literally context free, in the sense that the

same computations are required regardless of what went before it.

Also, when a job arrives it carries with it a great deal of informa-

tion about the nature of any previously stored files it may require.

By contrast, the vertebrate exists in an environment in which jobs

change slowly, each one blending into the other. A task can only be

considered in the context of the one which has preceded it. Finally,

while a task hardly ever carries with it the physiological analog of

the names of the files it is going to use, it is usually the case that

the initial retrievals need only be approximately correct.

Viewed in this way, the contrasts between a human and an electronic

I

54

computer are more understandable. Each is a response to the challenges

of a particular environment. But is one of then perhaps the best,

or at least an adequate, response to both environments? Since no one

proposes to abandon computers, is man obsolete? In the following

section we turn to this question, considering some problems of the

psychology of robots.

Robots are fun! They have served admirably in short stories,

novels, movies, television programs and have generated fascinating( 33 )research proposals v . More than fun is at stake, hoxjever, for

the government has invested over a million dollars in such research.

The main motivation for designing a robot is to provide for

humanlike execution of a task a person xrould find excessively dangerous

or uncomfortable. Extraterrestrial exploration, especially Martian

is a frequently cited glamorous example. There are many more prosaic

examples here on earth, such as the handling of radioactive material.

A secondary motivation is the design of a machine which could economical!

replace human workers. If one restricts the term "robot" to a general

purpose device which can be switched from one assignment to another,

this goal has not been reached yet. We do, of course, have a very large

number of machines each of which is designed to do just one task

formally done by humans—from setting pins in a bowling alley to serv-

ing as the short order cook in a hamburger stand.

We do not dispute the fact that if general purpose robots can

6 . ROBOTS

6.1 Some General Remarks

I

8

55

be built, they will be of great use in the space program. Consider

the exploration of Mars. The weight of a man and his life-support

systems exceed present payload capabilities by orders of magnitude.

Further, a round-trip is at present completely out of the question,

and ethics prevent sealing a man on a one-way trip, volunteer or not.

Thus, the observations and exploration must be made remotely. In

moon probes we have had considerable success with remote control of

probes. This will not worlc. in an interplanetary probe. The capacity

of a communication channel between an extraterrestrial probe and

Earth is very low, and the time taken for a signal to span the

distance from Mars to Earth and back may be on the order of half an

hour. By contrast, messages go to and from the Moon in about a second.

Further problems are presented by the rotation of Mars, which would

periodically position a surface vehical behind the planet, unless

the probe could be accurately placed very close to a Martian pole.

We do not usually think of special purpose devices as robots,

although the capability which some such devices have for being controlled

by a stored program points out the fuzziness of our definition of the

term. For example, there is a commercially available device which

has as its main components a flexible arm, a magnetic tape, and a small,

special purpose computer which can either (a) sense the position of

the arm, relative to the "body" of the machine, and record it on mag-

netic tape, or (b) read a position from the tape and move the arm to

it. This simple concept has produced a powerful device. To "program"

it all one need to do is switch the computing unit into the tape

I

56

record mode, grasp the arm, and force it to perform the desired move-

ments, such as placing a bottle cap on a bottle. When the machine

is later placed in tape read mode, the arm will recreate exactly those

motions through xrtiich it was lead the first time.

Such a device is obviously a general purpose machine. Given a

pair of more flexible arms, something which is technologically feasible

though perhaps not economically practical, xve could record the motions

needed to serve a tennis ball or bat a baseball. Perhaps we could

produce the long sought ideal, the perfectly consistent athelete who

used the same basic motions in game after game. But would this be

desirable? Surely the machine xrould never double fault in tennis...

unless the wind changed. Once this happened, the machine would conti-

nue to wave the racket in exactly the same way, x^ithout any adjustment

for the new position of the ball. Is this a robot or isn't it?

We shall arbitrarily say that such a device, however useful the

things it does, is not a robot. To us a robot must have at least three

capabilities; it must be able to move, to receive information from

a distance (roughly as we do in seeing and hearing) , and it must be

able to adapt to changes in its environment. It is doubtful that a

device this general is economically feasible for conventional commercial

use. Fortunately for the robot designer, there are tasks much less

limited by mundane economics. Perhaps it is appropriate to build

a robot for space exploration, or for the conducting of extremely

hazardous experiments here on Earth. We say "perhaps" because such

a device has yet to be built, let alone to be evaluated economically.

I

I

[i

57

As will be apparent, we have some skepticism as to whether it will

be in the next thirty years.

The low channel capacity of interplanetary communication necessi-

tates some analysis of the data before transmission in order to reduce

redundancy. The periodic occlusion of a surface probe by the mass of

the planet Mars necessitates storage of the data collected during the

occlusion, unless an orbiting relay is used, which would present occlu-

sion problems of its ox<m. Even if periods of silence were not enforced

by the planet's rotation, the long transmission delays between the probe

and Earth present problans for any probe intended to react to its environ-

ment. To be able to handle the unexpected, some decision-making capability

must be built into the probe itself. Thus our general purpose robot

is needed.

6.2 The Problem of Perception

Robotology has both scientific and technical problems. The dis-

tinction betxveen them is important. A scientific problem exists xjhen

we do not know the principles underlying a phenomenon. A technical

problem exists when we look for a way to achieve some end using knoxm

principles. Scientific problems are always stated so that they can be

solved, in principle, whereas a technical problem may in fact be

unsolvable. Intellectually, there is ample challenge for anyone in

science or technology. Nevertheless, technical problems are secondary

to scientific problems in the sense that they cannot be attacked until

the corresponding scientific problems have been solved. You cannot

apply an unknown scientific principle.

Robot builders face many technical problems, which now seem

58

difficult but not insurmountable. There is also a scientific problem

that requires solution before some of the most challenging technical

problems can be attacked, and before the solution of the problems

currently under intensive attack can be of any value. That is the

problem of perception.

Many of the uses currently proposed for robots require it to gather

visual information from its environment. This poses technical problems.

Analog accommodation, or focussing of the optical system, for example,

can be done by a system that tracks the conditions yielding the highest-

frequency components in a Fourier analysis of the output of a vidicon

tube. Something analogous to retinal disparity, which is sufficient

though not necessary for depth perception, can be determined by locating

the mode of the function obtained by convoluting the output of txro vidi-

con tubes located at an appropriate distance from one another. The

relative distance of objects could be estimated by the motion parallax,

also sufficient but not necessary for depth perception, resulting

vtfhen a vidicon tube is moved at right angles to a scene. More compli-

cated but equally obvious techniques could and have been, used to

define contours, corners, enclosed figures, and so on.

Ignored so far, however, are the effects of noise, and the fact

that the unit of analysis is seldom an entire scene, but rather some

part of the scene corresponding to a particular object in the environ-

ment. The treatment of these aspects of the problem is what separates

the man from the machines.

A common fallacy underlying many failures to get machines to do

fl

591

what man's visual system does with great ease is the assumption that

the information necessary to identify a pattern exists within the sti-

mulus itself. Closer to the truth, but also incorrect, is the assump-

tion that the necessary information exists xd.thin a particular set of

stimuli. Information processing systems of biological origin are not

mere passive devices for performing logical operations on the informa-

tion in a stimulus. They select and reject information, and, more

important, they contribute information of their oxm to that contained

by the stimulus. The processes that reject and supplement information

pervade the human visual system even to the receptors, and they operate

even on the simplest of stimuli.

The visual system obtains only two kinds of information from the

absorption of a photon: which receptor absorbed it, and approximately

when it was absorbed. Receptors differ in spatial location and in

the location of their absorption spectra within the visible spectrum,

cones having maximum absorption at wavelengths of 435 nm, 535 nm, or

570 nm, and rods at 510 nm ' ' . Then if two receptors

absorb a different number of photons within a particular time interval,

there Is no way of determining whether it is because a different

number of photons have fallen upon the two receptors, because the one

is more sensitive to the wavelengths of the photons that have fallen

upon it than the other, or because of a combination of these two factors

Suppose, for example, that the retina were uniformly illuminated with

light of short wavelength, such that only blue-sensitive cones were

excited. Exactly the same pattern of retinal excitation xrould be caused

60

by a speckled pattern of white light that illuminated only the blue-

sensitive cones. There is no way of distinguishing between the two

patterns of excitation; yet, uniform illumination of the retina with

light of short wavelength is never perceived as speckled white.

In a "sane" world a speckled pattern of white light that illumi-

nates blue-sensitive receptors only is extremely unlikely. This is

the point we want to make: the retina must "know" this. It must be

wired in such a way that this ambiguous stimulus is always perceived

in the more likely way. That is, retinal function must take into

account prior probabilities. In this case there is a very good reason

for this to be so, for it is impossible to pass through the optical

system of the eye a speckled pattern of white light that would illu-

minate only blue-sensitive receptors. In the entire evolution of man,

no organism is likely ever to have experienced such stimulation, nor

to have been required to respond to it accordingly.

In a preceding section we pointed out that the organization of

the retina changes, depending upon the amount of information in the

stimulus. This means that in order to make the most use of the infor-

mation received within any particular time interval, the retina must

be prepared for it before it is received. If the retina is organized

for detection of low-energy stimuli, and an information-rich stimulus

is received instead, it cannot reorganize rapidly enough to make the

best use of the Information contained in the stimulus. Similarly,

a retina organized to process information-rich stimuli is unprepared

for the detection of weak signals. Consider the simplest case of

%

61

viewing a light or dark field. The eye keeps a running estimate of

the amount of information (number of photons) it is likely to receive

within the next time interval. This is done in rather a sophisticated

way; for the intensity of the stimulus for which the eye is optimally

prepared is an exponentially weighted integral of the amount of light

that it has absorbed in the recent past; that is,

E(I) - Ce(t " V^dt ,

where E(I) is the expected retinal illuminance at time, t., ; I is the(79)

retinal illuminance at time, t; and tau is 15 mm for the rod system

(80, 81, 6) ,and 2 mm for the cone system . It is the necessity for

the eye to estimate E(I) that accounts for the phenomenon of dark

adaptation, and it is the equation above that accounts for the basic(82, 57)

form of dark adaptation curves

Thus, the retina introduces prior probabilities into the per-

ception of even the simplest stimuli. At all levels of the visual

system, and at all levels of perceptual complexity, information origi-

nating from within the organism itself contributes to the ultimate

perception of a particular stimulus. One can observe both in single(t w

nerve cells within the visual systenT and in the responses of the

whole organism, a great readiness to see contours, straight lines,

closed figures, and so forth. Many of these propensities have achieved

wide currency as the Gestalt principles of perception. At yet higher

levels of analysis, complicated contextual factors enter into

perception. These are too well known to dwell upon. They can

62

be seen in a vast variety of perceptual phenomena, from the perception

of speech to the change in identity of a figure in different contexts.

An example is shown in Figure I. Many others can be found in any

standard text (e.g., GalanterP°h " A spate of studies published

in the later 40's, under the banner of the "new look in perception,"

established that even the psychological state of an individual can

profoundly affect his perception of many kinds of stimuli. We see

best what we expect to see.

Finally, the visual system contributes information by imposing

order and meaningfulness upon poorly defined stimuli. For example,

a simple pattern of lines, such as the Necker cube shown in Figure 11,

is seldom seen merely as a pattern of lines in two dimensions, but,

rather, as the edges of a cube, in three dimensions. However, the

cube can be seen either as if viewed from above or as if viewed from

below; in fact, the perception of the cube usually oscillates perio-

dically between these two. Perception of one or the other, hoxrever,

constitutes a further reduction of entropy beyond definition of the

stimulus as a cube.

The preceding paragraph illustrates one of the most important ways

in which the human senses reduce the entropy of a stimulus. The pro-

cess of perception is less a search for meaningful information in the

stimulus than a classification of stimuli into meaningful categories.

Humans have built into them, through both genes and experience, a

predetermined set of percepts, each with its own prior probability.

Each new stimulus must be identified with one of them. What set of

■■%

iJ

63

percepts, then, should be supplied a robot intended to explore Mars,

and what should the corresponding prior probabilities be?

This brings us back to the main purpose for this section. We

have discussed some of the properties of human perception, and impli-

cit in this discussion is the hint that machines might possibly perceive

better if some of the principles that seem to govern human perception

were better applied to machines. Yet, applying such principles to

machines presents its own set of problems. How practical is this pro-

gram of "bionics"? What principles are there in biological visual

systems, the only existing systems which perceive, which can account

for their remarkable performance?

Basically, the visual system can be conceived of as a hierarchical

structure representing the information in the retinal image by increas-(43)

ing degrees of abstraction. ' Thus we may designate cells as occupying

level 1 if they register the amount of light falling upon a particular,

small area of the retina. The retinal receptors, therefore, occupy

level i, and for present purposes, optic nerve fibers and even thalamic

cells can be considered to occupy level ±.

That area of the retina which tends to affect the activity of

a particular neuron can be called the receptive field of that neuron.

Then, cells in the next higher level, iJL, behave as though each of them

were connected to a particular subset of cells at level jL, the receptive

fields of which are arranged along a straight line projection upon

the retina^ Thus, only illumination of the retina by a straight line

in the proper location and of the proper orientation can excite any

64

particular cell at level ii. Cells in level iii behave as though each

were connected to a set of cells in level 11 which have receptive fields

consisting of lines of uniform orientation but slightly different

location on the retina. Cells at higher levels are mere difficult

to characterize, some responding, for example, to figures with right-

angle corners in the proper location and orientation. *"' In primates

cells above level 1 exist only in the cortex and possibly th3superior

colliculus.

It is tempting to hypothesize a specially sensitive neuron for

every useful stimulus property. Consider the problem of specifying

necessary and sufficient neural events for recognition of an equilateral

triangle at any place In the visual field . Invariance of response with

respect to size and location could be obtained in a class of level iii

neurons receiving input from all level _ii_ units responsive to lines

at an angle of 0, (0 + pi/3), or (0 - pi/3), where 0 differs for dif-

ferent cells. Then the response of a cell receiving input from the

level iii cells could be independent of orientation of the triangle

as well. The trouble x^ith this approach is that it forces one to postu-

late an unrealistic number of neurons, exceeding the total number of

neurons in the brain! Further, the more specialized the hypothesized

cell, the greater proportion of time it will spend unused while neverthe-

less requiring space and nutrition. Such components do not survive

evolution, nor can they survive the economics of robotry.

It seems likely that there is an optimum level of abstraction,

a level that makes a good unit or basis for whatever further processing

i

65

the system does. Line- sensitive or contour-sensitive units are of ob-

vious utility, for the objects of interest to us are usually defined

by these components. Unless you know something about the environment,

or have a billion years to find out, it is doubtful that one would

guess what the ideal line of abstraction is.

How practical would it be to design a robot on the principles

governing visual function? The visual system depends upon parallel

processing. The peculiar advantage of parallel processing is speed,

a useful property for the nervous system because of. the sluggishness

of its logical elements. But this speed exacts a price in terms of

a great proliferation of connections. If a cell of level li requires

input from no more than 10 cells of level i, a conservative estimate,

each cell in level i^ must make connections with 300 cells or so of

level ii. Such multiplicity of connections poses a problem for a machine

constrained by limitations of size and mass. Further, the speed of

a machine's logical elements calls into question the need for parallel

processing. In viexj of the different kinds of constraints imposed

on brains and on machines, it seems unlikely that the mechanisms that

serve effectively for one will also serve effectively for the other.

At the Fall, 1967 Joint Computer Conference there was an interesting

discussion between designers of "commercial" and "academic" computers.

We have already described one of the commercial devices, which does

useful tasks in a dull way. One of us carried away the feeling that

a similar remark could be made about the robots built in academic and

scientific laboratories; they did trivial tasks in an interesting way.

66

Typically, the scientific "robots" were actually mechanical arms and

cameras serving as output and input devices for a conventional digital

computer. A good deal of progress was reported in performing such

tasks as co-ordinating visual input and mechanical output, so that( 97 )blocks could be stacked anyx^here within the camera's visual field

A more ambitious project involved a mobile carriage with visual and

tactile sensors mounted on it . Interestingly, its visual system

had at least the precursors of the feature detectors we have stressed

as being so important in visual perception. The carriage still had

to be wired to a very non-mobile digital computer. While this is ob-

viously not practical for the ultimate robot, and it is not clear that

miniturization techniques are so advanced that no problem exists,

the carriage plus computer arrangement Is a useful device for construc-

ting a laboratory in which to study computers.

Even in dealing x^ith computer extensions, however, formidable

technological problems arise. Designing adequate mechanical elements

for an arm, or processing commands to start and stop motors on a carriage,

are engineering and softx^are design questions x*hich require no mean

talent for their solution. Our concern is that progress on such

topics, although undoubtedly part of the robot problem, will lead

to the illusion that robots are about to be created. This is not

the case at all. After all technical problems are overcome, we will

still have to understand perception. It is not our contention that

this means work on robot technology should cease. What x^e do maintain

is that heavy investment in such work is not justified unless it can

i

■iI

F

3j;

V

67

be shown that there is parallel progress in our understanding of how

the robot is going to sense its world. If the robot is to see "as

a human sees," then we must find out how a human sees. If the robot

is going to sense its world in some other way, then we would like to

know what that way is going to be. These questions can be put off,

but they cannot be ignored forever.

6.3 Conclusions

If fact, the important question is whether or not machines of the

foreseeable future are at all capable of anything like perception,

as the word is usually used. On the one hand, attempts to get machines

to do even the simplest things, such as recognize letters of the al-

phabet after being given large samples of the type font, have experienced

a rather disappointing degree of success. On the other hand, studies

of human perception indicate that the processes involved are exceedingly

complex, drawing as they do upon vast amounts of information gleaned

from the context and from memory and bringing it to bear on the inter-

pretation of the stimulus in extremely subtle ways. It seems to us

unlikely, moreover, that much of the brain tissue devoted to such tasks

as perception is unneeded, and there are already in the literature

numerous pessimistic comparisons between the logical power per unit

volume of the human brain versus that of machines, and even these

underestimate the brain by orders of magnitude; for the unit of the

brain most nearly comparable to a logical element in a computer is

not the neuron, as is almost always assumed, but the synaptic bouton,

thousands of which can cover a single neuron.

68

There is still an argument for a simple proto-robot lacking

perception. Such a device could serve a useful function in controlling

some experiments in space and hazardous environment exploration,

even though one could think of things it could not do.

Let us close with a surprising, non-computer suggestion. Even

if it is judged that development of a perceiving robot is reasonably

feasible, its mass should be comparad not only with that of a man,

his life-support systems, and return vehicle, which still could be

less than that of the required computer, but should also be compared

with that of a pigeon with its own life-support system and without

a return vehicle. Pigeons have a highly developed visual system; they

have been trained to recognize all sorts of concepts, such as that( 41 )of a human being ; they can retain previously trained habits for

as long as six years without noticeable deficit ; they have been

trained to do human jobs such as pill and bottle inspecting and( 85 )

to guide air-born missiles . Of course we have no brief for the

pigeon. . .other infra-humans would do as well. The point is that bio-

logical devices are complex, sophisticated computers. We should try

to use them.

I

i

1:

FOOTNOTES

The preparation of this paper has been partially supported by

the National Science Foundation, Grant No. NSF 87-1438R, and

partially by the NINDB, Grant No. MH 15564-01

Page 1

Normally a cat's ear will respond electrically to a clicking

sound. There is evidence indicating that the response is reduc?.d

at the ear itself if the cat is watching a mouse

Page 13

Page 15 The hippocampus, a paleocortical structure first found in primi-

tive vertebrates, has been suggested as the site of working

(28)memory.

Recall that the sequence of events in this experiment is display

on, display off, reporting signal on. This is not perceived

as a discrete sequence of events. One of the authors participated

in a demonstration using the experimental apparatus for Sperling's

study. Subjectively, the warning system seemed to appear

shortly after the display went on, but before it went off.

This does not mean that there is no sensory specific buffering

Page 17

Page 18

prior to central buffering. There probably is. We have already

discussed evidence for visual buffering. Neisser (Chapter 8)

has reviewed the evidence for an auditory buffer with a similar

time span.

Page 19 The recognition of visual patterns, a classic example of something

people do well and machines do poorly, can be partly understood

in this light. A great deal of effort has been devoted to

designing automatic pattern recognizers which are capable of(78)

adjusting to any environment (e.g., Rosenblatt ). The point

we wish to make is that vertebrates have evolved in a particular

environment , and are prewired to recognize the patterns which

occur in it.

Footnotes (Cont'd.)

Page 25 This statement must be qualified somewhat, because the branching

search might be effected by the similarity between targets.

Thus in looking for a X, the presence of a horizontal bar on

the bottom (_) rules out the possiblity that an observed letter

matches the target, while if you were looking for a Z or a X,

the observed letter must be further analyzed. On the other

hand, the observation Inverted v at top E) would rule out

either letter. The point is that a coding process which compared

the input, in sequence, to templates of Zs and the Ks, would

require a linear increase in search time with an increase in

the number of target patterns. The model espoused here states

that the increase in search time xtfill be at most linear.

One can go beyond this data, speculating that the reliance on

auditory cues is a peculiarly human trait. Possibly it is due

to the central role of speech in human thought. It is not at

all clear how this suggestion would be tested. Clinical and

anecdotal evidence suggest that auditory (speech?) images are

not all that we store. It has been reported that if certain

regions of the brain are stimulated electrically during brain

surgery, the patient will report seeing again some past scene.

The subjective sensation is evidentally the one that is again

looking at the past, not that it is being reconstructed from

memory. Luria ; has described the case history of a

Page 29

professional mnemonist who was capable of amazing feats of memory

by depending almost entirely upon visual or physical experiences

Page 29

Page 31

Page 34

Page 35

\

Footnotes (Cont'd.)

(Cont'd.)

rather than abstract codings. Interestingly, Luria observes

that in day to day activity, this form of memory was often

not adaptive. Because he remembered everything, rather than

abstracting a coding of experience, the mnemonist found it diffi-

cult to concentrate his thoughts upon the more crucial features

of ' his environment .While we lean toward this theory, at least to the extent of

asserting that tag lists are an important part of the organization

of central short-term memory, the tagging model does not account

for all the data from dichotic listening studies. In particular,

it is difficult to hold a message arriving at one ear for more(15)

than a few seconds while the message at the other ear is reported

The times involved suggest that dichotic listening is at least

partly a peripheral storage phenomena.

A more colloquial, but less accurate, description of the Atkinson,

et al. model is possible. We can think of short-term memory

as a direct recall of a message. A retrieval from long-term

memory could be equated with a subjective feeling of "I'm pretty

sure it's this, although I don't exactly remember the message

that set the variable." This explanation is consistent with

but not implied by the Atkinson, et al. analysis.

Rapid mental calculating ability has sometimes been said to be

associated with low general mental ability. This does not appear

to be so. Gardner lists as an example of fast calculators:

John yon Neumann, a nineteenth century British civil engineer

Footnotes (Cont'd.)Page 35 (Cont'd.)

(perhaps the most spectacular calculator) , a professor of mathe-

matics, and a computer expert.

Page 36 At least one lightening calculator is known to make use of audi-

tory memory. When he makes mistakes, it is because the numbers

sound alike. On the other hand, another famous calculator states

that his memory is neither auditory nor visual.^31^

L

REFERENCES

1. G. B. Arden, Complex receptive fields and responses to moving objects

in cells of the rabbit's lateral geniculate body, J. Physiol. 166,

468-488 (1963).

2. R. Atkinson, J. Brelsford, and

for memory with application to

4, 277-300 (1967).

R. Shiffrin, Multiprocess models

a continuous presentation task, J.M.P.

3. R. Atkinson, and M. Schiffrin,

and its control processes, in

Human memory: A proposed system

'The Psychology of Learning and

Motivation" (K. Spence and J. Spence, cd.) Academic Press, New York

(1968).

4. F. Attneave, "Applications of Information Theory to Psychology:

A Summary of Basic Concepts, Methods, and Results," Holt, New

York (1959).

5. E. Averbach and A. CorielL, Short term memory in vision, Bell System

Technical Journal 40, 309-328 (1961).

6. H. B. Baiter and W. A. H. Rushton, The red-sensitive pigment in

normal cones, J. Physiol. 176, 56-72 (1965)

7. H. B. Barlow and J. M. B. Sparrock, The role of after-images in

dark adaptation, Science 144, 1309-1314 (1964).

8. L. Beach, E. Hunt, and W. Carter, Interactions betxreen memory

and decision making—in preparation.

9. M. A. Bouman, in "Sensory Communication" (W. A. Rosenblith, cd.)

pp;. 377-401, Wiley, New York (1961) .10. M. A. Bouman, J. J. Vos, and P. L. Walraven, Quantum theory of

luminance and chromaticity discrimination, JL_ Opt. Soc. Am., 53_,

121-128 (1963).

References cont'd.

11. L. Bourne, "Human Conceptual Behavior," Allyn and Bacon, Boston (1966)

12. G. S. Brindley, The discrimination of after-images, J. Physiol.

147, 194-203 (1959).

13. G. van den Brink, and M. A. Bouman, Quantum coincidence requirements

during dark-adaptation, Vision Res. 3, 479-481 (1963).

14. D. Broadbent, The role of auditory localization In attention and

memory span, J^ Exp. Psychol . 47, 191-196 (1959) .15. D. Broadbent, Immediate memory and simultaneous stimuli, Quart. J

Psychol. 9, 1-11 (1957).

16. D. Broadbent, "Perception and Communication", Pergamon Press, London

(1958).

17. D. Broadbent, Flow of information within the organism, J. Verbal

Learning and Verbal Behavior 2, 34-39 (1963) .18. P. K. Brown and G. Wald, Visual pigments in single rods and cones

of the human retina, Science 144, 45-52 (1964).

20. J. Bruner, Neural mechanisms in perception, Psychol. Rev. 64, 340-

-358 (1957)b.

21. J. S. Bruner, J. J. Goodnow and G. A. Austin, "A Study of Thinking,"

Wiley, New York (1956).

22. T. H. Bullock, Simple systems for the study of learning mechanisms,

Neurosciences Research Program Bulletin, Vol. 4, No . 2 (1966) .23. C. Cherry, "On Human Communication: A Review, a Survey, and a

Criticism," 2nd cd., MIT Press, Cambridge, Mass. (1966).

24. R. A. Cone, Quantum relations of the rat electroretinogram, J.

ggn. Physiol. 46. 1267-1286 (1963).

19. J. Bruner, On perceptual readiness, Psychol. Rev. 64, 123-152 (1957)a.

4

References Cont'd.

25. R. A. Cone, The early receptor potential of the vertebrate eye,

Cold Spring Harbor Symposium on Quantitative Biology, 30, 483-491

(1965).

26. R. Conrad, Acoustic confusions in immediate memory, Brit. J.

Psychol. 55, 75-83 (1964).

27. J. Crawford, E. Hunt, and G. Peak, Inverse forgetting in short

term memory, J. Expt'l.., Psychol. 72, 415-422 (1966).

28. R. J. Douglas, The hippocampus and behavior, Psychol. Bull. 17,

416-442 (1967).

29. P. Fitts, J. R. Peterson, and G. Wolpe, Cognitive aspects of

information processing: 11. Adjustments to stimulus redundancy,

J. Exp. Psychol., 65, 423-432 (1963).

30. E. Galanter, "Textbook of Elementary Psychology," Holden-Day,

San Francisco (1966) .32. W. Garner, "Uncertainty and Structure as Psychological Concepts,

Wiley, New York (1962).

33. D. Glaser, J. McCarthy, and M. Minsky, The automated biological

laboratory, in "The Biology and Exploration of Mars," (D. Pittendrigh,

W. Vishniac, and J. Pearman, cd.), National Academy of Sciences,

Washington (1966).

34. J. D. Gould, Visual factors In the design of computer-controlled

CRT displays, IBM Research Report RC 2016, IBM Watson Research

Center, Yorktown Heights, New Y0rk, (1968).

35. J. D. Gould, and W. L. Makous, Vision and lasers: Human factors of

laser displays, Information Display, in press.

31. M. Gardner, Mathematical games, Sci. American 216, 116-123 (1967)

References Cont'd.

36. L. Gregg, H. Simon, Process models and stochastic theories of simple

37. W. A. Hagins, Electrical signs of information flow in photoreceptors,

38. S. Hecht, S. Schlaer, and M. H. Pirenne, Energy, quanta, and vision,

39. C. Hempl, Fundamentals of concept formation in the empirical

sciences, Vol. 11, No. 7 of the International Encyclopedia ofUnified Science, Univ. of Chicago Press, 1952.

40. R. Hernandesr-Peon, H. Scherrer and M. Jouvet, Modification of

electrical activity in cochlear nucleus during "attention" in

unanaesthetized cats, Science 123, 331-332 (1956) .41. R. J. Herrnstein, and D. Loveland, Complex visual concept in the

pigeon, Science 146, 549-551 (1964).

42. W. E. Hick, On the rate of gain of information, Suart^ J^ Ex£.Psychol. 4. 11-26 (1952)

43. D. H. Hubel, Integrative processes in central visual pathways

44. D. H. Hubel and T. N. Wiesel, Receptive fields and functionalarchitecture in two nonstriate visual areas (18 and 19) of thecat, J. Neurophysiol. 28. 229-289 (1965).

45. D. H. Hubel and T. N. Wiesel, Receptive fields and functionalarchitecture, J^ Physiol. 195, 215-243 (1968)

46. E. Hunt, "Concept Learning: An Information Processing Problem,Wiley, New York (1962).

concept formation, J. Math. Psychol. 4. 246-276 (1967).

Cold Spring Harbor Symp. Quant. Biol. 30, 403-418 (1965)

It. Gen - Physiol. 25, 819-840 (1942) .

of the cat, J. Opt. Soc. Am., 53, 58-66 (1963)

References Cont'd.

47. E. B. Hunt, Simulation and analytic models of memory, J. Verbal

Learning and Verbal Behavior 2, 49-59 (1963).

48. E. Hunt, J. Marin, and P. Stone, "Experiments in Induction,

Academic Press, New York (1966).

49. R. Hyman, Stimulus information as a determinant of reaction time,

L. Exp. Psychol. 45 , 188-196 (1953) .50. H. Jakobson, The informational capacity of the human eye, Science

113, 292-293 (1951).

51. E. R. John, "Mechanisms of Memory," Academic Press, New York (1967)

52. E. R. Kandel and W. A. Spencer, Cellular neurophysiological

approaches in the study of learning, Physiol. Rev., 40, 65-134 (1968).

53. S. Kuno, The predictive analyzer and a path elimination technique,

Comm. A.CM. 8, 453-462 (1965)

54. J. Y. Lettvin, H. R. Maturana, W. S. McCulloch and W. H. Pitts,

What the frog's eye tells the frog's brain, Proc. Inst. Radio Engr. 47

1940-1951 (1959).

55. J. C. R. Licklider, "Libraries of the future," MIT Press, Cambridge,

Mass. (1965).

56. A. Luria, "The Mind of a Mnemonist," Basic Books, New York (1968).

57. W. L. Makous, L. D. Broxm, Systematic Changes in Time Course of .Dark adap

tation, Paper presented at the Psychonomic Soc. Meeting, Chicago (1968).

58. W. B. Marks, W. H. Dobelle and E. F. MacNichol, Jr., Visual pigments

of single primate cones, Science 143, 1181-1183 (1964).

59. H. R. Maturana, J. Y. Lettvin, W. H. Pitts and W. S. McCulloch,

Physiology and anatomy of vision in the frog, JL_ Gen. Physiol. 43

Supple., 129-175 (1960).

References Cont'd.

60. A. W. Melton.A. W. Melton, Implications of short term memory for a generaltheory of memory, J^ Verbal Learning and Verbal Behav. 2, 1-21 (1963)G. A. Miller, The magical number seven, plus or minus two: Somelimits on our capacity for processing information, Psychol. Rev.

61

63, 81-97 (1956).

62 B. Murdock, The retention of individual items, A Ex^ Ps^chol^ 62 ,618-625 (1961).

63 V. Neisser, Experiments in visual search and their theoreticalimplications. Paper presented to Psychonomic Society, 1964.V. Neisser, "Cognitive Psychology," Appleton-Century-Crof us,New York (1967).

64

65. G. Osterberg, Topography of tha layer of rods and cones in the human

C W. Oyster, and H. B. Barlow, Direction-selective units in rabbitretina: Distribution of preferred directions, Science 155, 842-843(1967).

67. W. Penfield and L. Roberts, "Speech and Brain Mechanisms," PrincetonU. Press, Princeton, (1954)

68. M. H. Pirenne and F. H. C. Marriott, in "Psychology: The Study ofa Science" (S. Koch, cd.) Vol. 1, pp. 288.361 , McGraw-Hill, New York(1959).

69. M. Posner, and R. Mitchell, Chronometric analysis of classification,Psychol . R eVjL U. 392--409 (1967).

70. L. Postman and L. Phillips, Short term temporal changes in free

retina, Acta Ophthalmoiogica, Suppl. 6. Copenhagen (1935)66. C. W. Oyster. anA ti n d^^i— nJ ..

recall, Ouart^ J^ Ex^ Ps^chol^ 17, 132-138 (1965).

81. W. A. H. Rushton, Cone pigment kinetics in the deuteranope, J^Physiol. 176, 38-45 (1965a).

1

References Cont'd.

71. H. Quastler, and V. J. Wulff , Human performance in information

transmission. Part one: Simple sequential routinized tasks.

Manuscript cited in Attneave, 1959.

72. M. R. Quillian, Semantic Memory, Technical Report AFCRL-66-189 ,Bolt, Beranek and Newman, Inc., Cambridge, Mass. (1966).

73. M. R. Quillian, Word concepts: A theory and simulation of some

basic semantic capabilities, Behavioral Sci. 12, 410-430 (1967).

74. S. W. Ranson, and S. L. Clark, "The Anatomy of the Nervous System,"

Saunders, Philadelphia (1953).

75. W. Reitman, Information processing models, computer simulation,

and the psychology of thinking, Mental Health Research Institute

Preprint //196, U. of Michigan, Ann Arbor, Mich. (1966).

76. F. Restle and D. Emmerich, Memory in concept attainment effect

of giving several problems concurrently, J_;_ Expt ' l . Psychol . 71,

794-799 (19C6).

77. C. Rosen and N. Nilsson, Application of Intelligent Automata to

Reconnaissance. Stanford Research Institute Technical Report,

Project SRI 5953 (1967).

78. F. Rosenblatt, "Principles of Neurodynamics," Spartan Books,

Washington (1962).

79. W. A. H. Rushton, Rhodopsin measurement and dark-adaptation in

a subject deficient in cone vision, J. Physiol. 156, 193-205 (1961)

80. W. A. H. Rushton, Cone pigment kinetics in the protonope, J^Physiol. 168, 374-388 (1963).

>i

References Cont'd.

82. W. A. H. Rushton, The Ferrier lecture: Visual adaptation, Proc.

R. Soc. (Lond.) 8162, 20-46 (l*6sb).

82. W. A. H. Rushton and G. H. Henry, Bleaching and regeneration of

cone pigments in man, Vision Res. 8, 617-631 (1968).

84. H. Simon and E. Feigenbaum, An information processing theory of

some effects of similarity, familiarization, and meaningfulness

in verbal learning, J^ Verbal Learning and Verbal Behavior 3,

385-396 (1964).

85. B. F. Skinner, Pigeons in a pelican, American Psychologist 15.28-37 (1960).

86. B. F. Skinner, "Science and Human Behavior," MacMillan, New York,

p. 71 (1953).

87. G. Sperling, The information available in brief visual presentations,

Psycho!. Mon-^a. 74. #11 (I960).

38. R. W. Sperry, in "The Neurosciences" (G. C. Quarton, T. Melnechuk,

and F. 0. Schmitt, cd.) pp. 714-722, Rockefeller Univ. Press,

New York (19 X).

89. W. H. Suiiiby fad I. Pollack, Short-time Processing of Information,

HFORL Report, TR-54-6, Ja?-.uary 1954.

90. T. Toiaita, A. Kaneko, M. Murakami and E. L. Pautler, Spectral

rrispor.se curves of single cones in the carp, Vision Res. 7_, 519-531(1967).

91. T. Trabasso and G. Bower, "Attention in Learning: Theory and

Research," Wiley, New York (1968) .92. L. Uhr, C. Vossler and J. Uleman, Pattern recognition over distortion

by human subjects and by a computer simulation model for visual

pattern recognition, J. Exp. Psychol. 63. 227-234 (1962).

References Cont'd.

93. T. Verhave, The pigeon as a quality-control inspector, American

Psychologist, 21, IC9-115 (1966).

94. J. Yon Neuman, "The Computer and the Brain," Yale U. Press, New

Haven, (1958).

95. G. L. Walls, 'The vertebrate eye and its adaptive r-diations," 2nd

cd., Hafner, New York (1963).

96. P. L. Walraven and M. A. Bouman, Fluctuation theory of colour

discrimination of normal trichromats, Vision Res. , 6, 567-586 (1966)

97. W. Wichman, Use of Optical Feedback in the Computer Control of an

Arm. Stanford Artificial Intelligence Project, Memo A. 1.56, (1967).

98. D. B. Yntema, Keeping track of several things at once, Human

Factors 5, 7-17 (1963).

99. D. Yntema and G. Meuser, Remembering the present state of a number

of variables, vL Exp. Psychol. 60 18-22 (1960) .100. D. Yntema, and G. Mouser, Keeping track of variables that have few

or many states, Jj_ pxvj Psychol . 63 , 391-393 (1962).

101. D. Yntema and F. Trask, Recall as a search process, J. Verbal Lrng.

and Verbal Behav. 2, 65-74 (1963).

IN BITS

1

I

■i

TABLE I

TASK INFORMATION TRANSMITTED1. Identification of pitch of a tone

(Audition)

2.5

2. Identification of loudness of a

tone (Audition)2.3

3. Identifying degree of saline

concentration (Taste)

1.9

4. Identifying position of a point

on a line (Vision)

3.25

HUMAN ABILITY TO IDENTIFY UNI-DIMENSIONAL STIMULI ( 61 *

,

i

i

i

!:

i

SENSORY CHANNEL

Taste

Vision (1)

Vision (2)

Audition

TABLE II

TASK

Judgement of saline concentration

Judgement of sucrose concentration

Saline + sucrose

Judging position of dot on line

Judging position of dot on square

Reading position of pointer on

one 360° dial

Reading positions of pointers on

two dials presented simultaneously

Loudness judgement

Pitch judgement

Pitch and loudness combined

INFORMATION

TRANSMISSION

1.70

1.69

2.25

3.2

4.4

4.2

6.3

1.8

1.7

3.1

INFORMATION TRANSMITTED BY MULTI-DIMENSIONAL STIMULI(32)(Based on summaries by Garner )

TABLE 111

BASIS FOR IDENTITY TIME REQUIRED TO DETjfr-j^jgmmTY

Physical identity

Name identity

Rule identity

TIME REQUIRED TO MAKE RULE IDENTITY JUDGEMENTS UNDER

VARIOUS CONDITIONS

(From Posner and Mitchell

i

TABLE IV

ORDER LEFT EAR RIGHT EAR

1 0 good

2 room 2

3 3 cool

EXAMPLE OF LIST OF TYPED ITEMS PRESENTED IN

YNTEMA AND TRASK'S DICHOTIC LISTENING STUDY

i

i

1

t

All

TABLE V

Attribute Recommended Value

Luminance

Wavelength

25-50 mL

Mid-spectrum

15-30Contrast ratio (max

Regeneration rate

Resolution

over mm lum)

60

50 points/inch, viewed at 24 inches

Character features

Height 12-15 mm. of visual angle

10 points, minimum

Width 75% of height

Maximum entropy

LEROY font best

Shape (font)

Interline spacing

Miscellaneous

30-50% of character height

Minimal density of information

Even illumination.

DESIRABLE CHARACTERISTICS OF VISUAL BISPLAYS(34 ' 35)

a sketching

d leave t

of the cc.icep

TABLE VI

Color

Trichromatic 4

Black-on-white (i ether than white-on-black) 7

8 graduations of brightness (going high enough)

Resolution (lines/inch)

5

400 4

200 6

9100

9Selective erasability, by computer and operator,

of each element

Controllable persistence

Lack of flicker

6

9

Hard copy of any frame available 9

Direct, simple retrieval and redisplay of hard copy 7

"Sketchpad" features 10

8Stylus with size, shape, weight, and feel of pencil

Reliability 9

8Ruggedness

10Economic feasibility

DESIRABLE FEATURES OF AN INTERACTIVE

CONSOLE ( '"Sketchpad" features assign to the computer tl.ose parts of the

and drawing skill that involve much practice and precision, a:

man responsible mainly for expressing the essential structure

he desires to represent.

i

TABLE VII

Position of keyboard of computer typewriter

Keyboard too high.

Raising chair to compensate for high keyboat! make:, knees hit table

Moving typewriter to typing table puts it too far from other controls

Angle of screen (nearly vertical)

Tiring to use light pen.

One tends to write too large.

Light pen

Too thick and heavy.

Parallax.

Printers

No lower case letters.

Too slow; should be at least 100 characters/secImcge quality

Flicker is annoying, tiring, and sometimes even sickening.

Insufficient contrast.

Inadequate character legibility.

Reflections from scope face a problem.

Hoods used to minimize reflections make access to screen difficult,

Turning lights down to reduce reflections makes print and type

script unreadable.

SHORTCOMINGS GF EXISTING INTERACTIVE CONSOLES

FIGURE CAPTIONS

Figure 1. Necker cube. Perception oscillates between that of a three

dimensional cube viewed from above, to one viexred from below.

Figure 2. Example of the effect of context upon perception. Although

the lower right figure in both parts of the figure is identical, in the

one case it is perceived as a bird, and in the other, as an antelope.

FIGURE 1

t

j