C . a i : A COMPUTING ENVIRONMENT FOR A1 RESEARCH

Overview, PMS and Operat ing System Cons idera t ions

b Y

G. B e l l (Co-chairman)

P . Freeman (Co-Chairman)

and

M. Barbacci S. Bhat ia W . B r oadley R. Chen L. Erman H. Goldberg W. Huen M. Knudsen D. McCracken A. Newel1 R. Reddy S. Rege G. Robertson

Department of Computer Science Carnegie-Mellon Un ive r s i t y

P i t t s b u r g h , Pa. 15213

This work was supported by t h e Advanced Research P r o j e c t s Agency of t he Of f i ce of the Sec re t a ry of Defense (F44620-70-C-0107) and i s monitored by the A i r Force Of f i ce of S c i e n t i f i c Research. This document has been approved f o r pub l i c r e l e a s e and s a l e ; i t s d i s t r i bu t ion i s un l imi ted .

TABLE OF CONTENTS

Page

Abstract ....................................................... i ............................................... Acknowledgments ii

......................................... Overview of the System 1

............................... Requirements for A1 Computing 1

................................ Response to the Requirements 4

..................................... The PMS Structure of C.ai 10

....................................... Computer Performance 14

............................... Interprocessor Communication 14

............................................. Primary Memory 16

........................................... Secondary Memory 22

............................................ Tertiary Memory 23

.................................. Operating System Computer 24

.................................................... Console 25

.................... AMOS: A Minimal Operating System for C.ai 26

Design Goals and Guidelines ................................ 26 ................................... Functions to be Provided 27

................................................ World Views 2 8

Structures Providing the Required Functions of AMOS ........ 29 Performance Monitoring ..................................... 34

Realization Considerations .................................... 36 Scheduling and Strategy for the Project .................... 36

................................. Contingencies and Research 37

................................. New Processor Alternatives 39

Necessity to Modify the Stanford A1 Processor .............. 40 ......................................... Economies of Scale 41

Conclusions ................................................... 43 Appendix 1: Design Constraints and Some Consequences ......... 44 Appendix 2: Outline of P.L* Processor Design ................. 47 Appendix 3: Outline of P.LISP Processor Design ............... 49

ABSTRACT

A computer for artificial intelligence research is examined. The

design is based on a large, straightforward primary memory facility

(about 8 million 74 bit words). Access to the memory is via at least

16 ports which are hardware protected; there is dynamic assignment of

the memory to the ports. The maximum port bandwidth is 8,600 million

bits/sec. Processors for languages (e.g., LISP) and specialized terminals

(e.g., video input/output) can be reliably connected to the system during

its operation. The approach is evolutionary in that high performance

processors, such as the Stanford A1 Processor, can be connected to the

memory structure, giving an overall power of at least 100 times a PDP-10

(and 200 to 300 times a PDP-10 for list processing languages) for 10

processors -- although 20 processors can be attached. Using this approach

we might expect 40 - 80 million PDP-10 operations/second. At the same time, special language processors (P.4) can be designed

and attached. These processors give even larger power increases, but

for restricted language use. Two processors, P.LISP and P.LJ: were

examined for the LISP and L* languages and are reported on separately.

A plan for building the machine in increments over the next three

to five years is examined. Specific schedules are proposed.

Concurrent with the operation of the machine, there should be re-

search into the design of hardware, software and theory of constructing

large ecale computing facilities with maximum modularity.

ACKNOWLEDGMENTS

The b a s i c PMS s t r u c t u r e of C.ai and t h a t of t h e L* and LISP language

processors was developed i n t h e Carnegie-Mellon Univers i ty Computer Science

Department, i n a p r o j e c t seminar on l i s t process ing machines. This r e p o r t

i s based on t h e working papers developed i n t h e CVAM seminar.

The group was in formal ly organized and met f r equen t ly a s a whole,

bu t t he main work f o r t h e r e p o r t s was c a r r i e d ou t i n t h e fol lowing sub-

groups :

CVAM:PMS - W. Broadley (Chairman), S. Bhat ia , and S. Rege

S p e c i f i c a t i o n of PMS, c i r c u i t r y , technology, and o v e r a l l performance.

CVAM:OS - P. Freeman (Chairman), S. Bhat ia , L. Erman, W. Huen, D. McCracken, R. Reddy and G. Robertson

S p e c i f i c a t i o n of an ope ra t i ng system.

CVAM:P.L* - D. McCracken and G. Robertson (chairmen), R. Chen

Design o f a hardwired processor f o r an L* kerne l .

CVAM:P.LISP - M. Barbacci (Chairman), H. Goldberg, and M. Knudsen

Design of a hardwired processor f o r LISP.

Many members of t h e Carnegie-Mellon Computer Science Department served

a s c r i t i c s and c o n s u l t a n t s on t h e des ign , e s p e c i a l l y A. Kendziora,

J. McCredie, A. Newell, R. Reddy and W. Wulf. Alan Kay of S tanford a l s o

i n t e r a c t e d wi th t h e group f r equen t ly and i n d e t a i l .

A d r a f t of t h i s document was presented a t a working meeting he ld a t

Bol t , Beranek and Newman i n Boston, A p r i l 26-27, 1971. Representa t ives

from the Advanced Research Projects Agency, Bolt, Beranek and Newman,

Carnegie-Mellon University, Massachusetts Institute of Technology,

Stanford Research Institute, Stanford University, and System Development

Corporation were present. Their comments and suggestions were of value

in preparing this final version.

Gordon Bell and Peter Freeman integrated the working papers into

this document and did the final editing.

iii

OVERVIEW OF THE SYSTEN

Consideration of the problem of designing and building an optimal

computer for ai research quickly leads one to the realization that there

may not be a feasible solution. The numerous constraints, wide variations

in computing style, and the impossibility of defining the ai problem nar-

rowly seem to make this a certainty. Thus, the major premise of the

design we are about to present is that if one wishes to provide ai re-

searchers with better computing tools, one must, in fact, provide an

environment in which many varied tools may be developed and used. Our

design should be viewed as a specification of such an environment.

REQUIREMENTS FOR A 1 COMPUTING

* In Bell and Newell's Computer Structures (McGraw-Hill, 1971) a

number of special function computers ranging from business to scientific

are described. The characteristics of machines used for ai research

appear to span this spectrum, exhibiting the max of each characteristic

attribute.

Memory Size

The primary memory is larger in an ai environment than with almost

all scientific computers because the local program data base is typically

larger than for scientific applications. Here we assume that the average

program size in this environment is 250,000 74-bit words (64 bits of

Jc The J?MS notation presented therein and used throughout this report is based on seven primitive component types: P-processor, M-memory, S-switch, L-link, K-controller, T-transducer, D-data operation. A computer composed of primitive components is represented by C; hence, C.ai for "ai computer;"

seems t o be adequate because dec i s ions can be bound i n sof tware and l a t e r

changed.

There i s almost uniform agreement ( a t t h e meetirgs on t h e ARPA l i s t

p rocess ing machine) t h a t a l a r g e (and probably l i n e a r ) address ing space

i s e s s e n t i a l . The o t h e r n o t i c e a b l e p o i n t of agreement i s t h a t t h e PDP-10

i n s t r u c t i o n s e t wi th on ly a few mod i f i ca t i ons i s a l l t h a t i s needed f o r

t he immediate f u t u r e .

There i s c l e a r l y a need f o r general-purpose processors wi th c l e a n

o rde r codes and s p e c i a l i z e d i n s t r u c t i o n s f o r c h a r a c t e r i s t i c a i p rocess ing

opera t ions . But, i n a d d i t i o n , t he magnitude and c l o s e l y def ined na tu re of

many a i t a s k s makes i t very d e s i r a b l e t o have h igh ly s p e c i a l i z e d processors

a s wel l .

The PDP-10 A s The Current a i Computer

Because t h e DEC PDP-10 i s t h e c u r r e n t most popular machine f o r ARPA-

supported a i r e sea rch , i t i s worth b r i e f l y examining t h e reasons f o r i t s

popu la r i t y .

1. P r i c e : It i s t h e on ly computer t h a t t h e group can a f fo rd . (A 360 Model 44 i s a l s o a candida te i n t h i s range t h a t has been overlooked and i t i s worth ask ing why.)

2. ISP: The i n s t r u c t i o n set i s very s t r a igh t fo rward , providing power b u t w i th none of t h e anomonalies i n address ing , i n s t r u c - t i o n s e t s i z e , d a t a types , etc. t h a t accompany most machines (e.g., 360, 1108 o r Sigma 7 family) .

PMS S t r u c t u r e : The machine has been e a s i l y approachable by a l l t o i n t e r connec t any k ind of dev i ce from fo re ign memory t o TV camera. Indeed, t h e PMS s t r u c t u r e , now e i g h t yea r s o l d , i s on ly being s l i g h t l y r ev i sed t o handle l a r g e r and f a s t e r memories. This e a s e of i n t e r f a c i n g was i n i t i a l l y used i n t e r - n a l l y by DEC t o admin i s t e r va r ious memory and p e r i p h e r a l des igns ; l a t e r , t h i s p o l i c y boomeranged by al lowing anyone t o connect any k ind of memory t o a PDP-10. I n c o n t r a s t , t h e IBM processor - memory i n t e r f a c e s a r e u s u a l l y so w e l l guarded and obscure t h a t

1. MJ - A s imple primary memory s t r u c t u r e w i t h 16 m u l t i p l e

p o r t s f o r connect ing a v a r i e t y of h igh speed processors

of t h e S tanford A1 processor des ign , s p e c i a l language

processors , secondary memories and s p e c i a l i z e d i / o

t ransducers .

a. A primary memory s i z e of 620 megabits wi th i nd iv idua l p o r t rates of 74, 148, 222 o r 296 b i t ~ / ~ o r t memory acces s (550 n s ) ; 135, 270, 405, o r 540 megabits/sec per p o r t ;

b. A t o t a l information r a t e t o a l l p o r t s of 2.1 t o 8.6 g igab i t s / s ec ;

c. Memory p o r t wid ths of 72+2 p a r i t y o r 64+10 s i n g l e e r r o r c o r r e c t i o n and double e r r o r d e t e c t i o n b i t s ;

d. The 16 p o r t s can be f u r t h e r demult iplexed t o provide more p o r t s ;

e. Each p o r t has a memory p o r t c o n t r o l f o r dynamically r ea s s ign ing 64k word blocks t o each processor and p r o t e c t i n g t h e memory from neighboring processors , The p o r t c o n t r o l a l s o i nc ludes s t a t i s t i c s d a t a ga the r ing and e r r o r con t ro l .

f . A mode of ope ra t i on which provides nea r ly 100% uptime .

2. MA - Secondary memory bandwidth t o a l low swapping. The loading

t i m e f o r a s i n g l e 100,000 word program would be roughly -14

sec. The wors t c a se swap t i m e would b e .3 seconds. Thus,

assuming f i v e drums, 15 programs could be swapped pe r second.

3. P - Mul t ip l e approaches f o r providing process ing power. These

range from a convent ional PDP-10 t o s p e c i a l i z e d hardwired l i s t

processors . This permi ts development of h igh ly f u n c t i o n a l l y

s p e c i a l i z e d processors .

t imes a PDP-10 f o r m u l t i p l e processors . The c o s t of t he se

processors appears t o be q u i t e low ($50,000-$100,000 each).

They a r e descr ibed i n two r e p o r t s of t h e CMU Computer Science

Department: I1C.ai (P.L*) -- An L* Processor f o r C.a4I1

D. McCracken and G. Robertson, and llC,ai (P.LISP) -- A LISP

Processor f o r C.ai:' M. Barbacci , H. Goldberg, and M. Knudsen.

The a b s t r a c t s of t he se r e p o r t s appear a s Appendices 2 and 3

of t h i e r epo r t .

3e. Another a l t e r n a t i v e would be t o c o n s t r u c t a c e n t r a l p rocessor

t o g i v e an i n s t r u c t i o n encoding improvement over t h e PDP-10.

For example, a processor based on t h e s t a c k and i n s t r u c t i o n

format of t h e PDP-11 might be d e s i r a b l e . This approach may

a l low high performance processors t o be b u i l t more e a s i l y .

4. Modularity i n PMS s t r u c t u r e . Here w e hope t o do s i g n i f i c a n t l y

b e t t e r than t h e PDP-10 by providing b e t t e r p r o t e c t i o n among

t h e processors a t t he memory po r t s . We b e l i e v e t h a t i t w i l l

never be necessary t o t u r n o f f computer power f o r any reason

(except f o r cool ing equipment f a i l u r e ) .

5. Uniform in t e rp roces so r comnunication. A second type of i n t e r -

f ace has been added which has a p ro toco l l i k e t h e PDP-11 and

a l lows intercommunication among t h e va r ious processors . Like

t h e PDP-10 i / o and memory busses , t h i s type bus should b e a b l e

t o be used over a s i g n i f i c a n t per iod of time. It a l lows t h e

t ransmiss ion of d a t a a t high r a t e s . Unlike most o t h e r busses ,

a. A large inventory of modules that would facilitate design of special experiments. The module types would include: caches, general purpose microprogram controls, arithmetic units, buffers (queues), port switches (to increase the number of memory ports), mapping, interfaces to.other computers, component exercisors, etc.

b.- Programmers and engineers to carry out many of the designs and assist in system integration of the devices.

The design as proposed would operate at a single central

site to give large memory and to decrease the operating cost.

However, the design and the construction strategy are such that

at ~ l m ~ s t anytime in the project the machine could be partitioned

physically for multiple site operation. The most practical

size for economy and reliability would no doubt require at least

two processors and 1 to 2 million words of primary memory,

9 . The system could be operational within one year at a central

site. At this time although any amount of primary memory could

be available, only two PDP-10 Tenex processors need be avail-

able. Over the next few years, more processors and memories

could be added.

10. Research should go along with making such a modular system

laboratory. The general direction would be to explore hard-

ware, software, and theory that made the interconnection of

modules of the above type easily interconnected. In this way,

a system of any type could be constructed easily.

16 Public Memory modules containing l a b o u t 8 x lo6, 74-bit words, MOS

Fa- cJCentral cross-point for Mp -P interconnection

16 memory ports each

Figure 1. PMS diagram (simplified-) of C.ai

L s 9L /providing up to 296 bits1550 ns

& multiplexed ports language processors + P. J?,

7 \pecialized i/o operating system secondary computers (C.amos and memory C. amos . spare) Jcentral trunk-switch

/ v I /

# 1/ 'I 1

for P intercommunication r

Links to ARPA network, tertiary memory, console, etc.

-13 - Publ ic primary memory1

' ~ u b l i c , primary memory; 223 - 224 words; 74, 148, 222, o r 296 b/w; 550 ns/w; MOS " c e n t r a l ; c ro s s -po in t swi tch ; M ~ - P C ~ M S d i a logues ; 16x16; ' j ~ e n t r a l , t runk swi tch ; i n t e r P d i a logues ; r 2 t runks i d e n t i c a l p r o t o c o l s " c e n t r a l , t runk swi tch ; P-K(Mp.port) d i a logues ; 2 2 t runks bK(Mp.port) Reloca t ion , p r o t e c t i o n , e r r o r c o r r e c t i o n , ' ~ ( ~ ~ e r a t i n ~ - ~ ~ s t e m ) - with l o c a l pr imary and secondary memory 7 ~ s ( s e c o n d a r y memory; drum; 1 .4*gb i t s )

1 ' ~ ( s ~ e c i a l i / o i n t e r f a c e s ; e .g. , TV) Y

t _L[1 - - . - d i r ec t memory p o r t ( d a t a , add re s se s )

@ or H7j---. c o n t r o l p o r t

4 rn.... - - p r o t e c t e d (u se r ) memory p o r t ( d a t a , add re s se s )

'0 E.G . , ,I ,Ck(minicomputer; ~ ~ ( 1 0 0 kwords); scope)

1

l l ' lYansducers and t e r t i a r y memory - managed by a Ck (e.g., t e r a b i t memory)

F igure 2 . PMS diagram o f C.ai

Table 1. comparison of C . a i wi th Other Computers

Mp .width Mp. s i z e Mp.i-rate ! Mp.i-rate ' Ms.i-rate (b/w) (mwords/sec)( (mbits/sec) I (mbits)

S tanford AI-10

Model 91

CDC 6600

C.ai

C. a i / 4

CDC STAR

ILLIAC IV'

1 S p e c i f i c a t i o n s taken from e a r l y ILLIAC I V paper by Barnes, e t a l . -

"L. Roberts - Data Processing Technology Forecas t , A p r i l 23, 1963. 3 ~ s s u m e s $ 8 ~ f o r memory,$bfor p e r i p h e r a l s and $ 3 0 0 ~ per processor . Adjust ing the memory s i z e t o t h a t of STAR, y i e l d s $7m

( t o t a l ) ; 1440 - 4320; 205 620 t o $ 1 b ; 2880 - 8640; 288 - 864.

T a b l e 2. C.ai Wmory C h a r a c t e r i s t i c s

Device

pho tomemory

moving -head d i s k (e .g . , Calcomp)

drum (e.g. , GI)

s h i f t r e g - i s t e r (AYS)

c o r e (Ampex)

Lockheed c o r e

410s A!! S

XOS (AMS)

a i p o l a r (Cogar)

S i p o l a r ROX

0 .,-I L) 0 G 5 w

t

t

s

s

l i , 0 4 9 i nnn

P

p

i

i

,."UU

' ~ s t i m a t e - Cogar h a s quo ted such a system a t $ 1 3 , 1 0 0 ~ f o r d e l i v e r y i n 1973. " ~ h e s e assume c u r r e n t d e n s i t i e s . We can s a f e l y assume double d e n s i t y , hence lower c o s t , more s t o r a g e , and h i g h e r t r a n s f e r r a t e s .

" t / t e r t i a r y ; s /secondary; p /pr imary; and i l i n t e r n a l p r o c e s s o r (program c o n t r o l , a c c u m u l a t o r s , e t c . )

1 1

i I I / :. 1x10 6 I i

i a I

t o t a l ~ o w e r ' f l o o r s p a c e s i z e module i-widrh ( b r t s ) I i . r a t e ( w a t t s ) f t 2 106b

I c o s t s (k$)

! (mb/s) i 1 I I

t . a c c e s s ( t . c y c l e )

0-5 5 +

20 ms

.7 1.1s (1.8 p s )

275 n s e c (650 n s )

500 n s e c (1.2 p s e c )

2 5 0 n s (400 n s )

4 0 n s

i.rate mb/s

3.3'

296 1 163 1 200 I *55 1 x16 2605 I 1 6 2 0 10,240 1 1.65 I 500

1 j

j 3 . 3 ~ 3 j 30 kw I i ! I 1 1x3 l (15 !cw/ I 10x20 i 200 x20 300 0.0075' 300

10 1 u n i t ) 1 = 200 1 = 4000 1 8 ms / 3.3

1 296 455 l 4 x 7286

i ! 200 kwi I 620 / 8070 1.3 1 500

I I I 1

I ! i F .J

1 3 1 p s 1 1 1 256 1 296 200 kwl ! 1400 1 1800 1 1.25 i 5x40 780 p s I 1 I i 0 . 7 1 2 0 0

&16/ drum

5&296 2960

(80 ns ) I

i I

50 1 20 kw ' 10x20 I 70x20 i ' 5x40

drum 1 ~ i ~ t ~ l / =2 00 i =I400 1 900 1 0.048 200 !

words I 1 100 - 2000 12 1 - 2 words 1 ! 1

4 0 ns (80 n s )

I

620 1 5 7 6 0 I I

$ 9 3 1 500

10

! 3947 .83

296 x16

2.5

10

2 96 x16

2 96

500 74 0 8600

I

1.52

2960 I

200 kw (.2 mw/ b i t )

100 - 2000

9500'

I j

Memory-Swi tch

- S (MP ~ e m o r ~ y ~ o r t

control ~(Mp.port)

physical address tables

Physical Address data ( fR--- \ -

control bits

relocaked block address

- - - Processor

p-address

control

error detectlcorrec -

error detec tlcorrec

gen.

il error _I control

- - - - -. - .- -- - - port -

excisor control

to Supervisory Computer - - - - -

Figure 3. K(Mp.port) Memory-port relocation, error-detection, error-correction and control.

t h e 8,192k word memory (74 b i t s ) would c o s t approx imate ly $9,500,000.

T h i s i s a p r o j e c t e d p r i c e , b u t i t i s p o s s i b l e t h a t memory p r i c e s w i l l

d e c r e a s e even more t h a n p r o j e c t e d . AMS i s t h e lowes t b i d d e r , b u t t h e r e

i s some q u e s t i o n a s t o whether they cou ld manufac tu re a 620 megabi t memory

i n t h e t ime frame d e s i r e d i n a d d i t i o n t o t h e i r c u r r e n t commitments. Cogar

h a s quoted a p r i c e of $13,100,000 f o r a s i m i l a r system f o r d e l i v e r y i n

1973. The memory p r i c e s a r e q u o t e s f o r memory systems w i t h TTL-compatible

i n t e r f a c e s . The power d i s s i p a t i o n f o r t h e AMS system i s 180 m i l l i w a t t s

p e r 1024 b i t s o r approx imate ly lOOkw f o r proposed system; s u p p o r t c i r -

c u i t r y w i l l d o u b l e t h i s t o 200kw.

The memory s w i t c h w i l l u t i l i z e MSI (medium s c a l e i n t e g r a t i o n ) l o g i c

whenever p o s s i b l e . The d a t a s w i t c h i s c u r r e n t l y e n v i s i o n e d a s u t i l i z i n g

t h e Texas Ins t rument SN74150N 16 b i t m u l t i p l e x o r w i t h a t y p i c a l d a t a

p r o p a g a t i o n t ime of 10 n s e c (assuming t h e d a t a s e l e c t l i n e s have been

s e t t l e d f o r 2 30 n s e c ) . By t h e t ime t h e swi tch i s c o n s t r u c t e d , f a s t e r

c i r c u i t s , such a s Schotky TTL, w i l l p robab ly b e a v a i l a b l e .

S i n c e t h e swi tch i s s o c e n t r a l , a d u a l c r o s s - p o i n t p robab ly should b e

used. Th is i s e s s e n t i a l l y a compound s w i t c h c o n s i s t i n g o f two c r o s s - p o i n t

s w i t c h e s and a (m+p) x S ( 1 - i n p u t , 2 -ou tpu t ) s w i t c h a s shown:

/ S ( c r o s s - p o i n t ; mxp)\ Mp m-inputs S(m;2) S ( p ; 2 ) p - i n p u t s P

\ s ( c r o s s - p o i n t ; mxp)/

The memory s w i t c h i t s e l f shou ld n o t exceed $200,000 i n c o s t ( p a r t s

and l a b o r ) . A qu ick c a l c u l a t i o n shows t h a t i n j u s t t h e d a t a and a d d r e s s

s w i t c h i n g l o g i c , 2800 SN74150N 16 i n p u t m u l t i p l e x o r s would b e needed a t

K. drum K. drum C(Ms) - --. . c (AMOS)

Ms.drum Ms.drum M s . drum M s .drum C(Mt)- M t

F igure 4. Eventual s econda ry - t e r t i a ry memory s t r u c t u r e .

TERTIARY MEMORY

Any s t o r a g e of programs and d a t a o u t s i d e t h e l o c a l C.ai environment

w i l l tend t o pu t a f a i r l y heavy t ransmiss ion load on t h e c u r r e n t , 50,000 kb

ARPA network. For example, w e have assumed average program s i z e s of

250,000 74-b i t words o r 18.5 m i l l i o n b i t s . The t ransmiss ion of a program

t h i s s i z e on t h e p re sen t ARPA network would r e q u i r e approximately 460

seconds. I f a s e s s i o n l a s t e d an hour, about 15 minutes of t h e hour would

be spent i n f i l e t ransmission. If 4 such u s e r s were on t h e system, then

t h e whole ARPA network would be swamped.

C lea r ly , some o n - s i t e permanent s t o r a g e must be provided. Programs

can r e s i d e on a t e r t i a r y memory u n t i l they a r e brought i n t o e i t h e r primary

o r secondary memory f o r more r ap id access .

Exact s p e c i f i c a t i o n of C.amos w i l l r e q u i r e f u r t h e r ref inement of

C.ai. It appears , however, t h a t a PDP-11 Model 45 s i z e machine w i l l be

s u f f i c i e n t i f enough o t h e r min is a r e used f o r M s , M t and network con t ro l .

CONSOLE

Scopes would b e used t o d i s p l a y t h e o v e r a l l a l l o c a t i o n of resources

t o t a s k s , and t h e s t a t u s of t h e computer. Severa l scopes might a l s o b e

employed f o r human i n t e r v e n t i o n r equ i r ed i n t h e management of t h e computer.

Each of t h e processors would occas iona l ly r e q u i r e a c e r t a i n amount

of console a c t i v i t y which would be done by smal l computers a s descr ibed

i n t h e s e c t i o n on t h e language processors . Due t o t h e n a t u r e of t h e

switching involved i n t h e i n d i v i d u a l p rocessors , i t might be p o s s i b l e t o

use only one small computer t o s e rve s e v e r a l l a r g e processors .

The func t ions provided by AMOS must be a minimal set con- s i s t e n t wi th managing t h e hardware resources of C.ai. Com- p l i c a t e d systems t ake a long time t o b u i l d and a r e more open t o problems.

The "users" of AMOS a r e t h e ope ra t i ng systems f o r each spec ia l -purpose P.R. Thus t h e t o t a l ope ra t i ng system i s a two-stage o b j e c t : an o v e r a l l ope ra t i ng system (AMOS) p l u s d i s t i n c t ope ra t i ng systems on each processor . I n most cases a human u s e r and/or h i s program see only one of t h e i nd iv idua l systems, no t AMOS.

S p e c i f i c a t i o n and cons t ruc t ion of t h e ope ra t i ng system f o r a P.R i s up t o t h e group r e spons ib l e f o r t h a t processor .

AMOS should usurp a s few des ign peroga t ives a s pos s ib l e . That i s , i t should i n f luence on ly minimally t h e des ign of ope ra t i ng systems and programs on i n d i v i d u a l P.fils. Fur ther , i t should no t g r e a t l y i n f luence t h e des ign of C.ai a s a whole i n o rde r t h a t it w i l l be p o s s i b l e i n t h e f u t u r e t o r e p l a c e AMOS wi th a completely d i f f e r e n t ope ra t i ng system. J(

It should be p o s s i b l e t o b u i l d very simple ope ra t i ng systems on t h e P . l 1 s i f des i red . They should no t have t o worry about phys i ca l l e v e l i / o and t h e i r communications wi th AMOS should be a s s imple a s pos s ib l e .

AMOS should be simple. This no t only a i d s c o n s t r u c t i o n t i m e and e f f o r t b u t a l s o understanding (an e s s e n t i a l po in t t o i n s u r e c o r r e c t ope ra t i on and usage).

FUNCTIONS TO BE PROVIDED

It i s e a s i e s t t o spec i fy what AMOS i s t o do by l i s t i n g t h e major

func t ions i t i s t o provide. E labora t ions of t h e s e func t ions w i l l b e

>k C.ai i s c l e a r l y

v i r t u a l machines a unique oppor tun i ty f o r implementing r a d i c a l l y new t h a t e x p l o i t i t s p a r a l l e l and f u n c t i o n a l l y s p e c i a l i z e d

p a r t s . It i s hoped t h a t t h e computer w i l l be a v a i l a b l e f o r r e sea rch a long t h i s l i n e . The understanding of such a machine, how t o break up a load computat ional ly , t h e c h a r a c t e r i s t i c s of t h e programs r u n on i t , e t c . i s s o meager a t p r e sen t t h a t i n i t i a l l y t h e only s e n s i b l e way t o u s e i t i s a s a c o l l e c t i o n of independent systems t h a t happen t o sha re some phys i ca l resources . AMOS and i t s hardware should not unduly impede re- search on more advanced modes of usage, however.

- t h e p o t e n t i a l f o r p rocesses on s e p a r a t e processors t o communicate and sha re resources ;

- a l a r g e o n - s i t e M s f o r temporary use and a very l a r g e M t f o r permanent s t o r a g e of information.

What an Operating System on a P.R Sees

- a device f o r a l l o c a t i n g and overseeing sha r ing o f Mp, M s , and Mt;

- a device t h a t w i l l move f i l e s between Mp, M s and M t upon r eques t ;

- a device t o handle t h e mechanics of t r a n s m i t t i n g and r ece iv ing informat ion over t h e ARPA network.

What AMOS sees

- P.A's competing f o r Mp, M s and Mt;

- r e q u e s t s t o sha re resources between processors ;

- l o g i c a l communication channels between P.R1s and t h e ARPA network t o e s t a b l i s h and main ta in ;

- f i l e s t o be c r ea t ed and moved;

- M ' s t o housekeep;

- account ing information t o be logged and displayed.

STRUCTURES PROVIDING THE RE$JIRED FUNCTIONS OF AMOS

D i f f e r e n t s t r u c t u r e s can be chosen t o provide t h e func t ions of AMOS.

Those w e have s e l e c t e d below seem t o be s u f f i c i e n t f o r t he t a s k and con-

s i s t e n t wi th our des ign ob jec t ives .

MP A l loca t ion

The opaqueness of how Mp a l l o c a t i o n s t o P.R1s a r e be ing used and

t h e i r s i z e (64K wds) impl ies us ing an extremely s imple algori thm. A

P.R w i l l send a r eques t t o AMOS over t h e bus t o a l l o c a t e o r d e a l l o c a t e

The r eques t can be made wi th a p r i o r i t y , thus al lowing swapping o r

paging informat ion t o be handled j u s t l i k e any o t h e r f i l e only wi th h igher

p r i o r i t y f o r performing t h e t r a n s f e r . Likewise, f i l e s can be t r a n s f e r r e d

from M s o r M t t o Mp. A l t e r n a t i v e l y , in format ion can be s e n t and rece ived

over t h e network d i r e c t l y t o a f i l e ( s ee below). F i l e s can be e rased by

r eques t . Note t h a t t h e P.k 's s p e c i f y where they want t h e i r f i l e s t o r e s i d e .

This seems e s s e n t i a l s i n c e on ly they w i l l know what they a r e being used

fo r . P r i c i n g s t r u c t u r e , time l i m i t s , and a l l o c a t i o n l i m i t s can b e used

t o i n s u r e proper migra t ion .

It i s assumed t h a t M s and M t p rovide hardware d e t e c t i o n of record

and f i l e ends s o t h a t t r a n s f e r s of p a r t i a l f i l e s may be made. On t h e

o t h e r hand, record t r a n s f e r may impose t oo much a d d i t i o n a l complexity on

AMOS.

Communication Over t h e Network

AMOS w i l l know noth ing about u se r s . I t w i l l have on ly l o g i c a l

channels t h a t i t can connect between a P.R and some e n t i t y t r a n s m i t t i n g

messages t o C.ai over t h e network.

AMOS may r e c e i v e messages on t h e network from e n t i t i e s f o r which

i t has no l o g i c a l channel s e t up r eques t i ng acces s t o a g iven P.R. The

P.R may have t o l d AMOS t h a t i t w i l l t ake a l l comers, on ly c e r t a i n ones,

o r t h a t i t wants t o be informed of a l l r eques t s f o r connect ion s o t h a t

i t can make a dynamic dec i s ion . I f t h e r eques to r cannot be a t t ached ,

h e w i l l be s o informed.

I n i t i a l i z a t i o n - C.amos w i l l have a n au to load bu t ton t h a t w i l l load i t s l o c a l memory

from a s t a r t u p d i s k wi th a program t o i n i t i a l i z e Mp bounds r e g i s t e r s

and load i t s main Mp from M s . Its b o o t s t r a p w i l l a l s o b e a b l e t o r e t r i e v e

from i t s l o c a l Ms va r ious debug, checkout, and recovery rou t ines .

C.amos w i l l be a b l e t o s t a r t u p any of t h e o t h e r P . l l s by a s i g n a l

over t h e bus. Once s t a r t e d , however, AMOS has no c o n t r o l over t h e P.1.

This means t h a t AMOS w i l l have a v a i l a b l e t h e ope ra t i ng system (o r a proper

boo t s t r ap ) f o r each P.1. I n some cases t h i s may inc lude loading a micro-

code s t o r e .

F i l e Movement

A l l f i l e ope ra t i ons a r e l o g i c a l (no t phys i ca l ) a s descr ibed above.

AMOS w i l l have one o r more minicomputers t h a t w i l l i n i t i a t e t r a n s f e r s

between memory h i e r a r c h i e s and perform housekeeping chores.

Resource Shar ing

The mechanism f o r sha r ing i s b a s i c a l l y t h e same f o r a l l resources .

Processor A ( t h e owner of a resource) t e l l s AMOS over t h e bus t h a t pro-

ce s so r B may have a given type of access t o t h a t resource. I f l a t e r , B

r e q u e s t s t h a t access , i t w i l l be gran ted (un less A has resc inded t h e acces s

r i g h t s ) .

I n t h e ca se of Mp t h i s i s implemented by s e t t i n g bounds r e g i s t e r s .

For f i l e s AMOS must keep l i s t s of processors (not processes ) t h a t can

acces s g iven f i l e s . It i s then up t o t h e processors t o c o n t r o l t h e acces s

of t h e i r i nd iv idua l processes .

(a) PC-Mp-Ms type of s t r u c t u r e ;

(b) reduce i t s own d a t a and keep c u r r e n t system information a v a i l - a b l e f o r AMOS;

( c ) w r i t e t o i t s own slow M s f o r l a t e r d i s p l a y and a n a l y s i s .

Some examples of information of i n t e r e s t a r e : (a ) K(Mp.port) e r r o r s

could b e counted, and t r ansmi t t ed t o C.amos, (b) t h e number o f memory

r e f e r ences could be counted and wa i t i ng t imes t abu la t ed , ( c ) a c e n t r a l

c lock may b e provided which a l l P ' s may access . A c e n t r a l t iming f a c i l i t y

might a l s o b e included a t t h e clock. I n o r d e r t o keep t h e t r a f f i c low,

a f a c i l i t y such a s t h e c lock might b roadcas t t h e time so t h a t each pro-

ce s so r could main ta in i t s own t i m e r s (which would undoubtedly be i n s o f t -

ware).

AMOS should have a number of sof tware monitors b u i l t i n t o i t s

modules. Such hooks are b e s t when implemented i n p a r a l l e l w i th t h e

ope ra t i ng system. Se lec ted in format ion would b e e i t h e r w r i t t e n by AMOS

o r read from r e g i s t e r s by C.pm. I f AMOS i s t o be a r e sou rce a l l o c a t o r ,

some P.R informat ion may be requi red . Information of t h i s type would

p l a c e c e r t a i n c o n s t r a i n t s on P .1 implementors, b u t t h e sha r ing of common

r e sou rces r e q u i r e s some s t anda rd i za t i on .

Each P.R should a l s o i nc lude i t s own hardware and sof tware measure-

ment devices w i th which AMOS can communicate. A mixture of sof tware and

independent hardware monitors i n t e g r a t e d i n t o t h e des ign of C.ai w i l l

a l low f o r f u t u r e s tudy of t h i s new s t r u c t u r e , and encourage growth based

on a knowledge of a c t u a l performance and u t i l i z a t i o n .

twice a s wide (74 p o i n t s ) , and must accept trp t o four words t ransrnl t ted

s e r i a l l y . We a l s o proposed t h a t t h i s switch s t r u c t u r e be d u p l i c:ltcd

c e n t r a l l y so t h a t 1/2 can f a i l o r be worked on wi thout b r ing ing t h e s y s t c r ~ ~

down.

Memory would be added s lowly, a s t h e process ing power i s increased.

An i n i t i a l con f igu ra t i on could be ope ra t i ng i n on ly one y e a r , composed of

t h e nucleus of t h e memory ( inc luding t h e secondary memory) t oge the r wi th

some small amount of p rocess ing power (two PDP-10 processors , running

Tenex). This con f igu ra t i on would provide immediate b e n e f i t t o some u s e r s

by having a l a r g e r memory. Even more memory could b e added i f needed.

By proceeding slowly t h e ope ra t i on of t h e computer could a l s o evolve

and t h e t i e w i th t he network could be made gradua l ly . During t h e second

year of ope ra t i on t h e f i r s t and second Stanford AT processors could be

i n t e g r a t e d t o o p e r a t e wi th t he memory system and t h e f i r s t two processors .

The neces s i t y f o r two processors i s based on t h e f a c t t h a t t h e f a c i l i t y

should be a b l e t o supply a cons t an t resource t o i t s u s e r s and, t hus , two

processors would be necessary t o meet t h e r e l i a b i l i t y requirement. A t

the end of t h e second yea r almost any con f igu ra t i on of p rocessors and

computers would be pos s ib l e .

CONTINGENCIES AND RESEARCH

The c e n t e r s e c t i o n of Schedule 1 d e a l s w i th t he schedul ing of spe-

c i a l i z e d processors f o r languages. Their p rogress would no t i n f luence

t h e main schedule l i n e t o any g r e a t e x t e n t , bu t could provide baclcup f o r

t he e x t r a computing power. I f s p e c i a l i z e d processors prove a s va luab le a s

we b e l i e v e they w i l l , then such a n approach would pay o f f and poss ib ly

negate t h e need f o r t h e l a r g e processors . Two of t h e cont ingencies

(KI10 and a small microprogram-based PDP-10) could no doubt be cont rac ted

t o DEC because they would be p a r t of t h e i r s tandard product l i n e .

NEW PROCESSOR ALTERNATIVES

The f i n a l l i n e on Schedule 1 shows what we might expect ( o p t i m i s t i c a l l y )

from a new processor based design. I n t h r e e years we might have such a

system ope ra t ing ready f o r sof tware debugging (assuming t h a t t h e des ign

and f a b r i c a t i o n had gone on i n p a r a l l e l w i th t h e hardware development).

I n poss ib ly a s e a r l y a s fou r yea r s , u s e r s might be on t h e system. (For

1 1 example, by comparison, t h e PDP-11 i s now about 2~ t o 4 2 yea r s o ld (de-

pending on when you count t h e s t a r t ) .

The problem wi th such a des ign appears t o be t h a t i t i s n ' t c l e a r why

any company would want t o b u i l d it. The only company which has both t h e

resources and t h e d i v e r s e product l i n e s t r a t e g y i s Honeywell. A GE645-

based processor has a l a r g e address space and poss ib ly might be considered.

There i s some evidence t h a t a s t r a igh t fo rward machine based on t h e

PDP-11 s t r u c t u r e , b u t wi th l a r g e addresses would be u s e f u l and could

provide b e t t e r encoding e f f i c i e n c y than a PDP-10. It might be much e a s i e r

t o b u i l d than t h e Stanford A 1 Processor , b u t i f i t were not p a r t of a

main product l i n e i t would be d i f f i c u l t t o g e t i t b u i l t . Buying computers

which a r e p a r t of another product l i n e al lows both a much lower c o s t and

h igher r e l i a b i l i t y (by no t having a one of a k ind) .

data structure with large pointers. While all of these methods may be

regarded as somewhat ad hoc, they should be sufficient. Thus, if we

compare the modified version with some of the alternative off-the-shelf

processors, making these modifications will probably make the PDP-10 no

worse than the alternatives. Some of the desirable modifications will

deal with list structure manipulation and typed variables. All in all,

it appears from the early explorations about modifications between

R. Greenblatt (M.I.T.) and the Stanford project, that they can be made

without unduly distorting the Stanford design.

ECONOMIES OF SCALE

We have not carried out calculations which would allow one to

decide whether it would be worthwhile making a system smaller than the

one proposed. The considerations for not scaling down the computer to

114 the size would be:

Fixed operating costs of the laboratory facility (overhead), engineers,, programers, cooling, lab supplies, space.

The design task is fixed almost independent of machine size. Thus, it would be desirable to get as much power as possible in a single place. This would give us the highest performance/ total cost.

Desirability of having a central facility with a very large amount of processing power attached to a single, large memory.

Desirability of having a single, very large memory. This would permit more users through better user-load-averaging. Also, larger programs could be run.

There may be some quantity discount for drums, disks and memory.

Higher reliability. All facilities would be duplicated in a larger computer.

CONCLUS IONS

We have given a n o v e r a l l argument a s t o t h e f e a s i b i l i t y and d e s i r -

a b i l i t y of bu i ld ing a computer t o be used i n a i research . It i s b a s i c a l l y

a very conserva t ive approach b u i l t on c u r r e n t computers t oge the r wi th

what we th ink can b e done e a s i l y wi th t h e technology. We have n o t assumed

very high performance processors . For example, one processor we th ink

f e a s i b l e i s t h e S tanford A 1 ve r s ion of t h e PDP-10. Such a processor should

not r e q u i r e more than two yea r s t o develop, and should b e r e p l i c a t a b l e f o r

i n t h e neighborhood of $100,000.

Given our o v e r a l l numbers, we th ink i t i s worth t r y i n g t o s p e c i f y

t h e next l e v e l of d e t a i l based on our evolu t ionary approach. We b e l i e v e

t h a t an approach t h a t d e p a r t s from a convent ional s t r u c t u r e (e.g., by

p l ac ing s p e c i f i c i n t e r p r e t a t i o n on address ing) w i l l bo th decrease t h e

performance and a l s o make t h e memory too s p e c i a l purpose, thereby e l imina t -

ing unspec i f i ed f u t u r e use t h a t might be made of t h a t l a r g e f a c i l i t y .

We th ink t h e r e a l po in t of t h e s t r u c t u r e i s t h a t i t should be simple,

y e t provide a s much p o t e n t i a l power (bandwidth) a s p o s s i b l e , and should

not be very presumptious about how p a r t i c u l a r f u t u r e processors would u s e

t h e f a c i l i t y . In p a r t i c u l a r , we s t r o n g l y b e l i e v e t h a t a d d i t i o n a l power

w i l l be gained by developing s p e c i a l purpose processors t o be used on

C.ai and t h a t t h i s course should b e thoroughly explored.

6. This

7. This

Allow all contractors to engage in certain kinds of improvements.

High reliability -- probably affects more users (say 50 N 100 versus 10 N 25 at a site).

Multiple users simultaneously.

is a machine for artificial intelligence research.

Consequences: Variety of specialized devices connected to it (e.g., robotic equipment); powerful arithmetic processing; large memory.

is a symbol and list processing machine.

Consequences: Access to very wide words so that different types of encodings are possible for various languages. The (memory size)/(memory accessed) ratio is probably large.

8. Examine all techniques,(especially those whichhave been found to work

in hardware and software):

Examples: cache (1 or more);

microprogramming ;

hash coding;

highly specialized processors.

9. Design should be highly likely to materialize.

Consequences: must be understandable;

maximum amount of project parallelism;

minimum amount of interaction among parts (hence well defined interfaces) .

10. Base system performance on easy-to-identify techniques.

Consequences: parallelism in word length, processors, memories ;

APPENDIX 2

OUTLINE OF P.L* PROCESSOR DESIGN

The fol lowing i s a n a b s t r a c t of t h e r e p o r t "C.ai (P.L.") -- An LJ:

Processor f o r C.ail' by D. McCracken and G. Robertson. It i s a v a i l a b l e

from t h e Carnegie-Mellon Computer Science Department o r t h e Defense

Documentation Center.

The r e s u l t s of a p re l iminary des ign s tudy f o r a s p e c i a l i z e d language

processor (P.A) f o r L* a r e presen ted . The o b j e c t i v e of t h e s tudy i s t o

g ive a n example of a s p e c i a l i z e d processor f o r C.ai.

The LYc processor i s t o run 20-30 simultaneous LJc u s e r s wi th very

l a r g e address spaces a t a speed improvement of b e t t e r than 10 times a

t y p i c a l PDP-10 L* system. I ts c o s t should be low r e l a t i v e t o t h e memory

r e sou rces of C.ai.

The des ign presen ted i s t h a t of a n L* c e n t r a l p rocessor (Pc.L*)

w i th a very low-level i n s t r u c t i o n set (about t h e l e v e l of t y p i c a l micro-

code). Pc.L* i s time-shared by a mini-computer t h a t s i t s t o t h e s i d e ,

s o t h a t each L* u s e r sees himself a s running on a base L* processor .

User contex ts a r e switched by swapping processor s t a t u s in format ion i n

Pc.L*.

Each L* u s e r has complete acces s t o t h e c e n t r a l p rocessor s t a t u s

through h i s address space. H i s machine code (microcode) can appear

anywhere i n t h e l a r g e address space, b u t executes ou t of a f a s t cache

memory. It thus runs a t microcode speeds. L* programs and d a t a being

i n t e r p r e t e d by t h e machine code a r e accessed e x p l i c i t l y from a second

APPENDIX 3

OUTLINE OF P.LISP PROCESSOR DESIGN

The following is an abstract of the report I1C.ai (P.LISP) -- A

LISP processor for C.ai" by M. Barbacci, H. Goldberg, and M. Knudsen.

It is available from the Carnegie-Mellon Computer Science Department

or the Defense Documentation Center.

A special processor designed for efficient LISP processing is

described. The processor is micro-programmable and makes heavy use of

a fast scratchpad memory with several special purpose registers (small

function units) and general byte manipulation capabilities. The approach

taken has been to avoid unorthodox schemes of implementation and employs

little in the way of unusually new (and untried) hardware.

Such a conservative approach will enable a fairly good implementation

in a reasonable time. One of the places where efficiency in list pro-

cessing (and in most programming applications) can be enhanced is in the

ratio of instruction fetchesldata fetches. To that end two things that

are not usually available were required: writeable (up-datable) micro-

code and recursive control of microcode. With them, it is possible to

implement the language interpreter as close as possible to the real

hardware machine. Such a machine could also be a "shell" language pro-

cessor. However, this was not a goal of the design, but rather a by-

product.

The microprogrammed processes include a storage-compacting garbage-

collector, which can be made to operate incrementally in parallel with

user-program execution. This option avoids interruptions in LISP execu-

tion for garbage collection.

I C . a i : A COMPUTING ENVIRONMENT FOR A1 RESEARCH

4 r > r s C R I F 1 T I L i N O T E 5 ( T y p e 0 1 r e p o r t and incltrrivt . dales)

S c i e n t i f i c I n t e r i m 5 A L l l k i O R l 5 ) (E'irht narnr, rniddl? inifinl, l as f name)

G. B e l l , P. Freeman, M. Barbacc i , S. B h a t i a , W . Broadley, R. Chen, L. Erman, H. Goldberg, W . Huen, M. Knudsen, D. McCracken, A. Newell , R. Reddy, S. Rege, G. Rober tson.

fi R E P O R T o n r r 7 8 . T O T A L N O . O F P A G E S 76. N O . O F R E F S

May, 1971 5 5 none ui,. C O N T R A C T o r ? G H A N ~ NO 98. O R I G I N A T O R ' S R E P O R T NUMBERIS)

~44620-70-C-01 07 t1. P R O J F C T' P I C

A0827-5 c ' . 9h. O T H E R R E P O R T N O I S I (Any other n ~ l m b c r s thnl mny br o s s i ~ n r d

th i s rrporl) 61 101D

This document h a s been approved f o r p u b l i c r e l e a s e and s a l e ; i t s d i s t r i b u t i o n i s u n l i m i t e d .

Ur 'PL .CM1 P i l A l l l b 1 O T r 5 1 2 S P O N S O R I N G M I L I T A R Y A C T I V I T Y 12, OTmR A i r Force O f f i c e of S c i e n t i f i c Rese 1400 Wilson Blvd.

1 ( ~ r l i n ~ t o n , V a . 22209

A computer f o r a r t i f i c i a l i n t e l l i g e n c e r e s e a r c h i s examined. The d e s i g n i s based on a l a r g e , s t r a i g h t f o r w a r d pr imary memory f a c i l i t y (about 8 m i l l i o n 74 b i t words). Access t o t h e memory i s v i a a t l e a s t 16 p o r t s which a r e hardware p r o t e c t e d ; t h e r e i s dynamic ass ignmentof t h e memory t o t h e p o r t s . The maximum p o r t bandwidth i s 8,600 m i l l i o n b i t s / s e c . P r o c e s s o r s f o r languages (e.g., LISP) and s p e c i a l i z e d t e r m i n a l s (e.g. , v i d e o i n p u t / o u t p u t ) can be r e l i a b l y connected t o t h e system d u r i n g i t o p e r a t i o n . The approach i s e v o l u t i o n a r y i n t h a t h i g h performance p r o c e s s o r s , such a s t h e S t a n f o r d A1 P r o c e s s o r , c a n be connected t o t h e memory s t r u c t u r e , g i v i n g an o v e r a l l power of a t l e a s t 100 t imes a PDP-10 (and 200 t o 300 t imes a PDP-10 f o r l i s t p r o c e s s i n g languages) f o r 1 0 p r o c e s s o r s -- a l t h o u g h 20 p r o c e s s o r s c a n be a t t a c h e d . Using t h i s approach we might e x p e c t 40 - 8 0 m i l l i o n opera t ions / second .

A t t h e same t i m e , s p e c i a l language p r o c e s s o r s ( ~ . i ) can b e des igned and a t t a c h e d . These p r o c e s s o r s g i v e even l a r g e r power i n c r e a s e s , b u t f o r r e s t r i c t e d language use . Two p r o c e s s o r s , P.LISP and P.L*, were examined f o r t h e LISP and L* languages and a r e r e p o r t e d on s e p a r a t e l y .

A p l a n f o r b u i l d i n g t h e machine i n increments over t h e n e x t t h r e e t o f i v e y e a r s i s examined. S p e c i f i c schedu les a r e proposed.

Concurrent w i t h t h e o p e r a t ion of t h e machine, t h e r e shou ld be r e s e a r c h i n t o t h e d e s i g n of hardware , s o f t w a r e and t h e o r y of c o n s t r u c t i n g l a r g e s c a l e computing f a c i l i t i e s w i t h maximum modula r i ty .