1

01011111111 ADVANGIntwo.

D r . g i c h o l a e g e g r o p o s i c eZempartment o f A r c h i t e c t u r eM a s s a c a m e e t t a ' e a r l i e r * o f77 M a a m e c h m e a t t s A v e n u eC a m b r i d g e . l a m e a c h e e e t t a I

Deer M i c k :

1 1 . 1 1 1 L 2 0 . S t o v e ma id is2 ifmw 4 . z _ m t o 6 . 6

Dr. Richard A. BoltPrincipal Research ScientistMassachusetts Institute of Technology

Acknowledgments

Realization of our spatial data-management project derives from manyminds and hands.

The Architecture Machine Group at theMassachusetts Institute of Technology,both as a collection of individuals and asa creative ambiance, generated theessential environment in which such aninvestigation could thrive. There doesindeed operate the "spirit of a place," dif-ficult to define, undeniable when present.

Systems implementation of spatialdata-management system concepts wascarried out under the leadership ofMr. William C. Done'son, who has per-formed throughout with exceptionalinitiative and ingenuity.

The energy and vision of Dr. Craig Fieldsof the Defense Advanced ResearchProjects Agency's Office of CyberneticsTechnology provided extraordinarily vig-orous and profitable sponsor interaction.We are all appreciative of his perceptivesupport.

My personal gratitude extends especiallyto Professor Nicholas Negroponte, Prin-cipal Investigator, whose élan and imagi-nation has touched every aspect of thisenterprise, indeed made it possible.

Richard A. Bolt

3

D r . S i c h o l n a U r r o p o o t eD n p a r t m o n t o t t A r h i t o c t u r e4 1 1 0 s 0 C h u s e t t s I n s t i t u t * a t77 M o s a m e c h w o o t t s A v e r n u e

C A M b r i d n e . 4 11 m . d s c h u m m t t o (

The Concept 6

The Setting 10

Finding Data 13

Perusing Data 28

What We Learned 50

Future Directions 56

D a n t S l i c k :

011POONME AtAtADICSO14111111

AALIONS

Ado Stoma s o l A i n b i z 1a l i m m m m O R I M M O t a r d o t s t a b e

4

Contents

Prologue

Such startling advances and cost reduc-tions are occurring in microelectronicsthat we believe future systems will not becharacterized by their memory size orprocessing speed. Instead, the humaninterface will become the major measure,calibrated in very subjective units, sosensory and personalized that it will beevaluated by feelings and perceptions.Is it easy to use? Does it feel good? Is itpleasurable?

It is this interface that will bring com-puters directly to generals, presidents ofcompanies, and six-year-old children.Thus, we ask the reader of this report tothink about its contents in two ways. Oneis to judge the specific value of spatialconstructs for nonspatial data. The otheris to appreciate the general intention ofmaking sound, visual, and tactile inter-faces to serve conjointly as the modesand media of interaction with a comput-ing resource, at once transparent andubiquitous. Such quality is no longer aluxury but a requirement.

Nicholas Negroponte

5

D r . I f i c h o l a e M e g r o p o o t eD e p a r t m e n t o f A r c h i t e c t u r eM o o s e c h o a e c t e I n s t i t u t e o f77 M a s s a c h u s e t t s A v e n u eC a r t d n n . M a s e s c h u e e t t s (

D o e r M i c k :

An n a m e t 4 t b i g 1A m t . o m o m C o o A m t . . r o b o

6

D E M M E A D V A I I I I M D1 4 1 0 11

A l l & WW2 The Concept

The basis of spatial data-management isaccessing a data item by going to whereit is rather than referencing it by name.

Consider what occurs when you retrievea book in your home library. You look atthe left-hand bookcase, second shelffrom the top, say, and scan about two-thirds down the shelf for a certain bookspine. Thus, the book is located not by itstitle but by its customary site in a spacedefined by other books and the book-case itself.

Consider how people ordinarily finditems on their desk tops: the appoint-ment book is up and to the right; the tele-phone is in the lower right corner; highpriority memos are kept in an "in" boximmediately to the left of the desk blotter;less urgent items are in the file folder, topmiddle of desk; and so forth.

The person who uses this desk hasorganized the layout of items in a more orless systematic way. He or she refers tothem constantly throughout the workingday: reaching in that direction, that far,up, down, to the right, to the left. Throughthis activity, a mental image of the layoutof the desk is elaborated in the "mind'seye." Additionally, through constant tac-tile interaction with the items, reaching forand touching them, a "motor memory" ofwhere things are arises as well. A scriptfor the act of retrieval becomes encodedinto the musculature, as it were, accord-ing to where the item is located.

The practical importance of this spatialprinciple of organization is illustratedwhen someone happens to disturb ourfamiliar arrangement of things. Perhapssome well-meaning soul tries to tidy upour seemingly chaotic desk top. Ratherthan being a benefit, the now "well-organized" desk is for us an organiza-tional disaster: the former "messy"arrangement was a familiar and well trav-eled ground whereupon we could godirectly to what we wanted. Such a lossof a familiar spatial layout can impressupon us the extent to which we take thissubtle but powerful principle of organiza-tion for granted in our daily lives.

An even more striking instance of the roleof spatiality in common experience is thecoherence and persistence of "imagery."Two individuals, arguing a topic in front ofa blackboard, will refer each other to dia-grams, equations, and terms on thebasis of where they had been written,even long after they have been erased!

It is surprising how pervasive the under-lying notion of spatiality is, even in sym-bolic modes of thought. Consider a file ofthree-by-five notecards, organizedalphabetically, but without letter tabsshowing. "Filed under the letter R" trans-lates into a tactile estimate of how fardown the row of cards we must reach.The same holds true for a dictionary with-out thumb indexes. When we want tolook up a word beginning with the letterR, we "guesstimate" where we shouldopen the pages of the book to begin oursearch. Thus, /3 is somehow a relativedistance, as well as a letter. There ismore spatiality implicit in what is ordinar-ily thought of as symbolic retrieval thanwe may realize.

In summary, it is precisely the insightabout how we tend to retrieve items fromdesk tops, from files and bookshelves,even from erased blackboards, that liesat the heart of the spatial data manage-ment concept: we find items on the basisof a more or less definite sense of theirlocation in a familiar space, which spacemay be actually present or remembered.

7

111111•Sliffin

Imp

I . Ammo omdmil161161111111Mak

8

This well-evolved human ability toorganize information spatially remainsessentially untapped in the realm ofcomputer-based information handling.Typically, in such systems, retrieval on asymbolic or name basis is the norm, andmust be the norm because the conven-tional keyboard interface is too limited achannel, the wrong mode and medium,to begin to offer the user a direct, palpa-ble sense of spatiality.

In the spring of 1976. the ArchitectureMachine Group at MIT proposed to theCybernetics Technology Office of theUnited States Defense Advanced Re-search Projects Agency (DARPA) a pro-gram of research organized under thetitle "Augmentation of Human Resourcesin Command and Control through Multi-ple Media Man-Machine Interaction...Intrinsic to the ensemble of studies out-lined in the proposal was a study recall-ing the ancient principle of using spatialcueing as an aid to performance andmemory: the "Simonides Effect."

Simonides was a poet of ancient Greecefamous for his ability to give long recita-tions entirely from memory. His secret,which as a teacher of rhetoric he sharedwith his students, was to tie each suc-cessive part of a to-be-rememberedpoem or speech to a specific localewithin the mental floor plan of either anactual or imagined temple. Around thefloor of the temple were statues, servingto augment the imagery of specificniches or corners.

To commit to memory a long poem orspeech, the orator would pause in hisimagination before the threshold, andwith this mental image in mind, committhe introductory remarks to memory.Then, for each successive subsection ofthe talk to be given, the orator wouldmentally walk from place to place withinthe temple. rehearsing the appropriatematerial before some specific piece ofstatuary. The resultant path about thetemple would then serve as a mentalschema to organize the speech upon itseventual recitation, the speech beingretrieved and reconstructed during ora-tion by way of an imaginary tour fromstatue to statue around the temple floor.

Another study outlined in that same pro-posal looked to the exploitation of virtualspatiality as a matrix for organizing infor-mation: the "virtual bulletin board." Thisterm refers to the formulation of animplicit space which occupies no "real"space but can nonetheless contain agreat deal, as well as offer the samespatial organizational cueing as to thewhereabouts of material that real worksurfaces do. The tacit size of a "field" ofinformation to be portrayed in computergraphics need not be limited by the sizeof the physical display. Since it is virtual,its apparent size and graphic organiza-tion are open to user definition in theunconstrained sense of Ivan Sutherland'scharacterization of a computer displayas "a window on Alice's Wonderland"("Computer Displays," Scientific Ameri-can, June 1970).

These two spatial themes specificallyresonated with Dr. Craig Fields, SeniorProgram Manager in the CyberneticsTechnology Office. As a result we startedto explore the potentialities of a spatiallyoriented data management system, "amultidimensional window into data." Theoutcome has been that over the past twoyears, under the aegis of DARPA, theArchitecture Machine Group has beenexperimenting with the creation of aninformation management system whosedistinguishing characteristic is that itexploits the user's sense of spatiality forpurposes of organizing and retrievingdata: a spatial data-managementsystem, or SDMS.

9

11•10IIMIP1111 111111111111116

ironalows1111.1MEMEIS E l lM U M leIninrosnirr• •••siorosurrar. morro-•—•—•Ewen Itimom

""11/11

alo alimus am6111 ain 11112016111AMEMIlli & • 1 1 • • Ji•

This page: The monitor tothe user's right as seen overthe arm of the user's chair.Fingertips of user's hand areon the small joystick. On thechair's arm, immediately infront of user's hand is a smalltouch-sensitive pad ready forcapture of fingertip gestureinput. The other arm of thechair is similarly instrumented.

Opposite page: The absenceof any desk-like appur-tenances situated in front ofthe user lends a tone of imme-diacy and directness to inter-action with data. At the upperright corner of the largescreen, one of the soundsystem's speakers is clearlyvisible.

The lapboard and stylus arenot featured in these illustra-tions. For a view of them, seeillustration at top of page 48.

10

The Setting

The setting of our experimental version ofSDMS is a multiple media informationplace. It can be construed as an imageof an office of the future.

The physical aspects of our "mediaroom" are easily described. The mediaroom is about the size of a personaloffice. The focal point of the room is awall-sized display screen. Directly infront of this large screen is an instru-mented Eames chair. On either arm ofthe chair is a small, pressure-sensitive

"joystick." Adjacent to the joystick oneach arm is a two-inch-square touch-sensitive pad. Also associated with thechair as complementary instrumentationis a ten-inch-square data tablet, with sty-lus; the tablet is upholstered to serve as alap pad. This tablet bears added instru-mentation: a small microphone mountedat its top edge to allow voice input.

Two small television monitors are situatedone on either side of the chair. Each ofthese monitors is touch-sensitive so thatthe user can point to data or can inputgestures indicating actions to be taken.

Finally, eight loudspeakers providesound output for the media room. Four ofthem are placed around the perimeter ofthe large-format display to the user'sfront. The remaining four are correspon-dingly arrayed to the user's rear. Octa-phonic sounds can be presented tothe user as coming from various appar-ent directions in the space of the room, incoordination with actions on the visualdisplay. The rate of playback is variableso that dynamic sound/motion effects,such as Doppler shifts, may be achieved.We call these "sound-synch graphics."

Notice that there is a conspicuous, butnot mandatory, absence of a keyboard.

The configuration described is onewhich evolved through previous work forthe Office of Naval Research (Contractnumber 0014-75-C-0460) and over thecourse of our experimentation, but thisprecise ensemble is certainly not the onlyone that might have been adopted. Thekernel notion of managing data spatiallyis not necessarily tied to a room-sizedterminal into which a user goes versus adesk-top arrangement in front of whichhe or she sits.

••• 111111111 11111•1111111110ir • I P.

"6' i r r 11•11111111111/t I t M I..."••••• ••• " M O P

moon. • "tot tmagmemmoommom •

4=111•1 • • • • • •

• • • • • • • . 1 1 • 1 1 a MAIM o n .Immo Lon k o r gm Imo inn

mom Almond•andio• %ago 111•••

A television projector in theroom adjacent to the mediaroom back-projects upon thelarge screen to the user'sfront.

12

The relaxed ambiance conveyed by thepresence of a chair of the Eames genrein lieu of something more utilitarian isintentional, however. It reflects convic-tions and positions about the nature andtone of human-computer interaction thatwe have attempted to actualize in themedia room setting. Just as the hands-on immediacy of touch-sensitive padssuggests a literal impatience with intan-gibles about data, so the decor as epito-mized in the selection of the style of chairrebuts the premise that system usersmust live in severe, ascetic settings.(See the Prologue to this report.)

We have attempted to create an interfacewhich is not a tiny, narrow-band "port-hole" into an information bank, that bankitself an abstractly addressed set ofintangibles. Rather, we have attemptedradically to recast the setting as an "infor-mational surround" wherein the user isdirectly engaged with data bodied forthin vision, sound, and touch, data inhabit-ing a spatially definite "virtual" world thatcan be interactively explored and navi-gated.

Finding Data

Spatial data-management involves thecreation of a plausible and commodiousvirtual- space at the computer interface,together with a way of getting around inthat space quickly and easily.

The spatial world of SDMS consists of asingle plane. This surface, called "Data-land." is presented to the user in twoaspects:

1it is continuously visible to the user in itsentirety in an "aerial," top-on view dis-played on one of the monitors, arbitrarilysituated to the user's right, called theworld view" monitor; and

2simultaneously, a small subsector of theDataland surface is displayed on the ten-foot diagonal screen to the user's front,vastly enlarged and with appreciablegain in detail.

The logical relationship between thesetwo views is akin to a mapping key, whichplaces a particular image within a largerlocale. The large screen effectively func-tions as a "window" and, as we shall seelater, a "magnifying glass" onto Dataland.

The "world view" monitor serves specifi-cally as a navigational aid to the user ingetting around Dataland. The large dis-play of whatever portion of Dataland is so

"close up" that the user would get losteasily if there were not always on view amap of the entire Dataland world. Asmall, highlighted, "you-are-here" rec-tangle on that display shows the user atall times the position of the large-screenwindow on Dataland.

To illustrate how an actual Datalandmight appear to a user seated in themedia room's control chair, let us assumea database already created. In practice,individuals would design and structuretheir own Datalands according to per-sonal preferences, tasks, and needs, justas the contents and organization ofanyone's desk top come to reflect somecombination of job requirements andindividual taste.

Dataland as it appears on the"world view" monitor to the

user's right. Notice highlighted"you-are-here" rectangle sit-uated over a map of north-eastern United States locatedin upper-left region of Data-land. "You-are-here" markeris translucent, under joystickcontrol, and indicates theregion of Dataland visible onthe large screen to user'sfront.

13

amon allinEnoo

••If PIP. 1

111

an• ammo, ••1111 l o a m mmilmmorE11111611 M a w A=

1111/• M I M I

14

N M I M I M I•••1111111IIIII 1 M INE 1 1 1

M p I m I I P • ••••• • • • • • • • IL •••••••••=weim."

Illustrations on this page andthe next depict relationshipbetweeen -world view" mon-itor and the large screen. OnDataland view, this page, thetranslucent "you-are-here"marker resides over lower-left-most item. The item, acalendar, appears on thelarge screen, much magni-fied and with finer detail(opposite page).

Now, what is visible to the user on thesurface of this example of Dataland?

Items of various sorts may be seen.Some are small photographs: pictures ofanimals, of people, even a miniatureLAN DSAT satellite photo of part of NewEngland. Other items look like smallmaps. We see what appear to be bookcovers, television sets, a business letter,even a small image of an electronichand-held calculator. Yet other itemsresemble emblems or "glyphs," not dis-similar from business logos.

How does one get around in such aninformational landscape?

User control of navigation around Data-land is provided by "joystick" action. Thesmall joystick in the right arm of the chairpermits a helicopter-like flight over thesurface. The joystick itself does notdeflect, but responds to pressure. It is

"self-centering" in that it registers zerodeflection when not touched. Push thestick "away" from you and the helicop-tering motion over Dataland is north, ortoward the top of the screen. Pull the joy-stick toward you and motion is south.Pressure to the right and left will directyou east or west.

-

1111 U1 :41111111 1•111•111i1.1,IMI I IMPUPP/ 111 I P S O .

1111111•91•11111 M E M fLOA. • . . . M E M

dqpRem.a•oto„ algeomoom•mmowe • 1

IMMO .10.=•=,

om. him •mmomm i

2 0 0 m . i m m o o I S M 1 1 0d a d i e i A W L S a on o g a i L e M I ' V I M I

16

.1111M111•1• • • • • I I M E

m o . • • • • • • , • • • •

The apparent rate of travel is proportionalto the pressure the user exerts on the joy-stick. At maximum pressure, it takesabout twenty seconds to travel acrossDataland, as illustrated in this exampledatabase. from its eastern to westernborders.

During flight, the user is given visualfeedback of two kinds:

1movement of the material across the

"window" of the large screen;

2tracking of the path of the window by thehighlighted rectangle on the auxiliarymonitor.

Let us say that you push the joystick tothe right and away from you. This actionwill set you helicoptering across Data-land to the "northeast." On the big screenyou will see material scudding by in asouthwesterly direction; that is, the intui-tive sense of movement will be that of awindow looking out over a world passingby below. Simultaneously, the highlightedrectangle on the auxiliary view at yourright hand will track your northeastwardprogress.

Complementing joystick control of flightover Dataland is "teleporation" by directtouch. Traversing relatively large dis-tances with the joystick can be frustrating;for the veteran user, this often represents

"dead time." Accordingly, the user isgiven the option of indicating where hewants to go directly on the touch-sensitive "world view." The highlighted

"you-are-here" rectangle will disappearfrom its current location, to reappearcentered at the spot indicated by thefinger touch. On the large screen. the

"magnified" view of the old location dis-appears, to be replaced by the just-indicated locale.

Spatialdatamanagementsystem

While its specific format is much a per-sonal matter, the graphic layout of Data-land can aid navigation significantly.First, the directness and simplicity ofDataland itself provides a unitary con-ceptual substrate: a large plane con-jointly given via two correlated views onthe small and large screens. Second, theabsolute and relative location of objectsand items is evident within the frame ofthe Dataland world. Absolute locationalcues are provided by the main "frame" ofDataland: the fact of edges, corners, up-down and left-right directionality. Anyitem is perceived and remembered in thecontext of and juxtaposed with certainother items. Third, the use of local back-ground colors affords a further visualsense of neighborhood, grouping certainitems together. Color backg roundingcan, of course, be "nested," color back-ground upon color background, as it is inour "rogues' gallery" of ArchitectureMachine Group personnel in the south-east corner of Dataland. Within the over-all rectangular area there is further groupintraorganization indicated by back-ground colors.

"Teleportation" about Data-land by direct touch: At top,user touch indicates itemwithin a group with orangebackground as what hewants to go to on Dataland.This item, a "book cover",fills screen to user's front asuser next indicates a locationto the right and somewhatlower (middle). "You-are-here" marker immediatelymoves to this new locale,and large screen view nowbears magnified view of thenewly selected item, a sche-matic floor plan of amuseum.

17

18

mInn

-••• ••• IRWIN.0,111O

IEL SIMENEllI IME l o m b e n e s ii lomonmpom 111111111111111111

fartriliallitgra WOW- • • • • 1 4 . 111 . 4 11 P P I I E R 4 P 0 • • •

lion Sumo moololl 6 E i m nmndolo •1111 Ellon I m o ilmemodno

idbo ialifininnillo M a n limo I S M SM U onimoden ANIMA MOMM E E I N I N I M E S S i l l M S S I

Faces which are unrecogniz-able and tiny on Dataland asseen on the "world view"monitor become detailed andclear when zoomed-in uponon the large screen.

The level of visual detail in items pro-vided on the auxiliary world-map viewdeserves comment. In everyday life, weneed some minimal amount of visualdetail to identify an item that we havenever seen before. If an item is small insize, or small because distant, it can lacksufficient shape or figural detail to enableus to discern what it is. However, recog-nition of already familiar items requiresless figural detail. We readily recognizefriends way down a street, at distanceswhere visual acuity is insufficient to pro-vide facial details. We recognize them onthe basis of gait, relative size, generalshape or -visual envelope," as well as bycontext, that is, by expecting a figurewith such-and-such-color clothes toappear in such-and-such a place.

Or. take the example of a Coca-Cola signdown the street. At a distance at which wecannot distinguish the individual lettersof the sign, we yet recognize it as being a

"Coke" sign. The fact that it appears overa drugstore or variety store supportscontextually this act of recognition.

A similar principle holds in the area ofaudition. On a certain popular televisiongame show ("Name That Tune"), per-sons who are particularly well informedabout popular music as well as alert cansometimes identify ("recognize") a songon the basis of no more than its first twoor three notes. Consider, for example, thefirst two notes of the melody ''Auld LangSyne." The clue that the song about to beheard is associated with a holiday, thedistinctive tonal interval of the first twonotes of the song, can be sufficient todefine the intersection of context (NewYear's Eve) and tonalities, as, indeed,

"Auld Lang Syne.

The recognition situation, as opposed tothe identification situation, is what weanticipate to be the operating conditionfor the typical user of SDMS. That is, thearray of highly familiar items in Datalandis "searched," not by a deep visual anal-ysis of every item before one's view, butby relatively "shallow" visual processing,quite adequate for the recognition offamiliar items in familiar places.

- -

• = • E M I .

OPP

• M I E R . . A . 4 4 6 0 1 6morrow • ',es 110,BritMINEEIMINEFV I a r e IOW • . . . S I P ? Ao,

"• • am

•=.41mm Ammo ben ammo o . • • • • . • • • , m o m p l imoo dimmommol. limom 4 i l l o o m m e l ea l l . ' e s molisimo • • • • • • • B E I . 1=11IME E M r a l e l l , 1 1 1 1 0M I M B • MOS a M O • M O M adlEill• • I n I I P L M E W S ,

20

If a user has been working with a certainpersonal Dataland configuration forsome time, it is reasonable that all thatshould be needed on the auxiliary viewof Dataland is sufficient detail to recog-nize items readily on that surface. Withconstant usage certain items couldundergo further reduction in visual detail,even to the extent of becoming simplecolored squares, circles, or triangles.This extreme represents a "cueing" situ-ation, rather than a recognition situation.

Qualitatively, the figural resolution ofitems in Dataland as they appear on theworld-map monitor fall in the intermedi-ate range on the spectrum: cueing, rec-ognition, identification. That is, they servereadily for recognition by an accustomeduser, but not necessarily for identificationupon a first-time look by a new observer.

Perceptibility indeed differs somewhatfrom item to item. For example, theLAN DSAT satellite image does look likesome bit of geography; but it is not clearwhether, from a farthest-out view, it is aLAN DSAT view, a map, or a drawing.The user who knows what it actually is ofcourse readily identifies it and refers to itas "the LAN DSAT photo." Or, considerthe set of photographs in the southeastcorner of Dataland. They are sponta-neously classifiable as photos, but evenfor those who are personal acquaint-ances of the people pictured, the photosare ever so slightly too small and unde-tailed to permit positive identificationwithout zooming in. Nonetheless, it isclear from "afar" to any observer that theitems are in fact photos of faces.

Sound, too, can be an invaluable naviga-tional aid. In SDMS, space as cued bysound can have an implicit extent muchgreater than that bounded by the visual

"window" onto Dataland given by thelarge-screen view. Off-screen voices,becoming gradually audible as oneapproaches some item or area, canserve to orient and guide.

Two examples of sounds that we haveemployed include:

1an off-screen voice, in stereo, reciting

"This is MIT.. .This is MIT...," the voiceleading the traveler in Dataland first tothe vicinity of, and then to, a certain"glyph" bearing the initials MIT, whichwas the source of the sound;

2a gray telephone with white touch-tonebuttons (not activated for "dialing," butpotentially so), which rings louder andlouder upon approach.

These and other sounds did much tocreate a sense of "region" and "neigh-borhood," a feeling of proximity, much asthe bells, hammerings, traffic noise, andplayground chatter of the real world fillthings out. At moments in navigatingacross Dataland, depending upon theroute taken, sound can be the sole direc-tional cue, should one momentarily notbe consulting the "world view" auxiliarymonitor but watching the large screen.

We note in passing that the user is giveninteractive control of the loudness level ofsound through gesturing upon either ofthe small, touch-sensitive pads on thearms of the chair. Circling with the finger-tip in a counterclockwise direction atten-uates the volume of sound, and a circlinggesture clockwise causes volume toincrease.

With respect to navigation around Data-land, we have so far talked only aboutmovement over the surface of the planeof data at a constant "altitude." How thendoes the user manage to get at the datadetected at a distance?

The Northeastern Coast ofthe United States seen fromafar. A satellite image of theNew England coastline isinset at the edge.

I n S U O M I & IlkilLagiallikiI M I E , , E I T g r o w & m u mwpers terare w a r e sma, 11111•11Mano

• MPOP••••••Mr ••••-4 104

dmo • • • • • = , , N F O 4 0 0 • • • • • o w. o m m

•Emipmm. I m o • • • • • . • A m m i*OE JimmonmEn Minip Amu. I l o a d m i l m ldIII-11 o n A N L & 40.11••• 1 1 1 . 1101•6Sal "E womb M U MMEll = M I S • —Nab •1111111111111i 11110Mi

Po M P M O M ,E111111111-••• .•••.••m

Zooming-in upon Dataland'smap of Southeast Asia: suc-cessive images on this pagedepict a monochromatic mapbecoming larger and "block-ier" in texture. The imagethen gains both clarity andcolor (opposite, top) forscrolling about (opposite,middle, and bottom).

22

While the right-hand joystick guides lat-eral and up-down movements, the joy-stick mounted in the left arm of the mediaroom's control chair permits the user to

"zoom" in upon Dataland for a closerlook. Pushing forward on the left-handjoystick will cause zooming in; pullingback will cause zooming out. Uponzooming in, the image of the item on thelarge screen gains in size dramatically.The relative size of the "you-are-here"highlighted rectangle on the "world view"auxiliary monitor grows correspondinglysmaller, yet remains on view so that theuser will always know his location in thespatial context of Dataland.

For example, on the "world view" auxil-iary map, we can recognize a small,monochromatic map of Southeast Asialocated in the western part of Dataland,at about mid-latitude. Coming in closerupon the map by pressing forward onthe left-hand joystick makes detail morediscernible, as well as causing the car-tography to gain color.

"'" - v a t . • - . 7 z - 4 1 0 0 0 : 5••••

J 1 5 ;

I•or

• • • • • • •"11. - - •

I,IIC l a

u - -•-vil

isaLirl.1 ' rumitiw-- '.":or • . . . A r s m eStr•ativims•mtairm...111 d m , .•

e. -9 t o = ; g o....a•sai - On

g a i t -

,;;;Ittow,

mem' 1•111:111111sam444P4

A

Notice that the enlargement of the mapimage takes place in "stages." That is,the image upon zooming in becomeslarger in discrete increments, simultane-ously more "blocky" in appearance. At acertain point in zooming in, the blockyimage is replaced by a clear map imagethe same size and in the same registeras the previous blocky one.

The increasing "blockiness" of the imageaccompanying the increase in size dur-ing zooming in, requiring eventualreplacement with a clearer image, is aconsequence of the digital nature of thetelevision image and of the way the

"zoom" effect is achieved.

23

IMPRIMP • • • • • • • M • • = • • • • • • • • = 11 • 111.•••• •••. . • - • • • •

• • L i • O N E , 1 1 1 • • • • • • • •• • • • • • • • • • I M • l i m m o 1• • • • • • • • • • • 1 1 • • • • • • • • • • • • •

• • • • • • • • • • 110=11P• • • • • • • 1 0 • • • • • • • • •

— • • • • • • . . r— = N B

• - w a • . n o 16. • • • • • • . 'gamut••••• • • • m o m . • • • • • • • • l b * Sos • " M a w a l bo•I 0 0 11 0 111 6 L a i n • N b a C o w m a n e lN o o w • I M B • • • •••1111•16 a N d w a s I NO n S a m m o S • • • V I M I M M O , I S I O M ,• M r

24

In broadcast television, the "information"for the image, namely the brightness(luminance) and color (chrominance)information for each scan line compos-ing it, comes over the airwaves as ananalogue signal. With digital television,however, the information which modu-lates the image either is generated bycomputer computations or comesdirectly out of the computer's memorystore. The color television images dis-played in SDMS, with important excep-tions to be noted later, actually reside asstored information in an area of computermemory termed a "frame buffer."

The digital color image in SDMS can beconceived of logically as composed ofan array of tiny dots called "pictureelements," or "pixels." Each pixel corre-sponds to certain numeric values locat-ed in addressable slots in the framebuffer. Given this mosaic, a ready way toblow up the image for a zoom-like effectis to repeat (in the sense of "displayingagain") each picture element, in both thehorizontal and vertical dimensions of theimage, first once, then twice, then threetimes, and so forth. The repetition ofpixels is patterned from center screenoutward in emulation of a central-axis

"optical" zoom. The result visually is anegatively accelerating rate of imageexpansion.

The basic Dataland image, as seen fromthe highest "ceiling" or altitude, actuallyincorporates one "repeat" to begin with,partly to avoid the rather large visual

"lurch" inherent in the quadrupling ofimage size that occurs when each pixelis first repeated once vertically and hori-zontally. One initial "repeat" makes for asmoother total effect even though onlyhalf of the television screen's real resolu-tion is being used.

The increasing "blockiness" of the imageupon zooming in derives precisely fromthe squarish pattern of repetitions of theindividual pixels, the basic pixel beingvisually just a small dot of color. To coun-teract this side effect of the way weachieve zoom effects, the now larger andblockier image can be replaced by animage of the same size and in the sameregister, but with increased detail andclarity.

In practice, the zoom procedure we haveadopted calls for the replacement of thescaled-up, "blocky" image at about arepeat count of seven, so that the loss inclarity with increasing blockiness is nottoo great. With successive rounds ofrepeat count incrementing followed byimage replacements, a zooming in proc-ess could be as "magnifying" as onewished, the practical limitation being theresolution of the source image information.

Other techniques of zooming are underinvestigation, in particular interpolativezoom, which removes blockiness andjumpiness but manifests the same deg-radation in resolution.

w 1 1 1 am K e l l e y

On the large screen to theuser's front, image of WilliamKelley of our laboratory's staff,at first small (opposite page),becomes larger and -block-ier" as the user zooms-in (topand middle, this page). At acertain point in approach, theblocky image is replaced(instantaneously) with ahigher resolution image(bottom).

25

- l o m m e • I m m o &• • • • • • • • • 0 1 0 0 0 1 0 • 4 •

•••• m o o a . , . • • • • • •bao a m . • • • • • • 4 m E 4

••44 4 • E . • • • • Am. •4•4 • • • • • • 11•444•4•• •44,•••4 .4 • • 4 . • • • 4••••• Mob • • • bers • • • • • • • asMO MB M O W N . S S S . 6 0 U M W . 44••1•1114 N M • • • " M S * m o o n anstima.m.• o n o m m o I m o M N , • • • • • • • • /144• your = W M . • 4 M E M MOO INt11111•1•11, • • • • • • • • S N I P • •I M • • • • • 4••• • • • 444. IE . IN OOP •• •••••••••••

At a distance, and at lowresolution, this image onDataland is clearly a letter.Letterhead logos, despitethe coarseness of scale,generally can serve asreliable cues for sender'sidentity.

26

While it might at first seem that such dis-continuities in image sharpness duringzoom can Only be disadvantageous,there are circumstances in which a lackof low-order visual detail followed by thesudden introduction of finer detail makesgood sense in terms of information pre-sentation.

In the upper right, or northeast, corner ofDataland is an area allocated to Architec-ture Machine Group "correspondence"with our research sponsor. DARPA.In this area is a facsimile of a letter, com-plete with letterhead and signature.When approached, the letter indeedlooks like a letter but is as yet unreada-ble. At a certain threshold nearness, thetext suddenly gains in clarity andbecomes perfectly legible.

To zoom in upon an item of interest is to"address" that item for purposes ofinspection. For some items, simply tocome up to them will cause them to gainnot only in size but in information:increased detail, finer delineation, color,and, in some cases, sound and move-ment. To approach them is to "activate'them. With other items, some further userinitiative such as "touch selection" maybe involved as a part of a more interac-tive process of perusal.

mem. a m m o .

• • • • 11 ,1 . -• , • • • - • • • • • • • • •

• • • • • • • • • • • • • , • • -• • • • • • • • • • • - , , ,

• • • • . , M N ,

1111111111•1

— S , 1 0 11 • • • •

, O a , d • . A • • 4 . ,i l m t m l , • • • • • , m i M D • e s * , , m e d ,

4 1 1 1 , , r o d e m V W . , , , • , 4 6 0 . , 0 1 • 1 1 1 1 6 , M N I N E • - • • • • • • • • • • •• 6 6 M o m . . • • • , , , " l e n o I M O S k o a l b S 1•1100•1111111110oda Ino .d ran i %Ai. a I M O I lEmpliten• • M i l • l i n I N .

-mal M S N . • . 1 1 • 1 • 1 1 S S 111111111I O W 01111111111•111 • l e M O W NIEUWE. • M O M i t I O N I M

10 I M P O P E N E M M O O l • •41•11111111: . M o i r, ,V i t l f /EP g o l i o n l i M I N I N I . O M , / . • a O W . 0 0• • • • • e l ...N., . 4 w o w * *

6 , , • • • • • • • •

•

* IPVINIMIE 1111.11111111P •Mar iEt t i r i t I M M O011P I M E P 4111,1ML • • • •

▪ 111011812 - -111111POIMIE * W a g w r i t'111 .1=MEND4 w i n n e s i o n

1.111.1.1111.0. % p o o rVIPIRIEL woomppww•• v . .

A n n ,

1MM, • • • • •

ak• Lim • • • • • • • 1 , ono w r , m o I •%id M i l o am amomdeummielit dib N M

M a l = M a k Mem 1.11111,0 1.116111.111. 4 . 1 0A N N A 11 . 1 . SOS V I M 111111111116• IBS M O Oone anomena am on arms tionsom monN M U N E I I M E •at M I M S 4411101111111* V I I 11111111• E l l O P 111111 * O I L •11111PNIP. B M E W 11▪ M O P 0 . = O M W N W U N .111111•111111111

OS= d i r AMIE. U P • • • W I E F I • •

IMMINNEM••••• IMP =MPiip•••

1•1111111111111111011 110111111111111smommoupon =NOP -4 10- •

lionowns l o w i l l o o innammo= I m o M a n n t o t b

DEFENSE ADVANCEDImo

A n t HOG

D r . a l c h o l s • I t e g r o p o n t oD e p o r t m e n t o f A r c h i t e c t u r eM a s s a c h u s e t t s I n s t i t u t e o f77 M a s s a c h u s e t t s Av e n u eC a m b r i d g e . M a s s a c h u s e t t s

Zooming-in upon the letterimage results in legible text.

27

28

Perusing Data

Logically distinct from finding a certainitem in Dataland is perusing that item inplace. Once the user has found the itemand zoomed in on it, the specific formthat perusal takes is conditioned by thenature of the data.

In direct contrast to material that is "outthere," visible, to be developed in fullsimply by approach, are materials whichneed to be "perused," where perusalmeans opening up a volume, starting afilm clip, selecting part of a set of slidesto be seen, and so on. The essential ideais that the user opens up and developshis interchange with the data more orless interactively, through direct touch ina "hands-on" manner. The dictionarysense of "perusal" is that of "survey orscrutiny, in some thoroughness or detail."It is this engaged quality of person-datainterchange that we intend by thisterminology.

Our notion of perusal includes a class of"data types" which, when addressed,prove to be unusually rich in motion,color, and sound. One of these new datatypes is yet an old and familiar one:the "book."

The user zooms in on what appears fromafar to be simply a rectangle. At closeenough range, the color patch gainsdetail in the form of some horizontalblack rows of "characters" implying aprinted title. Upon even yet nearerapproach, the black figures give way toreadable text, giving the title of the item,which is now clearly seen as implying a

"book-like" image.

The user, of course, knows that the rec-tangle is a "book," having put it there inthe first place. However, if he is a sub-scriber to a computer-network-based

"book of the month" service, and hasperhaps requested or has a standingorder for books, the book may be one hehas yet to see, having just appeared onthe Dataland surface in a certain area onthe data plane serving as a "book drop"for incoming communications which arebook-like.

Upon nearest approach, when the imageof the book is fully addressed in the vis-ual sense of filling the large screen, a

"table of contents" for the book appearson the second and so far unused colormonitor to the user's left. This table-of-contents view is one of a class of interac-tive "key maps" made available to theuser as adjuncts to certain data types, topermit direct user interaction with thedata to be perused.

In this instance, the key map allows useraccess to the interior of the "book" ondisplay, exactly as the table of contentsof a "real" book lets the reader (a) knowwhat the book offers by way of topicscovered, and (b) serves as a "dispatch-ing" table to send the reader directly tothe subsection of the book that is of mostimmediate interest.

• I 1 1 1

stem

S O M S

WORKBOOK

OVERVIEWDATA BASEMODEL 8 5WORLD V IEWKEY MAPSSOUNDU T I L I T I E SMORE

29

User interacts by touch witha "table of contents" keymap for book-like data types.

In close-up of this key mapbelow, note touchable "tabs"aligned with heading taglines.

Upon touch indicating chap-ter heading of SOUND (top),the heading expands (bot-tom) to offer sub-headingsunder SOUND.

30

Lk . ! E AT A S A S E

7.) Q I E W"APS

teZ,I T I L I T I E S"ORE

° S O U N D•

1 M E M O F e w , R I "

S 0 M S — E O U A L I Z AT /Omf —STEPEO v S—0EPTm OF SC..—CROSS TALF

2 ) HAP0mAF E P E P '71GT— m I EPuPTS—DOPPL EP ( F P S

3 ) S O U N D ' o p t , '—01Sk S I Z E—DATA PATES

Notice that the table of contents can"expand" to show a finer breakdown ofcontent area when the user selects amajor heading of interest. In concert withthe activity on the key map monitorscreen, the main large-screen view will

"go to" the section of the book selectedby the user through the interactivekey map so that the user may read thematerial.

There is yet further interaction with thedata type of book: that of turning itspages. This action is initiated by a spe-cific user action: a page-turning gesturegiven as a right-to-left, top-to-bottomstroke on either of the touch-sensitivepads on the arms of the user chair. Anypage can be turned back by a strokeacross the pad in the opposite direction.The accompanying visual action is thedisplay of an -animation" of a page actu-ally turning. The upper right-hand cornerof the page on view progressively "lifts"away and sweeps leftwards across thescreen, immediately revealing the newpage below.

ASCII TEXT DISPLAYThis i s a page o f ASCII t e x t

d isp layed by t h e new 85 f o n t system.The f o n t master i s s tored " o f f screen"s t a r t i n g a t page 0 o f t h e extendedmemory. A s t h e ASCII t e x t i s parsed,each c h a r a c t e r i s "moved", v i a t h emicrocode, t o t h e app rop r ia te spot i nthe c u r r e n t l y v i s i b l e space.

The f o n t s a re two b i t s pe r p i x e l .Associated w i t h each fon t i s al o c a t i o n t a b l e g i v i ng i t sc000rd inates on t h e i n v i s i b l eread- image. A l s o w i t h the f o n t i s aone d imens iona l kern ing t a b l e . T h ekern ing t a b l e i s s tored i n programmemory du r i ng t h e fo rmat t ing phaseof d i s p l a y.

The page i s f l i p p e d by scratch ing

This i sd isp layed by t h e new 85 f oThe f o n t master i s s toreds t a r t i n g a t page 0 o f t h e ememory. A s t h e ASCII t e x teach c h a r a c t e r i s "moved",microcode, t o t h e a p p r o p r ithe c u r r e n t l y v i s i b l e space

The f o n t s a re two b i t sAssociated w i t h each f o n t il o c a t i o n t a b l e g i v i ng i t sc000rd inates on t h e i n v i s i bread- image. A l s o w i t h t h eone d imens iona l kern ing t a bkern ing t a b l e i s s to red i nmemory du r i ng t h e f o r m a t t i

d i s p l a y.The page i s f l i p p e d by

p r o p r i a t e d i r ec t i on on at . i v e o y - p a d . A "page"

ofd isp l o wThe Eon p e r p i x e ls t a r t i n g a tmemory. A s t h e ASeach c h a r a c t e r i smicrocode, t o t he athe c u r r e n t l y v i s i b

The f o n t s a re tw • l a t eAssociated w i t h eac u r b i t s o fl o c a t i o n t a b l e g i v i f y l n g t h ec000rd inates on t h e n e d above,read- image. A l s o w l a t e areone d imens iona l ker a n dkern ing t a b l e i s s to p a g ememory d u r i n g t h e f i sof d i s p l a y.

The page i s f l i

, B y• p l a y

t o

he a p p r o p r i a t e d i r e c t i o n on ah - s e n s i t i v e joy-pad. A "page"

l y cons is ts o f two pages o fers , one occupying t h e low

b i t s ( o f t he 8 b i t p e r p i x e lgo) t h e o the r

two b i t s . B yt r i x , d i s p l a yone page t o

Stmemeachmicrocot he c u r r e n t 1

The f o n t sAssociated w il o c a t i o n t a b lc000rd ina tesread- image.one d imens ionkern ing t a b l ememory du r i ng

d i s p l a y.The page

the a p p r o p r i a t e d i rec t i on on at o u c h - s e n s i t i v e joy-pad. A "page"a c t u a l l y consists o f two pages o f

ha rac te rs , one occupying the lowrde r t w o b i t s ( o f t he 8 b i t pe r p i x e l

p l a y image) and the o therupy ing t h e next two b i t s . B y

ing t h e co lo r mat r i x , d i s p l a ybe swi tched from one page t o

reonkermeof

are a t emp la terder f o u r b i t s o fy modi fy ing t h ementioned above,he temp la te a re

adows back ande. A s t h e page

exposed i sshadows.

rent page i s

shadows are a temp la tehe h igh-o rder f ou r b i t s o f

image. B y modify ing t h echanges mentioned above,nts o f t h e templa te arethe shadows back and

the page. A s t h e pagege being exposed i s

h the shadows.the cur rent page i s

Fl Initial page of text (left, top)begins to "turn" (left, middle)upon user indication (astroke of the finger) ontouch-sensitive pad on armof user chair.

The next page of text isprogressively revealed asthe "page" edge sweepsleftwards, and is almostentirely on view near the endof the page animationsequence (right, bottom).

31

32

The image of the -televisionset- serves as locale markerfor television data types onDataland. On the tube face ofthe set can be a "programguide", or, as in the exampleon the following pages, a pic-torial label which becomesdynamic upon approach.

The intent is not to give a nostalgicimpression of the way books used tolook, like the electric fireplace with plasticback-lit logs. The purpose is quite func-tional. First, the page is a unit of appre-hension of the book; it is a "bite-sized"chunk or packet of the material that onefeels one can take in comfortably, incontrast to the stretching-ever-onward,rhythmless character of scrolled text.Second, the page serves as a progressmarker through the text in a "milepost"sense. With continuous scrolling, suchnatural, discrete units of progress are notexplicit in the medium, and, if felt neces-sary by the reader, must be effortfullyabstracted, resulting in a more burden-some tone to the reading task.

Thus, the "page-flipper" is not frivolous. Itprovides the user with feedback that apage has turned, the directionality of theanimation corresponding to forward orbackward. Specifically, it reinstates forthe user, in the context of the reading ofcomputer-television text, the functionalequivalent of what the fact of the pagegives to the reader of traditional pagedprinted media: the sense of knowingwhere you are in the material. More gen-erally, it is an instance of what we chooseto term a "media fiducial," a markingsystem, or built-in feature of any mediumthat serves the function of letting youknow "where you are" based on loca-tional cues arising out of the basic struc-ture of the medium itself.

One radical departure from the ordi-nary concept of the book, except asadumbrated in such page-corner flip-animations as this report has in its upperleft-hand page corners, is the fact thatwhile some page illustrations will be stillslides, other illustrations will be "movies,"or animated diagrams, with or withoutsound accompaniments. The possibilityof sound can, upon occasion, permit thebook to comment upon its own text andillustrations, and even read itself to you.

Another unusual data type residing inDataland is that of television. An instanceof this is the presence of a "glyph" on theDataland surface which is the image of aSony television. Zoomed in on, the televi-sion set becomes very large, its ownscreen filling the large screen directlybefore the user. What is at first a rathersketchy black-and-white image on thetube face of this "virtual television" isthen replaced with live action: "Columbo,"

"The Sting," or a documentary on how tostop smoking.

What may be seen on this television setin Dataland is anything that can be puton television, live or recorded, closed-circuit or broadcast. The user could, forexample, indicate in the context of a ses-sion with SDMS that he next wants to viewthe twelve o'clock news, or see a rerunof some footage that was taped earlier.

C ) • • • _ 7 .

Zooming-in approach to"television set" on Dataland(left column), enlarges thetelevision set's screen tobecome effectively as largeas the screen to the user'sfront (right column). The

-blocky" digital imagebecomes replaced by adynamic image in colorfrom videodisc, launchinginto continuous live action(opposite page).

34

35

36

Since recorded sequences that might beviewed on the television are processestaking place over time, the key map forinteraction with sequences on the virtualtelevision set makes available to the userways of controlling events that areoriented over a time span.

An example of such a key map is shownin the accompanying illustration. Centralto the key map is a "dial," whose circum-ference indicates the time span to becovered by the entire movie sequence,with a pointer hand showing the currenttime in passage. One revolution of thissemantic clock can be seconds, min-utes, or hours. The total time-to-go is dis-played by a digital counter on the keymap; minutes and seconds elapsed arealso displayed digitally.

Sliding-bar cursors toward the bottom ofthis key map allow the user to speed upor slow down the rate at which material isshown, as well as to have it shown in aforward or backward sequence. Thematerial can be simply a "movie," inwhich case it probably makes little senseto show backward slow motion. But ifthe material happens to be an instruc-tional movie, for example, about how totie a sailing knot, or do a sleight-of-handcard trick, then this interactive ability toregulate time can be invaluable. Thematerial to be shown in such a formatcan be, in general, any material that canbe viewed visually and sequentially:movies, television, animations, slidesequences, as well as any electronicproduct of television mixing and specialeffects.

User interaction with "dial"key map: user sets pace tobe somewhat slower thannormal (top. left). Sequenceis stopped (freeze frame) byputting finger on pointer ofdial (middle), and resumeswhen finger is lifted (bottom,right).

37

MCA Discovision IndustrialVideodisc Player—ModelP7820.

38

The visual repertoire of our virtual televi-sion screen, as well as other "slide col-lections" about Dataland, is markedlyaugmented by optical videodisc backup.Either side of this laser-scanned disk canhold up to 54,000 individual frames. Anyframe can be individually addressed(freeze frame) or arbitrarily selected (ran-dom access), or sequences of framescan be exhibited forward or backward atvarying rates. The optical videodisk asstorage vehicle has a dramatic costimpact on SDMS, inasmuch as theplayer costs less than $2500.

In a more general sense, the television"set" is an instance of any of a number ofvirtual "screens" which can reside in thegeography of Dataland. The type of vis-ual symbology or imagery surroundingany such screen indicates its usage: a

"drive-in screen for movies," for example.What we are discussing here is a generalcapacity for "filmic" types of data, liveand recorded, which are addressable inDataland in user-designated locales.

One of the most dramatic visual experi-ences on our experimental Datalandsurface is that provided by LAN DSATphotographs of about four thousandsquare miles of the New England coast,from the Boston area up into Maine.

In the northwest corner of our experi-mental Dataland, we have an image of thenortheast sector of the United States,with an inset area of the New Englandcoast. Upon traveling across Dataland tothis patch of New England, it is possibleto zoom in on any area.

The perusal of the LAN DSAT image hasits own appropriate key map. It takes theform of a view on the key map monitorof, in this case, the entire sector of NewEngland. The "key" consists of a high-lighted rectangle moving around thiskey map image, tracking the course ofthe joystick-guided view visible on thelarge screen. Like the world map view, itoffers direct teleportation about theLANDSAT image through touch.

•71

Zooming-in upon satelliteimage of Boston: In theseviews, approach is from thenorth over Cape Ann andMarblehead area. In first col-umn, top-to-bottom, blockyimage is replaced with finerresolution view (middle).while image expands andtravel proceeds southward.In top and middle panels ofthe second column,approach continues over thepeninsula of Nahant, with yetanother image replacement(bottom).

40

Nearing Boston, the triangu-lar runway patterns of LoganInternational Airport comeinto sight at lower left (leftcolumn). Finally, central Bos-ton and Charles River basinappear (right column, topand middle).

To this point, all images areretrieved from magneticmedia. The final image, amap, replaces the satelliteimagery; it comes off theoptical videodisc.

41

42

Zooming-in upon Boston. As the user zooms in by a certain amount,the increasingly blocky material isreplaced by a higher-resolution image inperfect registration. This replacementaction is performed a number of times asone comes in on any portion of the NewEngland area covered by the LAN DSATimage and gives a dramatic impressionof vast areas being contained within theconfines of Dataland. In the sequence ofpictures shown (coming over Marble-head toward Boston), the final limit of res-olution is the resolution at which theoriginal LANDSAT material was photo-graphed and digitized. Each point ofsource data in the stored image repre-sents about an acre in area.

Should we zoom in on the city of Boston,we would find that the LANDSAT imageof Boston is replaced at a certain level oraltitude of approach by a map of Bostonin the same size and register as theLANDSAT image it replaces. Then, as wezoom in on the Beacon Hill area of Bos-ton, the map image suddenly gives wayto a set of slide views of the central cityarea. The slide collection contains over200 slides of Boston as seen from vari-ous vantage points. There is a key mapwhich aids the user in perusing the col-lection of slides.

Close-up of slide collectionkey map used to accessvideodisc still frames.Region along spine ofstacked "slides" permitssliding of cursor arrow byfingertip. Alternately, thecaptions "Next slide" and

"Previous slide" can betapped for one-by-one viewing,or continuously touched forrapid sequential showing.

44

44,Nziat i,1*E1 Gala ii:M.1*

:It:1441MM

,

-

• OLIS- :

The interactive aspect of this key map isthat the user can slide a colored-arrow

"cursor" along the top edge of the diago-nal row of "boxes" representing slidesabout Boston available to be seen. Theanalogue for more conventional media isthat of remembering a slide tray. Movingthe cursor up and down c4uses slides tobe flashed in succession oNt he largescreen as indicated by the finger touch-ing and moving the display cursor.

Perusing slides of BeaconHill area of Boston: slides 40and 170 are shown beingselected out for viewing.

45

0 - -

The "active process,- in thiscase a calculator, is i l lustrat-ed by the photographicimage on the large screen.The user can avail h imself ofa closer, touch-sensit ive ver-sion, which can be invokedas a virtual machine. Otherexamples might include: atelephone, a dictaphone,even a television.

46

To recapitulate, the first key map wasthe "table of contents" of the "book,"allowing the user access to selected sec-tions of the text. The second type of keymap, exemplified by the time-counterdial associated with the "television set,"enables user interaction with what are inessence processes over time. The thirdtype, the LANDSAT image key map,functions with respect to the LANDSATimage as the "world view" auxiliary moni-tor functions with respect to Dataland atlarge: a "you-are-here" marker, respond-ing to user joystick navigation about aspace. The key map associated with the

"slide collection," residing at the map ofBoston within the New England LAND-SAT image, exemplifies the fourth type,used to manage data types which areessentially characterizable by "number":some ordered series of items, any one ofwhich the user may dwell on for someunspecified period of time.

We expect the repertoire of key maptypes to grow along with enlargements ofthe repertoire of data types, as new datatypes require ordering by some principlenot accounted for by any previouslydefined key map.

Further among data types in Datalandthat depart from the usual are those thatare processes. That is, SDMS's Datalandhas nodes on its surface, indicated byappropriate glyphs, which make certainfunctions available to the user.

A specious example is the existence inDataland of the image of a hand-heldcalculator. The user can zoom directly upto it. There then appears on the small keymap monitor a copy of this calculatorimage. The user can directly touch theimage surface on the monitor to inputarithmetic problems, just as with anactual calculator. This particular exampleof a process, perhaps trivial or "cute,"points the way to what will eventually bean interesting and powerful class ofdata types.

I 2 1 . 4 4 7 6 1

MR

' R A M S9f2

gE4g6211§§111

RIP

When making graphicalannotations user enterscommentary and jottingswith lapboard and stylus.

48

Noted was the idea that "perusing"implies the scrutiny in detail of items. Thisscrutiny in detail as an interactive proc-ess surely will trigger in the user theimpulse to annotate, to append jottings,to comment "marginally." The traditionalproperties of computer media typicallyfrustrate this impulse, one so readilyindulged in on paper.

As an antidote to this shortcoming, wehave introduced the capacity for makingwritten annotations in "transparent ink,"of varying line widths and colors. The ink,because transparent, does not obliteratematerial overwritten, and the notationsmade can be transient or permanent asthe user wishes.

Notes and commentary can be not onlyin written form, but auditory as well. Theuser can employ the stylus and tablet toindicate an "anchorpoint" on the Data-land surface for to-be-recorded voicenotes. The notes are played back subse-quently upon coming within some criticalrange of the site of the recording.

Sound can be itself a data type as well asa vehicle for annotations or navigationalaid. An instance of sound as data in ourinitial SDMS prototype was a digitallystored speech of Jerome Wiesner, Presi-dent of MIT. A portion of this speech wasactivated and played back upon approachto an image of Dr. Wiesner, the volumeincreasing as one drew nearer by zoom-ing in, or fading to the left or right as onewithdrew from face-center toward theright or left, respectively. The speechsound was, in effect, synchronized withthe location of the speech.

CNEFOMBE s o v A n c E D RESEARCHLace N I L S O N S O U L / id

AM I. 9 I G TO N v i It WAWA

D r. N i c h o l a s N e g r 0 0 0 n t eD e p o r t m e n t o f A r r h i t o c t u r e

M e e s e c h u m e t t e I n e t i t u t e o f I / w o n d e r : ,77 M e n e a c h u a t e t t e AVV21,40C e i b r i d n e . M e s e e c h u m e t t s 0 2 1 . 1 9

Dear I N A * :

Dr N i c h o l a s NearoponteDeparcamot o f A r c h i t e c t u r eMILSOAChumette I n s t i t u t e o f77 Nassachumette AvenueC a m b r i d g e , M e s e a c h u e e t t o

Omar b l e b s

Dr - M i xDspartWassect

o r et i t u t e o f

11 Vase u s e t t s hvenueCambridge, Maasachuset ts

Dear t , t ek!-

V

ADVANCED RESEARCHlase vion.11010 001111.1V

fltinSIA

Te c h n o l o g y

30v101::::DowiLso., 801•11-SVIM VGOMM

RESEARCH

ccTe c h n o l o g y

An octaphonic sound capability permitsthe creation in the media room of a veri-table "cocktail party" of sounds andvoices, any sound so re-created as togive it a spatially definite apparentsource in the room. Endowing soundsources with specific locales effectivelycreates a set of "channels," aiding sepa-ration and identification of a number ofsimultaneously heard voices. The possi-bility of "browsing" through sets ofauditory messages, redundantly cued invisual graphics, can make sound as adata type or means of annotation a richoffering to the user.

The ensemble of data types and asso-ciated interactions enumerated hereform an initial set and by no meansexhaust the long-term potentialities ofuser-system interaction. Before we turnto a consideration of what the future mayhold, however, let us set down some ofour major findings thus far about usingspace to organize data.

"Ink" is transparent, anddoes not obliterate materialbeneath. Note ink coloroptions, and color-strip

"highlighting" over addressin letterhead and in saluta-tion. Ink can be temporary,permanent, or placed inlogically separable layers.

49

101111•1111m-

IRPA

—%a

Diagram of spatial structureof Dataland in SDMS I, ourinitial SDMS prototype.

50

What We Learned

A major lesson of our experience inattempting to use space to articulateinformation was the virtue of straightfor-ward simplicity in the presentational as-pects of the user's "virtual" spatial world.

The system as outlined above did notspring into existence all at once butresulted from many years of Office ofNaval Research research in television-based computer graphics and an initialversion of a spatial data-managementsystem, SDMS I, developed over theproject's first year of DARPA support.

SDMS I was a "2-1/2-D" spatial world: aset of 2-D data planes, so linked as todefine a hierarchical, "laminar" space.The top level of this structure was Data-land itself: a large data plane with "items"of various sorts arranged on its surface,some grouped on common backgroundcolors. As described earlier, the largescreen in front of the user functioned as a

"window" onto Dataland, which appearedin its entirety on the small monitor.

Importantly, the scheme of the dataspace in SDMS I included a system of

"ports" on any data plane or layer, which,when zoomed in on, would give way to anext data layer, seemingly in back of or

"beneath- it. In the accompanying illus-tration, we can see "ports" leadIng fromthe top Dataland layer to different,second-level data planes. One of thesecond-level data planes in the illustra-tion is a terminal node, but the othersecond-level data planes themselveshave ports, each leading down to third-level data planes. The information con-tained on the lower planes was logicallysubordinate to the topic at the portthrough which the user dove.

Travel around this system was up anddown, chaining through the hierarchicalstructure of the nodes (ports) on the vari-ous data planes. One could travel deepdown into a certain data structure andthen "chain back" through the structureto get back to the point of entry.

52

Our SDMS I system was programmed toinclude up to twenty ports in any plane,and the "depth" of the structure in levelswas in principle indefinite. As a practicalmatter, in our demonstration version thedeepest "depth" in chaining down a hier-archy was about seven levels.

The spatial structure of SDMS I, however,generated a frustrating design paradoxfor any comprehensive data-world imagefor display on the "world view" monitor.

Although a number of candidate schemesfor mapping views were advanced,including quite ingenious neural net-likeschemes, some serious drawbackseemed always to attend them. Thosethat were visually simple and easy tointerpret were so at the expense of infor-mation content. Those that were compre-hensive became "baroque": the moreaccurately they portrayed the rathercomplex paths that some user might takethrough a data space consisting ofnested planes, the more the mappingview violated the principle that a displaysystem be visually straightforward andeasily interpreted.

The resolution of the dilemma was theinsight that both the finding of items andthe navigation in data space could bestbe served by a simple, single-planeDataland. This sort of space seemed tooffer all the primary benefits of locatabil-ity to be derived from the SDMS principlewithout having all the disadvantages ofcomplexity that a laminar space mightentail.

Thus, in our second prototype, SDMS II,the nested laminar space of SDMS Iwas replaced by a single, very largedata plane. When approaching an itemthrough the joystick "zooming" action,the user does not "pop through a port"down to another level of data space toobtain further information. Rather, oncethe viewer has come up close upon theitem, the action from his or her point ofview is then to "peruse" the item in place,as it were, with respect to the greaterspace of Dataland. That is, the perusal ofany item occurs without any sense of fur-ther "travel" in data space of the sortimplicit in going down through ports todeeper layers of data.

As a result, in SDMS II, locating an item,and perusal of that item once located,are, logically, two independent activities.In SDMS I, these activities were con-founded each with the other, in that asyou looked at more information, you alsopopped through down and deeper into

"location space" as well.

What about an SDMS in 3-D, with 3-D"flight"?

At our project's inception, the possibilityof creating a spatial data-managementsystem in interactive 3-D space in themanner of a flight simulator wasbroached. The idea was an intriguingone: flying in and around a space of dataitems under the guidance of a variety ofaviation-inspired navigational aids. Forpractical reasons, the idea was not pur-sued: acquisition of real-time flight simu-lation hardware, with visible surface por-trayal of a 3-0 world in color, would havefar exceeded cost parameters for ourproject. But, apart from costs, there arefactors intrinsic to dynamic 3-D flight tobe considered in relation to the ends tobe achieved in an SDMS.

While flight in simulated 3-D space canbe dramatic and compelling, to theextent that 3-D flight is convincinglysimulated, then to that extent the possi-bilities are expanded for becoming con-fused in that space. Spatiality, which inthe SDMS context is supposed to be inthe service of managing data, can thenitself become so rich in directionalitiesthat it generates sets of navigationalissues which in their turn demandresolution.

Beyond the issue of user orientation inreal-time 3-D flight is the problem of therecognition of objects in a dynamic 3-Dworld. The ability to approach any itemin a 3-D space, from any angle, canseriously compromise the ready inden-tification of that item. Things simply donot look the same from different angles.The perception that a certain item is the

"same" when seen from varying aspects(perceptual "shape constancy") isrobust, to be sure, but it can and willbreak down.

53

54

Not only does the internal structure of afigure affect its recognition, but the "top,"

"bottom," and "sides" as determined bythe viewer's relative orientation tend tocondition how it is seen, and whether it isrecognized or not. For example, even a2-D silhouette map of Africa turned 90degrees is much less readily identifiableas when seen in its usual north-south ori-entation (Rock, I., "The Perception of Dis-oriented Figures," Scientific American,January 1974).

To have all items always "face front"independent of the user's relative path ofapproach in flight is not necessarily acure for the object recognition problem.The appearance of items taken individu-ally is only a part of perceiving where youare and what you are looking at. Inter-item relational patterns, a significant con-textual basis both for recognizing thingsand for navigating in "neighborhoods" ofthings, become effectively different pat-terns when a grouping of items isapproached from different directions.

To summarize this point: the special vir-tue of a spatial data management systemlies not in a sense of space and flight fortheir own sakes but in the invocation of asufficient sense of space to activatethe native, highly evolved capacity ofhumans to organize their perceptualexperience spatially. The optimal spacefor an SDMS, then, is one that promptsspatial "schematizing" in the user, whileminimizing the potential for disorientation.

Experience indicates that the majority ofguest users trying out their hand at "fly-ing" across Dataland for the first timeinvariably depend on the auxiliary view tonavigate, using the large screen for

"confirmatory" glances that they havearrived where they set out to go. Theapparent aspect of Dataland is so largethat this auxiliary view is vital for success-ful navigation, even for the constant, well-practiced user who may have, over time,gained a very detailed mental picture ofthe "lay of the land."

It turns out to be the case that navigationover Dataland with the joystick is soeminently and inherently "response com-patible" that the novice user gains profi-ciency in flight in well under a minute.One of the prime attractions of such asystem is the readiness with which thenew user becomes a competent driver.

Accompanying the rapid acquisition of"piloting" proficiency is rapid assimilationby the novice of the graphic imagery ofthe Dataland surface in a "spatial" wayAlmost immediately, the new user beginsto refer to items by describing them as

"the letter to the right and north of thegreen square," or "the yellow rectanglesin the southeast corner." Although, ofcourse, this way of regarding data isintended in the system, the beginning ofdiscussing data in this mode by noviceusers is gratifyingly spontaneous.

Beyond sound as a data type and asan annotating modality, our work withsound as a "navigational aid" has builtan appreciation of its utility for guidanceand orientation, as well as for its enrich-ing function as a redundant cue for whatis going on visually. Those occasionswhen our own experimental sound sys-tem has been temporarily down haveconvinced us that we would be loath toreturn to a totally silent data world. Con-clusion: sound, well-orchestrated, is cru-cial. A spatial data-management systemmust have it.

While we learned much concerning spe-cific design features of our SDMS effort,realizations also emerged about furtherpotentialities of the spatial theme. Someof these indications for future directionsare set forth in the following section.

55

56

Future Directions

Progress in SDMS during our first twoyears of pursuing spatiality as a motif hasencouraged us to chart a prospectus ofnext steps that is advisedly extreme interms of positions taken.

Future directions elucidated include thoseof: topical spatiality, gaze-contingentdisplaying, eye-gesture coordination,voice-driving with eye contact, exploit-ing the user's near-field, user positionsensing through a space-sensored "labjacket."

The idea underlying "topical spatiality" issimple: the topography of Dataland isorganized on the fly, as a function of userselected axes. These axes are chosenby the user on the basis of what datadimensions are judged appropriate forsome specific task. In other words, unlikethe current arrangement, pictograms arenot stored and left in one place, but formpart of a changing 2-D space.

As an example, consider data typical ofa company's personnel file. Pictures ofemployees would reside on secondarystorage media, and Dataland wouldhave an indicative but blank array of rec-tangles. The user might then specify hishorizontal axis to be a measure of ageand the vertical axis as something likeyears of service, salary level, or manage-rial responsibility. Once selected, the pic-tures would be ordered on the fly in this2-D space, with overlap and clustering,most probably. The data would besearched with the knowledge that Mr.Smith (the company's founding father) issomewhere at upper right and the deliv-ery boy is in the southwest.

We propose to implement a topicalSDMS, within the existing system, using

7 b u i l d i n g types. The selection of datacomes from our 54,000-different-framevideodisc that was made in collaborationwith the MIT architecture slide libraryand contains about 20,000 slides ofAmerican Architecture. At this writing,those "frames" are additionally describedin a data structure which includes state,county, city, architects' names, dates ofconstruction, and twenty building types.The data lend themselves to traditionalspatiality (i.e., they can be viewed on amap of the US), but they also fall ontononspatial axes like date of construction,physical scale, degree of public access,and the like. In the case of date and size(X and Y, respectively), if we limit ourselection to post-Civil War, we can lookfor the World Trade Center in the north-east and Grant's tomb at the lower left.

s•

This theme will include unclustering pro-cedures that provide the proper chang-ing, nonlinear axes while the user zooms,in order to "spread out" overlappingdata. What we should learn are lessonsconcerning people's ability to createmental spaces and then to search them.

A critical focus of further researchinvolves a set of themes having to dowith the active engagement of the user inthe environmental space of the interface,expanding the spatial domain to encom-pass the real-space characteristics ofuser actions. The overall intention is torender the system cognizant of how theuser is operating in his personal space,as this space intersects with the sharedspace of the interface: directionalities inlooking, speaking, touching, gesturing,together with a system awareness ofuser position and proximities.

Among these themes is the awareness ofwhere the user is looking. Eye contact isan important adjunct in human interper-sonal exchange and, in general, a primeindicator of the focus of active awareness.

57

58

The opportunity here is for, among otherthings, an eye-controlled "virtual cam-era," that is, a display that points, pans,zooms, travels, as a function of where theuser is looking. In the context of theimagery in SDMS so far discussed, pos-sibilities might include even the trivialexample of a spatially dispersed set ofitems, perhaps faces, each of which

"talks" when eye-addressed by the user,the directed gaze used by the system asan implicit "command" or act of selec-tion. Another example would be animage, say a map, which locally expandsand gains detail as a function of the pointand duration of "looks".

The sense of space in and at the inter-face includes system awareness of theuser near-field. The tone of appointmentsin the media room, especially the open-ness of the chair, invites the user to getup out of the chair the more immediatelyto engage himself with the data. Ges-turing, arm waving, pointing, especiallyin conjunction with line-of-sight indica-tions captured through unobtrusive eyetracking, would endow the immediatespace between user and system withcommunicative meaning.

Space-ranging techniques that permitthe system to know where the user is inthe space of the interface, plus the user'sbody attitude, have intrinsic uses. A sys-tem of body-borne space sensors, unob-trusively disposed about a sort of "labjacket" as buttons, cuff links, or epaulettes,would establish a computer-interpretabledot figure demarking the momentaryposition and attitude of the user. Amongother functions, such knowledge canhelp to re-establish tracking contact of anerrant oculometer, making sensingwhere one is looking a more realisticproposition in a dynamic environment.

With body sensing in the user near-field,pushing items of virtual data about in anow-tangible 3-D space arises as a pos-sibility. Or, traffic-patrol-like direction ofitems streaming by in patterns for on-the-fly perusal. The dynamics of orchestra-tion. literally like those of conducting asymphony, arise as realities, especiallypertinent in an environment of spatiallydistinct sounds and voices in an octa-phonic surround. Pointing to virtualtalkers with indications to speak, to hurryup, to pause, becomes repertorial for theuser, now "fleshed out" at least minimallyin dot form in the augmented space ofthe interface.

Voice direction with eye contact, a natu-ral combination in the extra system realworld, persist in the realm of user-systeminterchange as a viable modality. Theability to voice-activate takes on moreprecise meaning and possibilities in con-junction with the ability to address byvoice and eye simultaneously in a coor-dinated fashion. To be able to point bygesture as well defines yet an additionaldimension of voice-and-eye dynamics.

These themes, somewhat synopticallyexpressed, implicate actions in anexplicit space, coextensive with the realroom space at the user-system interface,a shared space which can be artfullyintegrated with the virtual spaces whichhave been the focus of this, our work,so far.

At stake here is a user-system interface,a data space, so enriched and quick-ened that the human becomes morethan a dimensionless point, the space atthe interface something other than emptyvolumes. The long-term goal is theutmost in usability, where the user's nat-ural modalities and capacities for commu-nicative expression meet with perceptive,complementary system response.

59

,

Copyright ©1979 by theMassachusetts Institute ofTechnology. All rightsreserved. The contents ofthis publication may not bereproduced in any form with-out permission of copyrightowner.

Printed in the United Statesof America

Library of Congress Number78-78256

6 0

The views and conclusions in this docu-ment are those of the author, and shouldnot be interpreted as necessarily repre-senting the official policies, express orimplied, of the Defense AdvancedResearch Projects Agency, or the UnitedStates Government.

Work Sponsored byDefense Advanced Research ProjectsAgencyOffice of Cybernetics TechnologyCommand Systems Cybernetics ProgramContract number: MDA903-77-C-0037Contract period: 1 October 1976 through30 September 1978

Work Performed byArchitecture Machine GroupMassachusetts Institute of TechnologyPrincipal Investigator:Professor Nicholas Negroponte

Distribution:Unlimited

Key Words:Management Information SystemsComputer GraphicsMan-Machine StudiesCommunications Media

Printing:This volume has been printed in an edi-tion of 4000 copies. Supplemental fundsfor printing have been obtained from:Office of Naval ResearchIBM CorporationTektronix CorporationBell Northern ResearchMIT Industrial Liaison Program

Photography by Christian LischewskiDesigned by Betsy Hacker,MIT Design ServicesComposed in Helvetica Light andMedium by Typographic House, Inc.Printed by Rapid Service PressBound by Star Bindery