. ED 241/.720

AUTHORTITLE

INSTITUTIONSPONS AGENCY

REPOT NOPUB DATECONTRACTNOTEPUB TYPE

EDRa.PRICEDESCUIPTORS

DOCUMENT RESUMEI,

CE 038 439

Kieras, David E.; And Others,How Experts and NonAperts Operate Electronic 4.

Equipment frodAnstructiohi. Technical Report No.14.

N

.Arizona Univ., Tucson, liept.of Psy chology.Office of Naval Research, Arlington, Va. Personneland Training Research Programs Office.UARZ/DP/TR-83/0NR-141p Feb 84N00014-81-C-069947p.

Reports - Research Technical (143),

MF01/PCO2 Plus Postage.*Electronic Eggipment; Instruction; *InstructionalMaterials; *Media.Selection; .*Military Trafning;Outcoips of'Educiption; Postsecondary EducatiOn;.Problem Solving; Teaching Methods; *TrainingMeth ds

IDENTIFIERS *Exp rts; *Following Directions; Navy

ABSTRAGTThree. questions were addressed in a Navy experiment

in which subjects followed instructions to complete tasks involvingseveral pieces of electronic equipmedt. First, two instructionalformats were compared; a hierarchical menu format containing naturalchunks of instruction was not superior overall` to.a simple-step-by-step instructional format. The menu format was superior onlyif the subject was familiar with the type of delvice end was sometimessubstantially inferior otherWs*. Second, experts, were'compared tononexpertsl'found to be faslpr 6verall, and able t000perate equipmentwith fewer instructions in fhe menu condition. They were also fasterwhen complex physical actions were involved. Thus, there were bothspecific and general effects of expertise. Finally, evidence wasOpught that knowledge of how to operate equipment was schematic. Itwas expected that, when subjects in the menu format conditionoperated a device without selectinglany instructions to-read, theirsequelice of action should correspond to stereotyped schema-likepatterns. This Occurredt,only weakly, suggesting that even expertsoperate everyday deVices in a Problenr-solving mode, rather than byretrieved complete procedures. (Author/KC)

o

4

4.

(

1'

li***********k*************************************i********************* Reproductions supplied by EDRS are the best that an be made *

.*. from the original document. *

***************************************************.***********k********.

14f

$

1'

t

4.

1

a

:41fte * flow Experts And Noneiperts Operate

Electronic Equipment fron4 instructions

4

oMyid E. Kiera% Mark Tibbitslz

US DEPARTMENT OF EDUCATIONNANONAL INSTITUTE OF EDUCATION

E01.0-110*.Ag. AE5Ol1PCES TIO

CENTER .ERIC.

.'H tardueIi at4..., rAlgitre 'iCplegg2.1101,

y !A n,pmeelo4

a. -,0,r. S4$$.1gn MOS OK,'$ 0 OH.() PH

andiSusan ovair

I

.

TeChnical Report No, 14 (UARZ/DP/TR-83/0NR-14)ebrt.iary 10, 1984

. t,.. ,. .

«. . .,, 4;C

This research was supported by the' Personnel and Train Mg Research Programs,Office of Naval Research4cridbimet Number NO0b14-81=C-06,99, Contract'Co?

Authority ICIentification,Number INT 67-4p. Reproduotioninyvholeor in,paist isper ilted for any purpose of,Ahe U iffid States Government,

.

'Apprcsv9t1, for PublicIRelease;Distributiori Unlimited..

1.1 '

e 1 J

,.1

,

eg

$e

17

UNCLAS IFIED -SECURITY, CLASSIFICATION OF THIS PAGE (Whim Data entor.0)

4-

REPORLDOCUIAENTATIONPAGE .,olov

READ INST'RtIOTIONSINBEFORE COMPLETING FORM

T REPORT NuMSER e ,

No. 14 (OARZ/bP/TR-8370NR-14)2. GOVT ACCESSION Neel.

.

,RECIPIENT'S CATALOG NURSER

.

4. TITLE (rod SoblItto)-

. o ,

-How Experts and Nonexperts Operate .Electronic Equipment from Instructions

$., TYPE OF REPORT 12 PERIOD COVERED

Technical Reoport 2/10/84

Ilt PERF RISING ORO. REPORT HUMMER

7. AU T14014(0) . ,

David 1E. Kieras, Mark Tibbits, .

and Susan Bovaiv .

12. CONTM CT OR GRANT NUMBER(.)

N00014'033* 1-C-0691S

,.

S. PERFORMING ORGANIZATION NAME ANO ADDRESS

Deparrwent o Psychology 17- 'Univtrsity of Arizona ."

Tucson, AZ 85721 '

r'7

10. PROGRAM ONW=1.7""61153N1 RR 042-06RR 042-06-011 NR 667-473

I. CONTROLLING OFFICE NAME ANO ADORESS 4Personnel and Training Research ProgramsOffice of Nava). Research (Code 458)Arlington,' VA 22217

12. REPORT DATE , NZ*February 10, 196Z

13. Nu 131100 or P AGES

43

14 MOPHYORING AGENCY NAME Ili p. 00 ci ESS(St clIttoronl from Corate/UnS 0121c4)

. 1'

.

.0. s

IS. SECURITY CLASS. (ot rile report)

linClaSSified

lie, 13gatNICATION/OCOMPIGRAOINGS

16 OISTRIOUTION STATEMENT (of (ht. Report)

. . . ". .

Approved for Public Release; DistributiOn Unlimited"

.

n 171STAIDaTION ST ATEMENTW/ 00 obotroct .stored in block 20, ft di(torrart boa, Avert)r- ..:

r

. .

10 SUPPLEMENTARY NOTES 3". .

.

Requests for copies of this report should to addressed to David Kieras.

. .

.

is '

. .. .

IS. KEY WORDS (Crolhomroo torotr oleo ti fscous', apt NNW", by block nu oirpo ...

, .

Electronic Equipment,. Expertise, Instructions, Schemas .

r .

; ,-:,..'. ,

s

.

.

20 Awn r.c., (Confirm.* ell torero. Ws SI Rogow/0 mod/4001NY b7 Week nuolm)

Three questions were addressed in an experiTent in which subjects followedinstructions ,to complete tasks involving seveal pieces. of electroniceqtlipment: (1) Two instruction formats were compared: a-ierarchical menuformat containing natural chunks of instructions was not superior overallto a simple step -by -step instruction fdrmat. the manu fosmat was superioronly if the subject was familiar with the type of device, and was sometispes

J .

rte ta/3D I JAN 73 I. EDITION OF Nov it IS OPSOLIETE

S/11 010.2.034.6601 ' UNCLASSIFIEDSECURITY CLASSIFICATION OF THIS PAGE (Wawa Dom &moms)

a.

(

.k.

/EnTv CLIAretffitfi Or THIS PAGEOpten DSc* tneord)

$

substantially infer ior otherw ise. 2) Experts were comp ared to noneXperts,.and found to be fastek-overall, and able to operate equIpment with fewer

instructions in menu cdhditi9n. They,were also faster when complexphysical actions were involved. Thus, there. were both.specific"and generaleffects of expertise. (3) Evidence was sought that knowledge of how to'operate equipment was schematic. It was qxpectedthat when eubjects.in themenu format condition operated a' device without selecting any instructionsto read, their sequence of actions should cdrrespond,o) stereotyped schema--like-patterns. This occurred only weakly, suggesting that even expettsoperate everyday devices la a 'problem-solving mode, rather than by retrievedcomplete procedures.

1

h

"sr

p

sr,

e.

Amp

ft

mi

UNCLA9SIPTEDSECURITYsGLAIIIPIcATIOPI01 THIS PASE.(19)Iire Efted)

. 7

4s,.

Page 2 -

Abstract..:,.

. , . - .

.

ar ,

,

t' :Three questions were addressed -in an experiment in-'wnictisubjects followed instructipnt to complete tasks involving severalpieces of electronic equipment: (1) Two instruction formats ,were.compared: a hierarchical menu format containing nat4r4,1 chunks ofinstructions was not superior overall to a* simple ,step-by-step . ' '

instruction format. The. menu format was superior9only'kf thesubject was familiar with the type ofdevice, and was sometimes'

t substantially inferior otherVise. (2) Experts were.00Mpared tononexperts, and found to be faster overall, and able to operate,

. /' equipment with fewer -instroctions in the menu condition Th'ey

-were also faster whey Complex physical actiorls were 'involved.Thus, there were both specific and general effects of `expertise.(3) EVIdence xas sought that knowledge of how to operate quipmentwaa. schematic, It was expected thalt when subjects in thr menufOymat condition- operated a device without, selecting anyi'structions to read,their sequence of actions should-correspond,

, to stereotyped schema-like patterns.. 'this occurred only weakly,suggesting. that e'en experts everyday, :devices in aproblem-solving mode, rathel than by retrieved comp/ete

O. .

procedures. .. /

N .

- %

. ,

S

5

. a

I

1

e

t.

ACKNOWLEDGMENT

Page 3

'-Independent analyses of a portion of the data described' inthis /report. alte part, of a proposed Master's thesis by MarkTibbitts: The assistance of, Joy .Lafehr' and Vial Borders isgratefully acknowle4ed%

40.

tI

4

A

1

,)

ti

1-

.

How Experts and'Nonexperts Operate '

Electronic Equipment frdm Instru%tions

David E. Kieras Mark'Tibbits,and

Lean Bovairto

Page 4

'

This repoTt\descrtbes results from an experiment which 'Vasdesigned to assess three questions. about loW people operate a'piece of equipment from written instructions. The questions deal'with instruction format, expertise, and the organizationof prior

tkno4ledge, in'a task in which SUbjects. must follow a set ofinstructions in order to complete a task involving an electronicdevice.

I'. , .

,The first question is one of instruction format (see Smith & ,

Goodman, 1982)- This is the difference be,tween whe.ther the formator layout of the instructional material torces the user to executeeach step in order, or whether the instructions allow the user:to

. pick and choose the material to be read and executed. In,this,experiment, one group received step-by-step instructions that Mere '

presentedla single step at a time, and tkessubject had to readevery step. The other group received ailierarthicea menu of' .

'instructions, in which the subject could,either execute the taskwith only a high leveldescription, or could request more detail.In this way, the subject' would only have 4o .read the :- instructionsthat he or she. felt was necessary. to execute the task. Therationale of this manipulation is, that an expert subjeet couldtake advantage of the hierarcffeid menu format, because largeportions 9f the task would be familiar., ;However, a, nonexpert

.subject would have to, tread all of the instructions anyWay, so thedenu would not be of any great advantage. ,Purthermore, there

14- should be' relatively, little -difference between experts andOnexperts on step-by-step,indfruotioris, because in both cases allof the steps must be read. . .

, .

. t

. . .

is'The second question is .the nature -Of 'expertise effects.

/

While. expertise' has been'heavily studied (see Chi, Peitovich, &1 "I..aser, 1981;4- Chi 8T,IGlaser, in prep's), it his not beek, examined

.n one eiteit. of operating electronic equipment, a domain ofgreat pr tical importahce., Gel4rally, it is expectete that _,

experts would complete the tasks taster, and read fewer s ps intree menu condition. However, tilts could depend .on the deviceunder consideration. Only experts wouldtbe familiar with somedqvices* but even tne nohexsprts should be able/to operate otherdevices easily. Likewise, even nonexperts should 'know somethingsabout almost uny devic, such as hbw.to turn it' on. Thus, it wasexpected that there mould be n interaction Of subject expertise,experience with the,exadt dev and the nature of kitticularsteps in the intructions The basic question about expertise

to

4

Pagy 5

effects is whether th4re are general effects of expertise, or0

virether they are specific to the individual devices involved. /For .

this reason, several devices'of,widely differing-familiarity wereused. ,

**1

The third question concerns the nature of the prior knowledgethat subjects have about devices.- In Kieris (19'2) it wasdiuggested that knowledge of devices is organized in the 'fbrm, ofschemas. These schemas would. include knowledge not only of how torecognize a particular type of device, but also its typicalstructure and operating procedures. If device knowledge isorganized by schemas, there should be clea'r patterns in the data

-obtained in this experiment. Menu choices should follow patternsthat, would be expected from schematilc device knowledge. Ifsubfects operated the equipment entit.ely from prior knowledge,without no-W.-ding' instructions, which happened

i

in many% cases,. thentheir behavior shpuld follow some pattern that can be described in 1-terms of device schemas.

The tasic manipulations performed in this study were asfollows: .several devices were used, which included two every-daydevices, tvi,go devices familiar to only experts, and two noveldevices Millar to neither expertsior nonexperts. The subjectswere either experts, who typically had several years of workingexperiende in. 'electronics, or nonexperts, who were ordinarycollege students. A .questionnaire was used to confirm thesubject's classification, and to assess each subject's experiencewith the individual devices used in the experiment. The twoinstruction formats were either, a step-by-step 'format 'or ahierarchical' menu format. The terminal . nodes of the menuhierarchy consisted of the exact same individual instruction steps

yw as were Used in the step-by-step format. The variables measuredwere the total completion time for each task on each device, thecompletion time for each individual step in the step-by-stepinstructions, and in the menu condition,' the individual menuchoices, and tneir completion times: The subjects' behavior was

I recorded on videotape to allow detailed scoring on the subjects'activities vihile performing the tasks.

.

?.;

.Materials4

METHOD

Devices. The six devices used are described in Table. 1. Theradio, cassette recorder, VOM, and oscilloscope were of a standardmake. The,phi. phenomenon demonstrator was profestionally built,but in general construction'style it appeared to be a "home-brew"aM4t,edr,job. The physiological stimulator is a stsiodard piece ofapparatus in a physiological psychology lab, but as the ratings

.confirmed, it was es8entialay unfamiliar to all subjects. Noticethat all . of tne Lon-everyday devices were relativelyold-fashioned, being from the vacuum-tuba era: The, devices, wereprepared before presentation to each subject by setting allcontrols to incorrect positions's° that in order to, complete the

t

4.

1

(

. TabieDevices Used in the Experiment

Device NSdription

1. Radio ,A por.table AM-Wradiar- withibuilt-in AC,adapter, antenna, volume, tone, tuning,and band controls.

2. Recorder . A portatae audio cassette tape record r,with keyboard tape controls, red record,Interlock key, and volume control. Supplied

. cassette was not fully rewound....

*,

.3. VOM A standard volt-ohm-milliameter, with a. supplied resistor to measure.

. .

.4..0scilIescoye A dual-trace triggered-sweep oscilloscope....... with standard audio signal generator and

connecting cables:

4

,5. Phi Phenomenon A device that flashes two connected neonDemonstrator bulbs alternately at various rates and phase

relationships.

6. Physiological A large device with several dial - multiplierStimulator s.t.g-that produce yulsts of specified

magnitudeNrate,and duty cycles; a neon .

bulb is connects tto the output to indicatethe pulses.

S

.11

I

,

9

.

Page

. .

.

.task, each *control would have to be properly set. .

Instructions. A major goal in composing the insructionsl, wasto allow the menu and: the step-by-s iN instructions to' be easily-conipared to each other. This was done y preparing the materialsso that the terminal steps in tie menu instructions were 'exactlyidenti,cal to the steps comprlsing the step-by-step -instructions.,and were worded and displayed identically.

Thee menu instructions made up a hierarchy of natural "chunks"of the operating procedure. .Determination of the chunks was Aoneintuitively. It is clear-from some aspects of the results thatsome of the chunks chosen were in fact natural unts; however,'the data do not definitively qpnfirm theichunk classificaton..

4

Each set of.instructions began with a statement of the taskthat the subject had to accomplish. This main task statement wasspecific enough that the subject could, if ET-6rWe had adequateprior) knowledge, complete the entire 6task from just thisstatement. However, the main task statement did not deseqbe howthe controls on the device-had to be set or operated. Table 2lists theltasks hat were to be performed on each device, in the-same wording as /they were shpwn to subjects.

- Subjects

The nonexperts were recruited by 'campus and newspaperadiertisements, and were paid $5 for prticipating. As shown Ivthe experience questionnaires adm.inistere 9 the subjects, onlyone, expert subject was inadveftently recruited by this method.The expert subjects were recruited by advertisements directed atelectronics experts. i.n all cases, the subjects obtakned werehighly,experienced in electronics; the typical expert had severalyears experience as an xlecronics technician in the militant.Twenty subjects were recruited by each method, but in the analyses ,used below, the 'classification was corrected; to'yield nineteennonexperts and-twenty-one experts. Since earlier studies seemed

suggest that there were strong sex differences. amongIbneiperts, and fpmale electronics experts were extremely hardtolodte, all subjects Ilse& in this .experiment were pale.

Desi n. The instruction format condition was determined aterandom for -each subject. Eaph subject carried out-the six taskson tae six devices in the. same instruction format condition. The

device tasXe were done in a fixed order", which is the ordir inwhich" the devices ar'e listedlin Table 1. This order was chosgn topresent the tasks and devices in order of decreasing fadiliarity,and increasing apparent complexity 'within each .level 'of

familiarity.

r-f

Table 2

I

Task

1 . .

Main Task Statement

Listen .to Station KU!(T FM (90.5 FM) at 'medium volume:on ,

the portable radio'. . .

. .;

2. Record the words "testing.'. . 1, 2; 3" on the 'cassetterecorder,* and play the words back at medium vollime.

) 3. Measure the resiStpco of the resistor using the voltohmmeter.

4. Use the sigrial generator an the oscillascope to display.about two AC wave cycles on theloscillzscope screen. .

,

5. use the phi phenomenon demAstrator to flash the lightsat 5 CPS (cycles per second). /'

-Y.

6. Use the stimulatof to flash the neon light at a frequencyof 1,CPS"(cycles per second) with a flas4, duration of '

.7 seconds and a delay of .5 second's.

V

1

-'

I

4

Page ATM`

p

Apparatus and Procedure . .s.. : .

I -.. .:.

Each subject was run indi;/iduallY, and was seated in a 'smallrook before a table. On -the right-hand end pf the, table was astandard video terminal, on which a laboratory compute. displayedthe inztructions. TheJeft-hand portion of the table was occupied

' by the device. A videotape recorder recorded all of the'subject'sjold.tiv,i4r. The instructions were' predented.one_steg or 'menu at a

time_ with the subject tapping the space bar or Awing a' choicenumber to go on to tie nextdisplay. The laborktory computer

.. recorded the amount of time that the subject left each instructionstep or menu. on the screen. Due to the-nature of the equipment,

l'and the 'prohibitive 'scoring effort involved, it was not practicalto distinguish the time the subject spent reading from the timethe subject spent carrying out tie instructions. Thus., thelaboratory computeT"'was able 'only to 'record the cotplltion timefor. each step, defined as the tote' reading plu4 execution timefor the 'instruction step. Thealivideotape recording was used todetermine what subjects actually did on each step.

(....p

/

The Aevices were brought into the.room one at'a'time,'and thesubject then carried out the task 'on the dev,ice. When the subjedthad reached the end o' 'the 'instructiipliaTotre experimenter teturned.and checked that the task had been carried out.correctly, in termsof 4hefher the final correct result was achieved. The device was .

then xemoved, and a new device brought in. Sublects.who did notachieve the propel-44nel result were asked to repeat the task;however, the data frod these repeated,tasks were later dropped'from the 'analysis. -

.

Due to inadequate training of .the experimenters, on sometrials theequipment wap being moved in' and put of the Loom whilethe clock was running, .making thd completion time recd rd of thefirst won-Struction unreliable. It is belieht that' these "-eventsare not confounded with any of the experimental.-maniptilations, so ,

the analysis of'total completion time would be conservative due to.,-' the,extraneous variability. Examination of the video tapes shows

CItimere being brought in, and so these times reflect the tdtalwe t.

at the subjects -were uvisaliy inspecting. the deviceswhile .they

hat the subjects -interacted with the devices to corplete,t4e,

task.

0

Pt C\11of:4 .S U14§

Tota CoiplVion Ti

Analysis .The total dompIetion'timeofor each eubjecton each task.los calculated as the totalelapsedtime from thepresentation of the main task statement until the experimenterhadconfirmed that the task was completed torrect1 Data kom taskswers .dropp'ed in which the subject did the entire task liaqre .than--once, or failed to do the task at all correctly. 0,ut of the total.of.1t40 task attempts, 14 were thus dropped. Due to the unequalgroup, siz6s,, missing data,* and,. unbalanced 'device qxperience;!,

* "*

.. "

0

-Page 8-\

.

fac-bbr, the total times were Upalyzed using stepwi4 multiple0.regression.*

e subject's expertise groupt-instruction format condition,A .and subject's experience 'with the 'individual device Ore

re resented as dummy variables. he devibe experience variablewas based on the, questionnaires that each subject' filled out. If

q the subject indicated 4ny actual usage experience with the evice,4then -the devime experience dummy variable' received a value o one;otherwise a value of zero was .assigned. The *device factq. wasrepresented as a set of five dummy coded Variables with the radiobeing used' as the baselin0 Fbllowing,..the methdd suggested byPedhazur (1982) for mixed designs; a variable whose Value is the

,subject's mean total completion time over the six devices was ,

included. 'The betweensubjects factors and interactional wereentered first in the-equation, followed by the subject's mean timevariable, followed by all of. the withinsubject factors andsinteractions; Thp analysis. was hieletklical, in that main effects

...

were forced, into the equation lore interactions.. , ,

4

All ofthe interactions between ,subjedt experience, device,experience, anCinetruction,format condition were -represented, bON

lonly insfhotiOnformat condition and subject expertise group wereallowed to interact with the delqce factor; device experience was ,

not allowed to interact with thedeVice factor. The rationale forthis decision is that the device experience Variable is an*eadyspecific to...individual devices, so" interactions between individualdevice emerience and individual deVice dummy variables .would bei.difficult to ipterpret. .

%,..

lhNote .that sUpject expertise and epecific device experience inthese data are only. sli:ght-3)--correlated Cr=.13), and *the,..

interaction.bOween subject expertise and device experience wasnot signifibant. Thus these two factors peke practicallyindependent,,contrib4tiqns to the total cotpleticp times. Two ofthe devices were tamilier to everyone, and .two were'nfamiliar toalmoet everyone, resulting in these. two variables fbeing, nearlyorthogonal. o.o, 4 -.

4 : . $./

i .

-,.

ith a total of 23 variables in the equation and 163 degreesof freedom in the residual, 81.4 of variance in the total,completion tithes was accounted for. This extremely high figure ldde to two factors: the subject's mean completion t-ime accounted

$. for appr xlmately 15% Of the' variance, and the ,deviCe lectoraccounted for about 5O of the variance. .This is clearly due tothe,fac at the devices marled substantially in number of stepsin the tasks, and thus'the completion times v ry sysematical,lyover an extremely wide range. ThetelIects to b discussed belowwereh, all tested for significance at the . 5 level, using the"Ptoremoveu"statistic, which is a.conservativ, estimate of thesignificance of an individual variable as if were the last toenter the equation.

Nqb

13114

4

7

4

, I

.Main effects- Tible 3 shows the Means for the var ious maineffects thiT-Te7I significant. subjedt expertise variable wasquite `significant; eXperts Mere about one 'third .fdster incompletion time than: nonexperks; .There was no significant gaineffect of instruction format condition, oven, though the menucondition averaged about 30 seconds faster. This means that,counter -to intuition, the menu format wai not reliably superioroverall to the step -by -step format. This is probably. a result ofthe fact that while fewer steps were rtad in the menu condition,more maternal has tobe read in addition to the individual steps.The device experience faotor was significant;, being familiar with;a specific device led to a 30% improvement in"completion time. Aswourd be expected, there is Avery strong main effect.'of device's.

'-Interactions. The interaction bepween.device experience and

instruction format condition, shown in- Table 4, was significant.The menu instructions are actually "slower than the step-by-stepinstructions if the device is not familia.r, but substantiallyfaster than the .step-by-step instructions if the device isfamiliar. Thiis, not only do the menu instructions allow the userto take advantage of prior knowledge more than the step-by-stepinsieuctions, but the lack of prior knowledge means that.the extra"overhead" in menuinstructions, plus mistakes made as a result ofskipping instructions; actually slows down taskcomplwtion:

.

.

The interaction of instruction format condition, and device,. whose means are shown in Table 5, was significant. For theeradio,

recorder, and phi demonstratOr, the menu condition produced fasterresults than the step-by-step condition. However, the VOM,oscilloscope, and stimulator produced the opposite effect. Thisis probably due to the fact that thepe are devices which wereespecially difficult for nonexpgrts, exaggerating the effect" of

. -tile extra material in the menu format.,, Table 6, shows theinteraction between devices'andcsubject expertise group, which wasale() significant' Help it is clear tha7t the oscilloscope and VOMwere especially hard for the 'nonexperts compared to the experts.

Page 9

The three-way interaction between subject expdrIise,condition, and device was significant, and' illustrates the keyresult. The means are shown in Table 7,6.-i-which includes the

. percent gain resulting,from using the menu instructions instead ofstep -by- -step, pr nonexperts and experts on each device. Oneclear result is that the experts benefit from the use of the menuformat on all devices except fon the stimulator, where there is asubstantial impairment in performance. This is probably title to .

the fact that since this was a cbmplex and novel device, theexperts' Attempts to -operate it without reading much of theinstructions often led them-down "garden paths." For example, one

-

expert, e plugged indicator light into the yrong jack, and thenspending a long time trying to set the controls to light it. Withthe nonexperts, the two expert-familiar devices, the VOM, and theoscilloscope, produced much longer completion times in the pent; '

instructions compared to' the step-by-step. Since many of thenonexpert sub ects claimed experience-in usling the VOM, theirlonger compleVan times, in the menu condition may be similar to:

., ,

. . i

14

4

Table 3.Main Effectel in Total Time Data

Effect Means Sighifi4nce''1---

Expertise '2

lionexpertsEx'perts

Defice 'Efpci lentNOTI.=TimiliarFamiliar

.

Instruction formatStep-by-step .

Menu

.DevicesRadio #

Recorder .

V0M.

.Oscilloscopei Phi DemqnstFator

Stimulator11.

3t4.0214..4

- ... _,

319.4.223.0

274.4247.1

'

-.

14

or

..._

,_ --

4

**

_ ... ,

**

.

- NS1

137'1166.2256.8511:9 .

118.8343?

6

V.

eatm.

.1%. .

V. '1

;1.

r .

: Table 4 /

Total Time (Secs) as a Functionqf Device Expertise and Instruction Format

)

, .

,.-Device .Expe rience

4

1.

Not' eamiltar,Familiar

I

Instruction Format

Stepp by-step

307.8.254.4

Menu

331.0185.1

,.-

Table 5Totallime (Secs)'as a Funct ionof instruktion Format and Device

Format

..t

Device , 4Radio, Recrdr VOM OBcil PhiDem Stim

V 1.Step-by-step 185.7 , 201.3 230.1 504.3, 210..5 314.4.Menu 88.4 131.0 297.8' 521.9 167.0 375.5

Table 6Total Time (secs) as a Function,of Subject Expertise and Device

Device -

EApertise Radio Recrdr VOM Oscil PhiDem Stim

4ortexpert 165.8 148.7 342.8 649.046 230.3 -367.8Expert 111.0 145.7 175.8 396.41W 151.2 321.3

0.4

. . 16.

. 4111i

1'O

4

'Table 7Mean TOtal Time for each Device,.

Instruction Format, and Expertin arou'a

A

dir

'15,eviceSalo Recrdr VON

Jr

Oscie l PhiDep Stip,

Nonexpertr4Step-by-step 224.5 216.3 ; 270.8 604.3 260.4 t359.5Menu 100.9. 158.2 .462.8 723.6 196.9 378.0

. 6.

% Gain 5-5* 27% . -71% -20%. 24% '5%.

Experts P

Step -by -step; 147.0 186.4 189.5 404.4 160.6 : 269.2Menu 78.3 108.8 f50,5 387.4 142.6' 373.5

=, m% Gaip 47% 41% 17% 4%, 11%

A

74,

e

I

4.'

14

406

the "garden pat h" effect obtained for the. experts with thestimulator. Namely, a familiarity with a devige is adangerous thing;- 4,t can lead to 'lbnger completion times if .instructions are pot followed. The els;/ated time ..for thesesubjects in the menu.conditien with the osdillOscope is haider toexplain.

Conclusion. Theseiresults demonstrate that the ,virtues ofthe two ins6ruction formats-b.re-heavily dependent upon the user'sgeneral expertise and also the familiai-ity with title specificdevice. In general, the interactions seep.to be due stly to thespecific familiarity with the device, as opposed to .t e subject'sgeneral expertise. That is, the fact that the intertic ion between Ak

devite experience and ,instruction format was signific t, but" theinteraction betweeri subjects expOrtise and instructs n format wasnot, suggests that the advantage of menu instructions is a ma terof specific familiarity with the device,, and not generalexpertise,. Electronic experts may notdo better wi h the menuinstruction format unless they have specific familiarity with thedevice in question. Alternatively, if the device is unfamiliar,experts cap benefit from menu instructions if' lhe device issimple, such as the phi demonstrator, but not if it is compl9x?such as the stimulator.

A 's

- Page JO

0n the other hand, the sign ifiCant main 'effect of subjectexpertise; even with specific device experience taken intoaccount, is important. Experts were generally faster,' thannonexperts at operating the:equipment, regardless of itsfamiliarity. OWIsher aspects of tit results suggest that this is

. / due not just to faster execution of aetiohp, and also to betterorganized and more efficient actions as well.

Menu Choices

Number of frapes read. Table 8 shows the mean number offrameisTUTsNaTr--o-r instruction steps or menus) read in the menucondition fo.r each group and each device. For example, bothexperts and nonexperts read only one frame for the radio, 16melythe frame that contains the main task statement, but nonexpertschose to read an average of 50.4 frames of information for theoqcilloscope task, while experts read an average of only 1.2.THese data were subjected to a multiple regression analysissimilar e above one, with the factors being subjectexpertise, de ice etperience,- and devices, and interactions.ofsubject expertise with.device experience and indivi4ual deviceswere allowedy .The results are summarized in Table 9.

There were strong main effects of device, with the VOM,oscilloscope, and stimulator requiring many more frames than.theradio, which was taken as the baseline. The key results were thatneither subject expertise nor .'device experience, nor theirinteraction, were significant predictors of the number of framed:read, once the main effects of device and the nteraction ofsubject expertise with device were taken into account. As shownin Table 8, the VOM, oscilloscope, phi demonstrator,-;and

18.

,

/.

..

Table 8',

-0-,

. ..,

yean Number of Frames Read in the Menu Condition ,.,e M. for Each Eipetiie Group ..zAf ,, 4

...;

-..-,-.: -",

:0

Group Radio Recrdr VOM Osci). PhiDem Stm,

Nonaperts 1.0 1.0 32.5 50.4 18.2 47.8

Experts4?.` 1.0 1.0 ' 3.1 1.2 ; 3.4 2:1.9

Variable

(egress Analysis on Number oframeis Read in the Menu Condition

1 -,

,t

Colfacient

CONSTANT 1

-SUBJECT EXP.. 4 DEVICE EXP.

DEVICE 2 ,

DEVICE 3.DEVICE 4 ,,,t1

MICE 5 :14

9 DE E 6SUB XP X DEV2SUB EXP X DEY3SUB EXP 1 DEV4

.SUB EXP X DEV5 (

SUB EXP )C.DEV6 4;

*Notes .

,

R2 is .$5 with 12 variables and N=I07. Device 1 (Radio) is,- used as the baseline for dummy coding of Device factor.

...!_.

Std. Coef. . F -to- Remove

10.10.0 .0 0.00-9.1 3.99O. 0 O.

29.5 .516 45.4046.5 .839 " 113.19

% 9.1 .184 2.4938.7 751 43.98O.0 -.O. 0.00

-26.3 ° -.360 20.82-45.3 -:618 61.52-14.8 -.224 8.68-25..9 -.373 25.15

I

19

kPage 11

stipulator all 'required many fewer frames for experts than for

el

nonexperts. The main effect of device experience was marginallysignificant. Thus, it is clear from thele results that the menucondition allows exp its to benefit by permitting them to'readonly .a few frames.

-

Choice Patterns. The spedific pattern of frame choices foreach 57-fi- ce was consid red in terms of the menu hierarbhy for eachdevice. The intended organizatOn of the menu instructions wasthat the levels in the hierarchy would correspond) to the naturalchunks in the operatiori of the device. However, contrary to thegoals ofthe experiment, the evidence to support this claim isvery limited in these data. In. -order for there to be naturalchunks in the operation of the device, the device must be'' familiarto the subject. However, if the device was fairly familiar to the.subejecilk the subject would need to read very few trames, oftenonly the main task statement fram.e, and thus there would be fewchoices to reveal which -portions of the menu hierarchy werefamiliar and which-were not. Perhaps different devices would haveyielded more useful data.

okHowever, there were some interesting patterns in the choices.

Figure. 1 illustrates the best example. The figure shows the menuhierarchy for the phi demonstrator in simplified form. Theterminal portions of the tree consist of the sequence of actualsteps that were identical to the step -by -step instructions. Ineach box is shown the.proportions of nonexperts and.experts whotread the materliarin the box. Thus, for example, thi. top-levelbox corresponds to the frame that states the main, task. Almost,all subjects then read the main menu which contains four items:powering up the device, attaching the lights, .setting the mode,and adjusting the CPS dial. However, only 4O%. of the nonexpertsand only 10% of the experts felt It was. necessary to get the more,specific information about powering up the device,,and almost noneof the subjects required the step-by-step instructions about howto plug in the device and turncit on. The-other devices that alsoinvolved these steps also had this general pattern. Very fewsubjects, even nonexperts, required the specific 'instructions: onplugging in and turning on the device. This was true for theoscilloscope and signal generator combination, and also ,true fbrthe stimulator, which was a very complicated and unfamiliardevice.

I

Another effect that appears in Figure 1 is the tendency for,nonexperts to learn while doing similar acti/aties. Notice how50% of the offbjects required the step-by-step instructions forplugging in light A, but only 10% of them went on to read theinstructions for how to plug in light B. A similar effect appearsin the 'oscklrostope task, in which fewer nonexpert subjectsrequired the instructions .for plugging in and turninEcl-o-fr-"ViTe-second piece of equipment than for the tirst piece of 'e uipmenti'.The obvious implication of this effect is that subjects are notsimply executing `these instructions as they read them, and thenforgetting the instruction content when they proceed to trff.nextinstruction. Rather, they seem to be able to take tAe content of,

20

1.

4

:6

FLASH LIGHTS

AT 5 CPS

N: 90%E: 80%

POWER UP ATTACH LIGHTS SET MODE ADJUST CPS

N: 40%E: 10%

GET POWER TURN ON

N: 0% N: 16%

E: 0% E: 0%

I

N: 60%E: 102

LIGHT A LIGHT B

N: 90%E: 20%

RATE /PHASE

: 50%

07.

BA/AB

N: N: 90% NI 80%

E: 0% 'E: 0% E: 10% E: 10%

Figure 1. PrOportion of menu choices made at each menu level for experts (E)and nonexperts (N) on the phi phenomendn demonstrator.

41,

21

Page 12

, vit . N -

.

. . ,

one inst uction and genorafize it immediately to apply to asimilar situation. Hoiever, these results are too limited to shed,much light on' this issue. Further woR id-clearly needed.

.

Stip Completion Times.

,

malysis method. A regression analysis was done-toldeterminewhich factors predict-he amount of time taken to completeindividual steps in the step-br-step condition. The videotape,scoring was used to eliminate the times for individual "steps thatwer defective. In addition,"the time on the very first step, inthe instructions were not included since.idsome case g, theseti es were gontamindted as described above. ,This left a total of3008 *indpOldual step times for the .analysis. Each instructionstep was classified according to a set of categories, ,shown inTable. 10, which are the general ,types of actions stated by tin,instructions. These categories were each represented by a dummyvariable, with the, ISIMP category teilg uqed as the baseline. Thevideo tapes for each subject were sco ed according to the actionactually carried out by the subject on each step. The scoringcateg6ries for the actions are shown in Table 11. These were alsorepresented with, dummy variables, 'with SKIP being used as abaseline. In order to-examine the chunkingAmproperties of thestep-by-step instructions, the variab ME was defined, whichreflects the proportion of time th subjects in- the menucondition read the corresponding step.-,This variable fook on avalue that depended on whether the subject was an expert or anonexpert. If the subject was an expert"-then the value of MENUwas the proportion of experts that viewed the corresponding step,in the menu condition. Likewidel for a` nonexpert, the MENUvariable was the proportion of nonexperts that viewed thatinstruction. .

An additional variable that reflected' properties of theinstructions yas the number of words in each instruction. Thisveriable should hot be taken'to reflect comprehension time, sinceits coefficient is far too large;' rather, it provides a crudemeasure of the overall amount of inftirmation in the instruction.Additional variables tered into the analysis were the subject's

olftexpertise group, and tre ice experience variable, as describedabove. The subject xpertise variable was allowed to interactwith all of the instruction characteristic variables and theaction variables. As before, the order of entry in the stepwiseanalysis was hierarchical, and the conservative "F-to-remove"stativtio is reported. Tidally, since this was a mixed design,'the subject expertise' variable was entered into the equationfirst; followed by a subject mean variable, then by thewithin-subj cts variables;

Step time results. A summary of the analysis is shown inTable.12. Ircife that fhe coefficients must be interpreted in termsof the fact that all other factors are in the equation. There wasa substantial effect of subject expertise (SUBEKP), inthichexperts read the instructions on the order of 1.6 seconds fasterper .step than nonexperts. Also, the step times differed

22

}

4

. 4

..- Ableiv10

,' A

.

Dummy Variables Usedto Code Instruction Contehts- ..

Variable Descriptioizind Example.

. .

"V ,

ILOC Locate a Part of the device (locate the power switch)IADJ Setting a.control (turning knob to DC)

. .

ISIMP A simple ,action (flipping a switch) .

'EFFECT Adjusting a knob to produce a certain effect .

(zeroing' ohs scale)ICOMPH A complex physical action (plugging in a cord)siMPH A compleX physical.actionjamiliar to all expert .

, (zeroing a meter)

Table 11Dummy Variables Used to Code Subject's Actions

Variables Description 21: -0

.

DO Action same as instruction (

SKIP . go action carried outLOOK Subject looks at device .x.-

LOC Subject "ideates" a., part of device(e.g. touches it)" 0

ACT Subject engages, in some action other than above %

90o

e

23

".1

X.

4

Table 12Regreaathn' Analysis on

Completion Tides for each Step inthe Step-by-step Condition

Variable

CONSTANTSUBEXPSMEAN

Coefficient

1 .4563 '

.1. 0DO 4.11LOC 2.46.

ACT 7i30ILOC 1.86'IADJ 4.60!EFFECT, , 3".1°6

WORDS : .57ICOMPH .14.46IEXPH 6.42EXPCOM -5.52EX PIPE -5.78MENU - I.71'

Std. Coef.

.

-.110. 36.96242.29.

1. 77

449. 9.54

.139 - 4.55,WP .188 129.201 .126 5.43i .116 45.99

.093 18.92%339 366.98.247 154.99.156 42.66

-.065 11.19-.100 24.31'.091 8.59' .

F-to-Remove

Notes

R-2 is! .14075 with 19 variables and N=3008. Five variablesare not sWpwn because the F-ratios were nonsignificant. See textfor explanation of variableg. Values for SUBEXP are based on onlySUBEXP in the \equation, before SMEAN and the within-subjectsvariables are added.

I

V

4-

4 Page 13. .

/ .

subsiihtially depending ,eothi Ca010, ich actions that subjectsactually performed, and also in the properties.of t4e instructionsthemselves. This. .result in itself is. not too surprising. "However; it is noteworthythat two of the strongest (as shown byfthe standardized regression coefficients) instruction factors ate

4". the number. of words 'in the instruction 4WORDS); and Ofether'theinstruction required a complicated pflysic4 activity .(ICONTH).InstrUctions that required phSrsicalvactiliities that are faliliaronly to experts, such. as adjusting the zero adjust seceyi on. theV4M (IEXPH), also took significantly longer, even'though such

_

cab's were fairly rare.

.The key, results are the interactions. of_ expertise with two of.the instruction charactefistics, namely, complicated Physicalactivities (EXPCOM), and expert physical activities (EXPXPH). .

This suggests that not only are experts faster across the board,but they are especially fast at certain complicated physiCalactivities. Informal observation of the video tapes seems toconfirm this. NOnexpert subjects often spend a lot of .-time(tumbling with cords and connectors, while experts seem to know:exactly what they are doing in these physical activities, andprocied smoothly ancl,precisely.

An additional ke3 result is that the MENU variable wassignificant. The.coefficient means that with all othei fadtprs inthe equation, a step that.was always read ih the menu conditiontook about 1.7 seconds longer than one that was never read."'Assuming that the menu choices reflect' the familiarity ofprocedure "chunks," the amount of time takeli to completera step is ..

thus a function of its predictability on the basis of prior, 1.

knowledge.

, Knowledgebased Opera tion

In Kieras (1982) At was proposed that people's knowledge of,electronic devices is organizedas a-hierarchy of schemas, whichwould contain, among other things, schematic information on.how tooperate the corresponding class of devices. It is natiral,to.suppose that just es '- story schema ,avecifies the order ofappearance of items in a story; that a device schema would specify

4 the order of the steps for operating the device. Thus, whensubjects Operate a device baited only on their knowledge, therewould be a stereotyped sequence of behavior corresponding to theprocedurgi schema for operating the device. Some of the data fromtne menu condition was suitable for examining this issue; therewere' many cases where subjects attempted to operate the deviceafter receiving only the main task statement, without requestingfurther instructions.-

Analysis,method. The videotape record of the subjects'behavior ,was scored in terms of the individual activities thatsubjects performed, such as operating a certain control. Datawere dropped for subjects who got confused in the task or did itincorrectly in some way that w ?uld invalidate the data. Bothexperts and nonexperts all operated the radio and cassette

25

A

,- ,' .

4

I/ 4 , . .,

e ,. i i Page. 14..

4 10 r i

' '

, 4".4, .

recorder without any further instructiona iti-ihe: menwsonditign.Seven experts on -the, radio. and nitre . nonexperts.weve thus

r-available. AFor thkcassette4 reoorder there 'were...el/6.e AsOla . 4 I

behavior sequences from each' .group. With three other devi9est .-.

.

only experts operatId the device without instructions, For. :the.. ..

vOms. Qscilloscope fatid signal generator cOmbinati-en," 44d the phi'.° - -°--

phenomenon demonsttatoY, there we five, eight, .and 'eight such, .

subjeCts. . .. . . , . , 1

.

The method of -analyzing t is sewience data was t,c.),,, locate' °'sr

seqtAnces of Eittivities that ;ocCurre& at-lesetAttadt, and then'v,,,'express tht sequences. that sublebtV'performed with-as few terms to ..possible .by s referring to, these common sequences.. 1.19re

specifically, the seqftence data was represented ea a transition.,

4n'e'twork , tree diagram; in khigh the nodes tepresenl either, ;

individual actions for action "subroutines," and s.'' single ,path

.thr.ough the tree diagram represents the activities of a single ,..r subject. ,See Figure 2 for an example. Bach action l,represented

. by a two-letter symbol, and action Subroutines by -combinatfdris of , ..

'.these symbols. The depth of combination is 'indicated by 'the ,) -

notation; concatenated--symbols ara the shallowest level, with ..

brackets and parentheses indicating deeper levels orglibroutines..

In order to construct this transition diagram, all actionsexcept ,,specific centrol operations were deleted from the behaviorstream., Thus, for example, activities of lobating (touching) acont?Ol,"or looking at various parts of the device were ropped

. from the analysis./ The resulting pequences were Wen 'subjected to

4/Pa sprting process in which common sequences were identifiet and .

then the data regrouped' according -to the 'seqtAlitess, and theprocess repeated until no more sequences could be formed:

Once these sequences were defined, the behavior patterns for.all of the subjects could be rewritten as a tree diagram, in which 'ft

all subjects begin at. the origin and then branch out according tothe first action or sequence subroutine that they perform, 'and

'"uthen branch out fur,ther depending on their indpridual Actions.Since all subjects eventually did some; Sction that was different'from that done by. any other subject,-eventually the trees all had

' the game number of branches as there were subjects.,

Pattern results. In Figure ? is shown the top level diagramfor TEV'Tgquences for the nonexperts and expert's on tit-vradio.Notice how the nonexpert network seems to be "bushi'er" than theexpert network, and %also appears. to have more differentsubroutines. Beyond the preference for initially plugging in and .

turning op- the. radio., there seems to be little in the way of ad,interpretable pattern 4 the nonexpert sequences. However, thereis a basic pattern to the expert sequences. The subjects who.followed the- bottom two major branches first "set ue some pintion/of the ,radio before turning it on. The.subjectrfollowing to-upper branch turned on the radio immediately and then proceededmake a series of adjustments td it. Thus, even with as simpl adevice as a radio, there seem to be two major methods of operat ngit: the. first is setting it up and then turning it ofi, foillwed

26

v START

NOREfiERTi

(Pc -PoV1) ________,,,...Ea Pa -(Tni:i/W1)),%

0 .

a Pe.

V1 To .

(PcFe-(Tn-PoIl--Tn-lia)IN7::Tn Ra To -----itral -Val) ----7(Tral-TnV9-----::::. 4

V1 OtOPTriV1

'

a

STOP

PcPe

EXPERTS

START

4

(To -PoV1-Tn -Eal-----IFfs-TnV1) Tin--.4.-To 11 STOP4

Po l(Pe-ToV1)-Ba)-------Tra1 Li .

STOP

PoV1 ..-PPe-TaV1)-1447------V1:-(TnV1-TnV1)----ToVI--i-STOP.

Po PoV1 STOP

PoV1 -Tn-----(Pa-(inVi-TnV1)) STOP4

'

$(TollTn) STOP

PcPoV1Pe fn Em ilTn -or. STOP.'

t (Tn-VITn)----Ea .c STOP

4 4

PcFe (Po -V1Tn))

. STOPL

Tn ToPolo------V1Tn------Y1To STOP. -

...'PcEaPs ( Po -VITO ----111 , STOP

Pcaplug in power cordFsaselect PM. bandScaadjust tone control

r

27.

ToPoTn VI

PosTurn power onEaaextand antennaPfatorn power off

STOP

Vlaadjust volume dbo,trolTnaadjudt tuningRaaretract antenna

kFigt5e 2.. Behavior sequences for expert and noneipertisubjects

, operating the radio without instructions.

28

a

.

.

.)

Page 15 4 b. dO

o o,r

. vby adjusting it, and the second is turning it on, then-setting 'it.', .

up and adjusting it. 'Within each of these two major patterns.there are many minor i

.varlations.' :

. ! . . ,

apparent difference between experts and nonexperts.appears with the cassette' recorder in Figure 3.. Ovgrall, theexperts appear to produce Short"er and simpler sequences then' thenorie?tperts. Thust the experts in both the radio aqd the tape.recorder appear, to' have more consistent and shorter behaviorsequenceq. Some quantitative comparisons between,the expert and

. nonexpert transition networks were very.intriguing, but noneloof'ahem reached.statistical significance.

. v :r:r.

41.

. It should be noted that some of the complexity of .the tape'recorder 'behavior sequences,,is:probably due to the fact that the 0

tape cassette. .was. deliberately gilien to the subjects in acondition in which it was not fully rewound. Since the subjectsl-task was to record "testing 'one-twb-three on-the dupe andback, this confused some subjects if they rewound-the tape all theway back after recdrding as one, normally would. hus, some*.subjects, even experts, had to make more-than oneltttempt torecorl'thetape. Peihaps this Complexity is a reflection of the

-'fact that -the /tape recorder was not left in a schematic' state;that is, the norma -state for, a tape cassett6,is that-it is fully'rebound ;.4

.

impor'tant conclusion is. thpt if \Were 'is an -apparesp,° difference between' experts and nonexperts, even tri these 'everydaydev-ices, then experts are better even at operating everydaydevices than nonexperts. This presents a&serious problem forfuture studies of'electronics expertije, because it suggests verystrongly that nonexperts an not. be used a$ subjects tf such.studies even if very familiar-devices are used;

A further ,result that follows from eim examination of;theSetwo networks, and was also clearly apparent with,the °titer de/icesis that there is in fact very little stereotypy,,in th'e 'specific

. behavior sequences. Figure 4 presents the transition network forthe five experts' using the VOM. Notice ths4 the, numbe-r. ofsubroutines is quite small, and there is an almost immediatebranching of the tree into unique paths,' one'for each subject. ..-

Because of t1e 'extreme length of the for theoscilloscope and signal, generator combination, Figure 5 preisients atruncated and condensed version'of the full transition., network.For example, the term CRT means any control activities havingitodo with adjusting the CRT trace on the Oscilloscope, which couldinvolve any sequence of the five controls. Likewise, TB refers toany sequence involving adjustments to' the oscilloscope's timebase, which also. involved several controls. It should be notedthat eve R after this extreme condensation the pathp through the

',network b.gain branch into unique patterns very quickly'r The phiphenomenon demonstrator in Figure 6 'also shows a relatively quickbranching into unique paths.

t

(. NONEXPERTS

I

START

EXPERTS

as,

0

P.

.. .

DipTkSt-RvSti--Pliterl---FtPlts--RuP1Le--PfiiiirSt-CP1St-([RpTkSt-Res'A-P1LaSt)1TOP

)- 010kSt-RwSt)-P1,

PrPc-25IcC1 (P1Ls (ReSt-[(1/11454]-AitStil'

STOP

U(RpTkst...RwsT)-(1411,8-1r1St))-RvSt)----(Plts-Stj--((RpTkSt-ReSt)«.(PlLs-V1SSTOP

fReSt-U(RpTkSt-RvSt)-P1V1L8SX)-ReStli

P Pck.-E3 lc --ReSt---P1St -----[RpTkSt -V1] P1St RpTkgt----Ev---(Pllos-V1St)-ft

Nil 2j1cC1--1116t-----Rc-----Tk---St-----(BuSt-(PlIe-St))-----V1---Bat.---iL(RekSt -RmST) -OIL, 41St)) -Mt) -----T1 --a(RpTkAt -RI'S?) -AP11141St)1.1ReSt

w.

Pr---7---E1I0C1-----(P1Si..4(RpfikSt-RuStl-PltsSt)l---((iiipfikSt-ReSt)-(P1Ls-V1St)3-Rv30--- .

-(PILe-Y1 Stl-----[(Rokst..ayst)-(plIa-V1S03' STOP

-----01---Ic--C1-----Pc-------(RpTkStr41}-----ReSt---P1V1LeSt---(Rwit-(P1Ls-SI)) (iligkSt-RwSt)-p1T1LeStl STOP

STOP

STOP.

S ,TOP

f

RwSt---(P114-(St-ReSt))---(PlIo-St1-((lipTk4St-ReSt))-(PlLsr(St-ReSt))1-,14P1147,1Stl-Rw--4-STOP

C1-----(13-IcC1) ihrtPft-(St-RvSt))-(P1L19-(3t-RwSt)))-1(SpTk-(3t-RI(St))-(P1La-St)).sOP

R pT k- (St-Rv St ) )-P111. Le t STOP

P1-----St--PrPc-fRpTk-(St-ReSt) 3---(( 14-ISC1)-Reqt1---((P1Y1Le-(8t-ReSt))-(PiLs-Y1St) l--(14-IcC11-.

--1(R0Tk-(3t -RwSt)) -(P1Lo -St)) 'STOP

PrPc ((RpTk-(bt -ReSt)) -(P1Is -St)) STOP

-8t))- - STOP

(PrPc-(13310C1-RmSt))------((P1Ls-V1St)-ReSt)-(00k..(st.RwSt))-((PILs-Ylit)-Reit]-L-----7-16 STOP

. (EJI0C1-Rw3t)------------l(RpTk-(St41,20)-P1V1IsSt) STOP.

Pr=pleg cord into recorderRifsrevind keyVimvolitme controlOlmopen 114

Pc=plug cord tnto outlet` ctsinsert cassette.Stmatop key Plmplay keyleslisten to sound .Cloclose lid'Lb=look into battery compartment

Ejseject buttonRporncord & play keys.Tkistarksinto recorder

' Figure 3, Behavior sequences for expert.and nonexpert subjects operatingthe cassette tape recorders

. v3O'

31

EXPERTS

Ir!

START IbItc-------Tm ---C1-----TcTs ---Am ---(UlAm -Oils) ---U1 --An---CbCrRn

k.

%.I

U1 CbCrRn

>s,1,1grC1-44-14--IbirTarTsTetz

Irsinsert red leadOrsclip red lead.to residtorOrsuncilp red lead from resistorAmsadjuat meter zero screwTz=turn zero ohms knob

Figure 4.

4

Ibminsert black leadCbsciip black lead to resistorObaruncTIp black lead from resistoTesturn center rage selectorRnsread number from meter scale'

ts OZ. .

I

,//'1;sic*

1=clip leads to etherImunclip leadsxsunclip both le from resistor

ditaturn mode select r

STOP

!SOP

STOP

Behavior sequences for experts operating the VoltrOhm-Milliameter,

PERTS

."

Power All

START COR

J

CRT VI CRTTBCRTTBCRTTB ViiVITB CRT.

CRT---TB----CRT---...

lug-in-both CON ---GEN---005---Turn-on-osc CRTTB----Turn-ongen---...

over-up-gen 0311-: Power-up-macCRTCONCRT-2B ...

GENsadjust generator controls CRTaa0.4120 Ott controls TBvadjuilt time base oontrols

VImadjust vertical input.dontrola CON =e e dOnnectiousPower All =power up both unitg Pover-upmplug in and turn on

Figure.5. Highly condensed behavior sequences for experts operatingthe oscilloscope.

.

Page

Lack of fixed procedures. The fact that on the whple the eis very Tittle stereotyped. behavior seems to disconfirm, t ehypothesis suggested above, which is that device schema knowledgtightly specifies operating procedures for devices. However, ishould be pointed out that there are.some stmong consistencies inat least the initial stages of operating at least some of the \devices. For example, with the radio (Figure 2 ), all subjectsplugged in first: With the 'recorder, roughly half of theexperts and nonexperts plugged the device in'as the first step.With the oscilloscope, most of the subjects plugged in and turnedon both the oscilloscope and the signal generator before going anyfurther, but there weresome subjects that performed only part, of .

this operation before proceeding. Likewise, notice that manysubjects, after performing. the power-up operations, went on toconnect the twodevices together before proceeding, any further.Finally, with the demonstrator, again most of the subjects pluggedin the cord first, although some of them p'ug in all of the cordsand connectors before turning on the device.'

The VOM presents ari interesting contrastlEbecause it does nothave to be, plugged in and turned on. Notice that, there is verylittle stereotypy in the sequence of activities. OWe might' thinkthat inserting the test leads would be the natural first 'step, butonly two of the five subjects did this. Or one might think thatadjusting the meter to zero would' be a natural first step; onlyone of the subjects did so, although it should be noted that thisis not a routine operation in the normal use of a meter of thistype. Thus, it appears that there is some stereotyped behavior,but it is limited to some of the very initial stages of deviceoperation, and concerns mainly "power-up" procedures. If peopleindeed follow schematic procedures, these procedures are of such alimited and varied nature that characterizing them as. schemes isof little value.

$4

How sub"ects operate from memory. This lack of stereotypyrequires exp anation. Closer examination of the task situation of'operating a device from memory suggests that the expectation thatdevice operation would show stereotyped orders. is not reasonable.That is, although the devices were representatives of a veryfamiliar type of device, such as a radio, the likelihood, that anindividual, subject had actually had extensive practice withoperating, this particular make and model of device is essentiallyzero. To some extent, every device was a novel device to everysubject, Thus, none of the actual skills of operating the device_would highly automated, because this would only be the case ifone were familiar "with the specific location and properties of theparticular device. st Thus, subjects were essentially operatingthese devices in '4 problem- solving mode, instead of a memoryretrieval mode. Once the problem is looked at in this light, thelack of stereotypy in the behavior becomes clear.

In any actual device, there ere constraints that are imposedby the device on the order in which things are done. For example,on an oscilloscope, the intensity control must be adjusted beforea .trace Can be seen, and the oscilloscope.can not.be used until

lir

EXPERTS

START

atN\

PcPo TrTd--PbPa Td Pa (Ird-Pa) (Td-PrTd) ,STOP

(PrTd-Pa) STOP

Pc-PbP41------Povcz::[Td-PrPal-----(PrTd-r41-----thqs(Ta-PrP01------PrPa-----C1Pa-------STOP

Pa (Td-PrTd] C1Pa ClPa--L-C14

STOP4

c Paibk7.7.::TdPr:.---Po--Pa (PrTd-Pal-J.--(Td-PrFal----PrFa (PrPa(Td-PrPa))-----STOP

Po Td (prod -100.-----TdPr PrTd

PaPb Td- PoTo PcPo

STOP

FrPaPaPrPa----STOPTd PePo PaPb Pa (Td-BrTdj STOP

Pc =plug in power cordTd =turn dialTosturn power oft

Posturn power on Pqmplug in 1 ht A Pbsplug 4 light BPasPlip A/B mode PrsPlip ra /phase clscheck lights

'(k,

.Figure 6. Behaviorsequetices for experts operating the phi phenomenon demonstrator.

3

IPage 17

.the trace is visible. Before he intensity control can beproperly. adjusted, however, the oscilloscope must be turned on.Thus, for any device, there are some constraints on the order ofcertain coperitions. However, even for relatively simple devices,such as .a radio, these constraints in fact specify very little ofthe exact order of operation; many steps are independent of order

. given that the overall constraints are met. For example, theradio tuning can be adjusted at any time, but most Usefully afterthe radio is audibly playing. Thus, referring to Figure 2, thereare many different order's in which the expert subjects operatedthe controls on the radio, and there is a unique path for everysubject. However, all of the subjects succeeded in operating theradio, and typically with very little wasted time-or steps.

Conclusion. The best characterization of operating a pieceof. equipment from memory' seems to be that subjects perform'problem-solving by ,deterqining what constraints need to besatisfied along the way,, and then operating the controls in .s.manner that meets the constraints and accomplishes the task, butdoes not necessarily follow, any prescribed order. Since a majorconstraint is that the device be operating before it can beadjusted, there is a strong tendency for "power-up" step& to,bedone first. Since these data involve only a single observation oneach-subject in each device, it is impossible to tell whether eachsubject was.following an individual stereotyped sequence, whichseems unlikely. However, it is very clear that device operatingsequences do not have a major property of schemas, namely,'stereotypy of content.

The larger implication of this conclusion is that even thoughexperts can operate even .compaex pieces of equipment completelyfrom prior knowledge, they do not perform this by rote memoryretrievalt but rather by a very general problem-solving process.For example, the best characterization of what the experts didwith the oscilloscope is that once they had it plugged in, turnedon and connect id with a signal generator,.they made many pasdesover the controls making various fine adjustments in all'sectionsof the oscilloscope until they had achieved the final desiredresult. . Many of the operations were undoubtedly.redundant from astrictly technical point .of View. However, these generalptocesses are powerful enough that the experts. could operate thecompletely novel device, the phi phenomenon demonstrator, withoutany instructions, and quite often without any serious mistakes orwasted actions.

The general conclusion is that expertise does not consist ofa set of canned procedures for operating different devices, butratter of a set ofpowerful problem - solving heuristics which canbe applied to even novel devices, but which are not very efficienteven with familiar devices.

1'4

36"

Page 18

SUMMARY` rThe introduction listed thine questibrit.that this experiment

was designed to address. These concerned the instruction format,the nature of expertise effects, and the nature of the priorknowledge that people would have about electronic equipment. Thisexperiment yielded information about each of these three questionswhich can be summarized as follows:

Instruction Format. Contrary to intuition, the menu formatwas nt7FiTter overtil rthan the step -by -step format; which formatis supetidr lepends on the user's experience. Under, someconditions the specific experience with the actual device involvedcan be more importantthan the user's general expeitise. If the

... device is faailiar, the menu format helps, as would be expected,by r0Voing the amount of instructions hat must be read.Subjects tend not to read familiar steps ch as descriptions ofhow to power-up the equipment which everyon knows, nor do theyread descripti)nsof procedures that are very similar to ones theyhave just ompLeted. (i a device is not familiar, the user can goastray, d the'result may be much worse than using step-by-stepinstructio s in terms of total completion time.

Expertise Effects. Expertise had'both specific and generaleffects in these- results. Experts were faster overall, bothinthe menu and tie step-by-step conditions. But experience with thespecific device can be as important as the general experience.The experts were more efficient than the nonexperts in terms ofbeing abl'b to -...arry out complicated physical activities. Althougheverybody knows certain things about electronic equipment, such ashow to turn on a device, even on everyday devices the experts aremore efficient and more consistent in their activities tlian

nonexperts. .

Prior knowledge of electronic devices. It was prop sed thatsince people appareWTly have schema knowle electronicilevices, that they would also have knowledge schematic'procedures for operating devices. A primar' ch cteristic ofsuch schematis procedures would be a high degree . ofstereotypicality in, how the devices were operated when subjects,did not choose to read instructions. This expectation wascontradicted oy the' data; .there was very little stereotypedbehavior when subjects operated the devices stratlyen :the'ibasis,yf their prior knowledge. .

A more acz.urate assessment is based on making a distinctionbetween what people do when they have a highly automated skill atoperating a particular piece of equipment, and the. ability tooperate equipment in a more-normal setting in which every piece ofequipment is familiar, but not highly practiced. In this case,what subjects do is to engage in complicated problem-solvingstrategies, where the individual operating steps meet looseconstraints that are imposed by the nature of theidevice,,but donot otherwise fall into a strict stereotyped sequence. Thisproblem-solving strategy is very tobust but it is indonsistent

37

Page 19

between individuals and can be inefficient. experts clearly havemuch more powerful strategies than nonexperts for operatingdevices on the basis only of prior knowledge, but in the case ofunfamiliar eqOpment, their performance may actually beconsidfrably poorer than that of nohexperts who are followingstrict step-by-step instructions.

References

Chi, M. T. H., Feltovich, P. J., & Glaser, R. (1.981).

Categorization and representation of physics problems byexperts and novices. Cognitive Science, 5, 121-152.

Chi, M. T. H., & Glaser, R., (in press). Problem solvingabilities. In R. Sternberg' (Ed.), Human abilities:. Aninformation-processing approach. San Francisco: Freeman.

Kieran,, D. E. (1982). What people know -about electronic devices:A descriptive eriTiinCii, Report No. 12UARZ/DP[TR-82f0NR-12). University of rizona, Department ofPsychology.

Pedhazur,' E. J. . -(1982).- Multigle re ress on in behavioral'research (2nd ed.). New York: -Holt, ins a E-Winston.

ir

Smith, E. E. & Goodman, L. (1982). UnderStandin instructions:The role of explanatory material ec nical Rep t No. 59Bolt Beranek and Newman, Inc.

Arizona/Kieras (NR 157-473) 25-Jan-84

Navy

1 Robert AhlersCode N711Human Factors LaboratoryNAVTRAEQUIPCENOrlando, FL 32813

1 Dr. Ed Aiken'Navy Personnel R&D CenterSan Diego, CI 9215?

1 Dr. Meryl S. Baker

NdVy Personnel R&D CenterSan Diego,, CA 92152

1 Dr. Nick bondOffice of Naval ResearchLiaison Office, F:r LazoAPO aan Francisco, CA Q6503

1 Dr. Richard_ Cantone

Navy "esearch LaboratoryCode 7510,Washington, DC 20375

1 Dr. Fred ChangNavy Personnel R&D CepterSan Diego, CA 92152

1 Dr. Stanley CollyeiOffice of Naval Te hnology800 N, Quincy StreetArlington, VA 22217

1 CDR Mike CurranOffice of Naval Research800 N. Quincy St.Code 270Arlington, VA 22217

1 Dr. JudejranklinCode 751d-Navy Research LaboratoryWashington. DC 20375 -

1 Dr. Jim Hollan

Code 14Navy Perstnnel E z D CenterSan Diego, CA 92152

1 Dr. Ed HutchinsNavy Personnel &D Center

,San Diego, CA 952

.c

Page. 1 . -

Navy

1 Dr. Norman J. KerrChief of Naval Technical frainingNaval Air Station Memphis (75)Millington, TN 38054

1 Dr. Peter KincaidTraining Analysis & Evaluation GroupDept. of the NavyOrlando, FL 32813

1 Dr. William L. Maloy (02)Chief of Waval Education and TrainingNaval Air StationPensacola, FL 32508

Dr. Joe McLachlan ,*

Navy Personnel. R&D CenterSan Diego, CA 92152

1 Dr. James McMichaelNavy PersonnelR &D CenterSan Diego, CA 92452

1 Dr William Montague'NPRDC Code 13San Diego, CA 92152

1 Technical DirectorNavy Personnel R&D Center .**San Mato, CA 92152--

6 Commanding OfficerNaval Research LaboratoryCode 2627Washington, DC 20390

1 Office of Naval ResearchCod0433800 Ih.-Quincy SStreetArlington'. VA 22217

6 Perminel & Training Research GroupCode ill2PTOffice of Naval ResearchArlington, VA 22217

1 Office of the Chief of Naval OperationsResearch Development & Studies BranchOP 115Washington, DC 20350

.ci

Arizona/Kieras(HR 157-473)

Navy', .

25-Jan-84,

.1 Dr. Rtibert..G, Smpt

Office of Chief`of Naval Operationi

OP-987PWashington, .DC 20450

1 Dr, Alfred F:.;Modi, Director

,Training A:1,443'84 EvalusVon Group'Dept. of the Navy.Orlando, FL 32813

Dr.)tichard SnowLiaison ScientistOffice of Naval Research

Branch Office, LondonBox 39FPO New York, HY 09510

1 Dr. Richard SorensenNavy 'Personnel R&D CenterSan Diego, CA 92152

1 Dr. Frederick SteinheiserCHO OP115Navy AnnexArlington, VA 20370

1 Dr. Thomas Stipt'Navy Personnel R &D CenterSan Diego; CA 92152

1 Roger Weissinger-BaylonDepartment drAdmimistrative Sciences

,Naval Postgraduate School :

Monterey, CA 93940

1 Mr John H. Wolfe

Navy Persohnel R&D CenterSan Diego, CA 0,4C^

milm,

01

`Marine Corps

LH. William GreenupEducation4Advisor (E031)Education Center, MCDECQuantico, A.221341,

.

L

1 Special Aisistant for MarineCorps atters 4

I'MCode 1 OM'Offic of Naval esearch800 N. Quincy st;Arlington, VA-42217

'

, $

1 DR. A.L. SLA KYSCIENTIFIC ADV SOR (CODE RD-T)HQ,.U.S. MARIN CORPSWASHINGTON, bcf.2dso

Page, -2

40

1

1.

o

4

p

ArizonakKier0s (NR 0157-473)*

. Army

V1 Technical Director

U. S. Army. Research'Institute for theBehavioral and Social Sciences.5001 Eisenhower Avenue .

Alexandria. VA 22333

1 Dr. Beatrice J. Tarr .

U.. S. May Research InstitUte5001 Eisenhower AvenueAlexandria VA 22333

25 -Jan -64

1 Dr. Harold F. O'Neil. Jr:Director, Research LabArmy Research In titute'5001 Eisenhower AvenueAlexandria. VA 22333

1 Commander, U.S. Army Research Institutefor the Behavioral & Social Sciences

ATTN: PERI-BR (Dr. Judith Orasanu) ..

5001 Eisenhower AvenueAlexandria: VA 22333

1 Joseph Psotka, Ph.D.TTN: PERI-10

A m leseardh Institute5001 Eisenhower Ave.AlAxandria. VA 292ra

1 Dr, Robe Sasmor

U. S. Arid Research Institute for thet

Behavioral and Social Scienceitir

lr 5001 Eisenhower Avenue'Alexandria, VA 22333.

ftemmi

Air F rce

1 U.S Air Fore4 'Offiae -Of Scientific

esearch. L fe.Sciences,DIrectorate, NL

lling Air. Force Base

Washington, DC W332 I,

Dr. Earl A. Alluisi./ 'HQ. AFHRL (AFSC)

Brooks AFB, 7X.78235

, 1 Mr. Raymond E. ChristalAFHRL/MOEBrooks AFB, TX 78235

1 Bryan Dal4tlib

AFHRL/LRTLowry AFB, CO 8023p

1 Dr, Alfred Fregly,

AFOSR/NLBolling-AFB, DC 20332

1 Mk Genevieve HaddadProgram ManagerLife Sciences DirectorateAFOSR

Bolling ea. DC 20332'

1 Dr. T. M. LongridgeAFHRL/OTE

Williams AFB. AZ 85224

1 Dr. John TangneyAFOSR/NLBolling AFB. DC 20332

1 Dr. Joseph-4asstuke,AFHRL/LRTLowry AFB,CO 80230

pm.

/ F

41

r

irimgria/Kiei-as oiR 157 j1, 25-Jan-84..Ii

. DeAftment of Defense

12 Defense Techpioal'Inforiation Center. ,

Cameron Station, Bldg 5Alexindria, VA 22314.Attn: TC

- 1 Military 'Assistant tor .TraiTing and

Personnel TechnOlogy.Oftice of the Under Seepgary of Defensfor Research & EngineeringRoom 31)129,111e Patagoh.Wasiiington. DC 20101

1 Majgr Jack Thorpe .

DARPA1400 Wilson Blvd,

Arlington VA 22209-)

1 Dr. Robert A. Wisher

OUSDRE (ELS)'The_ Pentagon, Room 3D129Washington, DC 20301

O

.

4 Page 4.

Civilian Agencies

l'br.jOktricia A. ButlerNIE-BRN Bldg, Stop 1 71200 19th St., NWWashington, DC 20208'

1 Dr. Susan ChipmanLearning and DevelopmentNational Institute .of- Education1200 19th Street NW %ow,

Washington, DC 20208

1 Dr. Arthur% Melmed

724 BrownU. S. Dept. of Education-Washington, DC .20208

L

1 Dr. Andrew R. MolnarOffice of Scientific and Engineering

Personnel and EducationNational Science Foubdation

- Washington, DC 20550

1 Dr. Everett PalmerMail Stop 239-3NASA-Ames Research CenterMoffett Field, CA 94035

1 Dr. Mary StoddardC 10, Mail Stop B296Los Alamos National LaboratoriesLos Alamos. NM 87545

1 Chief, PsychologicalcReserch BranchU. S. Coast Guard (G-P-1/2/TP42)Washington, DC 20593

1 Dr. Frank Withrow , -

U. S. Office of Education400 Maryland Ave. SWWashington, DC 20202

. 1 Dr. Joseph L. Young, DirectorMemory & Cognitive ProcessesNational Science FoundationWashington, DC 20550

42

te

!hi

Arizona/Kieras (NR 157-473)

Private Sedtor

1 Dr. John R..Anderson -

Department of PsychologyCarnegie-Mellon UniversityPittsbqrgh, PA 15213

1 Dr. Alan Baddeley

Medical Research CouncilApplied Psychology Unit15 Chaucer RoadCambridge CB2 2EFENGLAND

1 Eva L. BakerDirector.

'UCLA Center .for the Study of Evaluation145 Moore HallUniONty of California, Los AngelesLos An les, CA 90024

25 4an -84

.

1:Mr. AvromBarrDepartment of Computer ScienceStanford UniversityStanford, CA.94305

1 Dr. Menucha BirenbaumSchool of EducationTel Aviv UniversityTel Aviv, Ramat Aviv 69978Israel

1 Dr. John BlackYale UniversityBox 11A. Yale StationNewHaven, C1 06526

1 Dr. John S. BrownXEROX Palo Alto Research Center3333 Coyote WadPalo Alto, CA 94304

1 Dr. Glenn Bryan6208 Poe RoadBethesda, MD 20817

Alb1 Dr. Bruce BuchtnanDepartment of Computer ScienceStanfordUniversityStanford, CA,94305

1 Dr. Jet me CarbonellCarnegie-Mellon UniversityDepartment of Psychology.Pittsburgh, PA '6213

4

v.

Private Sector ,

.1 Dr . "Pat Carpenter

pepartment of PsycholOgy .

Carnegie-Mellon University'Pittsburgh, PA 15213

1 Dr. Micheline Chi

Learning R & D CenterUniversity of Pittsburgh

3939.01HaraStreeiPittsburgh? PA 15213 je,e'

1 Dr. William Clincey,Departmelit or Computer ScienceStanford UniversityStanford, CA 94306

1 Dr. M(chael ColeUniversity of Californiaat Sat! 'Diego

Laboratory df ComparativeHuman Cognition - D003A

La Jolla, CA 92093

1 Dr. Allan M. Collins

40, Bolt Beranek & Newman, Inc.50 Moulton-S&eet sp

Cambridge, MA D213,

1 Dr. Kenneth B. Cross,"Anacapa Sciences, Inc.

P.O. Drawer Q,Santa Barbara: CA 93102

1 Dr. Emmanuel DonchinDepartment of PsychologyUniversity of IllinoisChampaign, IL 61820

it

1 Dr. Thomas M. DuffyDepartment of English.

rchnegie-Mellon Universityenley Park

Pittsburgh, CA 1213

1 ERIC Facility-Acquisitions4833 Rugby AvenueBethesda, MD 26014'

1 Dr. Anders EricssonDepartment of PsychologyUniversity of Colorado

\____ Boulder, CO 80309

. 43

I

irizalla/Kieeas fug 15^-473)

Privatt Sector '

1 Dr. Paml Feltovich'Department of Medical 'EducationSouthern'Illinois University

. School'of Medicine .

P.O. Box 3926,.'Springfield, IL 62708 0

' 1 Professor Reuven FeuersteinHWCRI Rehov KarmonBet HakeremJerusllem

25-Jan-84

Mr. Wallace FeurzeigDepartment of Educational Technology'Bolt Beranek &'Newman10 Moulton St.Cambridge; MA 02238

1 pi..Dixter FletcherUnUersity of OregonDepartment of Computer ScienceEugene, OR 97403

1.Dr4,:John R. FrederikAco,

Bolt Beranek.& Newman

5O Moulton Streetcambridge, !4A 02138

,

Dry. Michael Genesereth

DepartMent of Computer ScienceStanfoi-d University

Stanford, CA 94305

V-

. /4 Dr. Don Qentner'Center...1'0r Human Information Processing

Uni4ersity oSCaliforrife, San DiegoLa .Jolla, CA 92093

Dr. Dedre GentnerBoltBeranek & Newer)10 Moulton St.Cambridge, MA.02138

a

1 Dr. Robert, Glaser

Learning Relear41 & Development CenterUgiversity of Pittsburgh

3939 O'Hara StreetPITTSBURGH, PA 15 60

1 Dr. MarVin D. Glo k217 S One HallCornell' UniversityIthaca, NY 14853

AV

Ptivite SectorI%

1 DA. Josph Goguen

SRI International33 Ravenswood Avenut/Menlo Pailt, CA 9402k

1 Dr Daniel Gophers

Faculty of Industrial EngineeringIt& Management

TECHNIONHaifa 32000ISRAEL

1 Dr. Bert GreenJohns Hopkins UniversityDepartment of Psychology'Charles & 311th StreetBaltimore, MD 21218

1 DR. JAMES G. GREENO

LRDCUNIVERSITY OF PITTSBURGH

3939 O'HARA STREETPITTSBURGH, PA 15213

'"

1 Dr. Barbara Hayes-RothDepartment of Computer ScienceStanford University -

Stanford, CA 95305

10 Dr. Joan I. Heller

Graduate Group in Science andMathematics Education

c/o School of EducationAe. University of California,

. Berkeley, CA 94720

1 Dr. James R. RoffmanDepartment of PsychologyUniversity of Del .aware

Newark, DE 19711

4

'

1 American Institutes for Research1055 Thames Jefferson St., N.W..Washington, DC 20007

1 Glenda Greenwald, Ed.Human Intelligence NewsletterP. O. Box 1163Birmingham, MI 118012

1 Dr. karl HuntDept. of PsychologyUniversity of WashingtonSeattle, WA 98105

.44

Ak4A

zona/Kieras (NB 157-473) ' 25-Jan-84 '

Prqvate Sector

1 Robin JeffriesComputer Research CenterHewlett- Packard Laboratories

' 1501 Page Mill RoadPalo Alto, CA 94304

1 Dr. Marcel JustDepartment of Ps apologyCarnegie- Mellon University

Pittsburgh, PA 15213 f'

. 1.

1 Dr. WallereKin..su.h

Department of PsychollgYUniversity of ColoradoBou , SO RO1n2

1 Dr. D Klahr rDepartmen of PsychologyCarnegie- Mellon Univerqity

Schenley ParkPittsburgh, PA 15213

1 Dr. Stephen Kosslyn1236 William James Hall33 Kirkland St.

, Cambridge, MA 02138

1 Dr. Pat LangleyThe Robotics Institute-

. Carnegie-Mellon University- Pittsburgh, PA 15213

1 Dr. Jill LarkinDepartment of PsychologyCarnegie Mellon UniversityPittsburgh. PA 15213-

1Dr. Alan LesgoldLearning R&D Center '

University of PA.....scurgh

3939 O'Hira Str.letPittsburgh, PA 15260

1 Dr. Jim Levin.University of California

' at San DiegoLaboratory for Comparative

' Human Cognition - D003A

La Jolla, CA 92093

/ha

.'

Private Sector

1 Dr. Michael LevIne:.

Department of Educational Psychology'210 Education Bldg.University of IllinoisChampaign, IL -61801

1 Dr. Marcia C. Linn

Lawrence Hall of ScienceUniversity of.CaliforniaBerkeley, CA 94720

1 Dr. Don LyonAFHRL/OT (UDRI)Williams AFB, AZ 852251

1 Dr. Aay McClellandDepartment of PsychologyMITCambridge, MA 02139

1,Dr. James R. MillerComputernhought.Corporation1721 West Plario'HighwayPiano, TX 75075

1 Dr. Mark Miller ,ComputeriThought Corporation

1721 West Plano ParkwayPlano, TX 75075

1 Dr. Tom Moran'Xerox PARC3333 Coyote Hill RoadPalo Alto, CA 94304

1 Dr. Allen Munro

Behavioral Technology Laboratories1845 Elena Ave., Fourth FloorRedondo 'Beach, CA 90277

1 D. Donald A NormanCognitive Science, C-015Univ. ofPCalifornia, San DiegoLa Jolla, CA 92093

. 1 Dr. Jesse OrlanskyInstitute for Defense Analyses1801 N. Beauregard St.Alexandria, VA 22311

45 Ito

Arizona/Kieras (NR 157-473)

Private Sector

1 Dr. !limy PenningtonUnivensity of Chicago

Graduate Shoo of BLsiness1101 E. 58th St.'Chicago, IL 60637

likDR. PETER POLSONDEPT. OF PSYCHOLOGY

'UNIVERSITY OF COLORADOBOULDER, CO 80309

1 Dr. Mike PosnerDepartment of PsychologyUniversity of OregonEugene, OR 97403

1 Dr. Lynn RederDepartment of Psychology

CarnegierMellon.UniversitySchenley Park

PA 15213

1 Dr Fred ReifPhysics DepartmentUniversity of CaliforniaBerkeley, CA 94720

1 Dr. Lauren ResnickUGCUniversity of Pittsburgh

-3939 O'Hara StreetPittsburgh, PA 1521

1 Dr. Jeff RichardsonDenver Research InstituteUniversity of DenverDenver, CO 80208

'1" Mary S. Riley

0

25 -Jan -b4

. Program in Cognitive Science',enter for Human Information ProcessingIniversity of California, San DiegoLaJolla. CA 92093

1 Dr. Andrew M. RoseAmerican Institutes for Research1055 Thomas Jefferson St. NWWashington, DC 20007

1 Dr. Ernst Z. Rothkopf.Bell LaboratoriesMurray Hill, NJ 07974

Page

Private Sector

1:Dr. William B. RouseGeorgia Institute of TechnologySchool of Industrial & Systems

Engineering',Atlanta, GA 30332

1 Dr. David RtImelhart

Center for Human Information Processing.Univ. of California. "San Diego

La Jolla. CA 92093

1 Dr. Michael J. SametPerceptronics, Inc.6271 Variel AvenUeWoodland U113.4091364

1 Dr. Roger SchenkYale UniversityDepartiment of Computer Science

P.O. Box 2158.New Haven, CT 06520

1 Dr. Walter SchneiderPsychology Department

603 E. DanielChampaign, IL 61820

1 Dr. Alan SchoenfeldMathematics and EducationThe University of Rochester

Rochester, NY 14627

1 Mr. Colin SheppardApplied Psychology UnitAdmiralty Marine Technology Est.Teddington, MiddlesexUnited Kingdom

1 Dr. H. Wallace SinaikoProgram DirectorManpower Research and Advisory Services .

Smithsonian Institution801 North Pitt StreetAlexandria, VA 22314

1 Dr. Edward E. SmithBolt Beranek & Newman, Inc.50 Moulton StreetCambridge, MA 02138

46

Ar zona/Kieras DIR 157-473)

Private Sector 4.4

I Dr. Eliott SolowaYYalakUniversityDep4etment of Computer ScienceP%O. Bdx 2158New Haven, CT 06520

1 Dr. Kathol-T. SpoehrPsychology Apartment'Brown UniversityProvidence, RI 02912

1 Dr. Robert SternbergDept. of PsychologyYale UniversityBox 11Ar Yale Station .New Haven. CT 06520

1 Dr., Albert Stevens

Bolt Beranex & Newman, Inc.10 Moulton St.Cambridge, MA 02238

1 David E. Stone, Ph.D..Hazeltine Corporation

7680 Old Springhouse RoadMcLean. VA 22102

25-Jan-84

1 Dr. Kikumi TatsuokaComputer Based Education Research Lab252 Engineering Research LaboratoryUrbana, IL 61801

1 Dr. Maurice Tatsuoka220 Education Bldg1310 S. Sisxth St.

Champaign ,'1L 61820

1 Dr. Perry W. ThorndykePerceptronics, Inc.545 Middlefield Road, Suite 140Menlo Park, CA 94025

ter. Douglas TowneUniv,pf So. CaliforniaBehavioral Technology Labs1845 S, Elena Ave.Redondo Biaoh. CA 90277

1 Dr. Kurt Van LehnXerox PARC3333 Coyote Hill RoadPalo Alto, CA 911.304

47

.Page

Private Sector..

1 Dr. Keith T. WescourtPerceptronics, Inc..545 Middlefield Road, Suite 140Menlo Park, CA 94025

1 William B. Whitten

Bell Laboratories2D-610Holmdel, NJ 07733

1 Dr. MOM8S WickensDepartment of Psychology

Franz Hall1 University 'of Cal,ifornia

405 Hilgarde Avenue.40s Angeles, CA 90024

1 Dr. Mike Williams

IntIlliGenetics124 University AvenuePalo Alto, CA 94301

1 Dr. Joseph WOhl

Alphatech, Inc.

2 Burlington Executive Center111 Middlesex TurnpikeBurlington, MA 01803

a