COMPUERTECHINQUES FORCLUSER ANALYSISU)COORADO 83 … · I000O1.4-78-C-043 3 R. Michael Perry S. PERFORMING ORGANIZATION NAME AND ADDRESS 19. PROGRAM ELEMENT. PROJECT. TASK AREA a

AD-A28215 COMPUERTECHINQUES FORCLUSER ANALYSISU)COORADO 1/UNIV AT BOULDER DEPT OF COMPUTER SCIENCE R M PERRYAPR 83 CU-CS-244-83 NOVA 4-78-C 0433

-- I ND

(LIMB L.0.533.61111 '*'i~ L l

IL125 11.4 Ii

MICROCOPY RESOLUTION~ TEST CHARTNATIONAL BUREAU OF STANDARDS.1963-A

0 ,.!.w W

3-...

. . .. .. ..

. . . . 'PIP;' '~ I

-to *33

71 !4 I -X

333 V

C"WPUTrA TECANiIiULS FOR CLUSTR NA4YSIS

R. Michael Perry*

Cd-CS-2'i4-a3 April, 1983

*LDepartient of Computer Science, University of Colorado,

Boulder, CO O3U9

This rsearch was supported by OR grant number N00014-78-C-0433.

DTIC

SMAY 1719833This docmnont has been approved Afor pi:blic release and sale; itsdistibution is unlimited.

SECURITY CLASSIPICATION OF THIS PAGE (W*m D .Ia E te~ed. . .......__ _.. .. ......_ _-_

RE"D ISTRtJCTMOSREPORT DOCUMENTATION PAGE aZRoin couPLZrM MMw

1. RIPoT NUMSER OVT ACC EISOM N RECIPIENTIS CATALOG umRa

tA f __4_ix__/_f

4. TITLE (e Sumtl) S. TYPE OF REPORT I PEIO COVEREDiTechnical Report

Computer Techniques for Cluster Analysis Technical _Report

S. PERFORMING ORG. REPORT NUMBER

7. AUTNORfa) 6. #OM*,ACT ON GRANT MuMefa)fe

I000O1.4-78-C-043 3

R. Michael Perry

S. PERFORMING ORGANIZATION NAME AND ADDRESS 19. PROGRAM ELEMENT. PROJECT. TASKAREA a WORK UNIT NUMlERS

Department of Computer Science

Campus Box 430 NR 157-422University of Colorado, Boulder, CO 80309

It. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATEPersonnel and Training Research Programs April 1983

Office of Naval Research, Code 458 13. NUMEROFPAGESArlington, VA 22217 27

14. MONITORING AGENCY NAME G ADDRESS(0I differmt from Controlling Offic.) 1S. SECURITY CLASS. (of this lopoff)

Unclassified

IS,. DECLASSIFICATON/OOWNGRADINGSCHEDULE

1. DISTRIBUTION STATEMENT (of this Reo o).

-, .. d 0'o

4t4 C

17. DISTRIBUTION STATEMENT (of the al act entered In Block 20. It diffefrtl fro Report) . .

cc-le

IS. SUPPLEMENTARY NOTES N.t" .

IS. KEY WORDS (Conthnw on ees,e siel it nec*esr and Idetify by block number)

Cluster analysis, conceptualization, list structures, trees,list processing, metrics, distance relations.

20. ABSTRACT (Co uthn n r e ,., side It noeeowr amE dntify by Meek muw#6w)

Cluster analysis is a method for understanding the spatial arrangement ofpointlilke objects. It is practiced informally when stars are seen as forminggalaxies or grains in film are viewed as depicting an image. This paper describessome computer techniques for cluster analysis of a set of points when distancesbetween the points are known In general points that are close together will begrouped in the same cluster. Moreover, clusters of points can be treated as sin-gle points and grouped into higher-order clusters, thereby obtaining a hierarchi-cal arrangement that depicts large-scale features and fine detail as well. These

DD J 1473 EDITION Or I NOV S IS OBSOLETE

SECURITY CLASSIFICATION OF THIS PAGE (iWhm Des £teaeq

SCCUtTY CItASOFICATIOR OF TWIS PAG9(Phm Does go

techniques have been used in psychological studies of conceptualization andmemory retention at the University of Colorado. and the applications are brieflyreviewed.

S9CURITY CLASSIFICATION OF THIS PAGE(WhS Dalt anfemr4

Computer Techniques for Cluster AnalysisR. Michael Perry

AbstractCluster analysis Is a method for understanding the spatial arrangement of

pointlike objects. It is practiced informally when stars are seen as forminggalaxies or grains in film are viewed as depicting an Image. This paper describessome computer techniques for cluster analysis of a set of points when distancesbetween the points are known. In general points that are close together will begrouped in the same cluster. Moreover, clusters or points can be treated as sin-gle points and grouped into higher-order clusters, thereby obtaining a hierarchi-cal arrangement that depicts large-scale features and fine detail as well. Thesetechniques have been used in psychological studies of conceptualization andmemory retention at the University of Colorado, and the applications are brieflyreviewed. -

1. IntroductionOne of the basic mechanisms for understanding the world is to search for

groupings or arrangements among objects that are distinguished primarily byspatial location. Thus stars are seen as forming constellations, clusters andgalaxies. Grains in film similarly reveal an image even when they can be seenindividually. Much can be learned by the recognition of groupings or clustersamong pointlike objects, and techniques for automating this process thus are ofinterest.

This paper describes some computer techniques for cluster analysis of a setof points when distances between the points are known. These techniques weredeveloped in coninection with studies of human conceptualization and Mmoryretention at the University of Colorado. The algorithms and computer imple-mentation are described and their usage in the psychological studies is brieflyreviewed.

2. TheoryIn general in a clustering problem we are given a finite, nonempty set of

points S with an implied distance relation, and are asked to find a "clustering"under which points that are close together will tend to be in the same groupingor cluster. Thus the mutual distances are regarded as defining a spatialarrangement of the points. The purpose of clustering then is to furnish anunambiguous interpretation of the structure of the spatial arrangement. Theinterpretation in turn will depend on the method chosen for clustering, but itcan be hoped that significant features will not depend strongly on the methodthat is used.

Thus in particular the notion of "cluster" implies an arrangement or struc-turing of an "underlying set" of points chosen from S. into a set containing indi-vidual points or constituent clusters, with the additional requirement that theunderlying sets of different constituent clusters must be disjoint. For formalpurposes we define clusters over S inductively as follows.

1. Individual points p E S are clusters. The underlying set of a point p istaken to be Lei. and p is then said to be an atomic cluster and to have order 0.All other clusters will be nonatomic and will have order >0.

2. A nonatomic cluster C will have the form of a nonempty. noisingletonset of clusters of lower order, which in turn will be called its covatutwyts. Theone additional requirement will be that the underlying sets of these coutituentsmust all be disjoint. The clusters themselves will then be said to be diqoint. Theorder of C is defined as 1 + the maximum order of its constituents. 1he under-lying set of C is defined as the union of the underlying sets of its constiuents.

3. All of the clusters over 5 are obtained by applying rules (1) and (2)In summary, (I) individual points are clusters. (2) any nonempty. aonsingle-

ton set of clusters whose underlying sets are disjoint, is a cluster, and (3) theseare the only clusters. Thus a cluster is a tree with the branches unordered. Inparticular there are only finitely many clusters over S If S is a singleton thenthe only cluster Is the single member of S (but not S itself). For larger S, Sitself will be a cluster over S. in addition to its individual points. If S = J 1, 2, 31the clusters over S are 1. 2. , 3. ,21.|1, 3J. 2,. 1, I.2, 3f, ItI, 21. 31,.111. 31. 21.

11, 12, 311. For larger sets there are many more clustersGiven a cluster C over S. a subcluster is defined inductively as either C. or

if C is nonatomic. one of its constituents or a subcluster of one of Its consti-tuents. Thus in view of assumption (2) the underlying sets of two subdusters ofC must be disjoint, or one underlying set must be included in the other.

A ctusteris of S is a cluster over S whose underlying set is S. Thus in theabove illustration there are four clusterings of the three-element set S. and inaddition, six other clusters over S. Although many clusterings of anysizable Sare possible, the meaningful ones, from our point of view. are those in whichobjects that are close together are grouped into clusters of low order. More-over, to form higher-order clusters in a meaningful way it is necessaryto extendthe measure of distance, assumed to be given for the individual points, to arbi-trary clusters. In this way the entire set S can be formed into a cluster thatreveals large-scale structure as well as fine detail in the spatial arrangement ofpoints.

Something should be said about the allowable measures of distance,whether between points or clusters. In the most restrictive case the distancefunction d was required to be a metric, that is to satisfy the following propertiesfor points (or clusters) x, y and z.

1. d(z.) k 0, with equality holding if and only if z = V.2. d(z.y) = d' ..z),3. dxz! x.)d a.

However these properties were not always enforced, particularly in tie case ofnonatomic clusters. The minimum conditions that were always enforced were:

d(x.y) a 0, with equality holding if z = V.

That is, the distance was always required to be nonnegative and relexive, orindependent of the direction of measurement.

The methods of clustering of interest here, then, are distwni -baend.There are two main steps In formulating a method of this type: (1) establishingthe distance-based criterion under which two objects (whether points or clus-ters) will always be placed In the same cluster, and (2) extending the reasure ofdistance as far as necessary, so that a distance is defined between any two clus-ters that would be considered for Inclusion in a cluster of higher order.

-3-

Clustering then proceeds from the original, unstructured set of pointsPoints that meet the necessary criterion are grouped into clusters. One suchcluster may contain more than two points because clustering is transitive, thatis. if a and 6 are in the same cluster, and b and c are in the same cluster, thena and c must be in the same cluster. In general neighboring points will begrouped together. On the other hand a cluster may consist of a single point if nosuitable neighbor can be found In any ease, the points are grouped into clus-ters as far as the criterion allows, then the process is iterated, clusters beinggrouped into clusters of higher order in this manner the entire set of points isfinally structured into one large cluster Thus on each iterative step, if thereare at least two objects. that is, if the final cluster has not already been reached,at least two objects must be grouped into a cluster.

3. Methods for Distance-Based ClusteringIn the work reported here there were two main criteria for placing objects

in clusters during one iterative step of analysis. In the ftrst version two objectswere put in the same cluster if either object was a nearest neighbor of the otherIn the second version the two objects had to be mutuaL nearest neighbors to beguaranteed placement in the same cluster. In either case the distance measurehad to be extended to higher-order clusters. The main means of doing this wasto define the distance between two clusters as the separation distance betweenthe underlying sets, that is. the minimum distance from a point in one underly-ing set to a point in the other.

Other measures than the separation distance, which is not a metric, wereused on occasion It was found, however, that even when the properties or ametric were enforced the results were not much affected so that the separationdistance, which is easy to compute, became standard.

The methods of clustering, then, were primarily distinguished by how theyplaced objects in the same cluster during one step of analysis. The first versionwill be referred to as the nearest --neighbor method and the second (for reasonsgiven later) as contouring Both methods are illustrated in fig. 1, in which ahypothetical array of five points is to be clustered based on the usual Euclideandistance in the plane. This initial array is shown with the points labeled in (1a)The successive steps for the nearest-neighbor method are shown in (lb-c) andthe steps for contouring are shown in Id-f).

Thus the nearest-neighbor method takes only two iterative steps to com-plete the clustering, while three are needed for contouring. Moreover on eachstep of the first method any cluster is paired with at least one other one (since itmust have a nearest neighbor), thus all constituents of a cluster must have thesame order. Contouring, however, allows constituents to have differing orderThus in the clustering of (Ic) every ccnstituent has order 1 while in that of (If)the two constituents have orders 0 and 2.

-4-

4. .t 3z

C) 0ab c

d e f

Fig. i, showing the successive stages of clustering for two differentmethods (a) original array of points; (b-c) nearest-neighbor method; (d-f) con-touring.

The two clusterings, moreover, show significant differences in structureThus in (c) point I is paired with points 2 and 3 in one constituent. whilh in f)it is isolated as a constituent by it self In ') on tie oihr i. d. pil.. 1 .1,I .form one constituent while in k'f) two clusters consistinrg r'spect ively of 1pm:;and 3 and points 4 and 5. are formed into one constituent. All this is a conse-quence of the fact that point I is a considerable distance from the other points,even its nearest neighbor, point 3 Thus the first method is forced to grouppoint I with point 3, regardless of the distance, while the second method, by iso-lating point 1 and forming the other points into one constituent, is able to give abetter indication of the actual distances involved in the spatial arrangement.

In fact for the second method each cluster of any order has a "dispersiondistance" such that (') the constituents are within this distance of their nearestneighbors which ar- also within the cluster and (2) any other, disjoint clustermust be at a greater distance from any constituent and thus from the cluster asa whole The given cluster, then, is contained in a "contour" drawn at the disper-sion distance around its constituents, and thereby is isolated from all other dis-joint clusters, thus the method has been referred to as "contouring". In generalcontouring yields a more detailed structure that better reflects the largedisparity that may exist among the nearest-neighbor distances. Often, however,there is a great deal of fine structure for this method so that some "coarsening"-- removal of contours -- is helpful in visualizing the larger-scale structure.

.4. Algorithms

3oth methods of clustering can be carried out efficiently, that is. in timethat is polynomial in the number of points in the set, !S , with reasonableassumptions about the representation of the points in S and the difficulty ofcomputing distances between them. On each iterative step of clustering wemust determine the distance between clusters; clearly this is a polynomial-timeoperation since it depends only on the distances between points in the underly-ing sets, which are disjoint subsets of S. (A polynomial-time clustering was alsoobtained with other definitions of the inter-cluster distance.) For the nearest-neighbor method any cluster is joined with at least one other one on each itera-tive step, thus the number of clusters is reduced by at least half and the totalnumber of steps therefore is not more than log2 S j. With contouring it is possi-ble for only two clusters to be joined on each step so the number of steps couldbe as large as iS -1, but the timing will still be polynomial in 1 S . (In practiceit has not been excessive compared with the other method.)

An important feature of many of the clustering problems to date is thatmany of the point-to-point distances are infinite (in practice, a very largenumber) Thus a boolean relation is given such that points are a finite distanceapart. or are "connected" only if the relation holds between them. Typically theboolcan cornections are not much more numerous than the points themselvesbut they always form a connected graph, so that a path of connections can beformed between aiy two points. This means that any reasonable distance-basedclustering will have all constituents of any subcluster at finite distances fromtheir nearest neighbors.

The nearest-neighbor method is implemented as follows. On each iterativestep an array of clusters is formed. These clusters are to be grouped into clus-ters of the next-higher order. To each cluster in the array is associated its"adjacency list" -- those clusters at a finite distance from it. These clusters aresorted by increasing distance and those at the smallest distance -- the nearestneighbors -- are "marked" along with the original cluster, with a number denot-ing the cluster of next-higher order into which they will be placed. Meanwhile

PIP-

-B--

the original cluster is placed on a list assigned to this number This list, ineffect, is the cluster of next-higher order that is being formed.

The marking itself proceeds by examining each cluster in the array in turnIf a cluster has already been marked with a number ths number is retained,otherwise a new number is created Then the nearest neighbors of the cluster,are examined All of these should be marked with the sane number as the origi-nal cluster ['or those that are. nothing is done If arnv have not yet beenmarked at all, then they are assigned the new number and are added to theclu-ster list for this number, Ilowever, if a cluster has already been marked witha different number, then '%) the cluster list assigned to the old number isretrieved, (2) all clusters in this list. together with the neighboring cluster itself.are marked with the new number. '3) all these clusters are added to the clusterlist for the new number, and (1) the cluster list for thr old number is madeempty In this way clusters of the next-higher order aregradually built up and,whenever two clusters touch each other, they are coalesced into one.

When the process is complete the numbers denoting the higher-order clus-Lers are examined. For those having nonenipty lists rh hsts are retrieved --these are the clusters of next-higher order. If there is only one such cluster theanalysis is complete. Otherwise the iterative step is repe;ted with the new clus-ters

For contouring we could use the same approach, that is. form the new clus-ters the same way on each step, except that now the neihiboring clusters wouldhave to be mutual nearest neighbors, in keeping with the rationale for contour-ing. Instead we use a different algorithm which sometimes is much faster, andappears to be faster in general This algorithm is recursive rather than itera-Live Initially it is g v'n list of clusters to be formed into c clustering If this listis a singleton, that is. if it has only one element, then jist the one element isri-turned If it contains two elements or is larger but al the nearest-neighbordistances are the same, then the list itself is returned. Otherwise a nontrivialclustering is carried out.

First clusters arc formed into an array and to each is associated an adja-cency list as before. Next a "contouring distance" is chosen. Currently this isthe median of the nearest-neighbor distances of all clusters in the array (with agiven dis',mce being counted the number of times it occurs, rather than justonce), but other choices would also be acceptable, as in noted later. Any twoclusters that are not more than this distance apart are giouped together into acluster of next-higher order

The grouping into clusters is done the same way as n the nearest-neighbormethod except that different clusters are treated as "nearest neighbors", in thiscase, those that are not more than the contouring disLnce from each otherThus () not all clusters grouped together will be nearest neighbors and (2) notall nearest neighbors will be grouped together. A cluster that is more than thecontouring distance from its nearest neighbor will remin isolated. When thegrouping into clusters is complete, then, there will be sane newly-formed clus-ters of higher order and some isolated clusters that were .iot grouped.

Next the algorithm is applied recursively to each newly-formed cluster inturn using its constituents as the Initial list of clusters. In this way these Hu;-ters are structured as they will appear in the final clustering. Finally the algo-rithm is applied recursively again, this tLime to the list containing all the newlystructured clusters, together with the older, isolated ones. This results in thefinal clustering.

In particular it is easy to show that. unless the algorithm is given a single-ton list of clusters to start with, this case can never arise. Instead the

-7-

clustering will proceed recursively until each list to be clustered has all thenearest-neighbor distances the same and no further clustering is possible Forthe algorithm to work properly, however, the contouring distance must bechosen so that clustering will go to completion Any distance less th&' the max-imum nearest-neighbor distance, and not less than the minimum sud distancewill do. though certain distances are expected to result in faster clustering.

A particular case that illustrates how timing depends on the method is onein which the points, call them p,. p2. . p,. are structured so thal their dis-tances d(p,. pi i) form a decreasing sequence of (finite) values while all dis-tances d , p,) for ,i-j >I. are infinite. Thus p,1 will be the unique) nearestneighbor of p, whenever " < i < n but only p,, and p,, will be mutual nearestneighbors If these are formed into a (two-constituent) ctuster then Dnly it andN 2 of the remaining points will be mutual nearest neighbors, and so on

Thus if we applied the nonrecursive technique initially suggested for con-touring in %hich only the mutual nearest neighbors would be grouped on edhiterative step only one nevo cluster could be formed on each stcp and thenumber of steps needed wou!d be n- . Each of these ini turn would require timeproportLionad to the number of clusters so the overall timing would )e propor-tional to n 2

With the recursive algorithm, using the median of the nearest-naghbor dis-tdnce, as the contouring distance, the size of the problem is reduced by half foreach of the two recursive calls that follow the initial call, so the tirring will beproportional to nlog2 n The timing will vary if a difTerent choice of the contour-ing distance is used. For example. suppose it is chosen to be the kth imallest ofthe nearest-neighbor distances, with k/1n close to some constant c between 0and "(For the median c would be O.) The timing can be shown to .e propor-tional to nlogon where b = 1/ mazxc. -c , so that the ratio of thc timing tothat for the median case is 1o9n/ lo 2n. This ratio, consequently the timing, isminimized for the median case in which b reaches its maximum of 2.

This result, however, is problem-dependent For other distance mlations onthe n points other choices of contouring distances may give faster tinings Stillthe recursive algorithm, with some reasonable choice of the contouring dis-Lance, seems likely to have better worst-case behavior than the iterative algo-rithm

,Typicaliy with contouring there is much fine structure and the order of theclustering is much higher than in the nearest-neighbor case. Thus it s desirableto show several versions of the clustering representing differing amounts of"coarsening' or removal of contours from the original clustering. Thii will makethe larger sI riictures more apparent and also will live some indication of howmuch Ih clustering depends on small differences in the point-to-point distancesor how "robust" the clustering is. Thus if what is expected to be a srmll amountof coarsening in fact destroys most of the structure then it must. havedependedon small differences in the distances but if most of it survives then it is morerobust

The rationale used for coarsening is to start with the origina set of npoints, their nearest-neighbor distances, and the "fine-structure" clusteringgiven by contouring Next a contouring distance d is chosen as the kth smallestof the nearest-neighbor distances where k/n is close to a fixed value c. (Actu-ally we choose k so that (k-:)/ (n-1) is as close as possible to c.) Next all thenoratomic subclusters in the clustering are examined "top-down" -- that is,beginning with the whole clustering, proceeding to its constituents, then to theirconstituents and so on. Any subelusters in which the dispersion distance is notmore than d are "flattened" -- replaced by their underlying sets -- ard are then

marked ao mic i rorn this p)o ni. en thi- a ~wil be Lrea',ted is single points andwill riot be fuirther COei'-CC JWcd~ ing riiore, id\infed sLka 05 of the coarsening

!'o~r Me' ncxt Ltie hen. th viatI -r. 1 sndcustering will be furthercoarsecid u:scni the same miethod. i At\,v~r thft oitswill consist of the

Lop- Ire a eclusters -- I he-a I rat I,( rr Ittenic cr itre m urked such and arenot tihcltList ers of any 01her.- ri arked id otul c'I here wil be fewer such "points-

than on t he pre\,,Z)U> -ztc.P Fh ' c-es-nighibor dst ancirs arv determined andthe runt oiiri'.4, 1stcie R ed oi i before. using ,:I( -inle Vailue of (7 Nexttds en s(. ne I\ or ci d anY oc atr thiat arc notmiiokedici r r oi Oose ci*ersir-i u;stodre S ( t-han the contouririg?di-:t~iflcI' Al- i:bcu oee dt-A r. t o '. r- op-Ic, r -,At Corii a -;.o...isurs. anrd areniarkcd -1 (cOO t , Ff~i n-

li-. .mc' ( I-o I tt rAt d ifit? It u~i i , no fii-rcr that t- kintii te con-to~ iiii :"4 .. (I." 0 i4 ilt.c Lpr, ~co f I i -tirO ciu.stering In pair-tt k.RAr - tn '-'wt. M ecinv r a io i I: :eir r) Pc)o, di 1-vir; ttr arz

tu t v1 :, I,- rt I' I, r. tc 1 . Dc I) t' ~ cI f nr s ryia11(-r . it h

)k'O ti prfr ip 'mi:*c~i' nMo mi-I hod arde r t TvL r. ( tn :t ' ' I' %!i t), flew:, .: 't r 't -C ion Boit h

r I:' 1 ip p1ccco( icc' c':t i,( ci' .- prot--st. - wet! 1:, arrays of!cii 't, tc ttF'4~lrV' cg~z In rnar-

1,: f ''i u I t fur i-l Lir b ('t'.iiise it illowed-.V vcmi -c' Arrni.- t ic \c- nc dl\ -,,iintall c1 reat ed ind des-

;"' 'no er1)crne 'trAn/ LiPnderXin (-- \ i .--- il I - %ur;c- ':I de pending on the

pc:Ca kpq.AtI' typ-;l.t Li c ic Ct -thLi:rder- of --2 min for aset of ta'ii '0 o rt with cfatI',bc-i oiflct-ons. so IhIt most of thenc'':,t to :-ni A(ar Pc''' , rc i nit c Vii h ~r0 tim-e -,vout 20-30 min was

-ci f r * c *oir of potints, VW:tI oWr) c ,st~ance finit-e, since this made theri' i, ch i ir 'er Vo dotitu ihfPc iiiiiri, could be Improved, szo far this

A '--O '' 11a- to' :.oc' 'c'it rini.4es for clitc-ring, each ina i :zrc~thu m ore- , - eri ettno'j acid cocitouring- rnakngx; h -r cf0?' (1ji'- i i titinf ('on(cpticlizitiofl

:.r1 'I' nci t fl-iiorv !eteni O a; i ch( I c - )f t( oilnrido D'p~tr'tm vint of* .c i' in these st~u niomdi I obh'~ uo'~t, is heiicopters or cranies are

a- -ri 11c- c from- component, poxrts ",'pic ally ii object has - ibout ')0 parts IE]a chpcir nnco( flh-d& to one, or, more OtciPer -; Pats that are not physically con-cc- It :1 itre considered to be i Iinil .' di -mkance, apart except for a few eases inAlhi, f, P1 cii.1tf di-t'knci- wpiyo c--(--eir i aepted on grounds of svnm-flat r. "he numiber of conr-a etions in the object raniged fr-omT a few per-entmore ',o 'c0n ore than the numober of parts, thits most of the parts had only afew part.- connecting

;n the, most baJsic clustering problem we arc, given the order of request. ofhe part, when an )bject. is assembled rind are asked to find the "cone eptualiza-

Unrit- used in rnaking the nss cnibl\, E-ssen tially the cone pt ual zat; on is ahierairehicai stubc i vision of t hi object in w hi c h the incdivid nici connected parts, aregrouped into larger connected Units, and these into still larger units, and so onIn short, it is a distance-based clus-tering of the parts. Parts that are connected

and whose orders of request are nearly equal tre assumed to belong near eachother in the clustering, that is. as part of the same subcluster of low order 'Mhedistance betweer connecLed parts, then, is just the absolute value or thedifTerenc- in the orders of request Thus the parLs are treated as individualpoints arid the clustering is formed One variaLion of this analysis is to obtain a"consensus" conceptualization bY averainng the orders of request over severalassembly trials

A second problem concerns the clusterings that are obtained by the aboveanalysis when ,a number of subjects all assenible he same object under varyingconditions We %ould like to know if these ciustcringi fali into meaningfu pat-terns or hierarchies For instance we cdn ask whether there is essentially oneconCeptLuali/attLon wiLth n~or variitiuns or several essentially different concep-tualliations Thus a clustering of the coneeptualizations is called for

To do this we must define a distance between conceptuahzations. The waythis has been done is to consider the boolhan relation or set of connections inthe objtct The connections represented by pair : of parts, are numbered

2. rn Each connectin is represented by a pair Pl, P2J Given any clus-tring there *,!i exist a minimal subclaster C that contains both Pi and p, sothat p! and p. wil be in different ,-onstLituents -) F

Next we derine, the order of pI. p:IL as the order of C' This is the "bottom-tip or.-er sinc( it depnds on the order of the e.rnlituents of U An alternativeis tne 'top-down' order whien is tfle depth of C within the clustering, where theelustering itself will have depth tero its eonstituents depth one. their consti-tuer.t_ depthl two and so on AL inv rate for anv clustering we obtain an m-vector j :,' in) %here U; is the order of tihe ith connection It is notdifTiiall to T-he I hat in e.nt~i -e clusterrn : can be reconstructed from its assocl-ated v ,tur FJ',r eximp'e we ean ;tart with i(he pairs of lowest bottom-up orhiwhesI top-down order arid add pairs of succeeding orders until the entirearrangement of pairs is determined ) Thus two clusterings with the same vectormust be identical

F'; nalIv, the distance between two clusterings is defined as the distancebetween the associated vectors, tising a metric on an m-dimensional vectorspace, The metric that has been most useful is the Euelidean *-norm, that is,the sum of the absolute values of the differences of the corresponding terms oftwo vectors Tne bottom-up order and the *-norm distance were used exten-s:vetv in studics based on the nearest-neighbor method of clustering , ]. Forcontourinp, however, the top-do%n order is probably more appropriate since theclustenrins are not flat-bottomed . that is, constituents in a subeluster canhave variable order This means that the bottom-up order of a connecting paircan depend on a constituent that econtains neither point of the pair, somethingthat does not occur with the top-down order 'I he ltter, then, seems to offer amore reasonable indication of how *'similar" the components of a connection are

It should be noted that in this sort of analyss, regardless of how the orderis measured, all distances between points (in this case, clusterings) are finite'rhus there are large adjacency lists for the points and extensive computation isrequired for a moderately-sJzed problem, about ?5 rnin of CPtJ time on the VAXcomputer was needed ror one typical case involving 47 points.

Thus far two techniques for clustering problems have been described, bothdealing with cases in which the objects were correctly assembled. Two othertechniques have been developed for understanding what is involved when theassembly is incorrect. (Incorrect assembly occurred when objects were recon-structed from memory, without a correctly assembled object as reference)

i

-10-

In the first ease we are trying to determine how the dafferent connections inthe object influence the errors that are made Thus the connections are treatedas the "point." in a cluster analysis T'wo such connections are connected them-;elves if they have an endpoint in common Thus a boolean relation can bedefined on the set of connections 'wo connections that are not connected toeach other are at an infirLte distance apart: otherwise the distance will dependon how errors are made at these conn.ctions when assembly .1 the object isattempted

Thus it is assuned that. whenever the object is incorrectly assembled.thCre is at way of unambiguously deciding at which connections the errors aremade For the anal sis, then. a number of cases of an incorrectly-assembledobject are considered ro define the distance between two connections in theobject that have an endpoint in common we consider the sets Si and S2 of casesin which errors were made on the first or second connection, respectively Thedisteane. s then defined as S,-,2.1 SjL_.Sj. that is, as the ratio between thenumber of points in the symmetric difference of the two sets (where the sym-metric difference is the set of points in the union that are not in the intersec-tion) and the number of points in the union This distance can be shown to be ametric on finite, nonempty sets 21 In particular the distance is always <1; it isset to - by default if ;I ; 1 2 is empty

The distance is smaller if there is greater agreement between the errorsmade on one connection and those on the other. Thus the clustenng will tend toidentify groups of connections for which the pattern of errors is similar, such asthose on which nearly uli subjects made errors In this way it is hoped that con-nect ions that are consistenLly troublesome can be identified so that tnstruc-tional sequcnce. r'n be dcsigned to reduce the incidence of errors

The second technique., which is the final one considered here. is concernedwith identifying meaningful patterns of error making among different subjectgroups Subject., within a group assemble the object under similar conditions.I.'or exAmple, visual instructional material may be presented For each subjectwithin a .proup, then, we have the list of connections on which errors were madeI-",r xnv group we define an rn-vector 'v 1 , v, . ..) by v, = average number of,-rror. n,.edv on the ith connectLion = number of subjects who made the errorsdivided icv the number of subjects in the group The distance between groups isthen dcfirit'd i,-- the di-',ance between vectors, measured as before, and cluster-r4 (',,in proc.eed ,is in the earlier case

An ilustr.ton of this technique is shown in fig 2 The twelve points in eachC ,-4cririg represent tweivc subject. groups The object in this case has h8 con-ni.ctoni Thus each point ropresents a location in a 58-dimcnsional vectorp,%(c aii,, the point-to-point distances, which were given by the i-norm, are only

rowghh indicated in the figure. (2a) shows the clustering obtained by ther.-,rst-n,.ighhor method, while ',2b-d) show the results of contouring withImrceislr.,4 amounts of coarsening 'lhus (2b) has no coarsening, (2c) has amoderat, aniouiit, with the coarsening parameter c =0 b. and (2d) has the larg-cst inioijnt, with r.:= Certainlv the two methods show difTerences but there are

,isc similarities too In particular the nearest-neighbor method seems tocorrespond best to contouring with a moderate amount of coarsening.

a b

c d

Fig showiig iesulLs of clustering by the nearest-neighbor method (2a)and contouring with increasing amounts or coarsening (2b-d).

REFE~RENCESHaggett. 1P. Four principles for designing instructions. Technical Report

#121-ONR To appear.

21 Ehrenfeucht, A.. private communication.

Colorado/Baggett (NR 157-422) 7-Apr-83 Page I

Navy Marine Corps

I Baker, Meryl S. 3 5 1 GREENUP H WILLIAM (MCDEC) 123451 Bittner, Alvah 2 5 1 ONR 1OOM 12345I Bond, Nicholas A. 12345 1 SLAFKOSKY AL 12345I Bory, Alexander (AMMRL)I Cantone. Richard 345I CNET Liason, AFHRL 0 3451 Collyer, Stan 123451 CURRAN MIKE 12345I Duffy. Tom 23451 Federico, Pat-Anthony 0 2345I FORD JOHN 123451 Caynor, Mike 3451 HOLLAN JIM 23451 Hutchins, Ed 345I KERR NORMAN J 123451 Kincaid, Peter 345I MALOY WILLIAM L 123451 Montague, William 0 2345I NPRDC LIBRARY 123451 NPRDC TECHNICAL DIRECTOR 123456 NRL CODE 2627 123451 ONR 433 3456 ONR 442PT 12345I OP115 123451 Petho, Frank C. 012 45I POOCK GARY 123 5I Ricard, Gil 3451 Rimland, Bernie 012 51 ROSS, CARL 1 3 5I Scanland, Worth 01 3 51 Schiff. Irving1 Schiflett, Sam 0 451 Smith, Robert G. 0123451 SMODE ALFRED F 23451 SORENSEN RICHARD 12345I Steinheiser, Frederick 0 3451 Sympson, Brad 1 51 Van Natre, Nick 3I Wegman, Dr. Edward 1I WEISSINGER-BAYLON ROGER 2345I Wetzel, Douglas (NPRDC) 012 45I Wolfe, John H. 0123 5

Colorado/laggett (NR 157-422) 7-Apr-83 Page 2

Army Air Force

I ARI Technical Director 12345 1 AFHRL Technical Documents C n12345I FARR BEATRICE 2345 1 AFOSR Life Sciences 12345I KATZ MILTON S 12345 1 Air University Library 12345I O'Neil, Harry 345 1 Christal, Raymond E. 0123451 Orasanu. Judith 345 1 HADDAD GENEVIEVE 2345I Psotka, Joseph 345 1 Longridge, Tom 0 2345I SASMOR ROBERT 12345 1 PENNELL, ROGER 1 5I WISHER ROBERT 12345 1 REE MALCOLM 12 45

I Yasatuke, Joe 2345

Department of Defense Civilian Agencies

12 DTIC 12345 1 CHIPMAN SUSAN 23451 OUSDRE 12345 1 Selden, RamsayI Thorpe, Jack A. 345 1 USCG PSYCHOLOGICAL RESEARCH 12345

1 Weiss, Edward C.I Young, Joseph L. 012345

Colorado/Baggett (MR 157-422) 7-Apr-83 Page 3Private Sector Private Sector

1 Andersen, Erling B. 12345 1 Reckase, Mark D. 01 3 5

1 Annett, John 0 2345 1 RESNICK LAUREN 2345

1 Australian DoD Psychology 12345 1 ROSE iNDREW M 12345

1 Baddeley, Alan 2345 1 Rothkopf, Ernst 012345

1 Bejar, Isaac 12 5 1 Rouse, William 345

1 Birenbaum. Menucha 1 345 1 RUDNER LAWRENCE 1 5I Bund. der Verteidigung 012 45 1 RUMELHART DAVID 5

1 CARPENTER PAT 2 45 1 SAMEJIMA FUMIKO 123451 Carroll, John B. 0 2 5 1 Samet, Michael 345

1 CHASE WILLIAM 2 45 1 Schneider, Walter 0 23451 CHI MICHELINE 45 1 SEIDEL ROBERT J 123 5

1 COOPER LYNN A 2 45 1 Sinaiko, H. Wallace 0 345

I CROSS KENNETH 12345 1 SMITH EDWARD 2 45

1 ERIC 12345 1 SNOW RICHARD 12345

1 Feltovich, Paul 345 1 Spoehr, Kathryn T. 2345

1 Feurzeig, Wallace 0 345 1 Sternberg, Robert ( 2345

1 FIELDS VICTOR 1 3 5 1 Stevens, Albert 0 2345

1 Fletcher, Dexter 12345 1 Stone, David 01 345

I FREDFRIKSEN JOHN R 12345 1 Stout, William 1 5

1 Gentner, Don 2 4 1 SUPPES PATRICK 123451 Glaser, Robert 012345 1 SWAMINATHAN HARIHARAN 12345

I Clock, Marvin D. 0 2345 1 Tatsuoka, Kikumi 012345

1 GOGUEN, JOSEPH 2 45 1 Tatsuoka, Maurice 1 34

1 Gopher, Daniel 2 45 1 Thissen, David 01 5I Green, Bert 012345 1 Thorndyke, Perry 0 345

1 GREENO JAMES G 12345 1 Towne, Doug 0 345Il ayes-Roth, Barbara 2345 1 TSUTAKAWA ROBERT 1 5

I Hayes-Roth, Frederick 345 1 Vale, David 01 5I HUMPHREYS LLOYD 12 5 1 Weiss, David J. 012 51 Hunt, Earl 0 2 45 1 Wescourt, Keith T. 0 3451 Just, Marcel 2 4 1 WHITELY SUSAN 12 51 KEELE STEVEN W 2 45I KIERAS DAVID 23451 KINTSCH WALTER 45I Kosslyn, Stephen 012 45

1 Langley, Patrick 345I Lansman, Marcy 0 2 51 Lesgold, Alan 0123451 LEVINE MICHAEL 12345

I LEWIS CHARLES 123451 LINN, MARCIA C. 01 LINN ROBERT 12 5

I Lord, Frederic M. 01 51 McClelland, Jay 2 45I Miller, James R. 3451 MUNRO ALLEN 23451 Norman, Donald A. 0 23451 NOVICK MELVIN R 12345

I Orlansky, Jesse 0123451 PAULSON JAMES A 123451 PELLEGRINO JAMES 2 45I PENNINGTON, NANCY 3451 Petrullo, Luigi 012345

I POLSON MARTHA 2345

I POLSON PETER 2345

L ____ ___ -

Colorado/laggett (NR 157-422) 7-Apr-83 Page

Navy Navy

I Dr. Meryl S. Baker 1 Dr. Mike Gaynor

Navy Personnel R&D Center Navy Research LaboratorySan Diego, CA 92152 Code 7510

Washington, DC 20375

1 Dr. Alvah BittnerNaval Blodynamics Laboratory I Dr. Jim HollanNew Orleans. LA 70189 Code 304

Navy Personnel R & D Center

I Liaison Scientist San Diego, CA 92152

Office of Naval ResearchBranch Office, London I Dr. Ed Hutchins

Box 39 Navy Personnel R&D Center

FPO New York, NY 09510 San Diego, CA 92152

I Lt. Alexander Bory I Dr. Norman J. KerrApplied Psychology Chief of Naval Technical TrainingMeasurement Division Naval Air Station Memphis (75)NAMRL Millington, TN 18054NAS Pensacola. FL 3250R

I Dr. Peter KincaidI Dr. Richard Cantone Training Analysis & Evaluation GroupNavy Research Laboratory Dept. of the NavyCode 7510 Orlando, FL 32813Washington, DC 20375

I Dr. William L. Maloy (02)I Chief of Naval Education and Training Chief of Naval Education and TrainingLiason Office Naval Air Station

Air Force Human Resource LaboratoryOperations Training Division Pensacola, FL 32508WILLIAMS AFB, AZ 85224

1 Dr William MontagueI Dr. Stanley Collyer NPRDC Code 13Office of Naval Technology San Diego, CA 92152800 N. Quincy StreetArlington, VA 22217 1 Library, Code P201L

Navy Personnel R&D Center

I CDR Mike Curran San Diego, CA 92152Office of Naval Research800 N. Quincy St. I Technical Director

Code 270 Navy Personnel R&D Center

Arlington, VA 22217 San Diego, CA 92152

1 Dr. Tom Duffy 6 Commanding OfficerNavy Personnel R&D Center Naval Research LaboratorySan Diego, CA 92152 Code 2627

Washington, DC 203901 DR. PAT FEDERICOCode P13 1 Office of Naval ResearchNPRDC Code 433San Diego, CA 92152 800 N. Quincy SStreet

Arlington, VA 222171 Dr. John FordNavy Personnel R&D CenterSan Diego, CA 92152

Colorado/Baggett (NR 157-422) 7-Apr-83 Page 2

Navy Navy

6 Personnel & Training Research Group I Dr. Alfred F. Smode, Director,% Code 442PT Training Analysis & Evaluation Group

Office of Naval Research Dept. of the NavyArlington. VA 22217 Orlando, FL 32P13

I Office of the Chief of Naval Operations 1 Dr. Richard SorensenResearch Development & Studies Branch Navy Personnel R&D CenterOP 115 San Diego, CA 92152Washington, DC 20350

I Dr. Frederick SteinheiserI LT Frank C. Petho, MSC, USN (Ph.D) CNO - OPL1SCNET (N-432) Navy AnnexNAS Arlington, VA 20370Pensacola, FL 32508

1 Mr. Brad SympsonI Dr. Gary Poock Navy Personnel R&D CenterOperations Research Department San Diego, CA 92152Code 55PK

Naval Postgraduate School 1 Code 14Monterey, CA 93940 Navy Personnel R&D Center

San Diego, CA 921521 Dr. GiI RicardCode N711 I Dr. Edward WegmanNTEC Office of Naval Research (Code 41IS&P)Orlando, FL 32813 800 North Ouincy Street

Arlington, VA 22217

I Dr. Fernard Rimland (01C)Navy Personnel R&D Center I Roger Weissinger-BaylonSan Diego, CA 92152 Department of Administrative Sciences

Naval Postgraduate School

I Dr. Carl Ross Monterey, CA 93940CNET-PDCD

Building 90 1 Dr. Douglas WetzelGreat Lakes NTC. IL 60088 Code 12

Navy Personnel R&D CenterI Dr. Worth Scanland San Diego, CA 921-2CNET (N-5)NAS, Pensacola, FL 32508 1 Mr John H. Wolfe

Navy Personnel R&D CenterI Mr. Irving Schiff San Diego, CA 92152Dept. of the NavyChief of Naval OperationsOP 113Washington, DC 20350

1 Dr. Sam SchiflettAircrew Systems (SY7OE)

Naval Air Test CenterPatuxent River, MD 20670

I Dr. Robert G. qmithOffice of Chief of Naval OperationsOP-987HWashington, DC 20350


Marine Corps Army

1 H. William Greenup 1 Technical DirectorEducation Advisor (E031) U. S. Army Research Institute for theEducation Center, MCDEC Behavioral and Social SciencesQuantico, VA 22134 5001 Eisenhower Avenue

Alexandria, VA 22333

I Special Assistant for MarineCorps Matters I Dr. Beatrice J. Farr

Code 1OOM U. S. Army Research InstituteOffice of Naval Research 5001 Eisenhower Avenue800 N. Ouincy St. Alexandria, VA 22333Arlington, VA 22217

I Dr. Milton S. KatzI DR. A.L. SLAFKOSKY Training Technical AreaSCIENTIFIC ADVISOR (CODE RD-i) U.S. Army Research InstituteHO, U.S. MARINE CORPS 5001 Eisenhower AvenueWASHINGTON, DC 20380 Alexandria, VA 22333

1 Dr. Harold F. O'Neil, Jr.Director, Training Research LabArmy Research Institute5001 Eisenhower AvenueAlexandria, VA 22333

1 Commander, U.S. Army Research Institutefor the Behavioral & Social Sciences

ATTN: PERI-BR (Dr. Judith Orasanu)

5001 Eisenhower Avenue


I Joseph Psotka, Ph.D.ATTN: PERI-ICArmy Research Institute5001 Eisenhower Ave.Alexandria, VA 22333

I Dr. Robert SasmorU. S. Army Research Institute for theBehavioral and Social Sciences5001 Eisenhower Avenue


I Dr. Robert WisherArmy Research Institute5001 Eisenhower AvenueAlexandria, VA 22333

%

Colorado/Baggett (HRK 157-422) 7-Apr-83 Page 4

Air Force Department of Defense

I Technical Documents Center 12 Defense Technical Information CenterAir Force Human Resources Laboratory Cameron Station. Bldg 5WPAFB, ON 45433 Alexandria, VA 22314

Attn: TCI U.S. Air Force Office of Scientific

Research 1 Military Assistant for Training andLife Sciences Directorate, NL Personnel TechnologyBolling Air Force Base Office of the Under Secretary of DefensWashington, DC 20332 for Research & Engineering

Room 3D129, The PentagonI Air University Library Washington, DC 20301AUL/LSE 76/443Maxwell AFB, AL 36112 1 Major Jack Thorpe

DARPAI Mr. Raymond E. Christal 1400 Wilson Blvd.AFHRL/MOE Arlington, VA 22209Brooks AFB, TX 78235

I Dr. Genevieve HaddadProgram Manager

Life Sciences DirectorateAFOSRBoiling AFB, DC 20332

1 Dr. T. M. LongridgeAF11RL/OTEWilliams AFB. AZ 85224

I Dr. Roger PennellAir Force Human Resources LaboratoryLowry AFB, CO 80230

I Dr. Malcolm ReeAFHRL/MPBrooks AFB, TX 7P235

I Dr. Joseph YasatukeAFHRL/LRTLowry AFB, CO 80230

Colorado/Raggett (NR 157-422) 7-Apr-83 Page 5

Civilian Agencies Private Sector

I Dr. Susan Chipman I Dr. Erling B. AndersenLearning and Development Department of StatisticsNational Institute of Education Studiestraede 61200 19th Street NW 1455 CopenhagenWashington, DC 20208 DENMARK

I Dr. Ramsay W. Selden 1 Dr. John AnnettNational Institute of Education Department of Psychology1200 19th St., NW University of WarwickWashington. DC 20208 Coventry CV4 7AJ

ENGLANDI Chief, Psychological Reserch BranchU. S. Coast Guard (G-P-1/2/TP42) I Psychological Research UnitWashington, DC 20593 Dept. of Defense (Army Office)

Campbell Park OfficesI Dr. Edward C. Weiss Canberra ACT 2600National Science Foundation AUSTRALIA1800 C Street, NWWashington, DC 20550 1 Dr. Alan Baddeley

Medical Research CouncilI Dr. Joseph L. Young, Director Applied Psychology UnitMemory & Cognitive Processes 15 Chaucer RoadNational Science Foundation Cambridge CB2 2EFWashington, DC 20550 ENGLAND

1 Dr. Isaac Bejar

Educational Testing ServicePrinceton, NJ 08450

1 Dr. Menucha BirenbaumSchool of Education

Tel Aviv UniversityTel Aviv, Ramat Aviv 69978Israel

I Bundministerium der Verteidigung-Referat P II 4-

Psychological Service

Postfach 1328D-5300 Bonn 1F. R. of Germany

1 Dr. Pat Carpenter-Department of Psychology

Carnegie-Mellon UniversityPittnburph, PA 15211

I Dr. John r. trroll

404 Elliott Rd.

Chapfl Ilill, VCr' ?)514


Private Sector Private Sector

I Dr. William Chase I Dr. Don GentnerDepartment of Psychology Center for Human Information ProcessingCarnegie Mellon University University of California, San DiegoPittsburgh, PA 15213 La Jolla, CA 92093

I Dr. Micheline Chi I Dr. Robert ClaserLearning R & D Center Learning Research & Development CenterUniversity of Pittsburgh University of Pittsburgh3939 O'Hara Street 3939 O'Hara StreetPittsburgh, PA 15213 PITTSBURGH, PA 15260

I Dr. Lynn A. Cooper I Dr. Marvin D. ClockLRDC 217 Stone HallUniversity of Pittsburgh Cornell University3939 O'Hara Street Ithaca, NY 14853Pittsburgh, PA 15213

I Dr. Josph GoguenI Dr. Kenneth B. Cross SRI InternationalAnacapa Sciences, Inc. 333 Ravenswood AvenueP.O. Drawer 0 Menlo Park, CA 94025Santa Barbara, CA 93102

I Dr. Daniel GopherI ERIC Facility-Acquisitions Department of Psychology4833 Rugby Avenue University of IllinoisBethesda, MD 20014 Champaign, IL 61820

I Dr. Paul Feltovich I Dr. Bert GreenDepartment of Medical Education Johns Hopkins UniversitySouthern Illinois University Department of PsychologySchool of Medicine Charles & 34th StreetP.O. Box 3926 Baltimore, MD 21218Springfield, IL 62708

1 DR. JAMES G. GREENOI Mr. Wallace Feurzeig LRDC

Department of Educational Technology UNIVERSITY OF PITTSBURGHBolt Beranek & Newman 3939 O'HARA STREET1O Moulton St. PITTSBURGH, PA 15213Cambridge, MA 02238

1 Dr. Barbara Hayes-RothI Dr. Victor Fields Department of Computer Science

Dept. of Psychology Stanford UniversityMontgomery College Stanford, CA 95305Rockville, MD 20850

I Dr. Frederick Hayes-RothI Dr. Dexter Fletcher TeknowledgeWICAT Research Institute 525 University Ave.1875 S. State St. Palo Alto, CA 94301

Orem, UT 223331 Dr. Lloyd Humphreys

I Dr. John R. Frederiksen Department of PsychologyBolt Beranek & Newman University of Illinois50 Moulton Street Champaign, IL 61820Cambridge, MA 02138

NW ColoradolBaggett (NR 157-422) 7-Apr-83 Page 7


I Dr. Earl Hunt I Dr. Charles Lewis

Dept. of Psychology Faculteit Sociale Wetensehappen

University of Washington Rijksuniversiteit GroningenSeattle, WA 98105 Oude Boteringestraat 23

9712GC Groningen

I Dr. Marcel Just Netherlands

Department of PsychologyCarnegie-Mellon University I Dr. Marcia C. Linn

Pittsburgh, PA 15213 University of CaliforniaDirector, Adolescent Reasoning Project

I Dr. Steven W. Keele Berkeley, CA 94720

Dept. of Psychology

University of Oregon 1 Dr. Robert Linn

Eugene, OR 97403 College of EducationUniversity of Illinois

I Dr. David Kieras Urbana, IL 61801Department of Psychology

University of Arizona 1 Dr. Frederic M. Lord

Tuscon, AZ 85721 Educational Testing ServicePrinceton, NJ 08541

1 Dr. Walter KintschDepartment of Psychology I Dr. Jay McClellandUniversity of Colorado Department of PsychologyBoulder, CO 80302 MIT

Cambridge, MA 02139I Dr. Stephen KosslynDepartment of Psychology I Dr. James R. Miller

The Johns Hopkins University Computer*Thought CorporationBaltimore, MD 2121 1721 West Piano Highway

Piano, TX 75075

I Dr. Pat LangleyThe Robotics Institute 1 Dr. Allen Munro

Carnegie-Mellon University Behavioral Technology Laboratories

Pittsburgh. PA 15213 1845 Elena Ave., Fourth FloorRedondo Beach, CA 90277

1 Dr. Marcy LansmanThe L. L. Thurstone Psychometric 1 Dr. Donald A Norman

Laboratory Cognitive Science, C-015

University of North Carolina Univ. of California, San DiegoDavie Hall 013A La Jolla, CA 92093Chapel Hill, NC 27514

1 Dr. Melvin R. Novick

I Dr. Alan Lesgold 356 Lindquist Center for Measurment

Learning R&D Center University of Iowa

University of Pittsburgh Iowa City, IA 52242

3939 O'Hara StreetPittsburgh, PA 15260 1 Dr. Jesse Orlansky

Institute for Defense Analyses

I Dr. Michael Levine 1801 N. Beauregard St.% Department of Educational Psychology Alexandria. VA 22311

210 Education Bldg.University of Illinois

% Champaign, IL 61801



I Dr. James A. Paulson I Dr. William B. RousePortland State University Georgia Institute of TechnologyP.O. Box 751 School of Industrial & SystemsPortland, OR 97207 Engineering

Atlanta, GA 30332

I Dr. James W. PellegrinoUniversity of California, I Dr. Lawrence RudnerSanta Barbara 403 Elm AvenueDept. of Psychology Takoma Park. MD 20012

Santa Barabara, CA 93106I Dr. David Rumelhart

1 Dr. Nancy Pennington Center for Human Information ProcessingUniversity of Chicago Univ. of California, San DiegoGraduate School of Business La Jolla, CA 920931101 E. 58th St.Chicago, IL 60637 1 PROF. FUMIKO SAMEJIMA

DEPT. OF PSYCHOLOGY

I Mr. L. Petrullo UNIVERSITY OF TENNESSEE3695 N. Nelson St. KNOXVILLE, TN 37916ARLINGTON, VA 22207

1 Dr. Michael J. Samet

I Dr. Martha Poison Perceptronics, IncDepartment of Psychology 6271 Variel AvenueCampus Box 346 Woodland Hills, CA 91364

University of ColoradoBoulder, CO 80309 1 Dr. Walter Schneider

Psychology Department

I DR. PETER POLSON 603 E. DanielDEPT. OF PSYCHOLOGY Champaign, IL 61820UNIVERSITY OF COLORADOBOULDER. CO 80309 1 DR. ROBERT J. SEIDEL

INSTRUCTIONAL TECHNOLOGY GRO'P

I Dr. Mark D. Reckase HUMRROACT 300 N. WASHINGTON ST.P. 0. Box 168 ALEXANDRIA, VA 22314Iowa City, IA 52243

1 Dr. H. Wallace Sinaiko

I Dr. Lauren Resnick Program DirectorLRDC Manpower Research and Advisory ServicesUniversity of Pittsburgh Smithsonian Institution3939 O'Hara Street 801 North Pitt StreetPittsburgh, PA 1521 Alexandria, VA 22314

I Dr. Andrew M. Rose 1 Dr. Edward E. SmithAmerican Institutes for Research Bolt Beranek & Newman, Inc.1055 Thomas Jefferson St. NW 50 Moulton StreetWashington, DC 20007 Cambridge, MA 02138

I Dr. Ernst Z. Rothkopf 1 Dr. Richard SnowBell Laboratories School of EducationMurray Hill, NJ 07974 Stanford University

Stanford, CA 94305



I Dr. Kathryn T. Spoehr I Dr. Perry W. ThorndykePsychology Department Perceptronics, Inc.Brown University 545 Middlefield Road, Suite 140Providence, RI 02912 Menlo Park, CA 94025

I Dr. Robert Sternberg I Dr. Douglas TowneDept. of Psychology Univ. of So. CaliforniaYale University Behavioral Technology Labs

Box IIA, Yale Station 1845 S. Elena Ave.New Haven, CT 06520 Redondo Beach, CA 90277

I Dr. Albert Stevens I Dr. Robert TsutakawaBolt Beranek & Newman, Inc. Department of Statistics

.. 10 Moulton St. University of MissouriCambridge. MA 02238 Columbia, MO 65201

I David E. Stone, Ph.D. I Dr. David ValeHazeltine Corporation Assessment Systems Corporation7680 Old Springhouse Road 2233 University AvenueMcLean, VA 22102 Suite 310

St. Paul, MN 55114I Dr. William Stout

University of Illinois 1 Dr. David J. WeissDepartment of Mathematics N660 Elliott Hall

Urbana, IL 61801 University of Minnesota75 E. River Road

I DR. PATRICK SUPPES Minneapolis, MN 55455INSTITUTE FOR MATHEMATICAL STUDIES INTHE SOCIAL SCIENCES 1 Dr. Keith T. WescourtSTANFORD UNIVERSITY Perceptronics, Inc.

STANFORD, CA 94305 545 Middlefield Road, Suite 140Menlo Park, CA 94025

I Dr. Hariharan SwaminathanLaboratory of Psychometric and I DR. SUSAN E. WHITELYEvaluation Research PSYCHOLOGY DEPARTMIENTSchool of Education UNIVERSITY OF KANSASUniversity of Massachusetts Lawrence, KS 66045Amherst, MA 01003

1 Dr. Kikumi TatsuokaComputer Based Education Research Lab252 Engineering Research Laboratory

Urbana, IL 61801

I Dr. Maurice Tatsuoka220 Education Bldg1310 S. Sixth St.Champaign, IL 61820

1 Dr. David ThissenDepartment of PsychologyUniversity of Kansas

-Lawrence, KS 66044