FINAL RIPORT 01 013 CONTRACTXMII 100014-85-K-0113, 31 150-535

ROBT . TSUTAZAVA

, .T,, B., 196, DT IC: -.LECTE

*H DPrepared under contract No. 100014-85-K-0l13, SK 150-535

with the cognitive Science prosramOffice of laval Research

Approved for public release: distribution unlimited.Reproduction Is whole or part in pernitted forany purpose of the United States Government.

88 9 26 15

w - --

SECURITY CLASSIFICATION OF THIS PAGE

RNForm ApprovedREPORT DOCUMENTATION PAGE OMB No. 0704-01"

la REPORT SECURITY CLASSIFICATION lb RESTRICTIVE MARKINGS

Unclassified2a SECURITY CLASSIFIC;TIOni AUTHORITY 3 DISTRIBUTION/AVAILABILITY OF REPORT

Dpproved for public release; distri-2b. DECLASSIFICATION/DOWNGRADING SCHEDULE Ut ion unlimited

4 PERFORMING ORGANIZATION REPORT NUMBER(S) S MONITORING ORGANIZATION REPORT NUMBER(S)

None6a. NAME OF PERFORMING ORGANIZATION 6b OFFICE SYMBOL 7a NAME OF MONITORING ORGANIZATIONDepartment of Statistics (fapplicable) Cognitive Science ProgramUniversity of Missouri Office of Naval Research (Code 1142PT6c. ADDRESS (City, State, and ZIPCodre) 7b ADDRESS (City, State, and ZIP Code)222 Math Sciences 00 North Quincy StreetColumbia, MO 65211 rlington, VA 22217-5000

8a. NAME OF FUNDING/SPONSORING Sb OFFICE SYMBOL 9 PROCUREMENT INSTRUMENT IDENTIFICATION NUMBERORGANIZATION (If applicable) 100014-85-K-0113

8c. ADDRESS (City, State, and ZIP Code) 10 SOURCE OF FUNDING NUMBERSPROGRAM PROJECT TASK WORK UNITELEMENT NO NO NO JACCESSION NO

61153N RR04204 RR4204-0 4421-53511 TITLE (Include Security Classification)

Final Report on ONR Contract Number N00014-85-K-0113, NR 150-53512 PERSONAL AUTHOR(S)

Tsutakawa, Robert K13a. TYPE OF REPORT 13b TIME COVERED 14 DATE OF REPORT (Year, Month, Day) 15 PAGE COUNT

Fin FROM R n TO 30 88 Sep 30 1516 SUPPLEMENTARY NOTATION

17 COSATI CODES 18. SUBJECT TERMS (Continue on reverse if necessary and identify by block number)FIELD GROUP SUB-GROUP Bayesian IRT, calibration, ability estimation,

2PL, 3PL.19 ABSTRACT (Continue on reverse if necessary and identify by block number)

This is the final report on ONR Contract number N00014-85-K-0113,NR 150-535, Calibration of Test Items and Measurement of Abilities. Thereport reviews the results obtained during the period January 1985through September 1988. The results include the development of a generalBayesian framework for item response analysis and Bayesian estimation ofitem and ability parameters. A listing of technical reports andpublications resulting from the contract is also presented.

20 DISTRIBUTION/AVAILABILITY OF ABSTRACT 21. ABSTRACT SECURITY CLASSIFICATION- UNCLASSIFIED/UNLIMITED 0 SAME AS RPT 0 DTIC USERS

22a. NAME OF RESPONSIBLE INDIVIDUAL 22b. TELEPHONE (Include Area Code) I22c. OFFICE SYMBOLDr. Charles Davis 202-696-4046 pNR 1142CS

aDD Form 1473, JUN 36 Previous editions are obsolete. SECURITY CLASSIFICATION OF THIS PAGE

FINAL REPORT ON OUR CONTRACTNUMBER N00014-85-K-0113, MR 150-535

Calibration of Test Item and Measurement of Abilities

by

Robert K. TsutakawaPrincipal Investigator

SEPTEMBER 1988

Department of StatisticsUniversity of MissouriColumbia, MO 65211

II

-7i

Introduction

) The overall goal of this project is to develop new Bayesianprocedures for mental testing. A typical test, which is studiedhere, consist of k test items administered to n examinees. Thedata consists of an nxk matrix of binary responses indictingwhich of the k items are scored correctly and which incorrectlyby each of the n examinees.-

The statistical procedures are based on the assumption thatthere is a model which specifies the probability of a correctresponse to each item as a function of an unidimensionalability. Such functions are assumed to belong to certainfamilies such as the two-parameter logistic (2PL) or three-parameter (3PL) curves. These curves are identified byparameters called item parameters.\

When these models are used for testing, a set of items isinitially calibrated using a moderately large value for n (thesample size). The calibration consists of estimating the itemparameters. The calibrat d, curves are then used to ,_soreabilities of new examinees. , .

It is standard practice to ignore the uncertainty in the ritems once the calibration is complete and to estimate abilitiesassuming that the item parameters are known. This practice canlead to serious inferential errors in the measurement ofabilities. In particular, an interval estimate of an ability canbe too narrow giving a false impression of the accuracy of theestimate.

The sequential nature of first calibrating and then scoringmakes the Bayesian approach particularly appropriate. Accordingto this approach, an analysis is made of the uncertainties inthe estimated items at the calibration phase. This uncertaintyis then taken into account when abilities are measured. Theuncertainty in the measured ability is not only due to therandomness of responses from individuals with the same ability,but also due to the uncertainty in the calibrated items.

The Bayesian paradigm can be extended to on-linecalibration, where new items are introduced with items which havealready been calibrated. In this situation the uncertainties ofabilities based on the calibrated items are incorporated into theuncertainties of the new items. Again the typical standardpractice is to ignore the uncertainties in the abilities ofindividuals used for the calibration of the new items.

2

In order to develop this general Bayesian approach to mentaltesting, the research was divided into the following four topicsand the results are outlined below.

I. Development a general Bayesian framework for item responseanalysis.

II. Estimation of item parameters.

III. Estimation of abilities.

IV. On-line calibration.

I. Bayesian Framework

The general framework for Bayesian item response theory hasbeen described in Tsutakawa and Lin (Psychometrika, 1986). Giventhat the item response curves belong to a certain parametricfamily, a prior distribution for the item parameters are assumed.The joint likelihood function of ability and item parameters isbased on the assumption of local independence. The ability areassumed iid N(0,1). The marginal likelihood function is then theaverage of the joint likelihood function weighted by the N(O,1)prior. The marginal likelihood function is then multiplied tothe prior to get the (unnormalized) posterior for the itemparameters. The marginal posterior for the ability parameter canbe similarly expressed but is not easy to work with due to themultiple integrals involved.

II. Estimation of item parameters

The general approach developed for item parameter estimationis to use as point estimate the posterior mode and as measure ofuncertainty the posterior covariance matrix. The use of the EMalgorithm for computing the posterior mode is described inTsutakawa and Lin (1986). A novel feature of this paper is theuse of the ordered bivariate beta to form a prior distributionfor the item parameters in 2PL. This paper proposes the use of9the inverse posterior information matrix to approximate theposterior covariance matrix. It also illustrates the relativecloseness of estimated values in repeated samples, when comparedto standard methods such as LOGIST (Wingersky, Barton & Lord1982). L0

In Rigdon and Tsutakawa (JES 1987) an empirical Bayesprocedure is developed for the case in which both ability anditem parameters are sampled from population distributions withunknown hyperparameters. Here the EM algorithm (Dempster, Laird,& Rubin, JRSS-B 1977) is modified to simultaneously estimate the Code$hyperparameters. Simulations are used to show the robustness of Iorthis approach relative to marginal maximum likelihood in thecase of the Rasch model.

The' use of the Dirichlet distribution to form a priordistribution for item parameters in 3PL is studied in Tsutakawa(TR143, 1988). The conventional Bayesian approach assume priorindependence of parameters within items. This paper suggests asimple device to represent the prior dependence among parameterswithin items. The emphasis here is on looking at curves ratherthan parameters. Bayesian modal estimates are compared withLOGIST (Wingersky, Barton, and Lord, 1982) and marginal maximumlikelihood. The robustness of the Bayesian estimate relative toweights placed on the prior is also illustrated. One notablefeature of the Bayesian method is that there are much feweroutliers with unreasonable values.

III Ability estimation

Bayesian approximations to the posterior mean and varianceof ability are proposed and illustrated for 2PL in Tsutakawa andSoltys (JES, 1988). The standard empirical Bayes approximationsare posterior moments conditional on assuming the unknown itemparameters to equal those estimated at the calibration phase.The new approximation modifies this by adding terms representing,and correcting for, the uncertainties of the calibrated itemparameters. It is a special case of Lindley's (1980)approximation when the 3rd partial derivatives of thelogposterior vanish. The new approximation shows that theempirical Bayes approximation consistently underestimates theposterior variance. Other approximations, including those byLeonard (1982) and Tierney and Kadane (1986), have also beenexamined and found to require an excessive amount of computingand therefore not suitable for routine use in ability estimation.

The Bayesian approximation was then extended to 3PL inTsutakawa and Johnson (TR147, 1988). This paper demonstratesthat maximum likelihood and empirical Bayes, both of whichreplace unknown item parameters by those estimated, grosslyunderestimate the variance of the ability parameters. Thenumerical examples, upon which much of the conclusion is reached,is based on a sample of n-400. Although the discrepanciesbetween the procedures should decrease as n increases, there issome feeling at even at n-1600 the differences might not benegligible.

IV On-line calibration

Work on this topic remains incomplete due to the delayencountered in developing computer programs for Bayesian abilityestimation under 3PL. The delay was due to untimely personnelchanges, which required finding and training a new computerprogrammer each time a person left.

4

Summary

The development of Bayesian item response theory requiresconsiderable amount of computation and new techniques forapproximating posterior distributions. This researchdemonstrates that computational problems (though far from solved)can be dealt with by careful use of asymptotic approximations.It also demonstrates that reasonable prior distributions can beformulated in spite of the complexities of IRT models. But moreimportantly it shows the feasibility of developing acomprehensive and complete theory which can be adapted to largescale testing environments.

References

Dempster, A.P., Laird, N.M. & Rubin, D.B. (1977). Maximumlikelihood from incomplete data via the EM algorithm (withdiscussion). Journal of the Royal Statistical Society,Series B, 39, 1-38,

Leonard, T. (1982). Comment on "A simple predictive densityfunction", by M. Lejeune and G.D. Faulkenberry. Journal ofthe American Statistical Association 77, 657-658.

Lindley, D.V. (1980). Approximate Bayesian methods TrabajosEstadistica 31, 223-237.

Tierney, L. and Kadane, J.B. (1986). Accurate approximations forposterior moments and marginal densities. Journal of theAmerican Statistical Association 81, 82-86.

Wingersky, M. S., Barton, M.A., & Lord, F.M. (1982). LogistUser's Guide, Educational Testing Service, Princeton, NJ.

Technical Reports and Publications Prepared Under Contract

Tsutakawa, R.K. & Lin, H.Y. (1986). Bayesian estimation of itemresponse curves. Psychometrika 51, 251-267.

Rigdon, S.E. & Tsutakawa, R.K. (1987), Estimation for the Raschmodel when both ability and difficulty parameters arerandom. Journal of Educational Statistics, 12, 76-86.(previously distributed under Mathematical SciencesTechnical Report No. 133, Department of Statistics,University of Missouri.)

Tsutakawa, R.K. & Soltys, M.J. (1988). Approximation forBayesian ability estimation. Journal of Educationalstatistics, 13, 117-130. (Previously distributed underMathematical Sciences Technical Report No. 134, Departmentof Statistics, University of Missouri.)

5

Tsutakava. R.K. (1988). Dirichlet prior in Bayesian estimationof item response curves. Mathematical Sciences TechnicalReport No. 143, Department of Statistics, University ofMissouri.

Tsutakava, R.K. & Johnson, J.C. (1988). Bayesian abilityestimation via 3PL with partially known item parameters.Mathematical Sciences Technical Report No. 147, Departmentof Statistics, University of Missouri.

6

1 98/07,/0O6

University of Missouri-Columbia/Tsutakawa

Dr. rerry Ackerman Dr. R. Oarrell BockAmerican College Testing Programs University ot ChicagoP.O. Box 168 NORCIowa City, IA 52243 6030 South Ellis

Chicago, IL 60637Dr. Robert AhlersCode N711 Cdt. Arnold BohrerHuman Factors Laboratory Sectie Psycholoqisch OnderzoekNaval Training Systems Center Rekruterings-En SelectiecentrumOrlando, FL 32813 Kwartier Koninqen Astrid

Bru i jns traatDr. James Algina 1120 Brussels. BELGIUM1403 Norman HallUniversity of Florida Dr. Robert BreauxGainesville, FL 32605 Code 76

Naval Training Systems CenterDr. Erling B. Andersen Orlando. FL 42813-1100Department of StatisticsStudiestraede 6 Dr. Robert Brennan1455 Copenhagen American Collegie VeitinqDENMARK Protirams

P. 0. Box 168Dr. Eva L. Baker Iowa City. IA t;2243UCLA Center for the Study

of Evaluation Dr. James Carlson145 Moore Hall American College lestingUniversity of California ProgramLos Angeles, CA 90024 P.O. Box 168

Iowa City, IA 52243Dr. Isaac BejarMail Stop: 10-R Dr. John 8. CarrollEducational Testing Service 409 Elliott Rd., NorthRosedale Road Chapel Hill, NC 27514Princeton, NJ 08541

Dr. Robert M. CarrollDr. Menucha Birenbaum Chief of Naval OperationsSchool of Education OP-0182Tel Aviv University Washington. DC 20350Ramat Aviv 69978ISRAEL Dr. Raymond E. Christal

UES LAMP Science AdvisorDr. Arthur b. Blaiwes AFHRL/MOELCode N/12 Brooks A'B. fX /8235Naval Training Systems CenterOrlando, FL 32813-7100 Dr. Norman Cliff

Department of PsychologyDr. Bruce Bloxom Univ. of So. CaliforniaDefense Manpower Data Center Los Anoeles. CA 90089-1061550 Camino El Estero,

Suite 200Monterey, CA 93943-3231

7

1968/0Y/06

iJniversitv of Missouri-Columbia/Isutakawa

Director. Or. Fritz OrasQowManpower Support and University of Illinois

Readiness Program Department of Psycholoqv

Center for Naval Analysis 603 E. Daniel St.

2000 North Beauregard Street Champaign. IL 61820

Alexandria, VA 22311

Defense lechnical

Dr. Stanley Collver Information Center

Office of Naval rechnoloqy Cameron Station. Bldq 5

Code 222 Alexandria, VA 22314

800 N. Quincy Street Attn: TC

Arlington, VA 22217-5000 (1Z Copies)

Dr. Hans F. Crombag Dr. Stephen Dunbar

Faculty of Law 2248 Lindquist Center

University of Limburq for Measurement

P.O. Box 616 University of Iowa

Maastricht Iowa City, IA 5I2242

The NETHERLANDS 6200 MD

Dr. James A. EarlesDr. Timothy Davey Air Vorce Human Resources Lab

Educational resting Service Brooks AFB, 1X /8235

Princeton, NJ 08541

Dr. Kent Eaton

Dr. C. M. Dayton Army Nesearch Institute

Department of Measurement 5001 Eisenhower Avenue

Statistics & Evaluation Alexandria, VA 22333College of Education

University of Maryland Dr. John M. Lddins

College Park, MD 20/42 University of Illinois252 Eni]neerii1Q kes.arch

Dr. Ralph J. DeAvala Laboratory

Measurement, Statistics, 103 South Mathews 51.reet

and Evaluation Urbana, IL 61801

Benjamin Bldg., Rm. 4112UJniversity of Maryland Dr. Sus.in Lmbretson

College Park, MD 20/42 University of Kansas

Psycholoqv Departiment

Dr. Dattprasad Divqi 4,b FraserCenter for Naval Analysis Lawrence. KS 66045

4401 Ford Avenue

P.O. Box 16268 Dr. Georqe Englehard, Jr.Alexandria, VA 22302-0268 Division of Educational Studies

Emory University

Dr. Hei-Ki Dong 210 Fishburne Bldg.

Bell Communications Research Atlanta, GA 30322

6 Corporate Place

PYA-IK226 Dr. Benjamin A. FairbankPiscataway, NJ 08854 Performance Metrics, Inc.

5825 CallaghanSuite 225

San Antonio, TX 78228

8

1988/07/06

University of Missouri-Columbia/lsutakawa

Dr. P-A. Federico Or. Robert GlaserCode 51 Learning Research

NPRDC & Development CenterSan Diego, CA 92152-6800 University of Pittsburgh

3939 O'Hara StreetDr. Leonard Feldt Pittsburgh, PA 15260

Lindquist Centerfor Measurement Dr. bert Green

University of Iowa Johns Hopkins University

Iowa City, IA 52242 Department of PsychologyCharles & 34th Street

Dr. Richard L. Ferguson Baltimore, MD 21218American Colleqe TestingP.O. Box 168 DORNIER GMBHIowa City, IA 52243 P.O. Box 1420

D-7990 Friedrichshafen IDr. Gerhard Fischer WESI GERMANYLiebiggasse 5/3A 1010 Vienna Dr. Ronald K. HambletonAUSTRIA Universitv of Massachusetts

Laboratory of PsychometricDr. Myron Fischl and Evaluative ResearchU.S. Army Headquarters Hills South, Room 152DAPE-MRR Amherst, MA 01003The PentagonWashington, DC 20310-0300 Dr. Delwvn Harnisch

Universitv of IllinoisProf. Donald Vitzqerald 51 Gertv DriveUniversity of New England Champaign, IL 61820Department of PsychologyArmidale, New South Wales 2351 Dr. Grant HenningAUSIRALIA Senior Research Scientist

Division of MeasurementMr. Paul Foley Research and ServicesNavy Personnel R&D Center Educational restinq ServiceSan Diego, CA 92152-6800 Princeton, NJ 08541

Dr. Alfred R. Fregly Ms. Rebecca HetterAFOSR/NL, Bldg. 410 Navy Personnel R&D CenterBolling AFS, DC 20332-6448 Code 63

San Diego, CA 92152-6800Dr. Robert D. GibbonsIllinois State Psychiatric Inst. Or. Paul W. HollandRm 529W Educational Testing Service, 21-T1601 W. Taylor Street Rosedale RoadChicago, IL 60612 Prinoeton, NJ 08541

Dr. Janice Gifford Prof. Lutz F. HornkeUniversity of Massachusetts Institut fur PsychologieSchool of Education RW1H AachenAmherst, MA 01003 Jaegerstrasse 17/19

0-5100 AachenWES1 GERMANY

9

1988/0?/06

University of Missouri-Columbia/Tsutakawa

Dr. Paul Horst Or. G. Gage Kingsbury

677 G Street, #184 Portland Public Schools

Chula Vista, CA 92010 Research and Evaluation Department

501 North Dixon Street

Mr. Dick Hoshaw P. 0. Box 3107

OP-]3t Portland, OR 97209--3107

Arlinqton Annex

Room 2834 Dr. William Koch

Washington, DL 20330 i.- t Mea;. 3nl L/aI. Atr.

University of lexis-Au-tin

Dr. Lloyd Humphrevs , i ,i in J.rsi~v of JI in S

- ;u .?it- D.3Sed Lducat ion

1-hamp3iqg, LL bldeJO Resear.h Laboratory

Dr. Steven tiLuf, rthana. IL Li:')1

iUlniversity of Alberta Dr. Leonard Kroeker

Edmonton, Alberta Navy Personnel RW Center

CANADA TFW 2G5 Code 62

San DieQo. CA 92152-6800

Dr. Huynh Huynh

College of Education Dr. Jerry Lehnus

Univ. of South Carolina Defense Manpower Data Center

Columbia, SC 29208 Suite 400

1600 Wilson Blvd

Dr. Robert Jannarone Rosslvn. VA 22209

Elec. and Computer Eng. Dept.

University of South Carolina Dr. rhomas Leonard

Columbia, SC 29208 University of Wisconsin

Department of Statistics

Dr. Douglas H. Jones 1210 West Dayton Street

Thatcher Jones Associates Madison. WI 53705

P.O. Box 664010 Trafalga- Court Dr. Michael Levine

Lawrenceville, NJ 08648 Educational Psychology

210 Education Bldg.

Dr. Milton S. Katz University of illinois

European Science Coordination Champaign, IL 61801

OfficeU.S. Army Research Institute Dr. Charles Lewis

Box 65 Educational lesting Service

FPO New York 09510-1500 Princeton, NJ 08541-0001

Prof. John A. Keats Dr. Robert L. Linn

Department of Psychology Campus Box 249

University of Newcastle University of Colorado

N.S.W. 2308 Boulder, CO 80309-0249AUSIRALIA

10

1988/07/06

University of Missouri-Columbia/Tsutakawa

Dr. Robert Lockman Dr. William Montaque

Center for Naval Analysis NPRDC Code 13

4401 Ford Avenue San Diego, CA 92152-6800

P.O. Box 16268Alexandria, VA 22302-0268 Ms. Kathleen Moreno

Navy Personnel R&D Center

Dr. Frederic M. Lord Code 62

Educational Testing Service San Dieqo. CA 92152-6800Princeton, NJ 08541 Headquarters Marine Corps

Dr. George B. Macready Code MPI-20

Department of Measurement Washington, DC 20380

Statistics & EvaluationCollege of Education Dr. W. Alan Nicewander

University of Maryland University of Oklahoma

College Park, MD 20742 Department of PsychologyNorman, OK 73071

Dr. Gary MarcoStop 31-E Deputy lechnical Director

Educational Testing Service NPRDC Code 01A

Princeton, NJ 08451 San Diego, CA 9-lb?-6800

Dr. James R. McBride Director, Training Laboratory.

The Psychological Corporation NPRDC (Code 05)

1250 Sixth Avenue San Dieiqo, CA 92t52-6800

San Diego, CA 92101Directr. Manpower ana Person-,el

Or. Clarence C. McCormick Laboratorv.

HO, JSM upCM/MEpC r NPROt (Code 06)

2500 Green Bay Road .)a;( u iio. J-, i ,

North 1.hicao. It b0)064

-N uroanizational Systems Lab.

Educational Testing Service NPROX (Coie ()7)

16-T .-, . , --

Library, NPRDC

Dr. James McMichael Code P2O1L

Technical Director San Dieqo, CA 92152-6800

Navy Personnel R&D CenterSan Diego, CA 92152-6800 Commanding Officer,

Naval Research Laboratory

Dr. Barbara Means Code 2627

SRI International Washington, DC 20390

333 Ravenswood AvenueMenlo Park, CA 94025 Dr. Harold F. O'Neil. Jr.

School of Education - WPH 801

Dr. Robert Mislevy Department of Educational

Educational Testing Service Psychology & fechnology

Princeton, NJ 08541 University of Southern CaliforniaLos Angeles, CA 90089-0031

11

I

~~1988,'0!/O06

University of Missouri-Columbia/isutakawa

Dr. James B. Olsen Oepartment of Operations Research,WICAT Systems Naval Postgraduate School1875 South State Street Monterey, CA 93940Orem, UT 84058

Dr. Mark D. ReckaseOffice of Naval Research. ACT

Code 1142CS P. 0. Box 168800 N. Quincy Street Iowa City. IA 52243Arlinqton, VA 22217-5000(6 Copies) Dr. Malcolm Ree

AFHRL/MOAOffice of Naval Research, Brooks AlB, IX 13235

Code 125800 N. Quincy Street Dr. Barry RieqelhauptArlington, VA 22211-5000 HumRRO

1100 South Washington StreetAssistant for MPT Research, Alexandria. VA 22314

Development and StudiesOP OB7 Dr. Carl Ross

Washinqton, DC 203(0 CNLT-PDCDBuilding 90

Dr. Judith Orasanu Great Lakes NrC, IL b0088Basic Research OfficeArmy Research Institute Dr. J. Ryan5001 Eisenhower Avenue Department of EducationAlexandria, VA 22333 University of South Carolina

Columbia, SC 29208Dr. Jesse OrlanskyInstitute for Defense Analyses Dr. Fumiko Samejima1801 N. Beauregard St. Department of PsychologyAlexandria, VA 22311 University of lennessee

3106 Austin Peay Bldg.Dr. Randolph Park Knoxville. I'N 37916-0900Army Research Institute5001 Eisenhower Blvd. Mr. Drew SandsAlexandria, VA 22333 NPRDC Code 62

San Diego, CA 92152-6800Wayne M. PatienceAmerican Council on Education Lowell SchoerGED Testing Service, Suite 20 Psycholoqical & GuantitativeOne Dupont Circle, NW FoundationsWashington, DC 20036 College of Lducation

University of IowaDr. James Paulson Iowa City, IA 52242Department of PsychologyPortland State University Dr. Mary SchratzP.O. Box 751 Navy Personnel R&D CenterPortland, OR 97207 San Diego, CA 92152-6800

Dept. of Administrative Sciences Dr. Dan SegallCode 54 Navv Personnel R&D CenterNaval Postgraduate School San Dieqo, CA 92152Monterey, CA 93943-5026

12

I-w I W

t988/07/06

IJniversity of, Missouri-Columbia/rsutakawa

Dr. W. Steve Sellman Dr. Hariharan SwaminathanOASD(MRA&L) Laboratory of Psychometric and28269 rhe Pentagon Evaluation ResearchWashington, DC 20301 School of Education

University of MassachusettsDr. Kazuo Shigemasu Amherst, MA 010037-9-24 Kugenuma-KaiganFujisawa 251 Mr. Brad SympsonJAPAN Navy Personnel RPM Center

Code-62Dr. William Sim-, San Diego, CA 92152-6800Center for Naval Analysis4401 Ford Avenue Dr. John TangneyP.O. Box 16268 AFOSR/NL, bldg. 410Alexandria, VA 22302-0268 Boiling AIEB. DC 20332-6448

Dr. H. Wallace Sinaiko Dr. Kikumi latsuokaManpower Research CERL

and Advisory Services 2b2 Engineering ResearchSmithsonian Institution Laboratory801 North Pitt Street. Suite 120 103 S. Mathews AvenueAlexandria, VA 22314-1/13 Urbana, IL 61801

Dr. Richard E. Snow Dr. Maurice latsuokaSchool of Education 220 Educal:ion BldgStanford University 1310 S. Sixth St.Stanford, CA 94305 Champaign, IL 61820

Dr. Richard C. Sorensen Dr. David IhissenNavy Personnel R&D Center Department of PsychologySan Diego, CA 92152-6800 University of Kansas

Lawrence, KS 66044Dr. Paul SpeckmanUniversity of Missouri Mr. Gary lhomassonDepartment of Statistics University of IllinoisColumbia, MO 65201 Educational Psychology

Champain, IL 61820Dr. Judy SprayACT Dr. Robert IsutakawaP.O. Box 168 University of MissouriIowa City, IA 52243 Department of Statistics

222 Math. Sciences Bldg.Dr. Martha Stocking Columbia, MO 65211Educational Testing ServicePrinceton, NJ 08541 Dr. Ledyard Tucker

University of IllinoisDr. William Stout Department of" PsychologyUniversity of Illinois 603 E. Daniel StreetDepartment of Statistics Champaiqn, IL 61820101 Illini Hall725 South Wright St.Champaign, IL 61820

13

1988/07/06

University of Missouri-Columbia/isutakawa

Dr. Vern W. Urry Or. Douglas Wetzel

Personnel R&D Center Code 51

Office of Personnel Management Navy Personnel R&D Center

1900 E. Street, NW San Diego, CA 92152-6800

Washington, DC 20415Dr. kand R. Wilcox

Dr. David Vale University of Southern

Assessment Systems Corp. California

2233 University Avenue Depariment of Psychology

Suite 440 Los Angeles. CA 90089-1061

St. Paul, MN 55114German Military Representative

Dr. Frank L. Vicino ATTN: Wolfqang Wildgrube

Navy Personnel R&D Center Streitkraefteamt

San Diego, CA 92152-6800 D-5300 Bonn 24000 Brandywine Street, NW

Dr. Howard Wainer Washington, DC 20016

Educational Testing ServicePrinceton, NJ 08541 Dr. Bruce Williams

Department of Educational

Dr. Minq-Mei Wang Psychology

Lindquist Center University of Illinois

for Measurement Urbana, IL 61801

University of IowaIowa City, 1A 52242 Dr. Hilda Wing

NRC MH-176

Dr. Thomas A. Warm 2101 Constitution Ave.

Coast Guard Institute Washington, DC 20418

P. 0. Substation 18Oklahoma City, OK 73169 Dr. Martin F. Wiskoff

Defense Manpower Data Lenter

Dr. Brian Waters 550 Camino El Estero

HumRRO Suite 200

12908 Argyle Circle Monterey. CA 93943-3231

Alexandria, VA 22314Mr. John H. Wolfe

Dr. David J. Weiss Navy Personnel R&0 Center

N660 Elliott Hall San Diego, CA 92152-6800

University of Minnesota75 E. River Road Dr. George Wong

Minneapolis, MN 55455-0344 Biostatistics LaboratoryMemorial Sloan-Kettering

Dr. Ronald A. Weitzman Cancer Center

Box 146 1275 York Avenue

Carmel, CA 93921 New York, NY 10021

Major John Welsh Dr. Wallace Wulfeck, III

AFHRL/MOAN Navy Personnel R&D Center

Brooks AFB, TX 78223 Code 51San Diego, CA 92152-6800

14

II 1988,0//06

Universiiv of MSsour.-C.uiumbia/lsutakawa

Dr. Kentaro Yamamoto(3-TEducational Testing ServiceRosedale RoadPrinceton. N.J 08541

Dr. Wendy YenCTB/McGraw H i IIDel Monte Research ParkMonterey. CA 93940

Dr. Joseph L. YoungNational Science F-oundationRoom 3201800 G Street, N.W.Washinqton. DC 20550

Mr. Anthony R. ZaraNational Council of State

Boards of Nursing, Inc.625 North Michigan AvenueSuite 1544Chicago, IL 60611

Dr. Peter StoloffCenter for Naval Analysis4401 Ford AvenueP.O. Box 16268Alexandria, VA 22302-0268

15