DOCUMENT RESUME TM 018 884 AUTHOR Junker, Brian W ...DOCUMENT RESUME ED 348 399 TM 018 884 AUTHOR Junker, Brian W. TITLE A Note on Recovering the Ability Distribution from Test Scores

DOCUMENT RESUME

ED 348 399 TM 018 884

AUTHOR Junker, Brian W.TITLE A Note on Recovering the Ability Distribution from

Test Scores.INSTITUTION Carnegie Mellon Univ., Pittsburgh, PA. Dept. of

Statistics.SPONS AGENCY Office of Naval RPsearch, Arlington, VA. Cognitive

and Neural Sciences Div.REPORT NO ONR/CS-92-1PUB DATE Aay 92CONTRACT N00014-91-J-1208NOTE 36p.PUB TYPE Reports - Eval,uative/Feasibility (142)

EDRS PRICE MF01/PCO2 Plus Postage.DESCRIPTORS Computer Simulation; *Equations (Mathematics);

*Estimation (Mathematics); Item Bias; Item ResponseTheory; *Mathematical Models; *Scores; *StatisticalDistributions; Test Length

IDENTIFIERS *Ability Estimates; Local Independence (Tests);Population ParaMeters; Smoothing Methods; *ThetaEstimates

ABSTRACTA simple scheme is proposed for smoothly

approximating the ability distribution for relatively long tests,assuming that the item characteristic curves (ICCs) are known or wellestimated. The scheme works for a general class of ICCs and isguaranteed to completely recover the theta distribution as the testlength increases. The proposed method of estimating the abilitydistribution is robust to some violations of local independence.After an initial function inversion, the scheme can be inexpensivelyused to recover the theta distribution in Peon of several differentadministrations of the same test or several subpopulations in onetest administration. Moreover, this appi:oach could be used to recoverthe distribution of a dominant ability dimension when localindependence fails. The scheme provides a starting place fordiagnostics concerning assumptions about the shape of the thetadistribution or ICCs of a particular test. Work is currently underway to further examine and refine these methods using essentiallyunidimensional simulation data and to apply the estimator to realtests. Kernel smoothing is also considered. A 16-item list ofreferences, 10 tables, 6 graphs, and 2 appendixes that providedetails of the simulation and procfs are included. (RLC)

Reproductions supplied by EDRS are the best that can be madefrom the original document.

It**********************************************************************

D.- 4,

-"so vo

, .41.1*

A Note on Recoveringthe Ability Distribution

from Test Scoresby

Brian W. Junker

Department of StatisticsCarnegie Mellon University

Pittsburgh, PA 15213

May 1992

Technical Report ONR/CS 02-1

Prepared for the Model-Based Measurement Program. Cognitive and NeuralSciences Division. Office of Naval Research, under grant number N00014-91-J-1208. RIT 4421-560. Approved for public release; distribution unlimited.Reproduction in whole or in part is permitted for any purpose of the UnitedSt.qes Government.

3

REPORT DOCUMENTATION PAGEfarm Appia4V41CMS Pi& o1044118

New IMMIMM MOM* MIF TM MMIMM Of iMOMIIMIM ot I1SMEMS10 Onifte. 4 *Out DM vesamver. sioncaso Ithe stave let reivemov westgaseas.IMMO MIMMOMM

ementorM ANNO19 MU legglomononis sr sea aver me s s s ram

ais M00% 411$ ArtirsonDC U. ,

9ilevisues ow strowiammq no ems ovvele. amcanntaa at IMIeRMIMea. sIOCIWWWp 00119111111MS tatDan armor. loot 1204. attaatrat. vA 22.222,4303.

COMODIPOOO MO MMOmMOMIPMMMISSOP 91.9M0M010. conassatimmalinpatataaws vat ataaatt.to tvanotatee yovemea1nov lavvons. DIM(jOPOIN tor owenosspom 000111sorna

t01100 Olotwo mimememNW oe oft sweet wtta---t.trari gamtuattprows01011411111.11ffiessatos.

1. ARMY USE ONLY (Wye stow 12. REPORT DAT(I 20 May 1992

1 3. REPORT TYPE AND DATIS COMMI Technical

4. TITLE AND SUBTITLE

A Note on Recovering the Ability Distribution frma

Test Stores.

S. FUNGUS

G: N00014PE: 61153NPRI 304204

NUMBELS

-91 -.3 -1208

ORGANQATKIN

92-1

a. AUTHOR(S)

Brian W. Junker

TA: ER04204-01WIT; 4421-560-4

a PERRNIMINGMUDONTIMMER

011iCS

7. ratteasiaaa oaaarazanos Nahum aND apoasis4u)

Department of StatisticsCarnegie Mellon University .

Pittsburgh, PA 15213

9. SPONSORING/MONITORING AGENCY MAWS/ AND ADENUESS(ES)

Cognitive Science ProgramOffice of Naval Research (Code 1142C5)800 N. Quiney StreetArlington, VA 22217-5000

1 . SPONSORING/MONITORINGAGENCY REPORT NUMBER

11. SUPPLEMENTARY NOTES

Submitted for publication

12a. DISTRIBUTION t AVAILABILITY STATEMENT

Approved for public release; distribution 111,14m4ted.

12b. DISTRIBUTION CODE

13. ABSTRACT (Maxemum100 wows)

We propose a simple scheme for smoothly approximating the abilityfor relatively long tests, assuming that the ICC's are known orThe scheme works for quite a general class of item characteristicand is guaranteed to completely recover the 6 distribution as the.7, grows. After au initial function inversion, the scheme can beused to recover the e distribution in each of several differentof the same test (or subpopulations in one test administration).approach could be used to recover the distribution of a dominantwhen local independence fails. Finally, the scheme provides a startingdiagnostics concerning assumptions about the shape of the e distributionof a particulsr test. Work is currently underway to further examinethese methods using essentially unidimensional simulatimi,data,estimators to real tests.

distributionwell estimated.

curves (ICC's)test length.inexpensively

administrationsMoreover, thisability dimension

place fororICC's

and refineand to apply the

...-------..iS. NUMBER OF PAGES

27

...------..4. SUBJECT TERMS

Item response theory, kernel smoothing, latent traitdistribution, population assessment.

16. PRICE CODE

17. SECURITY CLASSIFICATIONOF REPORT

UNCLASSIFIED

1 . SECURITY CLASSWICATIONOF THIS F AUUNCLASSIFIED

.9. SECURITY CLASSIFICA IONOF ABSTF.ACT

UNCLASSIFIED

2.3. LiMITATION Of ABSTRACT

UL

_

VSN 15»C.,-01-Z80-5500 ,tanaara i-orm er -*--wmipes et, N SA Sod /3.I5?WIG,

1

AbstractWe propose a simple scheme for smoothly approximating the abslitr distribu-

tion for relatively long tests, assuming that the ICC's are known or well estimated.-The scheme works for quite a maul class of item characteristic turves (ICC's)and is guaranteed to completely recover the 9 distlibution an the test length,.1, grows. After an initial function inversice, the scheme can be inexpensivelrused to reamer the ê distribution in each of several different administrinionsof the simie test (or subpopulations in one test administratkm). Mereevert thisapproach could be used to recover the distnbution of a ciaminant abilitY dimen-sion when local independence fails. Finally, the scheme provides a starting placefor diagnostics concerning assumptions about the shape of the 0 distribution orICC's of a particular test. Work is currently underway to further examine an&refine these methods using estumtially unidimensional simulation data, and toapply the estimators to real. tests.

Keyword= Item response theory, kernel smoothing, latent trait distribution,population assessment.

The work reported here was initiated under the direction of Paul Rolland. while Junher walil a participantin the Educational Testing Service Sumner Predoctoral Research Program. Initial computer simulationswere performed by Dorothy Thayer at ETS; the simulations reported here were performed by Junker at th'.$University of Illinois and Carnegie Mellon University.

Junker: Recovering the Ability Distribution 2

I. The basic estimator

A principal application of educational testing is inferring the distribution of abilities in

various populations. This task is important for both users of these tests (in, say, comparing

various subpopnlations) and researchers and test developers (in, say, developing or using

item calibrationICC parameter estimationprocedures within the IRT framework).

Inference about the ability distribution from item response data goes back at least to

Lord (1953) who gives an interesting qualitative account of the possible distortions induced

by the traditional IRT model. With the rise in popularity of item response theory, IRT,

many methods for estimating the latent distribution have been developed.

Samejima and Livingsten (1979) fit polynomials to latent densities using the method of

moments. Samejima (1984) also fits 0 densities, given the MLE Ô, using specific parametric

familie. by matching two or more moments. Levine (1984, 1985) projects the (unknown)

latent distribution onto a convenient function space in the span of the test's conditional

likelihood functions and estimates the projection by maximum likelihood. Mislevy (1984)

assumes that the ability distribution is well approximated by a collection of masses centered

at points placed a priori along the 0 axis and estimates the sizes of the masses at each

point. More generally, hierarchical and/or empirical Bayes techniques may be used to esti-

mate parameters of the latent trait distribution if it belongs to a tractable family of priors.

These methods all rely upon local independence for their validity; moreover they tend to be

expensive in terms of computation and storage.

We will examine a simpler method of estimating the ability distribution which, in addi-

tion, is robust to some violations of local independence. Consider a set of .1 binary items

that may be embedded in a longer sequence or pool of items (X1, X2, X3, . .). Let e be the

latent trait of interest, let P1 (0), P2(0), . . , Pj(0) be the item characteristic curves, ICC's,


with respect to 0, and denote averages of items as Xj = Eif xi, and similarly for averages

-15j(0) of ICC& Under the usual local independence (LI) and monotonicity (M) conditions

of item response theory (e.g. liambleton, 1989), or more generally under Stout's (1990)

formulation of essential independence (EI) and local asymptotic discrimination (LAD), we

know that aj(Xj) ss. 7531(XJ) is a plausible point estimate of 8: ej(L) is a consistent

estimator of e under either set of assumptions. It immediately follows that the distribution

of &J(L)

FAO = t]

converges to that of 0 as well (e.g. Serfiing, 1980, p. 19). Now consider administering the

test Xj to N examinees, obtaining N response vectorsXL,' 2fArj and corresponding 0

estimates aj(,1i j), .N.7); a natural estimator of the 8 distribution is the "empirical"

distribution of these aj's

:1pAr.,(t) :4110,GLJ)5t)

{fraction of 01/MLA's 5. t}

where the "indicator function" ls takes the value.1 if S is true and 0 if S is false.

(1)

Theorem I Suppose (X1, X2, ...) is a sequence of items and 0 is a latent trait such that

EI and LAD hold. Define j 3(2( i) as above. If the distribution function

F(t) = Pfe < tl

is continuous, the empirical distnbution function PN.J(t) defined in (I), converges in proba-

bility to F at each t as both J -0 co and N o o.

As with the work of Stout (1990) and Junker (1991), the embedding in an infinite-length

item pool is partly a conceptual tool. In practice, one might check the EI condition using

Stout's (1987) test, and check the LAD condition by verifying that the average ICC for a

particular test was an invertible function.


In fact, the full strength of the LAD condition is not needed here. A weaker condition

that also gives the theorem is that, for all t2 > t1 there exists c(t1, t2) such that

Urn infi5j(C2) 7.7(ti) > f(tlt i2) (2)J

Similarly, the full strength of te EI condition is nct needed. It suffices to have, for all t,

lim Vat (Xj10 = t) = 0 (3)

Under the weaker conditions (2) and (3), the consistency of P.71(X1) as a. point estimate

for 0 may fail, but Theorem I still goal through The proof of Theorem I is ased on a

well-known exponential bound due to Dvoretsky, Kiefer and Wolfowitz (Settling, 1980, p.

59) on the error made in approximating Fj(t) with PN,j(t). See Appendix B for some details.

2 Two practical considerations

Note that the theorem does not in any way require that the ICC's have 0 and 1 as lower and

upper asymptotes. For example, if Pj has a lower asymptote c, i.e.,

liminfPJ(t) > c > 0, Vt E-..00

there certainly could be positive probability that some Sa's have X j < c. The only rea-

sonable thing for TT' to do with such an Xj is send it to oo, which ruins the estimate of

F.

But for any fixed 0, if e < iirn infj, RAO),

lim sup PI:Vj < elI 4..co

00

Um sup f PtXj ele = tldF(t)J ..ao 00

cv,

< Um supf P[XJ pi(o)le = ticiF(t)00

F(0),


after observing that 11j 5_ P3(0)10 t} 1{e


Theorem 2 Suppose XII X2, and 0 are as in Theorem I with ICC's P1(t), P2(07

with average TAO as before. and suppose

R1(t), R2(t), .

are another set of ICC's. unth average irij(t). Let15:7' and R be the corresponding inverses,

and let

aj(ei) =

Fix 0 such that 173:71R3(0) has a finite limit r(0). Then

FJ(0) = Prib(&) 5- 91 F(r(17))

(where F is the distribution of 0). If these hypotheses hold for every 9 and if r and F arecontinuous functions, then the convergence is uniform in 9.

The existence of the limit r(0) is a technical requirement that, like LAD, is innocuous in

the context of real, finite length tests. The most useful interpretation of Theorem 2 is that

IF3(0) FP5:-1Ti3(0)}1 0

as J oo, i.e., the distribution of 0 is estimated with a distortion 7534R3.. This followsfrom the theorem if F is continuous at r(9).

The proof of Theorem 2 expands on the technique used to prove convergence of FAO) to

F(0); see Appendix B. Just as in Theorem I it is also possible to show that the empirical

distributions1

PN.Atin=1

converge to F(r(0)).

The value of Theorem 2 is that if the function Pi-'(-173(6)) can be (partially) identified,

then the distribution of 9 j can still tell us a lot about the underlying 8 distribution. For

Junker: Recovering the Ability Distribution

example, if the "true ICC's" are P(0) and the 0 distribution is recovered with "estimated

R (0), with the estimated ICC's satisfying

frij(0) Rj(0)1 0

as J oo, then the estimated distributions Fj will converge to the true distribution F of

0, as long as the derivative P(0) is bounded away from zero at each U as J 0 co (this is

guaranteed by LAD for example).

Some knowledge of the underlying 0 distribution may even be available when the "true

ICC's" P.,(0) and the "recovery ICC's" RAO) do not match up asymptotically. For exam-

ple, it is easy to check numerically that for 'typical" parameter values, averages of logistic

ICC's are themselves approximately logistic (with parameters approximately the averages of

the discrimination and difficulty parameters of the individual ICC's). Thus for example if

the PAO) are Rasch (one-parameter logistic) and the estimation method for the "difficulty

parameters" bi is !mown, on average, to bias the k by some fixed but unknown additive

bias parameter (so that logit R1(0) 'Pe logil fl) then roughly 7;1(713(9)) a0 0,

with, a near 1, so that the location of the 0 distribution will be estimated wrongly but

the (shape) family to which it belongs may still be identified. Similar considerations apply

when the P2(0) are 3PL. and the R1(0) are 2PL: over the domain of 771,' (0), 75(L(0)) is

approximately linear.

3 Kernel smoothing

The basic estimator proposed ;n (1) is the "empirical distribution" function

ky.,i(t)1 +v..

NI

= EPN[x.,1{757,4 cil.n

4414F- tit e

r .t`

Junker: Recovering the Ability Distribution8

where

NPivfx." =j/e1) = v E 1 frst.F=3/nm. 4

is the natural estimator of the (discrete) distribution of Ari based on N observationsThe indicator function on the far right in (4) may be written

1 k{771(1/4

;Maker: Recovering the Ability Distribution9

Third, the observations Ty, ...,XNJ must betransformed by the nonlinear transfor-

mation 733'. This means that thegranularity changes- over the tinge of e and r.r; this

complicates practical calculations such as thoseleading to optimal rates for N1.1 and h.--

We now show that the weighted root mean square error(RMS).betvreen this estimator

and the true e distribution goes to zero as N,..1 co.The thesmem below is analogous to

Theorem I.

Thelon 3 Suppose X1, X2 . .and e are as in Theorem .1 with 1CC's P1(9),P2(0),...

Define kiqjh(t) as in (5), for a fixed kernel distributionfunction K. Then if the distribution

function F of 0 is continuous, and K has a finite first absolutemoment,

14RMS [EN.rh(t) F(t)]2g(t)dt1° - 0

as N oo,. --+ co and h --- 0, for any density 0).

(6)

Unlike most nonparametric density estimation results,there is no restriction on the rates

at which h 0, N 1. co or .1 oo. This ispartly because a distribution function is

smoother than, and therefore easier to estimate than, a density.The corresponding technique

for estimation of the 0 densitywould require h3 to tend to zero moreslowly than gij(Xj)

(3), for example, as well as further conditions onthe rates at which N and .1 tend to co.

Despite the fact that there are no rates in the theorem,devising h as a function of N and .1

to produce the "right" amount of smoothing is animportant issue to which we shall return

below.

The proof of Theorem 3 (see Appendix B) is based ondecomposing the RMS in (6) as

oe

RMS2 = {P[77(xj) hlt 5_ tll3g(t)dtL40t -7)] g(t)dtr° varK [ h+ (7)

0

Junker: Recovering the A ;Ality Distribution

where Y is a random variable with distribution K, independent of 0 and ail item responses.This technique can be modified to show that

EIPN,m(t) FM? 0 0for any t, and hence ENja(t) F(t) in probability, for each continuity point t of F. Forexample, thif provides another proof that our original estimator EN4 converges in probabilityto F. It would also be clear from the proof that the same smoothing could be applied withany consistent estimator jj in place 757:71(Xj).

From the decomposition of RMS in (7) into squared-bias and variance terms it seemsthat the optimal h should be more sensitive to J than N. Indeed, when I is small and Nis relatively large, the coarse g7-antlarity inherent in 3331(XJ) should predominate over thefiner granularity inherent in observing N examinees.A workable approach to setting h is to make a quick, crude estimate of the variance of eby assuming that Xj is uniformly distributed on the interval defined by the lower asymptotec and the upper asymptote d of NO) and then applying the formula

h = C. (Var ern(8)

which seems appropriate when it- has a variance (Silverman. 1986, pp. 45-48; Reiss, 1981).Our crude estimate of Var 0 is obtained by tabulating values of 161) = 7.7-4((j 1)/(1 + 2))for all j such that c < (j ± 1)/(4.1 + 2) < d, and calculatingVar 0)112 P.- (.7413)(interquartile range)

(following the relationship between interquartile range and standard deviation for the Normaldistribution). Preliminary trials with C = 1, 1/2.1/3, 1/4 in (8) indicated that C = 1/3produced the best RMS results.There is reason o believe tize, ,..licire of K should not be very influential on the RMS in(6) (Silverman, 1986, pp. 42-43). The K used in 011: dolions was

t 3K(t) u2) 104


-1t

14.(3t t3 + 2) , itt < 1

1 t > 1

This choice is conservative about the tails of the e distribution.

4 Computer simulation

(9)

The estimators proposed in Theorems 1 through 3 are less complicated than distribution

estimators currently in use in IRT. To help evaluate these estimators a pilot simulation

study was performed. In this simulation, item response data was generated using various

4 1 parametric moiels, and we attempted to recover the ability distribution using boththe smoothed and unsmootIled estimators.

Monte Carlo trials: M = 100Examinee samp e size: N = 5,000Ability distribution: Normal N(0, 1,

Bimodal Mixture 1. /V( 1.5, 1) + i N(1.5, 1)Discontinuous XI I

Test length: J = 10, 30. 60. 100ICC type: Rasch: Vs equally spaced from -2 to 2

3PL: Vs equally spaced from -2 to 2(ifs cycling through 0.5, 1.0, 1.5c-'s all set to 0.21

'Estimated': Generated with the 3PL ICC's above;Estimated with the ICC parameters:

81.--. N(b1,1/J)cti ,., Mai3O.25)

max{ N(0.2, 0.1), 0)(all independent).

Table 1: Monte Carlo simulation parameters.

The parameters of the pilot simulation are indicated in Table I. All possible combinations

1_5

hve 900 0

Junker Recovering the Ability Distribution 12

of these parameters were investigated. The choice of ability distributions was intended to

examine two °typical" and one "worst case" target distribution. While the standard normal

distribution is extremely smooth and has a bounded positive density the distribution of the

shifted chi-squared random variable x1-1 puts no mass below 9 = 1 and the density :lumps

from 0 to 1-fx) at 0 = 1. (This choice is not intended to be terribly realistic, but allmvs

us to explore the performance of our distribution estimator under adverse circumstances.)

Although the means of these distributions are both 0, the chi-squared distribution has twice

the variance of the normal. The bimodal mixture was chosen to represent a situation where

two radically different types of examinee take the test. Its standard deviation is also greater

than 1 (roughly 1.8).

The ICC's used were all subfainilies of the three parameter logistic (3PL) curves:

P2(t) = ci (1 ci)[1 expialit

In the case labelled "Rasch', ai 1,c1 0 and bi are as indicated. The same ICC'swere used to recover F as to generate the data. Indeed ey) is exactly the MLE for 9

under the Rasch model with known item parameters. Similarly for the 3PL case, whex all

the parameters were allowed to vary as indicated above; now xp is a somewhat inefficient

estimator of O. In the case labelled 'Estimated', the 3PL ICC's were used to generate the

data (113(0)'s in Theorem 2) but then their item parameters were deliberately contaminated

with noise to produce the -recovery ICCs" (.111(0)'s in Theorem 2) used to estimate F, to

roughly approximate the practical situation in which item parameters themselves must be

estimated from data. Thus the cases Rasch, 3PL, and 'Estimated' represent increasingly

hostile situations for the distribution estimator to work in.

Finally, the choice of N = 5.000 examinees was somewhat arbitrary. In preliminary runs,

N = 1,000 and N = 10,000 yielded measures of fit of the estimated ability distribution to the

true distributke quite comparable to those reported here. The main difference was in the

variaic of our estimated measures of fit. N = 5,000 was chosen because at that level the

161

Junket.: Recovering the Ability Distribution 13

variance is much better than at N = 1,000 and not much worse than that at N = 10,000.

The basic estimators used to compare recovery of F from case to case were the empirical

distribution function (EDF)

N= Tsr E ,/pcx,44.0_,st)

n=1.

and the kernel distribution estimator (KDE)

N

N (t) = KN n=1

where

ij(j) (Mal

=_p_I X + 11+ 2 I

(and K and h are as described in (8) and (9) above). Each of these distribution estimators

is consistent for the true e distribution, by application of Theorem 1 through Theorem 3.

For each simulated data set, sample means and standard deviations for estimates of

00142RMS = {Ef [Fest(t) F(0139(t)dt}

are reported. In addition, mean estimates of

MAX = Etsup{1Fett(t) oo t co)]

and the average value LOC = tn.,. at which MAX is attained axe reported. (Note: Feat

stands for either of the distribution estimators above.) In general the weighting function g

should be chosen to reflect our interests in the 0 distribution F: g should give more weight

to areas of F that should be well-estimated and less weight to areas of F for which we are

willing to tolerate less pod estimation. In these simulations, the weighting function g was

taken to be the standard normal density: some weight is given to estimating F well at all

O's, but more weight is given to estimating F well near 0 = 0. More details about these

distances and the methods of calculation can be found in Appendix A below.

17

I 0

Junker: Recovering the A bility Distribution 14

TestLength Estimator

RMSave SD

DeviationMAX LOC

10 EDF 0.04655 0.00002 0.11021 0.37694

KDE 0.02318 0.00003 0.03812 0.89134

30 EDF 0.01692 0.00001 0.04032 0.09754KDE 0.00887 0.00002 0.01447 0.23184

60 EDF 0.00984 0.00002 0.02510 0.07844KDE 0.00652 0.00002 0.01076 0.05334

100 EDF 0.00731 0.00002 0.01895 -0.02856KDE 0.00577 0.00002 0.00965 -0.07616

Table 2: e Pd N(0, 1 ), Rasch


R MS

ave SDDeviation 1

MAX LOC 1

10 EDF 0.07015 0.00002 0.15724 -1.00076K DE 0.05158 0.00003 0.09368 -1.23646

30 EDF 0.02794 0.00002 0.06418 -0.77476KDE 0.02176 0.00002 0.03755 -1.26626

60 EDF 0.01521 0.00002 0.03527 -0.46316KDE 0.01251 0.00002 0.02109 -1.05756,

100 EDF 0.01035 0.00002 0.02463 -0.33196 'KDE 0.00907 0.00003 0.01532 -0.80926

Table 3: 0 N(0.1), 3PL

s

I I I 1* I I II I



RMSave SD

DeviationMAX LOC

10 EDF 0.09665 0.00004 0.22175 -0.74996KDE 0.08412 0.00004 0 13431 -1.21956

30 0.05695 0.00004 0.11573 -0.67436KDE 0.05439 0.00004 0.08258 -0.89616

60 EDF 0.01835 0.00002 0.04188 -0.70396,

KDE 0.01645 0.00M3 0.02802 -1.10236100 EDF 0.01823 0.00003 0.03782

.

-0.49826

, KDE 0.01767 0.00004 0.02668 -0.79636

Table 4: e N(0,1), Estimated

From Tables 2, 3 and 4, it is clear that smoothing in the KDE is helping, especially with

short tests. In comparing Tables 2 and 3 it is clear that the presence of the nonzero lower

asymptote c is degrading the fits. This can be seen both in the reduced RMS values and '4n.the movement of LOC, the location of the maximum deviation between Fm and F, toward

negative values. Finally, comparison of Tables 3 and 4 indicates th using 'noir" ICC'ssomewhat degrades the recovery of F.

Figure 1 illustrates the performance of the estimators in Table F. The first three panels

are probability-probability (p-p) plots of the estimated 8 distributior ....ctical axis) apinst

the true 8 distribution (horizontal axis), for 10, 30 and 60 items. Each panel depicts etteof the 100 Monte Carlo trials for the corresponding line of Table 3 Tht rte-.1 functionsrepresent the EDF estimator and the smooth curve represents the KDE eswaator. Thecloser each is to the solid diagonal line, the better the true probabilitivs the e disSutionare estimated. In particular for 30 or 60 items, estimated probabilities are (;:.'.te close to

probabilities. The story is very similar for the performance of the estimators it. Tables 2, 5

and 6 (see also Figure 3). The fourth panel in Figure 1 compares the density derived from

the KDE estimator in panel three to with the true 49 density (some excessive '4umpinc......

the estimated density is attributable to the fact that the "window width" h v413 chosen to

19


Theta - Normal, 3PL.10 Items Theta - Normal. 3PL, 30 terns

0 0 42 0.4 0

Tao 011440uson

08

Theta - Nomad, 3PL, 60 Items

to

0 0 02 04 0 $

TM" Thee 01011141$0/1

I la

The These 010114004

Theta - Netted, 3PL. 60 kerne

0

rifts

a

Figure 1: p p and density plots of EDF and KDE estimators. EDF is represented by stepfunction, KDE by curve. In the last panel. the true density is the dashed curve and theKDE-based density estimate is the solid curve.


make a good distribution estimate rather than to make a good density estimate).

Pula - Normal, Estimated, 30 lama

02 0 4 0 6

Too Thset Diertmon

t .0

Theta - Normal, Estimated, 30 Items

4 2

Figure 2: g p and density plots of EDF and KDE estimators. EDF is represented by stepfunction, KDE by curve. In the second wenel, the true density is the dashed curve and theKDE-based density estimate is the solid curve.

Figure 2 illustrates the performance of the estimators in Table 4. The left panel is ap p plot of the EDF (step function) and KDE (smooth curve) estimators for 30 items, and

the right panel compares the corresponding KDE-based density with the true e density. In

the Monte Carlo trial illustrated, contamination in the parameters of the "recovery' ICC's

caused some bias and scale distortion in the estimated distribution, but the estimate stillcorrectly suggests that 0 has a Normal or bell-shaped distribution.

In Tables 5, 6 and 7, in which e is bimodal, the KDE estimator is still doing betterthan the EDF. It is encouraging to see that the orders of magnitudes of the RMS and MAX

measures of fit are the same as in the N(0, 1) case above. It is a little surprising that thefits can actually be better for the bimodal cases than the normal, but perhaps the greater

variability is working in our favor here: we are getting mom, extreme-ability examinees with

which to form F, and thus to estimate the tails of F. Finally, note that there is much less

21


difference in the fits of the 3PL and 'Estimated' 3PL cases.


RMSave SD

DeviationMAX LOC

10 EDF 0.04769 0.00003 Th.12379 -1.36996KDE 0.03678 0.03003 0.06299 -1.25226

30 EDF 0.01820 0.00003 0.04668 -0.61856KDE 0.01547 0.00003 3.02502 -0.42646

60 EDF 0.01107 0.00003 0.02710 -0.25206KDE 0.00995 0.00003

40.01622 -0.17576

,

100 EDF 0.00870 0.00003 0.01923 -0.03886KDE 0.00817 0.00003 0.01290 -0.13216

Table 5: e Bimodal, Rasch


RMSare SD

DeviationMAX LOC

10 EDF 0.05268 0.00003 0.12160 1.08084KDE 0.03612 0.00003 0.09342 -4.44996

,,

30.

EDF 0.02268 0.00002 0.05616 -0.66696KDE 0.01877 0.00W2 0.04229 -3.68386

60 EDF 0.01353 0.00003 0.03496 -1.24996KDE 0.01205 0.00003 0.02561 -2.75386

100 EDF 0.00998 0.00003 0.02457 -1.22086KDE 0.00924 0.00003 0.01860 -2.64946

Table 6: ê Bimodal, 3PL

Figure 3 illustrates the performance of the estimators in Table 6, for 60 items. Again.

the left panel is a p- p plot of the EDF (step function) and KDE (smooth curve) estimators

and the right panel depicts the KDE-based density estimate. Once again the estimated

distribution provides good estimates of probabilities under the true distribution, and the

corresponding density estimate tracks the two modes of the true e distribution reasonably

well.

22



RMSave SD

DeviationMAX LOC

10 EDF 0.Mi ifit 0.00005 0.14624 0.78714KDE 0.05101 0.00005 0.09497 -4.97589

30 EDF 0.03203 0.111005 0.08038 -2.37405KDE 0.02958 0.00005 0.06457 -3.38695EDF 0.01386 0.00003 0.03747 -1.11546KDE 0.01245 0.00003 0.02796 -2.63776

100 EDF 0.01120 0.00004 0.02776 -142786KDE 0.01055 0.00004 0.02134 -2.29616

Table 7: 9 m. Bimodal, Estimated

rola - Bimactai. 3PL. 60 hems mak - Bimodat. 3PL 60 Rims

TRIO MO DelftAMOR Theft

Figure 3; p - p and density plots of EDF and KDE estimators. EDF is represented by stepfunction. KDE by curve. In the second panel, the true density is the dashed curve and theKDE-based density estimate is the solid curve.

110

Junker Recovering the Ability Distribution20

In Tables 8, 9 and 10, note how gradual the decrease in MAX is; this can be attributal

partly to the fact that e "doesn't know" that F assigns no mass to the interval (-oo, -1)

and thus freely places j's there. so that Feat is grossly overestimatingF for 0 < -I. 'This

certainly explains why LOC is near -1. in a but one case. It seems remarkable that the

RMS should drop as muc4: as it does, considering the fact that the Normal weightingfunction

g assigns significant weight to the region near or below 8 = -1. Once againthem is little

difference between the 3PL and 'Estimated' 3PL eases. Finally, note that the EDF estimator

is doing better than the KDE estimator in many cases here. Our ad hocchoice of h is

probably failing u.s here by being too large to track the "sharp upturn' in F at -1.


R.MS

ave SDDeviation

MAX LOC

10 EDFKDE

0.099220.09241

0.000040.00003

,

0.233520.20600

-0.26996-1.00996

30 EDFKDE

0.054040.05508

0.000030.00113

0.146080.17924

-0.91796-1.00996

60 EDFKDE

0.038120.04010

0.000030.00003

0.159930.16010

-1.00996-1.00316

100 EDFKDE

0.029440.03215

0.000030.00003

0.152460.14717

-0.99996-0.99996

Table 8: e X.2 1, Rasch

5 Discussion

To implement this scheme in practice, one must numerically invert the average ICCP3 for

the test in question at or near the J4-1 possible values of T.T. After this, a table constructed

from the inversion can be used simply and cheaply to estimate 0 distributions for each

of several adrninistrations of the same test, or each of several subpopulations in asingle

administration. For shorter tests lengths the basic statistic ij may need to be resealed,

I I

Janke:: Recovering the Ability Distribution 21

Test RMS DeviationLength Estimator ave SD MAX LOC

10 EDF 0.11871 0.00004 0.30689 -1.00996KDE 0.10699 0.00004 0.28934 -1.00996

30 E1M 0.22700 -1.00996KDE 0.071 0.00004 0.23167 -1.00996

60 EDF 0.05291 0.00003 0.20477 -1.00996KDE 0.05408 0.00003 0.20211 -1.0M96

100 EDF 0.04153 0.00003 0.19628 -0.99996KDE 0.04365 0.00003 0.18294 -1.00976

Table 9: 0 X3 1, 3PL

Test RMS DeviationLength Estimator ave SD MAX LOC

10 EDF 0.11387 0.00005 0.30689 -1.00996KDE 0.10600 0.00005 0.33073 -1.00996

30 EDF 0.08264 0.00005 1 0.32359 -1.00996KDE 0.08161 0.00005 0.30244 -1.00996

60 EDF 0.05322 0.00003 0.20477 -1.00996KDE 0.05466 0.00004 0.21590 -1.00996

100 EDF ' 0.04303 0.00004* 0.20150 -1.00996KDE 0.04491 0.00004 0.20859 -1.00646

Table 10: 0 x2 - 1, Estimated

25

Junior: Recovering the A bility Distribution 22

as we have done with i()), to effectively estimate F. Kernel smoothing of the estimated

distribution (KDE) is also quite helpful. Work is currently underway (Nandakumar and

Junker, 1992) to further examine and refme these methods using essentially unidimensional

simulation data, and to app!y the estiinistors to real tests.

Because it is fest, this scheme could be a.lso be used for some diagnostic purposes. For

example, if ICC's were estimated under the assumption of a Normal underlying 0 distribution

and a 3Pli model, the KDE estimate of the 0 distribution could be plotted on a Normal

probability plot to examine jointly) the assumptions about distribution and ICC forms. Or

the 9 distribution estimates under two ICC estimation techniques could be compared to see

how well they agree: Quite different ICC forms or parameter sets could in principle lead

to very similar ei distributions; if so then for many purposes it would then be a matter of

indifference which ICC's were used, so considerations such as cost of ICC estimation, etc.,

could come into play. Finally, it may be possible to estimate the 0 distribution sufficiently

accurately with, say, Rasch ICC's (for which item parameters can be estimated independently

of the 0 distribution), and then use that estimate as part of a marginal xnaximum likelihood

approach to estimating item parameters in a 3PL model which more accurately models the

item response behavior.

References

Ash, R. (1972). Real Avalys s and Probability. New York: Academic Press.

Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee's

ability, in Lord, F. and Novick, M. (1968). Statistical Theories of Mental Test Scores.

Reading, Mass: Addison-Wesley.

Hambleton, R. K. (1989). Principles and selected applications of item response theory. In

Linn R.L. (ed.) Educational Measurment, Third Edition. New York: MacMillan, pp.

0 6


147-200.

Hoeffding, W. (1963). Probability inequalities for sums of bounded random variables. Jour-

nal of the Americ- Statistical Association, 58, 13-30.

Janker, B. W. (1991). Essential independence and likelihood-based ability estimation for

polytomous items. Psychometrika, 56,255-278.

Levine, M. (1984). An introduction to multilinear formula score theory. University of Illinois

and Office of Naval Research, Research Report 84-4.

Lord, F. (1953). The relation of test score to the trait underlying the test. Educational and

Psychological Measurement, 13, 517-549.

Mislevy, R. (1984). Estimating latent distributions. Psychometrika, 49, 359-381.

Nandakumar, R. and Junker. B. W. (1992). Estimating the latent ability distribution. To

be presented at the Annual Meeting of the Psychometric Society, Columbus OH, July

1992.

Rubinstein, R. Y. (1981). Simulation and the Monte Carlo Method Method. New York: John

Wiley and Sons.

Samejima., F. (1984). Plausatility functions of Iowa Vocabulary Test items estimated by the

simple sum procedure of the conditional P.D.F. approach. University of Tennessee and

Office of Naval Research. Research Report 84-1.

Samejima, F. and Livingston. P. (1979). Method of moments as the least squares solutior for

fitting a polynomial. l`niversity of Tennessee and Office of Naval Research, Rearch

Report 79-2.

Serfling, R. (1980). Appronmation Theorems in Mathematical Statistics. New York: John

Wiley and Sons.


Silverman, B. (1986). Density Estimation for Statistics and Data Analysis London: Oxford

University Press.

Stout, W. F. (1987). A nonpararnetric approach for assessing latent trait unidimensionaiity.

Psychometrika, 52, 589-617.

Stout, W. F. (1990). A new item response theory modeling approach with applications to

undimensionality assessment and ability estimation. Psychometrika, 55, 293-325.

Appendix A. Details of the simulation

For each simulated data set, M Monte Carlo trials were run (one trial entails sampling N examinees,

generating a 10 and J item responses for each examinee, and constructing the distribution estimates

PNJ and PNJA from these). In our simulation, Al was taken to be 100. In the discussion below,

Fe,t stands for eittor of the two distribution estimates tried.

For each triot two measures of fit to the true ability distribution F were reported. First, the

value of

= maxiiFest(ti) tot ...,t1200}

was calculated, for ti's ranging from 6 to 6 spaced at 0.01 intervals, as an approximation to

5 supilF,(t) )1;t E (--cV,00)}

as well as the value L = t at which 3 was attained. Second, an approximation to the squared

distance11co

,0[11,4(0 - F(t)12g(t)dt

was calculated. where the weight function g was taken to be the standard normal density. The

approximation used was the Monte Carlo approximation

:2 It'.- [Fest(To F(T012,

where are iid with marginal density g, and K = 500 for our simulation.


Finally, Monte Carlo sample averages

1 M1 M3 = E .LIELm, and r2 = E4g4 naini

were computed, as well as sample standard deviatiolis. 3 estimates EgbI estimates EN, and

I estimates fEti3J}h12 standard deviation for 7 was estimated using the delta method (Serfiing,

1980, p. 118).

Ef371 may be regarded as a reasonable approximation to MAX = EV]. Because of the dis-

cretization in calculating and L, Ern probably is not as good an indication of Ile true value

LOC = t where the distributions are firthest apart, but it may still be of some descriptive value.

Finally, {Er/211112 is exactly

RMS = E 1 1F.,t(t) F(t)rg(t)dt}1/2

The pseudo-random number generators used were linear congruential generatcra (see Rubin-

stein, 1981)

= (a c) mod rn,

using a = 75,c = 0, m = 231 for generating O's and a = 27 + 1, c = 1, m = 235 for generating

item responses. Normal observations were obtained from these unifolm observations by the polar

transformation

= y'-2log 111 cos 22rU2

Z2 = /-2iogY1siu2rU2

and the bimodal mixture and l


Appendix B ProofsProof of Theorem. 1: Observe that, for any c > 0,

PliftN,7(e) F(e)1 El P [IPN,J(49) Fj(e)i IF-7(e) ges)i

P [ItN,j(e) FJ(.3)1 c/2] (for large

< C e-2N(40?

for some universal constant C, and N large. (Serfiing, 1980, p. 59). This tends to zero as N oo.

0

Proof of Theorem 2: Observe that

KV (ri) 5_ 8] = Pr' .7 .S71.7(9)]

= PIPP(X.r) 5. T311149)]

= (Xi) + r(9) 75-7N.7(e) r(9)1.

By Slutsky's Theorma, since r(6) = liriij 757,1171(9) we know that 75(17j)-f-r(8) and 7571(i.r)

have the same asymptotic law, i.e. for any t.

Pli2;1(rj) r(8)- /77%(0) < ti F(i).

Then in particular for t =

FT;10-C./ + r(9) j(e)L(8) 5_ r(9)] Pir(8)).

The assertion about uniform convergence follows from a theorem of Polya 1080, .18).

Proof of Theorem 3: In the following calculation, it will be helpful to let Y be a random variable

with distribution K independent of 0 and all item responses. Squaring (6),

00RMS2 = E f iFNJatt) 110129(0dt

E E NT,2=0

2

t ( ).1)K pie til g(t)dt

Junker Recover* the Ability Distribution 27

00 r

=e- [bic1.94t )12 + [varianc(t)1} g(t)dt

{P1v[X* = K t -75771(.1Er.7=0 h

var PNEXJ = j/JVC-co

2

pfe tl} g(t)dt

ft-T;11/1h

= r {PIP-31 (X. j) + lay t] pie t)}2 g(t)dt

I {t 75.71(X,r)i}g(t)dtN n=i

hY < ti PIO < tn3g(t)dt

+ f VaxK[tN g(t)dt

1

(bias)jh (variance)N.m.

Note that (bias)2NA does not depend on N. As long as

EIY1 = julK(u)dti < oo,

we will have hY 0 in probability, so that by Slutsky's Theorem the distributions of 15,71(174 +hit

and P.71(Xt) will converge to the same thing, namely F(t) = P[O < tl, at every t (we are assuming

F is continuous) as J -4 oo and h and h -4 0. Hence the integrand of (bias)2Njh converges to

zero at each t, and if g(t) is a density it follows that (bins)tuk 0 as J -+ oo and h -* 0 (and Nis free).

On the other hand, for each fixed I, h, t the random variable

ft -75;1(11

is bounded between 0 and 1. hence if g(t) is a density we have for each fixed J and h

g(t)dt

K[t 1-7-'.11(X.r)}g(t)dt < 1.

Multiplying by 1/N it is clear that variance)NA 0 as N oo uniformly in J and h. Thisproves Theorem 3. 0

31

Of. Terry AdiermanEducadonat Psychology210 emu= Bidg.Universny of liftnoisChampaign. & 61801

Dr. Ronaid ArmstrongRutgers UniversityGraduate Schooi c4 ManagementNewark. NJ 07102

Dr. WUlism M. BartUnivensny of MinnesutaDeo. of Educ. Psychology330 Burton Han178 Pillsbury Or., S.E.Minneapolis, MN 55455

Dr. BM= fibromDefense Manpower Data Center99 Pacific St.

Suite 155AMtunerey, CA 93943-3231

Dr. Robert BiennenAmerman College Testing

ProgramsP. 0. Box 168lows City, IA 52243

Dr. John M. CarrollIBM Watson Research CenterUser interface instituteP.O. Box 704Yorktown Heights. NY 10598

Mr. Hua Hue ChungUniversity of WinoisDepartment of Statistics101 Mini Hat725 South Wright St.Champaign, IL 61820

Dr. Stanley ColtverOffice of Naval TechnologyCode 222800 N. Quincy StreetArlington. VA 22217-5000

Dr. Tummy DaveyAmencan Collage Testing ProgramP.O. Box 168Iowa City. IA 5224.3

Dr. Lou MelloCERLUniversity of Illinois103 South Mathews AvenueUrbana, IL 61801

Dr. Fritz DrasgowUniversny of IllinoisDepartment of Psycnotogy803 E. Daniel StChampaign. IL 61820

Dr. James Akins1403 Norms HalUniversity or FloridaGairesvide. FL 32805

Dr. Eva L BakerUCLA Canter kr the Study

of Evaluation145 Moore HallUniversity of CaliterniaLos Angeles. CA 90024

Dr. Isaac 8k4erLaw School Admiaskins

ServicesP.O. Brat 40Newtovnt, PA 18940-0040

Cdt. Arnoid BoberSectie Piwohologisob OndeizositRelouteengeEn 13441COMOOMUMKwarder Koningen AstridSruikettast1120 Brussels. BELGIUM

Dr. Gregory Genoa07841aGinwr4V12500 Garden ReadMonterey. CA 93940

Dr. Robert M. CanoeChief of Navel Opera:kmOP-01112Washington, DC 20350

Dr.. Norman CatDePtiftMere at PerflabgyUniv. of So. CdtorniaLos Angeles. CA 9008940131

Dr. Haas F. CrombegFaculty of LawUniverse! of LimburgP.O. Box 616MatastncMThe NETHERLANDS 62U0 MD

Dr. C. M. DayamDepartment of Measurement

Statistics & EvnitiatronCollege al EducationUnivorsay of MarylandCollege Perk, MD 20742

Dr. Dattpmad DivgiCenter tor Navai Analysts4401 Ford AvenueP.O. Sax 16288Alexanoria, VA 22302-0268

Defense TechnicalInformation Center

Camemn Station. Bldg 5Alexandria, VA 22314

RR

32

Dr. Eding B. AndersenDepammint al Steaks-Studeatizeide 61455 OapanhagenDEYSAMVC

Dr. Lava L BainesCrane of EducationUnhavaiW of Toledo2801 W. Sanaoft SkeetToledo, OH 43606

Dr. Menucha BirenhaumSchool of EducationTel Aviv UniversityRamat Aviv 69978mama

Dr. Robe! BreauxCode 01Naval Training Systems CenterOdando, FL 328264224

Dr. Joint S. Carroll449 Mott Rd, NorthChapel NM NC 27514

Dr. Raymond E. ChdatalLES LAMP Science AdvisorAFFNACELBrooke AFB. TX 78235

Maar.Manpower SuMort erldReadiness ProgramCorder for Navel Analysis4401 Ford AvenueP.O. Box 111268Alexandria. VA 223020268Ms. Carolyn a CM,Johns Hearin* UniversityDeranntern of PsychniogyCharles & 34th StreetBaltimore. MD 21218

Dr. Ralph J. DeAyalaMeasureromt. Statistios.

and EvaluationBenismin Bldg.. Rrn. 4112University of MarylandCollege Park, MD 20742

Mr. Hei-Ki DungBell Communications RosenfeltRoam PYA-1K207P.O. Box 1320Place:away. NJ 08355-1320

Dr. Stephen Dunbar224E1 Linder* COOK

*or MeasurementUniversity of Iowaboa Cit14 1A 52242

Dr. James A. Ear lasAir Force Human Ree0Lifeal LabBrooks AFL TX 78235

ERIC Facility-Accursitions2440 Resew= Blvd. Suite 550Rockville. MD 208504238

Dr. P-A. Fed.=Code 51NPRDCSan Diego, CA 92152-6900

Dr. Gerhard FocherUshiggasse 513A 1010 View*AUSTRIA

Mr. Paul FoleyNavy Personnel R&D CenterSan Diego. CA 92152-6800

Dr. Janice GifordUniversity of MeeeecnUsetteSchool of EducationAmherst, MA 01003

Dr. Sheme GottAFHRUMOMJBrooks AFB. TX 78235-5601

Prof. Edward HaertelSchool of EducationStanford UniversityStantort, CA 94305

Or. Grant HenningSenior Research SciermstDivision of MeasurementResearch arxi ServrcesEducational Testmg ServicePrinceton, NJ 08541

Dr. Paul W. HollandEducadonal Testing Service, 21-1Rosedale RoadPrinceton, NJ 08541

Dr. William HowellChief SCiertestAFHRUCABrooks AFB. TX 78235-5601

Dr. Susan EmbretsonUniversity at KansasPsychabgy Depanment426 FraserLawrence. KS NOS

Dr. Benjamin A. FabbanitOperationalTechnoiogiss Coip.5825 Callaghan. Sults 225San Antonio, IX 78228

Dr. Leonard Feb*Undquit Canter

for MeasurementUniversity of benIrma Cky, IA 52342

Dr. Myron Fisch'U.S. Army HeadquartemDAPE-MFFIThe PentagonWeshin91081, OC 20310-0300

Dr. Alfred R. Fre*AFOSRML Sidg. 410Baku; AFB, DC 203324448

Dr. Drew GamerEducatxmal Testing ServicePrim:won, NJ 08541

E 3ert GreenJoists Hopkins UnivensityDepartmere of Psych:lbwCharles & 3481SheetBaltimore. tiff3 21218

Dr. Ronald K. HarntaacetUniversity of MassachusettsLaboratory of Psychometric

and Evaluative ResearchHills South, Room 152Amherst, MA 01003

Ms. Rebecca HatterNavy Personnel R&D CenterCode 63San Dis9o, CA 92152-8800

Dr. Paul Horst6776 Street, It184Chula Vista. CA 92010

Or. Uoyd Hun:WaysUniversity of Sinai*Department of Psychobgy603 East Daniel StreetChampaign, L 61820

Dr. Growl Englehred. Jr.Mid= Educeatmei StudiesEmory tJolvansay210 figibuntsMints. GA 3011:2

Dr. Mantua J. Fern ConestogaCognitive & Insatictional Sciences2;9111 *nth Vernon StreetAilkoost, VA 22207

Dr. Maid L FergusonAmami College TeasingP.O. Box 168Iowa My, IA 52243

Pmt.Donsid RzgrealdUniveney of New EnglandDepsionent of Psyd10k4WMukluks. New Sauer Wales 2351AUS1RAUA

Et. Robert D. Gibbonsnooks State Psychiatric Inst.Rie 528W1801 W. Tayfor SheetMown IL 60612

Dr. Robert GlowLearning Research

& Chweitoment CenterUnlveraiW of PlItsbeigh3938 Mans StreetPlasittagh, PA 152110

Wheel HobanDORMER GMBHP.O. Box 14200-7990 Frisdrichaluden 1WEST GERMANY

Or- Datal/fn HarnischUnhorse! of INksais51 Getty DWGCharaPte0n, IL 61820

Dr. Thomas M. HirschACTP. 0. Box 168tows City, IA 52243

Me. Julia S. HoughCambridge Universrty Press40 West 20th StreetNew York, NY 10011

Dr. Steven Hunks3-104 Edw. N.Universrty of MartaEdniontork MantaCANADA TSG 2135

Dr. Huy*, HuynhCollege *I EducationUniv. of South CarotinaColumbia, SC 29208

Dr. Douglas H. Jones1280 Woodfem CourtToms River, NJ 08753

Dr. Milton S. KatzEuropean Science CoonlMatim

OfficeU.S. Army Research InstituteBox 65FPO New York 09510-1500

Dr. Soon-Hoon KimKedi924 Umyean-DangSeocrio-GiuSeoulSOUTH KOREA

Dr. Richerd J. KoubekDepanment of Biomedical

& Human Factors.139 embalming & Math Bldg.Might State UnivemkyDayton, OH 45435

Dr. Thomas LeonardUniversity of WlsomeinDemme& of Statistics12t0 West Dayton StreetMadison, WI 63705

Mr. Rodney LimUniversity of It boleDeosnment of Psychology603 E. Daniel St.

ChamPal9n- 11- 81820

Dr. Frederic M. LordEducational Testes; SetvicePrinceton, NJ 08541

Dr. Gary MarcoStop 31-EEducational Testing ServicePrinceton. NJ 08451

Dr. Clarence C. McCormickHO. USMEPCOMNIEPCT2500 Green Bay RoadNorth Chicago, IL 60054

Mr. Alan Meadcro Dr. Michael LAVInilEducational Psychology210 Education Bldg.University of IllinoisChicaao IL 61801

Imp! I 111 1111 I I I

Dr. Robed JannmoneElms end Computer Eng. DeptUnivemey of South Cambria001UMbilt. SC 29208

Dr. Brien JunkieCarnegie-WWII UniversityDernstment of StatisticsSchenker ParkPittsburgh, PA 15213

Prof. Jolm k KeatsDepestmina of PsychokigyUrftimity of NewocialeN.S.W. 2306AUS1RALIA

Dr. G. Gage KingsburyPortland Public &hornsReseerch and Evaluation Deparunent501 North Dixon StreetP. O. Box 3107Portland, OR 97209-2107

Dr. Leonard *eskerNavy Personnel R&D Center

Code 62San Diego, CA92152-6800

Dr. Michael LevineEducational Psychology210 Educelkin t509.tinker*, of WO*Champaign, L 81801

Dr. Robed L Urn,Campus Box 249Urkvereity of ColoradoBoulder, CO 80309-0249

Dr. Richard LuechtACTP. 0. Box 168Iowa City, IA 52243

Dr. Giessen J. ManinOffice of Chief of Naval

°emitters (OP 13 F)Navy Amex, Room 2=Washington. DC 20350

Mr. Chritemher McCuskerUniveisity of SkimsDeportment of Psychology603 E. Daniel St.Champaign, IL 61820

Dr. Timothy MillerACTP. O. Box 168bwa City, IA52243 3 4

Dr. Kumar Jorig-devUniversity of Wm*Deomment of Statratks101 Sini Hall725 South Wright SkeetChampaign, IL 61820

OT. Wheat KaplanOnce of Basic ResearchU.S. Army Reseankit buena.5031 Eisonhower AvenueAlexancida, VA 22333.=

Dr. Jere-ram KimDepartment of Psychology%Who Tennessee Stat.

UntinimityP.O. Box 522Murfreesboro. TN 37132

Dr. Willarn KochBox 7246. Mese. end Eval. Cir.University of Texae-AustinAustri, IX 78703

Dr. Jeny LehnueDeters* Manpavrer Data CenterSuite 4001600 Mem BlvdRoteiyo VA 22209

Dr. Chides LewisEducational Testing ServicePrinceton, NJ 08541-0001

Dr. Robert LockmanCenter to, Naval Analysis4401 Ford AvenueP.O. Box 18268Alexandria. VA 22302-0268

Dr. George B. MacreadyDsoartment of Measurement

Statistics & EvidustbnCollege of EducationUniversity of MarylandCollege Park. 11) 20742

Dr. James R. McBridefluirdiF06400 Elmhurst DriveSan Diego GA 92120

Dr. Robert McKkileyEducational Testing ServicePrinceton. NJ 08541

Dr. Robert MielevyEducationte Testing ServicePrinceton. NJ 08541

Dr. William thxdagueNPRDC Code 13San Diego, CA 92152-6800

Dr. Rama NandakumarEducational StudiesWi lard Hat Roam 213EUniversity of DelawareNewark. DE 19716

Dr. Hewitt F. (mg. Jr.&hind at Education - WPH 801Department of Educe timer

Psychology & TechnologyUniversity of Southern CaWomieLos Angeles. CA 90089-0031

Dr. Judith OrasenuBasic Research Oft*Anny Research Menke.5001 Eisenhower AvenueAlexandria, VA 22333

Wayne M. PatkinceAmerican Council on Educed=GED Testing Sarital. Suite 20One Dupont Mole. NWWashington, DC 20036

Dr. Mark D. ReckaseACTP. O. Sox 168Iowa City. iA 52243

Dr. Cad RossCNET-PDCDBuilding 90Great Lakes NTC. IL 60088

Mr. Drew SandsNPROC Code 62San Diecye, CA 92152-6800

Dr. Dan SegallNavy Personnel R&D CenterSan Diego, CA 92152

Dr. Randall ShumakerNaval Research LaboratoryCode 55104555 Overlook Avenue, S.W.Washington. DC 20375-5000

Dr. Judy SprayACTP.O. Box 168Iowa City. IA 52243

Ms. Kathleen MorenoNeel Personnel R&D CenterCode 62Sass Diego, CA 921524800

Lbrary, NPRDCCode P2011-San Diego, CA 921524000

Dr. James B. OlsenWICAT Systems1875 Sarah State StreetOrem, UT 84058

Dr. Jesse Orienekykistitute for Odense AnalysesMI ttBI.grdSt.Alesmstirts, VA 22311

Dr. James PaubwrDepanment of PsychtdogyPortland Slate UniversityP.O. Box 751Porthmd. OR 97207

Dr. Malcolm ResAFFIRLAIOASmoke AFB. TX 78235

Dr. J. RyanDeperummt of EducationUrtitfOrlfit f Of South CarolinaColundsis, SC 29208

Lowed &hoerPayarsoiogicai & Ouamitative

FoundationsCollege of EducalkmUniverse), of IowaIowa City. IA 52242

Dr. Robin ShealyUnlveray of WinoisDepartment of Statkitioa101 Mini Hail725 South Wrigiri St.ChamPaillh. a. 61820

Dr. Richard E. SnowSchooi of EducationStanford UniversityStanford, CA 94305

Dr. Martha StacidngEducationai Testing ServicePrincerces, NJ 08541 35

Headwaiters Marino CorpsCode 10140Washkeyon. DC 20380

LihntrienNamd CeMer for Applied Resew=

in Atietshd IntelligenceNaiad Resew* tAboratoryCods 5510Washington, DC 203754000

0151* of Nand Reessoli,Cade 1142C8

800 N. Ouincy StreetArlington, VA 222174000

Dr. Peter J. PeshleyEduistionai Testing ServiceRosedale RoadPrinteson. NJ 08541

Dept. of Mtn' SciencesCode 54

Nat* Postumduate SchoolMonterey. CA 939434026

Me. Stove ReissMC aka HalUtdversity of kgretarsola75 E. War ReedMinneepods, WI 554554344

Dr. Funs** SamerimaDePertment of PsychologyUniversity at Tennessee3108 Austin Petty Bldg.Knoxvdie, TN 37916-0900

Dr. Mary Schram4100 ParkeideCarlsbad. CA 92008

Dr. Kazuo Shigemasu7-944 Krigenume-KeiganFulsaws 251JAPAN

Dr. Richard C. SorensenNavy Personnel R&D CenterSan Disgo. CA 921524800

Dr. Peter StabffCenter tor Naval Analysis4401 Ford AmmoP.O. Box 16268Aiseendrks. VA 223024268

Dr. %aim StoutUniversity of Nino lo[imosa of Statistics101 ISM Nal725 South Wright St.Champaign, 4. 61320

Dr. John TangneyAFOSRiNL. Bldg. 410Boling AFB. DC 203324448

Dr. David ThomDepanment of PsychoiogyUniversity of KansasLawrence, KS 66044

Dr. Robert Taut:wawaUnivinvey of MissouriDapartmant of *Metes222 Math. Sciences Bldg.Columbia, MO 55211

Dr. Frank L MoineNavy Personnel R&D CenterSan Diego. CA 92152-6800

Dr. ating-filei WogEducational Tsang ServiceMal Stop 0 -TPrinoston. NJ 09541

Dr. David J. WeiseN660 Menai'Unkrersity of Minnesota75 E. River RoadMinneapolis. Mt4 55455-0344

Dr. Douglas Wets&Cods 51Navy Personnel R&D CanterSan Diego, CA 92152-6800

Dr. Brum WillamsDepartment of Educational

PsychologyUnkersity ot IBMsUrbana, IL 81801

Dr. George On131' LabersioryMemorial Sloan-Kettering

Came Comer1275 York AromaNew York, NY 10021

Dr. Wendy YenCTB/McGrawDel Monte Rosearth ParkMoreeny, CA 93940

Dr. Hems= SwarnatimanLationatil of Psychomeno and

E R esearchSchool of EducationUnimetty of MassectumettaAmherst, MA 01003

Dr. leant TampicoEducational Teetkig ServiceMal Stop 03-TPrinceton. NJ 08541

Mr. Thomas J. ThomasJohns Napkins UntyingDepartment ri PsychologyCharles & 34th &maBaltimore., MD 21218

Dr. Legard TuckerUnwiring of OwlsDepanmel of Psychology603 E. Daniel StreetChampaign, L 61820

Dr. Howard WeinerEducational Testing ServicePrinceton. NJ 08541

Dr. Thomas A. WannFAA Academy AAC9341)P.O.. Boa 23062Cidahoma City, OK 73125

Dr. Ronald A. WeitzmanSox 148Carmel, CA 93921

Dr. Rand R. ManUniversity of Southem

CataniaDepanmere of PsychologyLos Angeles, CA 900804061

Dr. Ma WingFederal Aviation Administration800 kidependence Ave. SWWashington, DC 20591

Dr. %lace Wulfedr. UINavy Personnel R&D CenterCode 51San Digo, CA 921524800

Dr. Joaloh L YoungNationfid Sconce FoundationRoom 3201800 Smet. N.W.Washington, DC 20550

Mr. Brad SympsonNavy Pamennei R&D ComerCodb62Sao Diego. CA 9215241100

Dr. Mambo TamaraEdutistional Twang ServiceMad Stop 034Prinotion. NJ 09541

Mr. Gory ThomasonUniversity of WOOEd:Mond PaychoeigyChampaign. IL 611120

Dr. Dint ValeAstressonent Systems Cana-2233 University AvenueSults 440St. Paul, MN 55114

Dr. klcasel T. WailerUneremay 01 Wisconsin-MilwaukeeEdmond Partiotogy DepartmentBox 413Milwartiers, WI 5=01

Dr. Brian WatsmNumR1101100 $. WarthkeitonAismatiie, VA 22314

MelorJohn WelshAFFIRLAIDANBrooks AFB, TX 78223

Gomm Mary ReoreeentsaveAT1Tt lAbloarq Wildgnibe

StralkateasamtD-6300 Bonn 2

4000 Brandimins Street NWWashington. DC 20016

Mr. John IL WaftNoy Personnel R&D CenterSea Digo, CA 921524800

Dr. Kentaro Yamamoto02-TEckatational Tomtits; ServiesRosedale RoadPrinceton, Nif 03541

Mr. Anthony R. ZaraNational Council of State

Ikons of Nursing. Inc.625 Noah Midtipm AvenueSue 1644-ChIcaor). IL 60611

Documents

DOCUMENT RESUME TM 018 884 AUTHOR Junker, Brian W ...DOCUMENT RESUME ED 348 399 TM 018 884 AUTHOR Junker, Brian W. TITLE A Note on Recovering the Ability Distribution from Test Scores