This article was downloaded by: [University of Birmingham]On: 10 October 2014, At: 22:04Publisher: RoutledgeInforma Ltd Registered in England and Wales Registered Number: 1072954 Registered office: Mortimer House,37-41 Mortimer Street, London W1T 3JH, UK

Educational PsychologistPublication details, including instructions for authors and subscription information:http://www.tandfonline.com/loi/hedp20

Educational data analysis for policy decisionsMelvin R. Novick aa College of Education , The University of Iowa , Iowa City, IA, 52242Published online: 01 Oct 2009.

To cite this article: Melvin R. Novick (1977) Educational data analysis for policy decisions , Educational Psychologist, 12:2,138-145, DOI: 10.1080/00461527709529169

To link to this article: http://dx.doi.org/10.1080/00461527709529169

PLEASE SCROLL DOWN FOR ARTICLE

Taylor & Francis makes every effort to ensure the accuracy of all the information (the “Content”) contained in thepublications on our platform. However, Taylor & Francis, our agents, and our licensors make no representationsor warranties whatsoever as to the accuracy, completeness, or suitability for any purpose of the Content. Anyopinions and views expressed in this publication are the opinions and views of the authors, and are not theviews of or endorsed by Taylor & Francis. The accuracy of the Content should not be relied upon and should beindependently verified with primary sources of information. Taylor and Francis shall not be liable for any losses,actions, claims, proceedings, demands, costs, expenses, damages, and other liabilities whatsoever or howsoevercaused arising directly or indirectly in connection with, in relation to or arising out of the use of the Content.

This article may be used for research, teaching, and private study purposes. Any substantial or systematicreproduction, redistribution, reselling, loan, sub-licensing, systematic supply, or distribution in anyform to anyone is expressly forbidden. Terms & Conditions of access and use can be found at http://www.tandfonline.com/page/terms-and-conditions

Educational Psychologist, 1977 Vol. 12, No. 2, 138-145

EDUCATIONAL DATA ANALYSISFOR

POLICY DECISIONS1

MELVIN R. NOVICKThe University of Iowa

ABSTRACT

The field of educational psychology as an academic and professional discipline is oneof great breadth. While it focuses on the practicalities of the classroom and instructioncenter, it has its roots in the theoretical areas of learning and motivation theory andassorted other specialized areas of the behavioral sciences. It has as companiondisciplines the fields of educational measurement and statistics, with which it oftenshares a common departmental home within a college of education. It also has a greateror lesser affiliation with traditional departments of sociology, anthropology, andeconomics. In this paper, it is suggested that we need to draw more fully on theseindividual disciplines, and in a more integrated way, if we are going to come to gripswith some of the major policy questions that currently face American education. Theseissues are primarily problems in educational psychology, but ones that can be solvedonly through a multidisciplinary approach.

The past half decade has seen our educational system impacted by litigation. Suchlitigation typically concerns itself with decisions that have to be made on policyquestions: litigation concerning dress codes, suspension rules, admissions pro-cedures, busing, tenure, and the right of teachers to.bargain collectively—a rightwhich has benefited so many other professional and trade groups. It is indeedinteresting to note that many of these issues are now thought of as "policy" questions,subject to public accountability and litigation, where they might earlier have beenthought of as subject only to administrative discretion.

Policy issue, policy question, litigation, accountability: why this new vocabulary?Why is everything now a policy issue? Why is it becoming more and more necessaryto consider broad classes of decisions rather than individual decisions? The answer is,I think, that the American educational system is increasingly being forced to make itsdecisions in public, and to be accountable to the public for these decisions.Educational administrators have lost much of their ultimate authority, in the sensethat they now are subject to a degree of accountability that previously did not exist.Decisions now increasingly must be presented in an open book for all to see, to audit,and to challenge, and with increasing regularity these challenges are made not in theclassroom or the school meeting room, but in the courtroom. Administrators arebeing forced to answer the question "Why?" and to do so in open court.

The disturbing truth is that the question "Why?" is seldom answered very sensibly.To bus or not to bus, for example, is a much discussed issue. But I an unaware of anystatistically monitored, psychologically valid investigations that have confronted1 An invited address presented to Divisions 15 and S of the American Psychological Association,September 6, 1976. Research supported in part by NSF grant #EPP73-00164. The author's address isCollege of Education, The University of Iowa, Iowa City, IA 52242.

138

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

MELVIN R. NOVICK.

this question in any specific instance in such a way as to produce results. Rhetoricabounds, but data analysis and reanalysis is weak, and on my reading, mostinconclusive. Some seem to feel that busing is useful always; others, never. I am notsure, but I think the answer is sometimes, perhaps often, but I don't know when, andI haven't seen any data yet that begins to inform me on this issue. My thesis here isthat we must determine under what conditions busing is useful. I suggest thateducational psychologists as a group, with assistance from specialists in educationalmeasurement and statistics, economics, sociology, and anthropology can help toanswer such questions. Indeed some may say that they have a professional obligationto try.

There is always, however, the reticence of the scientist to come out of thelaboratory and into the field of public policy issues, and it is easy to excuse one'sreluctance to attack such big issues by claiming that the technology is not yet there,that we don't know the answers to all the questions, and that we are unprepared. Ihope to show you here that at least a statistical methodology does exist for puttingtogether solutions to such policy questions and that if educational psychologistschoose not to participate in such policy discussions they cannot justify theirreluctance by claiming that such methodologies do not exist.

To begin to understand how we might approach major policy issues such asbusing, let us consider a sequence of topics in which educational psychologists arenow able, with varying degrees of sophistication, to combine powerful methods ofdata analysis with psychological insights to provide rational, publicly auditable,decisions. 1 shall, of course, begin with a field in which application is relatively easy.

The field of individually prescribed instruction is one in which decision makinghas moved from the status of an educational folk art to that of a scientific endeavor.A half decade ago, Millman (1972) studied the operating characteristics of certaincommonly accepted decision rules and showed that they had many unsatisfactoryproperties. Hambleton and Novick (1973) first considered the problem from a fullydecision-theoretic point of view and showed how to combine probabilistic assess-ments with judgments as to potential gains and losses, so as to specify rationallydetermined cutting scores. The paradigm is a simple one—a student can be declared amaster and move to the next unit of instruction, or he can be declared a non-masterand sent back to begin the current unit again. The true state of a student is alsoconsidered dichotomous. He either is or is not a master. Thus the problem isconsidered in the context of the well known four-fold table, where one can writedown four numbers representing the losses that would be incurred for correct andincorrect decisions. Given prior experience with comparable students and the direct .observations on the particular student (the test score), the posterior odds that thestudent is a master can be calculated. A standard approach is to take a zero loss for acorrect decision, a loss, a, for a false positive, and a loss, b, fora false negative. It isthen easy to show that the setting of a cutting score depends on a comparison of theratio a/ b, the relative loss associated with the two kinds of incorrect decisions, withthe posterior odds that the student is a master. This is, of course, comparable to arational determination of the alpha and beta errors of classical statistics, byconsidering relative losses for the two kinds of decisions, but in this case prior orbackground information is also included in the data analysis. From this simple kindof analysis, cut scores can be obtained for any specified test length and specifiedminimum criterion mastery level, and indeed the question of test length can bestudied, as it was by Novick and Lewis (1974). The issue considered here is a policy

139

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

EDUCATIONAL POLICY DECISIONS

issue in a narrow sense in that one is setting down policy for deciding when to send astudent forward and when to hold him back. The determination of what constitutesmastery is not one without educational-psychological implications. Indeed it is notan issue on which I would expect to get agreement among educational psychologists.There are those who believe that minimum performance levels should be set veryhigh and that, therefore, cutting scores should be set very high. On this view it issupposed that any points not now mastered will be forever lost. Others might takequite a different view of the educational process, and believe that relatively lowmastery levels would suffice. In the belief that high mastery should not be required,they would argue that students will pick up what they missed now while studyingmore advanced materials. Only sufficient mastery should be demanded to permit astudent to consider new material intelligently. What is the answer? Which position iscorrect? I don't know, and am disturbed that I am unaware of a literature that hasbeen cited to justify the relatively high performance levels now demanded. Should wedemand such high levels? I think the answer is that it depends. Sometimes high levelsof mastery are required, other times, I think, they are not. And this can be determinedonly by a careful study that links performance level on one module with subsequentperformance on other modules. This kind of study can, and should, be done;typically it is not done. But I am pleased to say that I have recently considered somework which begins to at least develop the mathematics of a model for making thiskind of assessment. Perhaps the existence of such a model will influence educationalpsychologists to attempt this kind of analysis. The answer can come only from a bodyof theory and literature of experimental confirmation. Educational psychology,measurement, and statistics are all required.

Despite the relative simplicity and affective neutrality of this situation, we canbegin to see the basic elements of a policy issue evolving, and the proposal of asolution that depends upon a statistical methodology incorporating assessed beliefsconcerning probabilities, and losses or utilities. We can also see that these utilitiesmay reflect differences in conception based upon educational psychological fact ortheory, with implications for the educational process which may be subject to reviewby administrators, by parents, and possibly even by the courts. We have not yet, tomy knowledge, had a court case in which high cutting scores have been challengedand charged to be the cause of holding back disadvantaged students, but I would notrule out the possibility that such litigation might someday confront us. Should we befaced with such litigation, I would hope that a body of research would be available tosupport whatever policy decisions had been made.

Now let us move to an area of greater controversy—that of educational selection.Here we are confronted with a clear matter of educational policy and need toconsider the potential gains and losses of any policy decision that we contemplatemaking. We need to consider the certain fact that whatever policy we do adopt, it willbe challenged in the public arena. At issue is whether or not some students, becauseof minority or other identifiable status, should be given certain preference in educa-tional selection over some other students who do not have such status. Typically, theproblem is stated in terms of minority and majority groups where the term minorityis a synonym for the word Black. You may recall that in the case heard before theSupreme Court, DeFunis v. Odegaard (1974), the University of Washington wascharged with discriminatory action in providing preference for less qualified Blackstudents as against more qualified white students. Specifically, DeFunis charged thathe was excluded from law school because of a policy based on racial preference and

140

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

ME1.V1N R. NOVICK

that he therefore suffered discrimination. While the issue of DeFunis has not beencompletely resolved, the day of resolution seems to be approaching as the case ofBakke vs. the Regents of California (1976) heads toward decision in the CaliforniaSupreme Court with appeal to the United States Supreme Court highly likelywhatever the initial decision. In the related area of employment selection, there havebeen numerous cases in which there has been some attempt to determine when and towhat extent applicants might be treated differently because of characteristics otherthan those that relate to job performance.

Until recently the concept of group parity first introduced by Thorndike (1971)and later supported by Linn (1973) and particularly by Cole (1973) seemed to enjoydominant support. One year ago I delivered a paper co-authored with Dorsey D.Ellis, Jr. (Note 1), of the Law School of the University of Iowa, on the topic ofeducational and employment selection in which we outlined what we consider to be aconstitutionally and psychologically valid approach to educational and employmentselection. Our position was and is in opposition to group parity concepts on theground that racial or ethnic identification is irrelevant and that preference should begiven on the basis of individual disadvantage on the theory that government has anobligation to provide equality of opportunity irrespective of race or ethnicity. I judgethat a poll of the APA audience that heard that paper at that time would have givenus a vote of about 5% in support of our legal position. Recent court decisions,however, have been consistent with the constitutional position we adopted and welook for a definitive ruling in the near future. We were gratified to see that the newproposed EEOCC (Equal Employment Opportunity Coordinating Council) guide-lines for employment selection make no use of group parity/quota concepts.

The position taken in the Novick and Ellis (Note 1) paper was a reaffirmation ofone stated in two papers in the Journal of Educational Measurement. The Winter1976 issue of that journal consisted of a paper by my former student, Nancy Petersenand myself, replies by Lee Cronbach, Robert Linn, Nancy Cole, et al., and RichardDarlington, and a concluding piece by myself and Petersen. In the first paper weexamined the various group parity models posed by Thorndike, Linn, and Cole, andshowed them to be inconsistent with each other and with any usual form of statisticaldecision theory. We argued that only a formal decision theoretic approach to theproblem of culture-fair selection could begin to produce a coherent and viablesolution. In their replies, only Cole took exception to our arguments against groupparity, while Cronbach, Darlington, and Linn supported our position to a greater orlesser extent. And indeed Thorndike was quoted as specifically supporting thePetersen-Novick position:

I am in general agreement with the point of view taken in the paperthat there is no really meaningful working solution to the test biasproblem unless one takes into account the utility (positive or negative) ofthe various outcomes... basically, one's value system is deeply involvedin one's judgment as to what is "fair use" of any selection device.

Please note, I am not suggesting that the other contributors to the Journal ofEducational Measurement discussion specifically supported the threshold utilitymodel discussed by Novick and Petersen. Indeed they did not. But each approachedthe problem as a problem in decision theory including Cole, who attempted toreformulate group parity as a decision theory application. Thus there emerged animplied consensus that the approach should be decision theoretic. This I take to be aquantum gain on the previous approaches that attempted to provide an axiomaticsolution.

141

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

EDUCATIONAL POLICY DECISIONS

The problems of educational and employment selection and the desire to attainequal opportunity without regard to ethnic or racial affiliation presents the educa-tional psychologist with a very difficult problem. But we do not believe that thisproblem is unsolvable. The problem now as we see it, is one of measurement,specifically the measurement of disadvantage. A crude measurement would involvethe division of students into advantaged and disadvantaged groups, and this wouldlead to the kind of four-fold table analysis that we are familiar with when dealingwith racial and ethnic groups. Disadvantage, however, is not an all-or-none concept,it is a question of degree, and what we would seek is some scaled continuous measureof disadvantage which would be used as input to the selection process.

When we turn to the task of measuring disadvantage, we find that we do not startthis project without benefit of any prior investigation. Indeed we know a great dealabout disadvantage. We know that intellectual development is related in a significantway to birth order, number and spacing of children in the family, intellectual level ofthe family, and the intellectual level of the peer and school group. Current measuresof disadvantage used in scholarship and compensatory education programs also usefather's and mother's occupation and the availability of specific objects in thehome—books, study area, etc. We perhaps do not know just how important each ofthese variables is, nor how to scale the set variables. We do know however thatdisadvantage can accrue to a child for reasons far beyond family and peer groupintellectual level. We know very well that the ugly child is typically very highlydisadvantaged because of his treatment by his peers and by his teachers. I would alsoargue that the exceptionally attractive child or young person may also bedisadvantaged in the sense that he may receive too much attention and too muchreward, without any learning and performance. It is very likely that some, butcertainly not all beauty queens and football heroes have stunted developmentbecause of the over-attention that they receive because of qualities which may not bevaluable to them in later life. Is it not possible that we exploit some of our students inways that might not generally be thought of as exploitation or disadvantage?

There was a recent court case in which the parents of a child sued a school districtfor damages because their child, upon graduation from high school, was unable toread at the sixth grade level. The argument was that the school had a responsibility tosee to it that the child attained the level commensurate with his grade. That suitfailed. However, in other contexts such a suit might not necessarily fail. And it isconceivable and, it seems to me, at least logically if not legally sound, to argue that aschool district that gave a student high grades because he was a football hero mightbe morally and perhaps financially liable for the lack of education that studentsuffered. My point here is not to start us on a new wave of litigation, but simply to getus to begin to think about what disadvantage really is and to understand that it maynot, and probably does not, necessarily restrict itself to skin color. Indeed, it may wellbe that in many circumstances what we judge to be racial discrimination could nowbe more accurately described as a cultural discrimination which does not exist forBlacks who conform to middle or upper class standards. I do not suggest that this isat all desirable, but only that the issue of disadvantage and discrimination is muchmore complex than some would have us believe.

In the Novick and Petersen (1976) and Novick and Ellis (Note 1) papers, weargued that it should be a fundamental tenet of society that we have as an objectivethe equalization of educational opportunity for all persons irrespective of racialidentification. We presume that such a goal in itself has high social utility and that, to

142

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

MELVIN R. NOVICK

some extent, in varying degrees depending on circumstance, we would be willing tosacrifice a certain amount of investment capital, and perhaps a certain amount ofefficiency of operation, in order to attain this equality of opportunity. When thisprinciple is accepted, we come face to face with the technical question of deciding justhow important equality of opportunity is. How much do we value it? What trade-offsare we willing to accept? Obviously, we are willing to accept more trade-offs ineducational selection than in employment selection. And within employmentselection, we are willing to accept a greater loss of efficiency in less demanding skillsthan in more demanding ones. We may or may not be willing to accept very muchdecrement in efficiency in medicine, but that surely is a debatable point on which apolicy decision needs to be made.

Once we have made the decision to invest some additional capital in equalizingeducational opportunity we then propose to offer compensatory education to thosewho qualify for it. What this means is that we have made a policy decision to investand to expend some of our money in a form of education that is more costly than theusual mode of instruction. The federal government has been doing this for muchmore than a decade and of course the question arises as to whether sufficient gain isbeing gotten for the expenditure of millions of dollars in the many federally fundedcompensatory education projects. To help answer this question the Office ofEducation and the National Institute of Education have conducted studies in whichtreatment and control groups have been compared. The treatment group gets theexpensive compensatory education and the control group gets a regular form ofeducation. Following such a study the data must be analyzed to determine whetheror not our money has been well spent. The federal agencies must advise Congress asto whether they should allocate another 100 million dollars, or 500 million orperhaps a billion dollars for compensatory education; or whether the money isvaluable only to the teachers, administrators, and researchers who receive paymentsfrom these funds.

I do not wish to discuss here all of the technical problems that makes studies ofcompensatory education difficult. The almost complete inability to create com-parable treatment and control groups, the inability to keep additional funds out ofcontrol groups, the inability to monitor treatments to make sure somethingmeaningful is really being done with the money: these things, and many more, makeit extremely difficult if not impossible to conduct meaningful field trials. But let ussweep all that aside. Let us suppose that we have in the field created a legitimateexperiment. Let us suppose that we have two groups of closely matched dis-advantaged students and that one receives an expensive treatment and the other acontrol treatment. Further, let us suppose that the treatment group advances at therate of .8 of a grade equivalent score during the year, whereas the control groupadvances at the rate of about .65 of a grade equivalent score during that year. Thelatter score is what might be expected, the former is what some persons might hopefor. Now we must ask ourselves, is this increment in grade equivalent score of .15years worth the money that we have invested? This is a scientific and a policyevaluation that must be made. It tells us nothing to do a t test and reject thehypothesis of no difference, and to conclude that there is a significant differencebetween treatment and control group. Someone must make the evaluation as to whatconstitutes a meaningful and useful difference. In none of the evaluation literature Ihave seen have I ever found any statement as to what constitutes an acceptableincrement for the funds expended. Suggestions have been made on several occasions

143

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

EDUCATIONAL POLICY DECISIONS

that it would be a useful exercise to interrogate individually and in confidence someeducational psychologists who might be knowledgeable as to what amount ofdifference makes a difference. Support for this kind of work is growing, but veryslowly. No reasonable answers to the questions set by the Congress and by educatorscan be answered until this point is directly addressed by educational psychologists.

Let me summarize to this point. Most important educational questions involvepolicy issues to be settled by administrative decision but subject to public and judicialreview. The determination of a correct decision depends upon value judgments thatmust be made explicitly with the certain knowledge that different persons will havedifferent values. These value judgments typically cannot be made without profes-sional guidance which links values of present outcomes to future outcomes. Parentsand other concerned lay-persons including members of Congress cannot themselvesjudge the value of an increment of. 15 in a grade equivalent score. They need help.Educational psychologists can supply much of that help, but as I indicated earlier,the services of many other professional persons may be required.

If my exhortations have been convincing, you will now want to have a peek intomy laboratory. You will want to know how many electrodes I shall implant in yourbrain and in how many issues of the Washington Post I shall publish the difficultevaluations you make that, on later analysis, turn out to be incoherent. The answer isnone and none.

In my laboratory is a cathode ray tube terminal (a TV screen) (Novick, 1975) andin front of it is a standard typewriter keyboard on which you can hunt and peck witha character or line eraser just a finger-tip away. You will not need to know how toprogram the computer. It will converse with you in the English language, thoughyour own responses will be either binary zero-one's or at most numericaljudgment entries, e.g., .04, .76, .92. Now that you have entered my laboratory,consider the following problem. You are a university admissions officer. Oneopening remains. You must choose between an applicant who will end up with agrade point average of 2.5 for sure and another who will get either a 2.0 or 3.0. Youdon't know which, but you judge the probability that he will get a 3.0 to be p. Now,ceteris paribus, if p is close to one you will prefer the second student (3.0), i.e., youwill take a chance, but if p is close to zero you will prefer the first student (2.0). Acontinuity argument leads to the assertion that there will be a unique value of p thatwill make you indifferent between the first student and the second. Please enter thatnumber and don't be too concerned, you will have as many opportunities as you liketo change your mind. Now let's set the utility of a GPA of 0 at 0 and the utility of aGPA of 4.0 at 1.0. Let us further proceed to ask you to compare the one-apartgambles, .5 for sure against the gamble with outcomes 0,1; 1.0 for sure against .5, 1.5,and so forth. Your responses to these seven questions are sufficient to determine yourutility function for GPA. In practice we ask additional questions, fit a utility functionby least squares, and allow the evaluator to reassess until he or she is happy with theutility function specified. I hope this description convinces you that the process ispainless. Actually most evaluators find it interesting and even fun. We have used thisprogram with Ivy League professors, state university professors, community collegeprofessors, and students, and typically get specifications that both we and theevaluator judge as being accurate (Note 2).

Now suppose we were to ask you to make such a judgment concerning the utilityof various possible increments in grade equivalent score following a compensatorytreatment. You may now view the interrogation process as being pleasurable or at

144

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4

MELVIN R. NOVICK

least painless, but you may view the public statement of such utilities to be potentiallyembarrassing. Obviously you will not then cooperate. Therefore I propose that weadopt one of several strategies that will guarantee anonymity.

The first is know as the Delphi procedure. Members of an evaluation team areinterrogated separately. Then the various evaluations are discussed by the groupwithout attribution of a specific evaluation to its source. Reevaluation then occursindividually and these final evaluations are used individually to suggest decisions, orif a concensus is reached a single evaluation is used. It is entirely adequate for ourpurposes, if several utility functions are specified (Note 2).

Through a similar process prior distributions can be obtained through individualinterrogation, discussion without attribution, and then reassessment. Again, one ormore specifications can be provided the decision-maker who on the basis of thesespecifications makes a decision. That decision will be considered to be auditable as tomodel, prior distribution, and utility function, and the decision-maker will need to beprepared to defend the basis for his decision. In some cases he may not be free tochoose some of the utility functions offered. If our reading of the law is correct, thenutility functions that are differentiable because of race or ethnicity are not legallyacceptable while those that are differentiable on the basis of a prior state ofadvantage are.

It is, I think, an exciting time for educational psychology. There are newchallenges, new ways to contribute both to science and to society. I hope this paperwill serve to stimulate some of these possible contributions.

REFERENCE NOTES

1. NOVICK, M. R.,& ELLIS, D. D., JR. Equal opportunity in educational and employment selection.Unpublished manuscript, 1976.

2. NOVICK, M. R., D E K E Y R E L , D., & ISAACS, G. L. Manual for the Computer-Assisted DataAnalysis (CADA) Monitor—1977. In preparation, 1977.

REFERENCES

COLE, N. S. Bias in selection. Journal of Educational Measurement, 1973, 10, 237-255.CRONBACH, L. J. Equity in selection—where psychometrics and political philosophy meet. Journal of

Educational Measurement, 1976, 13, 31-42.DARLINGTON, R. B. A defense of rational personnel selection, and two new methods. Journal of

Educational Measurement, 1976, 13, 43-52.HAMBLETON, R., & NOVICK, M. R. Toward an integration of theory and method for criterion

referenced tests. Journal of Educational Measurement, 1973, 10, 159-170.LINN, R. L. Fair test use in selection. Review of Educational Research, 1973, 43, 139-161.LINN, R. L. In search of fair selection procedures. Journal of Educational Measurement, 1976, 13, 77-88.MILLMAN, J. Determining test length, passing scores and test lengths for objectives-based tests. Los

Angeles: Instructional Objectives Exchange, 1972.NOVICK, M. R. A course in Bayesian statistics. American Statistician, 1975, 29, 94-97.NOVICK, M. R., & LEWIS, C. Prescribing test length for criterion-referenced measurement, CSE

Monograph Series in Evaluation, 3: Problems in Criterion-Referenced Measurement. Los Angles:Center for the Study of Evaluation, UCLA, 1974.

NOVICK, M. R., & PETERSEN, N. S. Towards equalizing educational and employment opportunity.Journal of Educational Measurement, 1976, 13, 77-87.

PETERSEN, N. S., & NOVICK, M. R. An evaluation of some models for culture-fair selection. Journalof Educational Measurement, 1976, 13, 3-30.

THORNDIKE, R. L. Concepts of culture-fairness. Journal of Educational Measurement, 1971, 8, 63-70.

145

Dow

nloa

ded

by [

Uni

vers

ity o

f B

irm

ingh

am]

at 2

2:04

10

Oct

ober

201

4