On Equivalence Constructed- Response Multiple-Choice Tests

355

On the Equivalence of Constructed-Response and Multiple-Choice TestsRoss E. Traub

Ontario Institute for Studies in Education

Charles W. Fisher

Far West Laboratory for Educational Research and Development

Two sets of mathematical reasoning and twosets of verbal comprehension items were cast intoeach of three formats—constructed response,standard multiple-choice, and Coombs multiple-choice—in order to assess whether tests with iden-tical content but different formats measure thesame attribute, except for possible differences inerror variance and scaling factors. The resulting12 tests were administered to 199 eighth-grade stu-dents. The hypothesis of equivalent measures wasrejected for only two comparisons: the con-structed-response measure of verbal comprehen-sion was different from both the standard and theCoombs multiple-choice measures of this ability.Maximum likelihood factor analysis confirmed thehypothesis that a five-factor structure will give asatisfactory account of the common varianceamong the 12 tests. As expected, the two majorfactors were mathematical reasoning and verbalcomprehension. Contrary to expectation, only oneof the other three factors bore a (weak) resem-blance to a format factor. Tests marking the abili-ty to follow directions, recall and recognitionmemory, and risk-taking were included, but thesevariables did not correlate as expected with thethree minor factors.

A question of enduring interest is whether

tests that employ different response formats,

but that in other respects are as similar as pos-sible, measure the same attribute. This ques-tion has been asked of constructed-response ascompared with multiple-choice tests (Cook,1955; Davis & Fifer, 1959; Heim & Watts, 1967;Vernon, 1962) and of multiple-choice tests hav-ing standard as compared with nonstandardformats (Coombs, Milholland, & Womer, 1956;Dressel & Schmid, 1953; Hambleton, Roberts,& Traub, 1970; Rippey, 1968). Results of avail-able research suggest that the distributions ofscores on tests employing different formatscannot be assumed to have the same mean andstandard deviation, even when the tests are ad-ministered to the same group of examinees orto groups that differ only because of randomallocation of examinees to groups. In addition,the reliability and criterion correlation coeffi-cients associated with different response for-mats cannot be assumed to be the same. Theseresults are not, of course, sufficient evidence to

reject the null hypothesis that tests with dif-ferent formats measure the same attribute. It is

possible to account for differences in meansand standard deviations through appeal to pos-sible differences in the scales of measurementassociated with different formats; and dif-

ferences in reliability and criterion correlationcoefficients can be attributed to possible dif-

Downloaded from the Digital Conservancy at the University of Minnesota, http://purl.umn.edu/93227. May be reproduced with no cost by students and faculty for academic use. Non-academic reproduction

requires payment of royalties through the Copyright Clearance Center, http://www.copyright.com/

356

ferences in the relative amount of error vari-ance associated with the different test formatsand also to possible differences in the scales ofmeasurement such that one scale is a nonlineartransformation of the other.

For several years now, statistical procedureshave been in existence for testing the null

hypothesis of equivalence of measures. Earlyexemplars of these procedures (Lord, 1957;McNemar, 1958) were somewhat difficult to

use. More recently, however, Lord (1971) haspresented a statistically rigorous test based onwork by Villegas (1964) that is relatively easyto employ. This test is of the hypothesis that&dquo;two sets of measurements differ only becauseof (1) errors of measurement, (2) differing unitsof measurement, and (3) differing arbitrary ori-gins for measurement&dquo; (Lord 1971, p. 1). Clear-ly, this test accounts for all the previously de-scribed reasons for differences between themeasurements yielded by two different test for-mats except for those differences caused by thefact that one scale is a nonlinear transforma-tion of the other.

In addition to Lord’s procedure, recent de-velopments in factor analysis make it possibleto test hypotheses about the relationshipamong measurements arising from tests withdifferent formats. These developments, sub-

sumed under the heading &dquo;confirmatory factoranalysis,&dquo; have been made principally byJ6reskog (1969, 1971 ) and McDonald (1969).The purpose of the present investigation was

to test the equivalence of three response for-mats, each applied to items from two differentcontent domains. The formats were (1) con-

structed-response, (2) standard multiple-choice, in which the examinee is instructed tochoose one option per item, the one he thinksis correct, and (3) nonstandard multiple-choice, in which the examinee is asked to iden-

tify as many of the incorrect options as he can.This latter procedure was described byCoombs, Milholland and Womer (1956) and ishereafter called the Coombs format or the

Coombs procedure. The two content domains

were verbal comprehension, as defined opera-tionally by questions on the meaning of words;and mathematical reasoning, as defined opera-tionally by questions about a variety of mathe-matical concepts and skills, and by problemswhose solutions depend on the ability to applya variety of mathematical concepts and skills.The motivation for studying the equivalence

of measurements arising from different re-

sponse formats was to gain some further under-standing of partial knowledge. The standardmultiple-choice format does not assess and

credit partial knowledge-the kind of know-ledge that enables an examinee to respond at abetter-than-chance level to items that cannot

with certainty be answered correctly. The

Coombs format nullifies this criticism because

it enables an examinee to gain partial credit byidentifying one or more of the incorrect op-tions to an item, even though not all of the in-correct options are identified. What remains atissue, in the face of this logical analysis, is

whether measurements based on the Coombs

format reflect the same attribute as measure-

ments based on the standard multiple-choiceformat. For example, it might be the case thatthe longer and more involved instructions asso-ciated with the Coombs format introduce the

factor of following directions’ into the meas-urements, a factor that might not be present inmeasurements based on the standard multiple-choice format with its simpler instructions.A comparison of the Coombs and standard

multiple-choice formats appears interesting inits own right, but both these formats can beviewed as ways to simplify and objectify thescoring of constructed-response items. To theextent that this view of objectively scorabletests is accepted, interest extends to a compari-son of measurements derived from all three

1The ability to follow directions has been called integrationand defined as the "ability simultaneously to bear in mindand to combine or integrate a number of premises, orrules, in order to produce the correct response" (Lucas &

French, 1953, p. 3).



357

formats. Again, the issue is whether the meas-urements derived from a constructed-responseformat reflect the same attribute as measure-ments derived from objectively scorable for-mats. For example, items that are designed totest factual knowledge and that involve theconstructed-response format can be answeredby the exercise of recall memory. The sameitems, when cast into a multiple-choice format,can be answered by the exercise of either recallor recognition memory. In addition, a multiple-choice format is more clearly subject to the in-fluence of risk-taking (guessing) behavior thanis a constructed-response format.

In the case of the constructed-response for-mat, an examinee can guess only if he makesthe effort to generate a response. This factalone operates against risk-taking behavior. Inaddition, the set of possible responses is prob-ably quite large, although for any examinee itconsists of only those possibilities he can gen-erate and this number is not necessarily large;the larger the set of possible responses, the lesslikely the examinee is to guess correctly andtherefore have risk-taking influence his test

score. On the other hand, in the case of mul-

tiple-choice formats, the set of possible re-

sponses is small in addition to being preciselythe same for every examinee. This means that

the probability of a correct guess is sufficientlylarge for risk-taking to influence test scores sig-nificantly. Fortunately, the topic of risk-takingon multiple-choice tests has been the subjectfor considerable research, and measures of in-dividual differences in risk-taking have beenproposed (see, for example, Slakter, 1967;Swineford, 1938; Ziller, 1957); hence, it is a

factor that can be included as an independentvariable in research studies.

In summary, the main purpose of the presentstudy was to test the equivalence of measure-ments obtained using constructed-response,standard multiple-choice and Coombs re-

sponse formats. In addition, the study was de-signed to identify format factors and to studythe association between these factors, if found,

and the psychological attributes of following-directions ability, recall memory, recognitionmemory, and risk-taking.

Method

Instrumentation

To attain the main purpose of this investiga-tion, it was necessary to impose two constraintson the measures devised for each content do-main : (1) The content of the measures for onetest format had to be as similar as possible tothe content of the measures for another testformat. This constraint was satisfied by usingthe same set of item stems for all three test for-mats and the same item response options forboth the standard multiple-choice and Coombsformats; (2) The number of measures per re-sponse format had to be at least two in order to

implement Lord’s (1971) procedure for testingequivalence. This constraint was satisfied byforming two sets of verbal comprehensionitems and two sets of mathematical reasoningitems. The two sets of verbal comprehensionitems were drawn from a pool formed by theitems marking the verbal comprehension fac-tor in the Kit of Reference Tests for CognitiveFactors (French, Ekstrom, & Price, 1963); thetwo sets of mathematical reasoning items weredrawn from a pool consisting of the mathe-matics items in Forms 3A and 4A of the 1957edition of SCAT (the Cooperative School andCollege Ability Tests, 1957), the items markingthe general reasoning factor in the Kit ofReference Tests for Cognitive Factors (Frenchet al., 1963), and the items in the CanadianNew Achievement Test in Mathematics (1965).The large pools of verbal comprehension andmathematical reasoning items were pretestedin their standard multiple-choice formats,under instructions to answer every item withno penalty for wrong answers, to approxi-mately 100 students at the same eighth-gradelevel as the students who subsequently partici-pated in the study. These pretest data were



358

used to compute indices of item difficulty (thepercentage of correct responses) and item dis-crimination (the item-total biserial correlationcoefficient). (The total score used in the compu-tation of a biserial correlation coefficient was

the sum of scores on all the items included in

the pool for a given content domain.) The twosets of items drawn from the verbal compre-hension pool each contained 50 items; the

two sets of items from the mathematical rea-

soning pool each contained 30 items. The itemsets for a content domain were matched for

pretest indices of difficulty, with the averagedifficulty being .50 in each case. The item setswere also matched as closely as possible forvalues of the pretest indices of discrimination.

The secondary purpose of the study was toseek response format factors and, if such fac-tors were isolated, to study the degree of asso-ciation between the factors and measures of

possibly related psychological attributes. Thesearch for format factors, given the design ofthe study, took place among the covariancesbetween measures having the same format butdifferent content. In other words, a factor de-fined by the constructed-response format wasconceived as one that would be associated with

the constructed-response measures of both theverbal comprehension and mathematical rea-soning domains of content and not with thestandard multiple-choice and Coombs meas-ures of these domains. Format factors asso-

ciated with the standard multiple-choice andCoombs formats would be similarly defined.The variables of following directions, recall

memory, recognition memory and propensityfor risk-taking (on multiple-choice tests) weremeasured for the purpose of studying the asso-ciation between these variables and format fac-

tors, if such factors were identified. The abilityto follow directions was measured by two in-struments that had been used previously byTraub (1970) and prepared as adaptations of atest devised originally by J. W. French. Twomeasures of recall memory were employed,both of which were taken from the tests of as-

sociative memory contained in the Kit of Ref-erence Tests for Cognitive Factors (French etal., 1963). Recognition memory was assessedby two measures, both adaptations of materialsdeveloped by Duncanson (1966). The fourthvariable, risk-taking, was measured using an in-strument developed by Traub and Hambleton(1972). The rationale for this instrument wasproposed by Swineford (1938, 1941).

Design and Subjects

Lord’s procedure for testing the equivalenceof measures and the method of confirmatoryfactor analysis are applicable to data obtainedfrom a single group of subjects. In this study,usable data were obtained on 199 eighth-gradestudents (93 females), with a mean age at test-ing of approximately 13 years, 8 months (thestandard deviation of the age distribution was

approximately 8 months). During the1971-1972 academic year, these students at-

tended one of the two junior high schools thatcooperated in the study. Both schools were lo-cated in East York, a borough of MetropolitanToronto. The neighborhoods served by theseschools were described by the school prin-cipals as a mixture of lower and middle classes.

In any study involving tests that differ only inresponse format, care must be taken to mini-mize memory effects. This was done in the

present study by scheduling the tests so thatthere was a two-week interval between eachadministration of the same set of items and byadministering the constructed response for-

mats first; Heim and Watts (1966) found thatcarry-over from one administration of a set of

vocabulary items to a second administration ofthe items in a different format was markedlyless when the constructed-response format pre-ceded the multiple-choice format than whenthe reverse order was followed.

It was anticipated, and subsequent eventstended to confirm, that motivating students towork all versions of the verbal comprehensionand mathematical reasoning tests would be a



359

problem. The following steps were taken tominimize this problem: (1) students were toldat the first administration that the study wasdesigned to find out whether people score

better when a test involves one kind of re-

sponse format than when it involves another,that they would be tested periodically over aperiod of weeks and that their scores on thetests would be sent to them individually (thispromise was kept in that copies of individualreport forms were delivered to the school whenthe scoring had been completed); (2) the stand-ard multiple-choice format was introducedwith the comment that it would give the stu-dent a chance to improve his performance onthe constructed-response tests; (3) the Coombsformat was introduced as another chance to

improve on past performance.Two other critical conditions of the test ad-

ministrations were the scoring instructions andthe time limits provided for the administrationof each test. On all tests employing a con-structed-response format-two verbal compre-hension, two mathematical reasoning, two fol-lowing directions and two recall memorytests-students were informed that the numberof correct answers would be the score on thesetests and that, on the verbal comprehensionand mathematical reasoning tests with a con-structed-response format, it was to their bene-fit to show all their work because partial creditcould be obtained for work done on questionsanswered incorrectly. Six of the remainingmeasures, four verbal comprehension and

mathematical reasoning tests and two tests ofrecognition memory, were presented in a mul-tiple-choice format and were scored for thenumber of correct answers. In view of this, thestudents were instructed to answer every ques-tion and to guess if necessary. The four tests

presented in the Coombs format and the meas-ure of risk-taking involved rather elaboratescoring instructions with a complex system ofrewards and penalties. The students were in-formed of the scoring system in each case andseveral examples were considered to demon-

strate the potential effect of the scoring sys-tem.

Time limits for the tests are reported inTable 1. These limits were established on thebasis of pilot administrations of the tests and(except in the case of the memory tests) weregenerous, even for the Coombs format, whichwas most time-consuming, so as to achieve es-sentially power conditions. The time limits forthe tests of recall memory were those specifiedby French et al. (1963); the limits for the recog-nition memory tests were set on the basis of

pretest results to achieve a satisfactory distri-bution of scores (i.e., a distribution with a

range of approximately three standard devia-tions on either side of the mean).

Scoring

Special keys were prepared for the con-

structed-response versions of the verbal com-prehension and mathematical reasoning tests.These keys, which indicated to the scorer howto award partial credit for certain wrong an-swers, were applied to responses obtained froma pretest and were revised as required in thelight of apparent inadequacies. The final formsof the keys were applied by independentscorers to a random sample of 50 constructed-response answer booklets from one school forForm B of the tests for both content domains.The correlation between the scores assigned bythe scorers was .97 for the verbal comprehen-sion test and .98 for the mathematical reason-

ing test.All other tests used in the study could be

scored objectively.’

A Note on Sample Size

The sample size of 199 represents approxi-mately one-half the total number of eighth-grade students attending the two cooperating

2Copies of tests and scoring keys are available from theauthors on request.



360

schools during the time the data were beingcollected. The data from the other studentswere discarded for one of several reasons: (1)some students were so-called &dquo;New Canadians&dquo;and had difficulty understanding written

English; (2) some students were absent fromschool on one or more of the seven days onwhich the tests were administered; (3) somestudents attempted fewer than ten of the ques-tions on a test or marked their multiple-choiceanswer sheets following a clear pattern unre-lated to the pattern of correct answers andwere judged to have paid little attention to thetask; (4) some students were observed to copyanswers from other students during one ormore of the testing occasions. The frequencyof occurrence of reasons (3) and (4) was zerofor the first two testing occasions but over thenext four occasions, when the test items were

repeated in the other formats, this frequencydeparted quite substantially from zero. The oc-currence of this type of behavior indicates the

difficulty that is encountered in sustaining stu-dent motivation when tests are administered

repeatedly.

Results and Discussion

Basic Statistics

Means and standard deviations for all 19

measures, coefficients a for the 12 mathemati-cal reasoning and verbal comprehension meas-ures, and intercorrelations among all 19 meas-

ures are presented in Table 1. ’

Alpha coefficients are not reported for theseven marker variables; the calculation of a is

impossible for the measure of risk-taking andcannot be justified for speeded tests such asthe tests of recall and recognition memory. De-spite this, the results suggest that the reliabil-ities of at least the memory tests were relativelylow. The evidence for this suggestion consistsof the correlations between the pairs of testsdesigned to measure the recall and recognitionmemory factors. Although these pairs of testsare not parallel in content, their intercorrela-



361



362

Table 3

Results from the Tests of the Null Hypothesisthat Two Measures are Equivalent

Note: M is the determinant of the 2x2 matrix computed fromthe equation M = A -1.39W, where: A is the 2x2 matrix of among-persons sums of squares and cross products and W is the 2x2matrix of within-persons sums of squares and cross products,and both A and W are derived for a particular comparison fromthe figures reported in Table 2; 1.39 is the 99th percentileof the F distribution, with 199 and 199 degrees of freedom(see Lord, 1971).

aFor key to comparison codes, see Table 2.

bTechnically, the decision described as &dquo;Accept H &dquo; should read&dquo;Do Not Reject H &dquo;.

0

o

tions are much lower than would be expectedfor tests that reliably measure the same ability.

Equivalence of Measures

All possible pairs of tests having the samecontent and different formats were assessed for

equivalence using Lord’s (1971) procedure.’

The basic data required to make the tests arereported in Table 2 and the results of the testsare reported in Table 3. For measures of

mathematical reasoning, the hypothesis of

equivalence could not be rejected for any ofthe three possible contrasts of test formats. Formeasures of verbal comprehension, the

hypothesis of equivalence was rejected for twocontrasts-constructed-response vs. standard

multiple-choice and constructed-response vs.

Coombs. It was not possible to reject the nullhypothesis of equivalence for the contrast be-tween the standard multiple-choice andCoombs formats.

3The strategy of testing all possible pairs of instruments forequivalence can be criticized because the tests that aremade are not linearly independent. In this specific in-

stance, however, a better strategy, one that would avoidthis criticism, did not suggest itself.



363

To ascertain whether factors associated withtest format would override those associatedwith content, three other pairings were con-sidered. In each case, response format washeld constant and content was varied; that is,the constructed response versions of mathe-matical reasoning and verbal comprehensionwere tested for equivalence, as were the stand-ard multiple-choice and Coombs versions ofthese tests. The hypothesis of equivalence wasrejected for all three of these comparisons.(Basic data and results of the tests are also re-

ported in Tables 2 and 3.)The foregoing results indicate that the tests

of mathematical reasoning measured the sameattribute regardless of response format, where-as the attributes measured by tests of verbalcomprehension varied as a function of re-

sponse format. A conception of mental func-tioning that would account for this finding isthe following: (1) Determining the correct an-swer to a mathematical reasoning item involvesworking out the answer to the item regardlessof the format of the test. The work is recordedin the case of constructed response items and is

used as a basis for choosing a response in thecase of the standard multiple-choice andCoombs formats; (2) Determining the correctanswer to a verbal comprehension item in-

volves recalling definitions when a con-

structed-response format is used. When stand-ard multiple-choice and Coombs formats areused, however, it is only necessary to recognizethe correct answer, and recognition involvesruling out implausible response options. Thisconception of the differences among the men-tal operations that are employed in workingmathematical reasoning as compared withverbal comprehension tests implies that themain advantage of the Coombs response for-mat-the possibility of revealing partial know-ledge by identifying one or more response op-tions as incorrect but not identifying all the in-correct options-would be utilized to a greaterextent with the verbal comprehension than themathematical reasoning items. Statistics con-

firming this implication are reported in Table4.4 4

Format Factors

The intercorrelations among the 12 meas-ures of mathematical reasoning and verbal

comprehension were subjected to confirma-tory factor analysis using the COSA-I programof McDonald and Leong (1975) in an effort toidentify format factors. A format factor is a

factor associated with tests employing thesame response format, regardless of test con-tent. In line with the hypothesized existence offormat factors is a five-factor structure involv-

ing two correlated factors-one markingmathematical reasoning, the other markingverbal comprehension-and three orthogonalformat factors, one for each of the three re-

sponse formats included in the study. It provedpossible to obtain a satisfactory fit of a five-fac-tor structure provided that, in addition to thespecified five factors, the possibility of corre-lated unique factors among all six measures ofmathematical reasoning and among all sixmeasures of verbal comprehension was al-

lowed.’ For this structure, the approximate X2statistic arising from the goodness-of-fit testwas 21.297 which, with 11 degrees of freedom,has a probability of chance-occurrence underthe null hypothesis of slightly more than .03.Estimated values of the parameters of thisstructure are given in Table 5.

4The frequency of partial knowledge responses might rea-sonably be expected to increase as test difficulty increased.Differences in test difficulty do not, on average, appear toaccount for the present results. The mean scores on the

multiple-choice versions of the mathematical reasoningand verbal comprehension tests are approximately equal toone-half the total number of items in the tests.

5The analysis that was done was not factor analysis in theclassical sense of the term. It may be described as the con-

firmatory analogue of interbattery factor analysis (Tucker,1958).



364



Any attempt to deviate from the structure inTable 3 by fitting fewer factors resulted in sig-nificantly larger XZ statistics; the increase in thevalue of X2 was approximately equal to twicethe increase in the number of degrees of free-dom associated with each decrement of one inthe number of factors from five to two factors.

Any attempt to equate to zero some or all ofthe correlations among unique factors for testshaving the same content resulted in the occur-rence of a &dquo;Heywood case,&dquo; in which the esti-mated unique variance of at least one of the 12tests was a nontrivial but meaningless negativenumber.

There are several points worth noting aboutthe structure reported in Table 5:

1. The two main factors were mathematical

reasoning (Factor I) and verbal compre-hension (Factor II). As expected, these

factors were highly intercorrelated.2. The sets of items comprising Forms A and

B of each content domain were far from

parallel, regardless of the format in whichthey were presented. Had these item setsbeen parallel, the factor coefficients andunique variances of Forms A and B for agiven format and content domain wouldhave been the same, but when this kind of

constraint was imposed on the structurefitted to the data, the result was very un-

satisfactory.3. The fitted structure ignored a substantial

amount of the variance held in common

among the six tests of verbal comprehen-sion. This is clear from the size of the off-

diagonal entries in U for the six verbal

tests. These entries were considerablylarger on the average than the correspond-ing entries for the six mathematical reason-ing tests.

4. It was hoped that Factors III, IV and Vwould look like format factors. In order to

have the appearance of a format factor,the coefficients of the four tests sharingthe same response format should all have



366

the same algebraic sign and be largeenough in absolute magnitude to be dis-tinguishable from zero. The only factor ofthe three that comes close to satisfyingthese conditions was the third which can,

perhaps, be called a constructed-responsefactor. The coefficients on the fourth and

fifth factors, however, did not meet theconditions for format factors.

As a further guide to interpreting the factorstructure reported in Table 5, an &dquo;extension

analysis&dquo; (Lord 1956, pp. 40, 42) was performedin which least squares estimates of the coeffi-

cients of the seven marker variables on the five

factors were obtained. These coefficients are

reported in Table 6. Several observations aresupported by the numbers given in the table:

1. The tests of following-directions abilityhave sizeable coefficients on the mathe-matical reasoning and the verbal compre-hension factors (I and II, respectively).These tests do not, contrary to expecta-tion, have substantially larger coefficientson the fifth factor-the one defined bytests with the Coombs format-than theyhave on the third and fourth fac-

tors-those defined by the constructed-re-sponse and standard multiple-choice tests,respectively.

2. The results for the tests of recall memoryare interesting in that they have positivecoefficients on the mathematical reason-

ing factor and negative coefficients on theverbal comprehension factor. The positivecoefficients on mathematical reasoningmay reflect nothing more than that the tworecall memory tests required examinees toform associations between pictures or ob-ject labels and numbers. It is possible,however, to use these results as partial sup-port for the previously described theory ofexaminee behavior on constructed-re-

sponse as compared with multiple-choicetests. According to the theory, examineesrespond to mathematical reasoning items,

regardless of test format, by doing theoperations needed to derive answers to thequestions. This is an activity which pre-sumably would draw heavily on recall

memory. The factor structure providessupport for this suggestion. The theoryalso predicts (a) a positive association be-tween recall memory and constructed-re-

sponse tests of verbal comprehension and(b) a positive association between recogni-tion memory and multiple-choice tests ofverbal comprehension. Because the verbalcomprehension factor in this study ismarked by both constructed response andmultiple-choice tests, it is difficult to pre-dict just what associations there should bebetween the verbal comprehension factorand the tests of recall memory and recog-nition memory. The obtained negative co-efficients for recall memory on the verbal

comprehension factor are something of apuzzle-why should performance of thesetests be hampered by recall memory?-butthe positive coefficients for recognitionmemory on this factor are not surprising,although their size is smaller than might beexpected.

3. Neither the recall memory nor the recogni-tion memory tests had coefficients the size

they were expected to have on the factorsmarked by tests with different formats, i.e.,high coefficients for recall memory on thethird factor and high coefficients for re-cognition memory on the fourth and fifthfactors.

4. The positive coefficient for the measure ofrisk-taking on the verbal comprehensionfactor is most probably a reflection of thefact that the content of the risk-takingmeasure consisted of vocabulary items.The negative coefficients for this measureon the first, fourth and fifth factors are notso large as to suggest an important nega-tive association between risk-taking be-

havior and the abilities defined by thesefactors.



367

Table 6

Estimated Coefficients for the Seven Marker Variables on theFive Factors Defined by the Tests of Mathematical

~

Reasoning and Verbal Comprehension

Note: The coefficients reported in this table were estimated as follows:G = Q’HS ( SH’ HS ) 1

where H and S are matrices reported in Table 5 and Q’ is the 7 by 12matrix of cross-correlations between the set of 7 marker variables and

the set of 12 mathematical reasoning and verbal comprehension tests (seeentries in the last 7 rows and the first 12 columns of the intercorrela&dquo;

tion matrix reported in Table 2).

Conclusions

The main conclusion of this study concernsthe equivalence of measurements arising fromtests based on the same content but employingdifferent formats. When content was held con-

stant and allowance was made for differencesdue to errors of measurement and scale para-meters, i.e., units and origins, the tests of

mathematical reasoning that were employedwere equivalent regardless of format, but thetests of verbal comprehension were not. In par-ticular, the free-response tests of verbal com-prehension seemed to measure something dif-ferent than standard multiple-choice and

Coombs tests of this ability, although the stand-ard multiple-choice and Coombs formats them-selves yielded equivalent measures of verbalcomprehension. This finding, if found to be

generally true, has obvious methodological im-plications for educational and psychologicalresearchers. The design of instruments to

measure verbal comprehension must be donewith the full awareness that different formats

may well yield measures of different abilities.

This same concern is apparently not necessaryfor tests of mathematical reasoning.The foregoing, major conclusion of the

study cannot go unqualified. In this study, allthe pairs of tests with the same format, regard-less of whether the content consisted of mathe-matics questions or vocabulary items, were notstatistically parallel (i.e., they had different

means, variances, reliability coefficients, andintercorrelations with other variables). Furtherevidence of the lack of parallelism was ob-tained from the factor analyses in that a paral-lel forms factor structure did not provide asatisfactory fit to the matrix of intercorrela-tions among the twelve mathematical reason-

ing and verbal comprehension tests. The re-sults of the only factor analysis that gave satis-factory results indicate that the unique factor(including error of measurement) for one formof a test was correlated with the unique factorfor the &dquo;parallel&dquo; form of the test. The statis-tical test of equivalence provided by Lord as-sumes the existence of &dquo;replicate&dquo; measure-ments-truly parallel tests would provide repli-cate measurements-having errors of measure-



368

ment that are uncorrelated across replications(Lord, 1971, p. 2). Nothing seems to be knownabout the robustness of Lord’s test when this

assumption is violated.The second, and very much weaker conclu-

sion of this study is that evidence was obtainedof the existence of a constructed-response for-mat factor. The evidence for this factor is weak

because all the coefficients were small in abso-

lute magnitude and the factor did not have theexpected associations with the marker vari-

ables, although the constructed response testof mathematical reasoning and verbal compre-hension had, as expected, positive coefficientson this factor.

The primary reason for undertaking the

study-to identify format factors and gain anunderstanding or explanation of these factorsby relating them to marker variables for follow-ing-directions ability, recall and recognitionmemory, and risk-taking-appears to have

been unjustified. It was not possible to identifyformat factors that were clearly marked andthat accounted for a substantial amount ofvariance common to the tests having the sameformat regardless of content.

References

Canadian New Achievement Test in Mathematics.Toronto: Ontario Institute for Studies in Educa-tion, 1965.

Cook, D. L. An investigation of three aspects offree-response and choice type tests at the collegelevel. (Doctoral dissertation, Ohio State Univer-sity). Ann Arbor, MI: University Microfilms,1955, No. 12, 886.

Coombs, C. H., Milholland, J. E., and Womer, F. B.The assessment of partial knowledge. Educa-tional and Psychological Measurement, 1956, 16,13-37.

Cooperative School and College Ability Tests.

Princeton, NJ: Educational Testing Service,1957.

Davis, F. B., and Fifer, G. The effect on test reliabil-ity and validity of scoring aptitude and achieve-ment tests with weights for every choice. Educa-tional and Psychological Measurement, 1959, 19,159-170.

Dressel, P. L., and Schmid, J. Some modifications ofthe multiple-choice item. Educational and Psy-chological Measurement, 1953, 13, 574-595.

Duncanson, J. P. Learning and measured abilities.Journal of Educational Psychology, 1966, 57(4),220-229.

French, J. W., Ekstrom, R. B., and Price, L. A. Kitof reference tests for cognitive factors (Rev. ed.).Princeton, NJ: Educational Testing Service,1963.

Hambleton, R. K., Roberts, D. M., and Traub, R. E.A comparison of the reliability and validity oftwo methods for assessing partial knowledge ona multiple-choice test. Journal of EducationalMeasurement, 1970, 7, 75-82.

Heim, A. W., and Watts, K. P. An experiment onmultiple-choice versus open-ended answering ina vocabulary test. British Journal of EducationalPsychology, 1967, 37, 339-346.

Jöreskog, K. G. A general approach to confirma-tory maximum likelihood factor analysis. Psy-chometrika, 1969, 34, 183-202.

Jöreskog, K. G. Statistical analysis of sets of con-generic tests. Psychometrika, 1971, 36, 109-133.

Lord, F. M. A significance test for the hypothesisthat two variables measure the same trait exceptfor errors of measurement. Psychometrika. 195722, 207-220.

Lord, F. M. A study of speed factors in tests andacademic grades. Psychometrika, 1956, 21. 31-50.

Lord, F. M. Testing if two measuring proceduresmeasure the same psychological dimension. (Re-search Bulletin RB-71-36). Princeton, NJ: Educa-tional Testing Service, 1971.

Lucas, C. M., and French, J. W. A factorial studyof experimental tests of judgment and planning.(Research Bulletin 53-16). Princeton, NJ: Educa-tional Testing Service, 1953.

McDonald, R. P. A generalized common factor

analysis based on residual covariance matricesof prescribed structure. British Journal ofMathematical and Statistical Psychology, 1969,22, 149-163.

McDonald, R. P., and Leong, K. COSA-I program.Toronto: Ontario Institute for Studies in Educa-tion, 1975.

McNemar, Q. Attenuation and interaction. Psycho-matrika. 1958, 23, 259-266.

Rippey, R. Probabilistic testing. Journal of Educa-tional Measurement. 1968, 5, 211-215.

Slakter, M. J. Risk taking on objective ex-

aminations. American Educational ResearchJournal. 1967, 4, 31-43.

Swineford, F. The measurement of a personality



369

trait. Journal of Educational Psychology. 1938,29. 295-300.

Swineford, F. Analysis of a personality trait. Journalof Educational Psychology. 1941, 32, 438-444.

Traub, R. E. A factor analysis of programmed learn-ing and ability measures. Canadian Journal ofBeha vioral Science, 1970, 2, 44-59.

Traub, R. E., and Hambleton, R. K. The effect of

scoring instructions and degree of speedednesson the validity and reliability of multiple-choicetests. Educational and Psychological Measure-ment, 1972, 32, 737-757.

Tucker, L. R. An inter-battery method of factoranalysis. Psychometrika, 1958, 23, 111-136.

Vernon, P. E. The determinants of reading compre-hension. Educational and Psychological Meas-urement, 1962, 22, 269-286.

Villegas, C. Confidence region for a linear relation.Annals of Mathematical Statistics. 1964, 35, 780-788.

Ziller, R. C. A measure of the gambling response-set in objective tests. Psychometrika, 1957, 22,289-292.

AcknowledgementsThis project was supported by a research grant

.from the Office of the Coordinator of Research andDevelopment, OISE. The authors are indebted to1). Mr. Gordon Brown. Principal. St. Clair JuniorSchool. Mr. Frank Gould. Principal. We,stwoodJunior School, their teaching staffs and eighth-grade,students for cooperating in the study: 2). MaryCockell. Liz Falk. Colin Fraser. Mohindra Gill.Lorne Gundlack. Gladys Kachkowski. Kuo Leong.Joyce Townsend. Pat Tracy, Wanda Wahlstrom andDawn Whitmore. each of whom gave assistance at,some phase of the ,study; 3). J. P. Duncanson forpermission to reproduce materials used in one testof recognition memory: and 4). Educational TestingService. Princeton. N.J. for permission to reproduceitemsfrom two forms of the 195 7 editio n of SCAT.

Author’s Address

Ross E. Traub, The Ontario Institute for Studies inEducation, Department of Measurement, Evaluationand Computer Applications, Educational EvaluationCenter, 252 Bloor Street West, Toronto, Ontario,Canada M5S 1V6.



Documents

On Equivalence Constructed- Response Multiple-Choice Tests