J . Patrick Rooney Golden Rule Insurance Company

Gregory Anrig, President of Edu- cational Testing Service (ETS), has now labeled as a “mistake” the agreement reached in 1984 between ETS, the Illinois Department of In- surance, and Golden Rule Insurance Company. The agreement estab- lishes guidelines for insurance agent license tests in Illinois that are in- tended to reduce racial differences on those tests. In light of Mr. Anrig’s recent labeling of it as a “mistake,” the questions put to me for this ar- ticle are: (a) What caused Golden Rule to bring the lawsuit? and (b) What is our current opinion of the settlement vis-a-vis Anrig’s labeling it a mistake in the January 1987 issue of the APA Monitor?

Why We Sued For Golden Rule, it all started one

night in December 1975 when we were visited by the man who had acted as coordinator for the Illinois Department of Insurance in the de- velopment of ETS’s new agent li- censing exam, which had been inau- gurated in Illinois in October 1975. He came to see us because one of the corporations in our manage- ment group had been conducting a small agent license prep course.

The coordinator for the Illinois In- surance Department wanted us to hire him and put him in charge of our test preparation course. He ex- plained that the passing rate on the new ETS test was only 31%-lower still on retakes. (We later learned

Summer 1987

that, in order to achieve even that passing rate, the passing grade was set at 40% correct answers. Previ- ous Department of Insurance license tests had a 60-70% passing rate, at 70% correct answers to pass.) The new ETS test, he said, was having a devastating impact on the insur- ance industry. There were at least 10,000 persons annually applying for insurance licenses and taking the test in Illinois, a big market for an expanded prep course.

The bottom line was: If we would hire him and put him in charge of our test preparation course, we could all make a lot of money from an expanded prep course-aided by his special credentials.

His presentation had a traumatic effect on me. That was 1975, and I could still hear the peace protest music of Country Joe McDonald: “There’s plenty of good money to be made supplying the Army with the tools of the trade.” The economic presentation made to me on that night had the same theme: There’s

J . Patrick Rooney i s Chairman of the Board and Chief Executive Oj$cer 0fth.e Golden Ruk Insurance Company, G o l h Rule Buildings, The Waterfront, Indi- anapolis, I N 46224-4199.

9

plenty good money to be made off of the misfortune of all these peo- ple who were being excluded.

I had an unusual background that enabled me to raise questions with which the typical insurance chief ex- ecutive officer might not have been concerned. I’d been previously in- volved in advocating equal access for minorities. As a result, the issue I raised that night with the man from the Illinois Department of In- surance, and in all the subsequent discussions, was not how to make the exam easier, but “What about the impact on minorities?” Both that night and in all subsequent communication, no one was willing to discuss safeguards against dis- crimination.

Our black regional managers in Chicago told us the new ETS license exam had caused them simply to give up on trying to get new black candidates into the insurance busi- ness. They commenced recruiting only people who were already li- censed. From what we could learn, we believed that the new test was for all practical purposes excluding blacks entirely from the occupation of insurance agent.

In February 1976, a t a Depart- ment of Insurance-sponsored meet- ing in Chicago, there did come an immense uproar from the insurance industry about the low passing rate. At that meeting, our questions about racial impact were met with stone- walling. Though many insurance companies and Insurance Depart- ment officials were concerned about the test difficulty and the low pass- ing rate, we could elicit no concern over racial exclusion. I t was then that we decided to sue.

In May 1976, ETS modified the test. The passing rate rose to the 70-75% range; industry complaints stopped with the better passing rate. Nothing was done about the minority concerns.

A Brief Synopsis As a preamble to my current opin-

ion of the settlement, it may be use- ful for the reader to receive a brief synopsis of the litigation and the steps leading to the settlement.

After our suit was filed in July 1976, a temporary injunction was issued in August, stopping the test -not on the substantive issue, but on the legality of the fee procedure. 10

TABLE 1 Passing Rates on Illinois Agent License Exam

Original Test I (October 11, 1975-August 28, 1976) White Minority

~ -

Test results in sample (includes tests after the May modification) 621 116

Percent passing 54.59 39 66

Modified Test II (December 4, 1976 and later) White Minority

Test results in sample 869 157 Percent passing 77.10 51.59

Four months later, the Illinois Legis- lature repaired that. Meanwhile, ETS took two actions: (a) negotia- tion with Golden Rule to settle the suit and (b) work to develop a new test.

The settlement negotiations failed. ETS refused to be bound by any meaningful guidelines for minimiz- ing discriminatory impact; the plain- tiffs would not settle without bind- ing guidelines.

ETS developed a “new” test; the Illinois Legislature cured the fund- ing impediment; the “new” test commenced in December 1976, and ETS sought dismissal of the suit as now moot because of the “new” test. The Circuit Court granted the dismissal.

Golden Rule refiled the suit cover- ing both tests, “old” and “new.”

In 1978 some preliminary discov- ery was obtained that developed data indicating discriminatory im- pact. Data indications are repre- sented in Table 1.

ETS and the Illinois Insurance Department sought a second dis- missal and got it. The Circuit Court ruled that the complaint failed to allege intentional discrimination, and that ETS was merely a private contractor for the state (hence not subject to constitutional due process and equal protection limitations). The Appellate Court unanimously reversed and remanded for trial (Golden Rule Life Insurance Com- pany et al., 1980). The Appellate Court’s decision established two sig- nificant legal precedents:

1. That ETS and the Illinois Di- rector of Insurance could be liable for intentional racial discrimination, in violation of the Equal Protection Clauses, upon a showing, among other things, that they knew the ex- aminations had a racially disparate effect but failed to take any correc- tive action in the face of that knowl- edge.

2. That ETS, by virtue of becom- ing an integral part of the state- licensing function, was subject to the constitutional limitations of the Equal Protection and Due Process clauses.

Both ETS and the Director sought review by the Illinois Supreme Court, but the Supreme Court denied their petitions.

By now we had a lawsuit that was 4 years old. The plaintiffs were out to prove intentional discrimination in the development of the licensing test; an immense amount of money had already been spent by both sides. But no depositions or inter- rogatories had been done; the case was a long way from trial.

Steps to Prove Discrimination Proof of intentional discrimination

by looking at test questions is doubt- ful. How does one look at a test question and come up with evidence on which experts can agree that “white people on the whole will do better than blacks on this question; black people on the whole will do more poorly than whites on that test question”?

Instead, proof of intentional dis-

Educational Measurement: Issues and Practice

crimination would be based on the following: (a) Did the testing com- pany have knowledge that it might be discriminating? (b) Did the testing company take adequate precautions (such as job analysis and pretesting) t o prevent discrimination in connection with the test construction? and (c) What is the empirical evidence of disparate impact, and the degree of it?

With regard to point (a), the plain- tiffs expected to prove knowledge by ETS from the notice the suit it- self represented and the data sam- ple gathered in 1978.

As for point (b), the plaintiffs ex- pected to prove culpable behavior by ETS because of ETS’s failure to take precautions to prevent discrim- inatory impact. Based on knowl- edge gained in the discovery pro- cess (when ETS officials and former officials were questioned under oath with a court reporter recording re- sponses), the plaintiffs were pre- pared to prove at trial that:

1. ETS had not even complied with its own guidelines to avoid dis- crimination, nor was it complying with other professionally accepted guidelines.

2. Even though ETS had the means to collect and analyze data on the passing rates of blacks and whites, the testing firm did not do

3. Few, if any, members of racial minorities helped develop the tests.

4. Test questions were not pre- tested for their effect on minorities before being used in scoring.

5. ETS did not conduct a job anal- ysis of the insurance agent’s job be- fore developing the licensing test, and when it did in 1980, the firm did not incorporate the results into the test until 1982 (the seventh year of testing).

There remained (c) the matter of proof of disparate impact. ETS had not collected the racial data on the test applicants in Illinois. Some in- formation could be garnered by sort- ing test results based on pictures of applicants that were required as part of the test application form. That’s how the rudimentary data reported above were gathered in 1978. Not a very satisfactory meth- odology.

Much to the surprise of the plain- tiffs, we learned that the Social

so.

Security Administration has racial information on all of us, correlated with our social security numbers. Further, the Social Security Admin- istration had sometimes been will- ing, for litigation such as this, to use its computer records to sort data by race.

This became the crucial point in the discovery process-preparatory for a trial. The plaintiffs asked the Social Security Administration to do the sort. The Illinois Department of Insurance vigorously opposed the

1 agreed-not an entirely magnanimous gesture, for it seemed likely that experts (ours and theirs) might work out a more techni- cally sound resolution than a judge might impose.

Social Security Administration’s co- operation in processing the data on race for the lawsuit. Nonetheless, the Social Security Administration agreed that it would do the sort.

By then it was late 1983. We were advised by our counsel that trial would finally come within the next year. The judge said he wanted to retire but wished to finish this case off before he did so-a favorable sign for getting a trial.

“Working It Out” Enter Gregory Anrig in January

1984. Anrig requested to meet with me and came to Chicago for the pur- pose. A pleasant person, Anrig said he has normally been on my side of the racial issue; he postured this as not his dispute, in that it originated before he became president of ETS, and indicated he’d like to settle it if I could agree with him to instruct our lawyers to “work it out.”

I agreed-not an entirely mag- nanimous gesture, for it seemed likely that experts (ours and theirs)

might work out a more technically sound resolution than a judge might impose. The judge was still a risk; and if the “work it out” settlement wasn’t acceptable, we could still go on to trial.

In 1987, Anrig now writes in the APA Monitor that the settlement was a mistake.

If so, it was a “mistake” made on Mr. Anrig’s initiative. I t was initi- ated only after the plaintiffs ob- tained the agreement from the So- cial Security Administration to process the data by race for trial- data that would be crucial for proof in the trial.

Anrig (1987) writes in the Moni- t o r : “Our legal experts were confi- dent that the case would ultimately be decided in favor of the Illinois Department of Insurance and ETS” (p. 3). What else can we expect him to say?

The settlement itself was as much ETS’s as Golden Rule’s. They may not like it today when they’re fac- ing other jurisdictions that want to hold ETS to account by the same standards, but it was theirs as much as ours. One of the benefits to ETS is that it was a mechanical (mathe- matical) solution that permitted them to keep intact their aversion to any of the EEOC Guidelines-the guidelines that the rest of us have to live with.

What do I think of the settlement today in light of Gregory Anrig’s new decision that the settlement was a “mistake”?

First, it wasn’t my settlement; I have no pride of authorship. I t was a settlement of the experts-theirs and ours. My initial preference after Anrig’s 1984 “work it out” proposi- tion to me was to return to the EEOC Guidelines as a binding stan- dard for the test construction. I was told that ETS had repeatedly re- fused proposals to be bound by the EEOC Guidelines. The mathemati- cal settlement ensued-what *4nrig now calls the “Golden Rule settle- ment” when he labels it a “mis- take.”

In forming my judgment today, there’s one additional significant piece of information. The settle- ment agreement required ETS to make annual reports to the public on the passing rates by race. In 1986 ETS made that report cover- ing 1985, the last year of testing

11 Summer 1987

TABLE 2 1985 Passing Rates

Life Exam Accident and Health Exam Number Percent Number Percent tested passing tested passing

Black 1,091 59% 734 41 % White 6,944 83 % 4,935 74 %

Total 8,035 5,669

Note. From “1985 Illinois Candidates for Licensure in Life Insurance and Ac- cidents & Health Insurance,” a report prepared for the Illinois Department of Insurance, 1986, Princeton, N J : Educational Testing Service, Center for Occupational and Professional Assessment.

before implementing the reforms mandated by the 1984 settlement (see Table 2). 1985 was the 10th year of ETS testing.

Statistical tests of the equality of passing rates show that these dif- ferences each exceed 14 standard deviations (SO). Since a difference of 1.96 SD is statistically signifi- cant, and the probability of a dif- ference of 7 SD is approximately 1 in 1,000,000, these differences clearly cannot be explained by chance. (For minority groups other than blacks, the results are similar.)

The results of Social Security pro- cessing of earlier data have not been made public; ETS has opposed release of that data. I have access only to data ETS was required by the settlement agreement to keep and release covering 1985.

My Current Opinion ETS is the largest, most presti-

gious of the testing companies-the IBM of the testing industry. That places a special responsibility on ETS: Failings cannot be attributed to either lack of resources or lack of competence.

Granted the magnitude of the dis- parate racial impact (which ETS dil- igently failed to look at prior to the requirements imposed by the settle- ment), and granted all the precau- tions ETS failed to take toward pre- venting avoidable discriminatory impact (we alleged “intentional dis- crimination”), then yes, I believe it is a good settlement that should be imposed upon Educational Testing Service in every possible situation.

12

The essence of the settlement is: 1. Test questions dealing with the

same subject matter are to be com- pared based on black-white differ- ence in correct answer rates. The questions with the Ieast difference are required to be used first. (The settlement actually created a mech- anism of Type I and Type I1 ques- tions, but that was merely a method of sorting questions with least racial difference and most racial differ- ence. The distinction between Type I and Type 11 was those with less than and those with more than 15% differential passing rate. That amount of differential is simply what was settled upon in this case; the critical commitment was the measurement of the pass-fail rate on each question dealing with the same subject matter and the obliga- tion to use first those with least dif- ferential impact.)

2. The settlement was meant to be workable, not a settlement that would put the testing company out of business. Therefore, it permits the use of Type I1 questions (those with greater racial differences) if no other questions exist. The settle- ment mandates the continuous de- velopment and pretesting of addi- tional questions.

3. The settlement agreement creates no absolute standards. Since questions are compared on their rel- ative racial differences, the settle- ment is not only workable but also should have the effect over time of reducing unnecessary racial differ- ences.

4. Questions used are required to

have an overall passing rate of 40%, just above the guess rate. Even so, the settlement permits the use of questions with a lower than 40% pass rate if there are no other ques- tions available. This settlement was imposed on a testing program that had been inaugurated some years earlier without pretesting and that had an initial pass rate of only 31% -probably zero or close to it for blacks. In the context of that his- tory, the requirement to give prior- ity to questions with a 40% pass rate seems reasonable.

5. The settlement obligates the testing company to gather and make public the data on the racial impact of the test as a whole and the im- pact of individual questions.

One of the important advantages to the settlement is that it is under- standable. It can be understood by a judge, which is crucial to achieve enforcement, should that be neces- sary. Another advantage to the set- tlement is that the concepts allow for a variation in the mathematical standards if a different context jus- tifies a standard that is either more or less strict. The 15% differential could instead be 10% or 20%. The 40% overall correct answer rate on questions for priority use could be varied-and certainly should be varied if the number of possible answers were either smaller or greater.

The settlement is an important step forward. It should not be dismissed as a “mistake.”

Seen from ETS’s standpoint, the mistake may be that i t is a standard to which Educational Testing Ser- vice can be held accountable. What’s worse for ETS, it’s a standard that lay people and judges can under- stand. That is why it is an important milestone in the quest for fair, un- biased testing.

References Anrig, G. R. (1987, January). “Golden

Rule”: Second thoughts. APA Morii- tor, p. 3.

Educational Testing Service. (1986). 1985 Illinois caxdidates for lieensure in l$e insurance and accidpnt & health insimirzce. Princeton, NJ: Education- al Testing Service, Center for Occupa- tional and Professional Assessment.

Golden Rule Life Insurance Co. et al. v. Mathias e t al., 86 Ill. App. 3rd. 323 (1980).

Educational Measurement: Issues and Practice