MENOAn admission test experiment at the Vrije Universiteit Brussel

Raoul Van Esbroeck

23/11/2006 2

Vrije Universiteit Brussel

Pag.

Why an admission test in Flanders?

• Open admission• High failure rate• Costs for society• Costs for the individual student• Need for more efficiency• Admission test a possible option

23/11/2006 3

Vrije Universiteit Brussel

Pag.

Why the MENO-test?

• UCLES (University of Cambridge Local Examination Syndicate)

• Skills crucial to success in HE• Limited to 50 multiple choice items

• Critical Thinking (CT) with 3 subscales• Problem Solving (PS) with 3 subscales

• Good reliability and validity• Limited bias (language, age, ethnic origin)• Gender and SES bias?

23/11/2006 4

Vrije Universiteit Brussel

Pag.

Admission test experiment

• OBPWO project (Promoter: Prof. M. De Metsenaere)

• Questionnaire• Social and cultural variables• MENO- test

• Administered in class during first week • Cohort of entering Freshmen in 95-96• Respondents: 85% of the target group (N =

1000)

23/11/2006 5

Vrije Universiteit Brussel

Pag.

Does MENO allow correct classification?

• Errors in classification• False positives (FP)• False negatives (FN)

• Total number of errors% refused %FN %FP10 6 8725 18 6350 39 39

• Conclusion: too many errors

23/11/2006 6

Vrije Universiteit Brussel

Pag.

Has the test an added value?

• Can the test predict academic success better than SE results?

• A model with 4 variables:• SE option, class rank, graduating on time• MENO

• Explains 18.7% of variance• MENO represents only 1.75% in this model

• Conclusion: only limited added value

23/11/2006 7

Vrije Universiteit Brussel

Pag.

Bias?

• There is a gender bias • Girls would be more refused than boys• FN error is higher with girls• FP error is higher with boys

• There is a SES bias• Lower SES will be more refused than higher SES

groups• FN error is higher with lower SES groups• FP error is higher with higher SES groups

23/11/2006 8

Vrije Universiteit Brussel

Pag.

Increase of success rate?

• There is limited increase

• If 25% of the lowest MENO-scores are eliminated success rate increases from

49% to 58%

• What about errors?

23/11/2006 9

Vrije Universiteit Brussel

Pag.

Conclusion on MENO

• MENO is not an acceptable admission test• Too many errors in classification • Limited added value• There is gender and SES bias• The gains are limited

• Using this test would be unethical

23/11/2006 10

Vrije Universiteit Brussel

Pag.

Are other tests better?

• There are indications that the MENO in the UK is also biased

• The Chrysostomos test (P. Janssen, KULeuven) also

faces classification errors• A “Pass” (50%) score on the test leads to 37.6% of classification errors• A 60% score on the test leads to 30.9% of classification errors (with

about 5% of FN)

• Similar problems exist in other countries where admission tests are used

23/11/2006 11

Vrije Universiteit Brussel

Pag.

General conclusion

• There is no need for admission tests because • There is limited added value• Too many errors are made • There is bias• The gains in success rate are limited• The costs for the tests are extremely high

• There is no real societal benefit if FN are to be avoided

• There will be too many wrong decisions if you want to have real gains

23/11/2006 12

Vrije Universiteit Brussel

Pag.

Personal reflection

• The use of general admission tests for selection is unacceptable and unethical

• Such tests may be excellent for guidance purposes