Overreach in High-Stakes Testing the Case of Chile's National Univeristy Admission Test (PSU)

PowerPoint Presentation

Overreach in High-Stakes Testing: The case of Chiles National University Admission Test (PSU)

9th ITC Conference, San Sebastian, Spain

July 5, 2014

Mladen Koljatic

Monica Silva

Richard P. Phelps

2

Topics of the presentation

Types of admission tests

The PSU context and problems

Lessons to be learned

The role of international testing associations

Types of admission tests

Aptitude Tests or Achievement Tests in the Context of Admission?

Where school inputs vary widelyacademic achievement tests are likely to measure the opportunity to learn more than the ability to learn. [Developing] nations concerned about picking their future talent must consider the possibility that an aptitude testmay be more able to overcome the local differences in school quality (Heyneman, 1987, p. 253)

SOURCE: Phelps, Standardized Testing Primer, 2007

Aptitude Tests or Achievement Tests in the Context of Admission?

Achievement tests are fairer to students because they measure accomplishment rather than ill-defined notions of aptitude; they can be used to improve performance; they are less vulnerable to charges of socioeconomic bias; they are more appropriate for schools, because they set clear curricular guidelines and clarify what is important for students to learnthe movement away from aptitude tests is an appropriate step for U.S. students, schools and universities (Atkinson, 2001)

Purpose of Admission Tests

To predict college success

without providing an unfair or unjustified advantage to any particular group of test-takers

SOURCE: Phelps, Standardized Testing Primer, 2007

2. The PSU context and problems

School Context: 2 curriculums
Curricular BranchType of SchoolScientificHumanistic (SH)Technical-Professional (TP)MunicipalPrivate-SubsidizedPrivate- Paid

The context of Chiles PSU

PSU focused on measuring the SH curriculum

Students who attend the TP tracks

the poorest in the nation

over 90% wish to pursue tertiary education

social mobility highly related to university education in Chile

* Source: Mineduc, 2009


PSU test scores used

Admission

Financial aid

Government subsidies to universities

very high-stakes associated to test scores

Students who attend TP often have no choice

Universities offer academic degrees and also technical and professional careers (such as occupational therapy, lab technician, enology)

All applicants are required to take the admission tests (PSU) for programs offered by public universities

Test scores used to assign financial aid and scholarships

Click to edit Master text styles

Second level

Third level

Fourth level

Fifth level

11


Centralized admission system to all public funded universities (+ some private universities)

PSU is run by a department within the largest public university (U. de Chile)

Admission system requires:

PSU test scores

High-school GPA (HSGP)

Class ranking

No other criterion is considered for application to higher education institutions

History of Chiles PSU

From 1966-2003 : aptitude-type test (similar to SAT)

From 2004- the present: PSU

Achievement tests: measure 100% of SH curriculum

Mathematics

Language

Science (Chemistry, Physics and Biology)

Social Sciences (History, Geography and Economics)

Change of tests aimed at evaluating outcomes of an educational reform

From 1966-2003

aptitude-type test modelled after SAT 1 and complemented by Subject Tests (like SAT 2 or AP) for a small number of careers that required knowledge of advanced contents

From 2004-

a set of mandatory achievement tests (Prueba de Seleccin Universitaria, PSU) that attempted to measure 100% of S&H curricular contents in Mathematics, Language, Science (Chemistry, Physics and Biology) and Social Sciences (History, Geography and Economics)

Change of tests was aimed at evaluating the outcomes of a recently implemented educational reform of secondary education (as well as selecting students for university education) a sort of SAT and NAEP combined


Second level

Third level

Fourth level

Fifth level

13

Project to Develop the PSU TESTS

Supported and promoted by the Ministry of Education and council of rectors of public universities (CRUCh)

Developed using public funds

Project led by an economist and a social psychologist with no training or expertise in the development of high-stakes tests

Multiplicity of Purposes for the PSU

Measure the implementation of a new curriculum

Fairly measure mastery of one of the two national curricula (SH)

Incentivize high schools to implement the new curriculum

Incentivize high school students to study more

Predict overall success of students in universities

Predict success across very different types of university programs

Reduce socio-economic disparities in performance

No single test can possibly do all of this.


Second level

Third level

Fourth level

Fifth level

15

Evidence of the tests technical quality should be provided for each purpose (St.13.2)

as well as adequate monitoring of potential negative consequences (St.13.1)

Multiplicity of Purposes for the PSU

How the PSU Runs

Source: adapted from the Pearson Report (2013)

Council of Rectors of Chilean Universities

"owners" of the PSU

designated by CRUCh as supervisors and official evaluators of the PSU PRESIDED BY CREATORS OF THE TEST

agency responsible for developing test items, test assembly, tests administration, test scoring, for CRUCh and associated universities

Main funder of the PSU since 2007.

It holds the purse for CRUCh universities, since they are funded through public monies

CRUCH

CRUCHS COMMITTEE OF TECHNICAL

ADVISORS FOR PSU

U. de Chile

Ministry of Education

Official Chilean Evaluations of the PSU

Eight evaluative reports from CRUCh between 2004-2010:

Pseudo-evaluations: overly optimistic reports aimed to construct a positive image of the test

Do not report any analyses for the TP group

When reporting gaps, omit the TP data from the analyses (without saying so)

Deny the existence of a growing score gap between private paid and municipal school groups, especially those attending the TP track

CRUCh denies access to data bases to independent researchers

19

International Evaluations of the PSU

ETS (2005)

Pearson (2013)

20

Efforts to suppress information

2005: ETS Report not released, according to U. Chile because of secrecy clause imposed by ETS

2012: New audit. CRUCh intended:

Auditors to report to expert team chosen by CRUCh

to curtail analyses for TP students

... to control and edit final report

21

2007: U. de Chile sued to release ETS report

A group of faculty and students represented by a pro-transparency nonprofit organization go to court

judge rules in favor of U. de Chile

Efforts to reveal information

22

Efforts to reveal information

ETS report is finally released in 2012 thanks to TRANSPARENCY LAWS passed in the country

From 2007: 5 years of pressure by university student leaders, faculty and leaders of NGOs for new audit

2012: Funded by the Ministry of Education (maximum amount suggested by CRUChs CTA: US$400,000)

Pearsons Final Report of the PSU (2013): Recommendations

PSU needs to be revised to advance equity for TP groups

reduction in contents

or study alternatives for the equitable assessment of CH and TP students

PSU is below predictive standards of other international tests

belying claims from CRUChs experts as to the predictive capacity of the PSU

24

Pearsons Final Report of the PSU (2013): SERIOUS DEFICITS

Inadequate use of PSU cut scores to assign financial help and scholarships

Inadequate use of scores to pass judgment on the quality of schools

25


Faulty piloting of test items

DIF analyses does not include factors as socioeconomic status, high school curricular branch (S&H and TP)

Inadequate difficulty level of the PSU Math Test (detected in ETSs 2005 audit)

Actual reporting of PSU Science score deemed untenable: Science PSU requires not one, but three scores

26


Inadequate level of analysis expected when equating a high-stakes assessment: misleading information regarding year-to-year equating and pilot calibrations. THESE ACTIVITIES ARE NOT TAKING PLACE EVEN THOUGH THE DOCUMENTATION PROVIDED SEEMS TO INDICATE THEY ARE. (p.43)

PSU equating is below international standards

ETC

27

Pearsons Evaluation of the PSU (2013)

Comprehensive

Pearson met with stakeholders other than CRUCh

Incorporated some analyses for the TP group (at no extra cost)

Specified new studies that needed (e.g.,, to guide reduction of contents)

However

Recommendations are not prioritized

Highly technical report, not aimed at policy makers

Did not look at effects on enrollment

Pearsons Final Report of the PSU (2013):

Revealed how easy it is to mislead the public when it comes to technical matters

CRUCh deceived the public as to the quality of the tests, in terms of prediction and fairness

CRUCH does NOT reduce contents in the PSU, against Pearson's reccomendation

CRUCh's new commitee of technical advisors (CTA) is controlled by an associate of the PSU test-makers

29

Aftermath of Pearsons Report...

Past Problems = Present Problems

Vested interests

No real changes in the governance system

Lack of transparency

No accountability

Main problems remain unsolved

30

Aftermath of Pearsons Report

Students demand an end to the PSU elimination of the test

Only HSGPA and class ranking as selection criteria

Moratorium not only to the PSU but to other educational tests as well

31

3. Lessons to be learned

32

Changes in tests

Some nations that are considering moving towards achievement tests might benefit from looking at the Chilean experience to avoid a proliferation of purposes, test corruption and abuse

Changes entail risks

-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test

-There is a need for a governance scheme that guarantees independent evaluations in order to protect applicants rights to be tested by good and fair tests

-Without transparency and full access to data it is tempting to those in power to produce pseudo-evaluations that construct an overly positive image of the quality of the new test

- Independent faculty that dispute pseudo-evaluations

-There is a need for a governance scheme that guarantees independent evaluations and applicants rights to be tested by good and fair tests


Second level

Third level

Fourth level

Fifth level

34

Data access is not only a problem in developing nations

The SAT item-level information is not frequently available to independent researchers who are not affiliated with ETS or the College Board

Source: Santelices & Wilson, 2010.

Governance Issues in Order to Prevent Test Abuse

Test-developing agency should be professional and independent

Account for funds received and quality of work

Report to external boards of experts that can demonstrate independence from the agency

Periodically audit tests for quality and fairness

Audit reports & data bases should be public

--> TRANSPARENCY AND ACCOUNTABILITY

4. The role of international testing associations

Changes in tests

Question to ask:

How can international organizations such as ITC play a more active role in promoting good consulting practices by test experts in developing nations?

Is there a larger role for testing associations such as ITC in the promotion of good practices worldwide?

The ITC Guidelines on Adapting Tests

The ITC Guidelines on Test Use

The ITC Guidelines on Computer-Based and Internet-delivered Testing

The ITC Guidelines on Quality Control in Scoring, Test Analysis and Reporting of Test Scores

The ITC Guidelines for Test Review?

The ITC Guidelines for International Consulting?

Overreach in High-Stakes Testing: The case of Chiles National University Admission Test (PSU)

Mladen Koljatic

Monica Silva

Richard P. Phelps

9th ITC Conference

San Sebastian, Spain

July 5, 2014

Sorry, but this presentation will be in English.

However, these slides will be made available later in both English and Spanish.


Second level

Third level

Fourth level

Fifth level

41

Predictive validities of the PSU

(CRUCh vs Pearson estimates)

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013; CTA

Predictive validities: ACT and PSU

SOURCE: ACT, Research Summary Services, 1997_1998; Pearson, Final Report Evaluation of the Chile PSU, January 2013

Predictive validities: SAT and PSU

SOURCE: Pearson, Final Report Evaluation of the Chile PSU, January 2013

44


(BUSINESS MAJORS)



(Education)


46

Percentage-point drop in enrollment of municipal school graduates attending two top universities

UC U. de Chile

PAA (adm. 2003)1732

__________________________________

PSU (adm.2004)17 29

PSU (adm.2005)16 25

PSU (adm.2006)14 20

PSU (adm.2007)13 --

PSU (adm.2008)11 --

PSU (adm.2009)11 --

PSU (adm.2010) 12--

Source: official institutional info

Drop in enrolled students from Municipal Schools, other CRUCh Universities

Fuente: E. Simonsen, 2009

Estimated average 10% drop in enrollment from students from municipal schools at top CRUCh Universities in three years NO measures taken

porcentaje matrculados municipal

0

10

20

30

40

50

200320042005200620072008

PUC

U CONCE

USACH

PUCV

Grfico2200320032003200320042004200420042005200520052005200620062006200620072007200720072008200820082008
PUC

U CONCE

USACH

PUCV


16

48

37

28

16

48

35

26

16

47

35

27

15

46

33

26

13

44

34

24

11

41

34

23

Grfico118185084308662013134118151083798693382837415348366634629922385716686270688719132426117414772408776792406020612788066935342452721575588139102831247852401968430811632624785
year

number

number of test takers

Grfico620022002200220022003200320032003200420042004200420052005200520052006200620062006200720072007200720082008200820082009200920092009
total

municipal

subsidized

private

year

total number of test takers

181850

84308

66201

31341

181510

83798

69338

28374

153483

66634

62992

23857

166862

70688

71913

24261

174147

72408

77679

24060

206127

88066

93534

24527

215755

88139

102831

24785

240196

84308

116326

24785

Hoja1Total number of test takersNmunicipalNsubsidizedNnprivate paid2002181,85084,30866,20131,3411818502003181,51083,79869,33828,3741815102004153,48366,63462,99223,8571534832005166,86270,68871,91324,2611668622006174,14772,40877,67924,0601741472007206,12788,06693,53424,5272061272008215,75588,139102,83124,7852157552009240,19684,308116,32624,785225419

Hoja2PUCU CONCEUSACHPUCV200316483728200416483526200516473527200615463326200713443424200811413423

Hoja2000000000000000000000000
PUC

U CONCE

USACH

PUCV


0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Hoja3

Documents

Overreach in High-Stakes Testing the Case of Chile's National Univeristy Admission Test (PSU)