Benő Csapó http://www.staff.u-szeged.hu/~csapo/

Theoretical Perspectives and Practical Constraints in Designing a Technology-

Based Matriculation Examination

UNIVERSITY OF SZEGED INSTITUTE OF EDUCATION

Forum Criteriorum International seminar on criteria-based assessment

29.-30.9.2015, Finlandia Hall, Helsinki

Outline

• Traditions and developments in the Hungarian Matriculation Examinations, and the problems of the present system

• Designing a new Matriculation Examination system

• Constraints of renewing the examination system

Traditions and developments in the Hungarian Matriculation

Examinations, and the problems of the

present system

Traditions and developments in the Hungarian Matriculation Examinations

• First introduced in 1849 • Almost unchanged for 150 years

– oral and written components – main subjects: Hungarian language and

literature, Mathematics and History • Reforms started in 1995 • A new system introduced in 2005

Characteristics of the present ME

• Two levels (basic and advanced) • Three mandatory subjects: Hungarian

language and literature, Mathematics and History

• Large number of electives (>100) • Can be taken at several languages

Problems of the present ME

• Few students take the advanced level – universities accept basic level – extra points are given for advanced level

• The difference between the two levels are not exactly defined

• Too many electives • Overall results can not be compared • Problems with objectivity, reliability and

validity • Expensive

Number of students taking advanced level examinations

0

2 000

4 000

6 000

8 000

10 000

12 000

His

tory

Mat

h

Hun

garia

n

Biol

ogy

Engl

ish

Ger

man

Che

mis

try

Phys

ics

ICT

Fren

ch

2006 2007 2008

Number of students taking basic and advanced level written examinations

in 2013

Subject Basic level

Advanced level

Advanced level in %

Mathematics 77123 3726 4,61

Hungarian 80020 1717 2,10

History 78014 5678 6,78

Distribution of the basic level Mathematics written examination scores

0%

5%

10%

15%

20%

25%

30%

0-9% 10-19%

20-29%

30-39%

40-49%

50-59%

60-69%

70-79%

80-89%

90-100%

2009 2010 2011 2012 2013

Designing a new Matriculation Examination system

What can we learn from the mistakes?

Csapó, B. (2009). The second stage of public education and the Matura. In Fazekas, K., Köllő, J., & Varga, J. (Eds.). Green book for the renewal of public education in Hungary (pp. 81-104). Budapest: Ecostat.

Starting a newer renewal

Function of a Matriculation Examination

• Certification, closing examination at the end of the 12 years of primary and secondary education – summative aspects

• Entrance examination for higher education – readiness, predictive aspects

Developing a Matriculation Examination system

• Determining the examination subjects • Framework development based on a

research-informed conception of knowledge • Scale development for capturing a broad

range of knowledge and skills • Choosing the assessment medium • Instrument development, item development • Piloting • Implementation

Determining the examination subjects

• Hungarian language and literature • Mathematics • History (and social studies, civics) • Science • English

Why are these necessary? Why not more?

Framework development

• What to assess? • Main options

– Disciplinary knowledge, curricular contents • mapping the learning materials into assessment items

– Application of knowledge, transferring knowledge into new situations

• contexts of application – Psychological approach

• impact of education on the development of general cognitive skills

The assessments are based on a three-dimensional framework

PISA TIMSS Diagnostic

- application - content - content (literacy) - reasoning - reasoning - application - application

Generalizing the results of cognitive research: a three/dimensional

conception of knowledge

Psyhological dimension: Internal reference Basic skills – general abilities continuum Cognitive skills, competencies etc.

Curricular content: External (professional) reference Expertise (disciplinary/professional dimension) Expert knowledge, professional knowledge

Application: External (social) reference General literacy: social, cultural, ”lay”, “civic” dimension. PISA reading literacy, mathematical literacy, scientific literacy. Technical literacy, musical literacy, ICT literacy etc.

Goals of education and organization of knowledge

Cultural determination

BASIC SKILLS – GENERAL ABILITIES

Disciplinary determination

Psychological determination

GENERAL LITERACY

CONTENT KNOWLEDGE – EXPERTISE PISA mathematical and

scientific literacy

1st and 2nd IEA Science and Mathematics Study

PISA 2012 Problem Solving

Scale development

• Capturing a wide range of knowledge and skills

• Stability of scales over years • Anchoring, IRT scaling • Multistage testing, adaptive testing • Determining minimum competencies

– normative approach – criterion-referenced approach

• Grading, setting cut scores

Choosing the assessment medium

• Advantages of technology-based assessment • Delivery methods, advantages of online

testing • Determining the testing venue: schools vs.

testing centers • Choosing the testing device: keyboard vs.

tablet

Instrument development, item development

• Developing an online assessment platform – see the eDia platform – web: edia.hu

• Item development – training of item-writers

• content expertise • technological expertise • assessment expertise

– mapping the frameworks into item banks

http://edia.hu

English

http://edia.hu/?q=en

Publications / Books

Demo test

Piloting and implementation

• Determining empirical parameters of the items

• Establishing scales • Determining the predictive validity of the

assessments: longitudinal studies

A feasibility study at the University of Szeged in September 2015

• Assessing ca. 1500 students entering the university

• Administering 6 online tests – Hungarian language and literature – Mathematics – History – Science (biology, physics and science) – English – Dynamic problem solving

Experiences of the feasibility study

• No difficulties with the assessment • Good reliability coefficients • Students received detailed personalized

feedback within one week after the testing • University and faculty level reports were

prepared in one week after the assessment

Reliability of the tests

Test Cronbach-alpha

Hungarian 0,90

Mathematics 0,89

History 0,93

Science 0,88

English 0,96

Problem solving 0,88

Correlations of the tests

Test Hunga-rian

Mathe-matics History Science English

Mathematics 0,434

History 0,598 0,409

Science 0,375 0,529 0,395

English 0,307 0,341 0,337 0,399

Problem solving

0,125 0,492 0,192 0,401 0,227

Correlations of the tests with the matriculation examination results at

basic level

Matriculation subject

Test

Hunga-rian

Mathe-matics History Science English Problem

solving

Hungarian 0,378 0,071 0,220 0,219 0,162 n.s.

Mathematics 0,291 0,656 0,233 0,426 0,241 0,414

History 0,395 0,219 0,503 0,312 0,202 0,109

Science 0,203 0,241 0,257 0,271 n.s. 0,221

English 0,369 0,318 0,312 0,351 0,632 0,225

Constraints of renewing the examination system

Practical constraints in renewing the Matriculation Examinations

• Political and policy issues • The power of traditions • The Matriculation is part of the educational

culture • Lack of knowledge about the new possibilities • Short tem interests of stakeholders • Teachers’ fears of loosing influence

www.staff.u-szeged.hu/~csapo

Thanks for your attention!

10:30-11:15

• 45 min