TIMSS 2003: CROSS COUNTRY COMPARISONS INFORMING

RESEARCH Vijay Reddy

Human Sciences Research Council

7 April 2006

TIMSS Example 1

Performance of selected Countries

Singapore 97 (0.5) Korea, Rep. of 93 (0.8) Japan 93 (0.9) Hong Kong, SAR 93 (0.9) International Avg. 85 (0.2) Botswana 81 (1.3) Morocco 81 (2.2) Jordan 78 (1.9) Saudi Arabia 78 (2.3) Palestinian Nat'l Auth. 78 (1.8) Philippines 77 (1.6) Egypt 67 (2.1) Tunisia 59 (1.9) Ghana 55 (1.8) South Africa 52 (1.7)

TIMSS EXAMPLE 2

Performance of Selected Countries

Netherlands 97 (1.0) Sweden 96 (1.1) Estonia 96 (1.2) Singapore 95 (1.1) Lithuania 95 (1.0) International Avg. 77 (0.3) Egypt 48 (2.5) Philippines 42 (2.8) Botswana 40 (2.6) Saudi Arabia 35 (2.6) South Africa 30 (2.7) Ghana 24 (2.4)

GroupA 22B 41C 25D 11

B WA 17 58B 45 15C 27 17D 12 11

Structure of presentation

1. What is TIMSS?

2. South African performance in TIMSS 2003

3. Cross-country comparison in TIMSS.

4. Where to next?

What is TIMSS?

What is TIMSS?

• TIMSS tries to ascertain what do students know in mathematics and science and thus there are instruments to measure learners disciplinary knowledge and skills.

• In addition the study tries to explain the outcome and therefore collects data on the context (national, home and school)

TIMSS 2003

• Study is conducted at the grade 4 and 8 level – South Africa participated at the grade 8 level.

• 50 countries participated.

• Data was collected at the end of 2002 –about 9000 learners.

SOUTH AFRICAN PERFORMANCE IN TIMSS 2003

CountriesYears of

Schooling*Average

AgeScience Achievement Distribution

HumanDevelopment

Index**

Singapore 8 14.3 578 (4.3) h 0.884Chinese Taipei 8 14.2 571 (3.5) h –

¿ Korea, Rep. of 8 14.6 558 (1.6) h 0.879Hong Kong, SAR 8 14.4 556 (3.0) h 0.889Estonia 8 15.2 552 (2.5) h 0.833J apan 8 14.4 552 (1.7) h 0.932Hungary 8 14.5 543 (2.8) h 0.837Netherlands 8 14.3 536 (3.1) h 0.938United States 8 14.2 527 (3.1) h 0.937Australia 8 or 9 13.9 527 (3.8) h 0.939Sweden 8 14.9 524 (2.7) h 0.941Slovenia 7 or 8 13.8 520 (1.8) h 0.881New Zealand 8.5 - 9.5 14.1 520 (5.0) h 0.917Lithuania 8 14.9 519 (2.1) h 0.824Slovak Republic 8 14.3 517 (3.2) h 0.836Belgium (Flemish) 8 14.1 516 (2.5) h 0.937Russian Federation 7 or 8 14.2 514 (3.7) h 0.779Latvia 8 15.0 512 (2.6) h 0.811Scotland 9 13.7 512 (3.4) h 0.930Malaysia 8 14.3 510 (3.7) h 0.790Norway 7 13.8 494 (2.2) h 0.944Italy 8 13.9 491 (3.1) h 0.916Israel 8 14.0 488 (3.1) h 0.905Bulgaria 8 14.9 479 (5.2) 0.795J ordan 8 13.9 475 (3.8) 0.743International Avg. 8 14.5 474 (0.6) –Moldova, Rep. of 8 14.9 472 (3.4) 0.700Romania 8 15.0 470 (4.9) 0.773Serbia 8 14.9 468 (2.5) i –Armenia 8 14.9 461 (3.5) i 0.729Iran, Islamic Rep. of 8 14.4 453 (2.3) i 0.719Macedonia, Rep. of 8 14.6 449 (3.6) i 0.784Cyprus 8 13.8 441 (2.0) i 0.891Bahrain 8 14.1 438 (1.8) i 0.839Palestinian Nat'l Auth. 8 14.1 435 (3.2) i 0.731Egypt 8 14.4 421 (3.9) i 0.648Indonesia 8 14.5 420 (4.1) i 0.682Chile 8 14.2 413 (2.9) i 0.831Tunisia 8 14.8 404 (2.1) i 0.740Saudi Arabia 8 14.1 398 (4.0) i 0.769Morocco 8 15.2 396 (2.5) i 0.606Lebanon 8 14.6 393 (4.3) i 0.752Philippines 8 14.8 377 (5.8) i 0.751Botswana 8 15.1 365 (2.8) i 0.614Ghana 8 15.5 255 (5.9) i 0.567South Africa 8 15.1 244 (6.7) i 0.684England 9 14.3 544 (4.1) h 0.930

Benchmarking ParticipantsBasque Country, Spain 8 14.1 489 (2.7) h –Indiana State, US 8 14.5 531 (4.8) h –Ontario Province, Can. 8 13.8 533 (2.7) h –Quebec Province, Can. 8 14.2 531 (3.0) h –

* Represents years of schooling counting from the first year of ISCED Level 1.

** Taken from United Nations Development Program's Human Development Report 2003 , p. 237-240.¿ Korea tested the same cohort of students as other countries, but later in 2003, at the beginning of the next school year.

( ) Standard errors appear in parentheses. Because results are rounded to the nearest whole number, some totals may appear inconsistent.

A dash (–) indicates comparable data are not available.

AverageScale Score

0 100 200 300 400 500 600 700 800

95th

5th

Country average significantly lowerthan international average

Country average significantly higherthan international average

h

i25th 75th 95th

95% Confidence Interval for Average (±2SE)

Percentiles of Performance

Figure 4.1 Distribution of Science Achievement

264

227

314

366

468 467

0

100

200

300

400

500

National Mean Ex DET Ex HoR Ex HoD Ex HoA International Mean

Avera

ge m

ath

em

ati

cs s

cale

sco

res

389

333 304

265 264 261 255 251

223 2170.69

0.68 0.680.66

0.640.62 0.63

0.6

0.77

0.75

0

50

100

150

200

250

300

350

400

450

WC NC GP FS National MP KZN NWP EC LP

Ave

rag

e m

ath

emat

ics

scal

e sc

ore

s

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75

0.8

0.85

0.9

Hu

man

Dev

elo

pm

ent

Ind

ex 2

002

KEY FINDING

• Scores are a reflection of the inequalities in the country.

• “Quality” and “Inequality”

What do we learn from cross-country comparative research?

• Cross country analysis allows one to compare scores of learners, who have similar characteristics, from different countries.

• The comparison can be useful to generate or exclude hypotheses that might influence learner achievements.

Performance of African countries on TIMSS

Av.Scale

Score

Pop(mil)

Life expectancy

NERPrim

NERSec

GNI per capita

Egypt 421 (3.9) 66.4 69 90 78 1470

Tunisia 404 (2.1) 9.8 73 97 68 1990

Morocco 396 (2.5) 29.6 68 88 31 1170

Botswana 365 (2.8) 1.7 38 81 55 3010

Ghana 255 (5.9) 20.3 55 60 30 270

South Africa 244 (6.7) 45.3 46 90 62 2500

Example 1

Highest educational level of either parent and average mathematics scale scores

Finished university or equivalent or higher

Finished post secondary education (but not university)

Finished upper secondary schooling

Finished lower secondary schooling

No more than primary schooling

South Africa 11% 352 (16.6)

13% 294 (10.0)

30% 266 (6.0)

18% 244 (4.0)

28% 223 (4.4)

Botswana 10% 411 (7.1)

14% 378 (4.0)

16% 361(4.0)

20% 366(3.5)

41% 356

Ghana 10% 320 (8.1)

17% 296 (6.7)

22% 292 (5.8)

37% 261(5.7)

14% 246 (5.5)

Singapore 16% 651 (3.3)

4% 624 (5.9)

21% 621 (3.6)

48% 600 (3.9)

11% 571 (6)

Norway 66% 485 (2.6)

16% 459 (5.7)

12% 451 (4.9)

4% 419 (9.4)

2%

International Average

28% 503 (0.9)

17% 480 (0.9)

28% 463 (0.7)

15% 434 (1.1)

12% 410 (1.4)

Example 2

Extent language of the test is spoken at home and science scores

Always Almost always

Sometimes Never

South Africa 18% 347(17.3)

9% 310 (13.8)

57% 225 (4.3)

15% 153 (5.2)

Botswana 5% 374 (13.5)

6% 412 (10.9)

80% 366(2.5)

9% 316(6.5)

Ghana 23% 260 (7.0)

10% 279 (10.9)

63 % 263 (5.8)

5% 155 (11.5)

Singapore 23% 613 (3.9)

19% 603 (3.9)

49% 557 (5.1)

8% 545 (6.7)

Norway 85% 498 (1.9)

10% 489 (6.1)

3% 435 (10.6)

1% -

International 68% 482 (0.8)

11% 483 (1.0)

17% 442 (1.5)

4% 389 (2.4)

Example 3

Highest educational level of science teachers

Beyond first university degree

Finished university or equivalent

Finished post-sec but not university

Finished upper secondary school

Did not complete upper secondary school

South Africa

7 21 69 2 0

Botswana

4 34 61 1 0

Ghana

0 9 79 12 0

Egypt

8 92 0 0 0

Tunisia

81 17 1 0 0

Morocco

2 32 16 44 5

International 22 57 18 3 0

Class size and math scores

Class size

1-24 learners

25-32 learners

33-41 learners

>41 learners

South Africa 45 4 309 (35.8)

14 290 (23.8)

30 265 (11.7)

52 249 (8.7)

Botswana 37 1 -

14 392 (9.1)

60 360 (3.7)

25 362 (4.1)

Ghana 37 16 232 (7.4)

18 249 (8.9)

29 292 (9.0)

37 289 (9.1)

Egypt 38 3 422 (13.8)

9 428 (11.3)

61 403 (4.3)

27 407 (7.5)

Tunisia 34 1 -

26 404 (3.6)

71 412 (3.2)

2 -

International 30 29 461 (1.9)

35 473 (1.4)

24 470 (2.1)

13 448 (1.7)

Economic disadvantage and math score

GNI per capita in dollars

Schools with 0-10% economically disadvantaged learners

Schools with 11-25% economically disadvantaged learners

Schools with 26-50 % economically disadvantaged learners

Schools with > 50% economically disadvantaged learners

South Africa 2500 3 479(44.9)

2 -

9 334 (25.7)

85 237 (3.4)

Botswana 3010 15 385 (11.0)

22 375 (6.2)

25 363 (3.2)

38 354 (3.2)

Ghana 270 4 295 (24.1)

8 308 (14.9)

18 286 (9.0)

71 264 (5.9)

Egypt 1470 11 448 (11.8)

24 410 (8.2)

23 393 (6.3)

42 392 (5.5)

International Average

22 496 (2.1)

26 476 (1.3)

21 460 (1.5)

31 439 (1.3)

Teachers and principals perceptions of school climate

High Medium Low South Africa PPSC

TPSC 7

10 45 46

48 44

Botswana PPSC TPSC

1 3

31 29

68 68

Ghana PPSC TPSC

13 17

68 54

18 30

International Avg.

PPSC TPSC

15 10

67 60

18 30

School attendance

High GSCA Medium GSCA Low GSCA South Africa 6 50 44 Botswana 5 62 33 Ghana 8 69 23 International 23 58 19

What research now?

• Importance of schools in poorer societies.

• Understanding how concepts are developed inside classrooms.

• Educational/subject disciplinary researchers need to come to the fore.