PISA - University of Macau

OECD Programme for

International Student Assessment 2012

BOOKLET #

Macao-China Main Study PISA 2006

Date of Test /

/ 2006 Day Month

2012 M

aca

o-P

ISA

Macao-PISA 2012 Report

Digital Assessment of problem solving,

mathematics and reading literacy performance

of 15-year-old students from an international

comparison perspective

Educational Testing and

Assessment Research Centre

Faculty of Education

University of Macau

December, 2014

Project Consortium:

Australian Council for Educational Research (ACER, Australia)

cApStAn Linguistic Quality Control (Belgium)

Deutsches Institut für Internationale Pädagogische Forschung (DIPF, Germany)

Educational Testing Service (ETS, USA)

Institutt for Læ rerutdanning og Skoleutvikling (ILS, Norway)

Leibniz - Institute for Science and Mathematics Education (IPN, Germany)

National Institute for Educational Policy Research (NIER, Japan)

The Tao Initiative: CRP - Henri Tudor and Université de Luxembourg - EMACS (Luxembourg)

Unité d'analyse des systèmes et des pratiques d'enseignement (aSPe, Belgium)

Westat (USA)

- 0 -

- 1 -

Macao-PISA 2012 Report:

Digital Assessment of problem solving, mathematics and reading

literacy performance of 15-year-old students from

an international comparison perspective

Kwok-cheung Cheung, Pou-seong Sit, Soi-kei Mak & Man-kai, Ieong

Educational Testing and Assessment Research Centre

University of Macau

Macao, People’s Republic of China

December, 2014

- 2 -

Foreword

The Programme for International Student Assessment (PISA) was launched by the Organisation

for Economic Co-operation and Development (OECD). It is a worldwide study conducted on a

regular basis, designed to understand 15-year-old students’ basic competencies and the factors

that influence their learning, as well as to compare and evaluate the relative effectiveness of the

education provided by the participating countries and economies.

Having participated in PISA 2003, PISA 2006, PISA 2009 and PISA 2012, the Macao Special

Administrative Region will again take part in PISA 2015. As of PISA 2006, the Education and

Youth Affairs Bureau of the Macao SAR Government has commissioned the University of

Macau to conduct academic research on PISA and the work related to the execution of PISA in

Macao.

By gauging the scholastic performance of students, the PISA results serve as a reference for

educational administration authority when formulating education development plans and

policies, as well as carrying out education reform and other related work, in order to improve

the quality of the education being provided. The results also provide important information for

participating schools in review of their curriculum, teaching, student evaluation work, and the

like, so as to enhance the effectiveness of their education management.

- 3 -

Acknowledgements

Successful completion of the Macao-PISA 2012 Study was contingent on:

Commissioned with financial support by the Education and Youth Affairs Bureau of

Macao Government;

Guidance and resource support by the University of Macau authority, especially

Research and Development Administration Office of University of Macau;

Academic and technical support by the Educational Testing and Assessment Research

Centre, Faculty of Education, University of Macau;

Cooperation of secondary schools participating in the PISA 2012 Study;

Active participation of students and their parents in responding to PISA 2012 tests and

questionnaires;

Cooperation of schools providing the testing venues: Colegio de Santa Rosa de Lima

(English Secondary), Escola Luso-Chinesa Técnico-Profissional, Escola Primaria

Oficial Luso-Chinesa Sir Robert Ho Tung, Keang Peng Middle School, Pui Ching

Middle School, Sheng Kung Hui Choi Kou School (Macau), The Affiliated School of

the University of Macau, and Yuet Wah College.

- 4 -

Table of Contents

Acknowledgements 3

Table of Contents 4

List of Tables 6

List of Figures 8

List of Appendices 9

Executive Summary 10

Chapter 1 Conduct of Enquiry 13

1.1 Introduction 13

1.2 Sample design 14

1.3 Digital problem-solving assessment framework 15

1.4 An example of a test unit of digital problem-solving assessment tasks 17

1.5 Description of the proficiency levels of the digital problem-solving, mathematics,

and reading scale 18

Chapter 2 A Profile of Digital Literacy Performance for 15-year-olds in Macao 25

2.1 Macao 15-year-olds’ digital literacy performance 25

2.2 An international comparison of performance in the three digital literacy domains 29

2.3 Implications for Macao’s stake-holders when mode of PISA assessment changes

from print to the digital 32

Chapter 3 Relationships between Digital Literacy Performance and ESCS for Macao

Schools 42

3.1 Plots of digital literacy performance with ESCS in the Macao sample 42

3.2 Relationships of school digital literacy performance with school ESCS 43

Chapter 4 The 15-year-olds’ Performance on the Released Digital Assessment Tasks 47

4.1 Digital problem-solving tasks 48

4.2 Digital mathematics tasks 60

Chapter 5 Three Quality Education Indicators for Improving Digital Problem-solving 72

- 5 -

Ability of Grade Repeaters in Macao Schools

5.1 Assessment of Perseverance, Openness for problem solving, and ICT use at home

for school-related tasks 72

5.2 Correspondence analysis of “Perseverance – Give up easily” 73

5.3 Correspondence analysis of “Openness for problem solving – Easily link facts” 75

5.4 Correspondence analysis of “ICT use at home for school-related tasks – Internet

for schoolwork” 78

5.5 Implications for the grade repeaters in Macao schools 80

References 84

Appendices 85

- 6 -

List of Tables

Table 1.1 Characteristics of schools in Macao’s PISA 2012 digital assessment

sample

Table 1.2 Number of 15-year-olds sampled and tested in Macao’s PISA 2012

digital assessment

Table 1.3 Proficiency level descriptions of the digital problem-solving scale

Table 1.4 Proficiency level descriptions of the digital mathematics scale

Table 1.5 Proficiency level descriptions of the digital reading scale

Table 2.1 Macao’s 15-year-olds’ digital literacy performance results

Table 2.2 Distribution of Macao 15-year-olds’ proficiency levels on the digital

literacy scales

Table 2.3 Problem-solving performance of 15-years-olds in PISA 2003 and

PISA 2012

Table 2.4 Mathematical literacy performance of 15-years-olds in PISA 2012

Table 2.5 Reading literacy performance of 15-years-olds in PISA 2012

Table 2.6 Distribution of mathematical literacy proficiency levels of Macao’s

15-year-olds in PISA 2012

Table 2.7 Distribution of reading literacy proficiency levels of Macao’s

15-year-olds in PISA 2012

Table 2.8 Distribution of digital problem-solving proficiency levels of

Macao’s 15-year-olds in PISA 2012

Table 3.1 Digital literacy performance and ESCS of participating schools

Table 4.1 Blueprint used in the design of digital problem-solving assessment

tasks

Table 4.2 PISA released digital problem solving task – CLIMATE CONTROL

(Q1)

Table 4.3 PISA released digital problem solving task – CLIMATE CONTROL

(Q2)

Table 4.4 PISA released digital problem solving task – TICKETS (Q1)



Table 4.7 PISA released digital problem solving task – ROBOT CLEANER

(Q1)


(Q2)


(Q3)

- 7 -

Table 4.10 PISA released digital problem solving task – TRAFFIC (Q1)



Table 4.13 Blueprint used in the design of digital mathematics assessment tasks

Table 4.14 PISA released digital mathematics task – CD PROCUCTION (Q1)



Table 4.17 PISA released digital mathematics task – STAR POINTS (Q1)




Table 4.21 PISA released digital mathematics task – BODY MASS INDEX

(Q1)


(Q2)


(Q3)

Table 5.1 Correspondence analysis of Perseverance – Give up easily: Row &

column profiles

Table 5.2 Correspondence analysis of Openness for problem solving – Easily

link facts: Row & column profiles

Table 5.3 Correspondence analysis of ICT use at home for school-related

tasks – Internet for schoolwork: Row & column profiles

- 8 -

List of Figures

Figure 1.1 Main features of the PISA 2012 digital problem-solving assessment

framework

Figure 1.2 An example of a released PISA 2012 digital problem-solving test

unit (MP3 PLAYER)

Figure 2.1 Percentage of 15-year-olds at different digital problem-solving

proficiency levels across grades in the Macao sample

Figure 2.2 Percentage of 15-year-olds at different grade levels across digital

problem-solving proficiency levels in the Macao sample

Figure 2.3 QQ Plot of print versus digital mathematics literacy in PISA 2012

Figure 2.4 Percentage of 15-year-olds for Macao and selected economies at

each proficiency level of mathematical literacy in PISA 2012

(Digital vs. Print)

Figure 2.5 QQ Plot of print versus digital reading literacy in PISA 2012

Figure 2.6 Percentage of 15-year-olds for Macao and selected economies at

each proficiency level of reading literacy in PISA 2012 (Digital vs.

Print)

Figure 3.1 Plots of digital literacy performance with ESCS

Figure 3.2 Plot of school digital problem-solving performance with school

ESCS

Figure 3.3 Plot of school digital mathematics performance with school ESCS

Figure 3.4 Plot of school digital reading performance with school ESCS

Figure 4.1 PISA released digital problem solving task – CLIMATE CONTROL

(Q1)

Figure 4.2 PISA released digital problem solving task – CLIMATE CONTROL

(Q2)

Figure 4.3 PISA released digital problem solving task – TICKETS (Q1)



Figure 4.6 PISA released digital problem solving task – ROBOT CLEANER

(Q1)


(Q2)


(Q3)

Figure 4.9 PISA released digital problem solving task – TRAFFIC (Q1)



Figure 4.12 PISA released digital mathematics task – CD PROCUCTION (Q1)

- 9 -



Figure 4.15 PISA released digital mathematics task – STAR POINTS (Q1)




Figure 4.19 PISA released digital mathematics task – BODY MASS INDEX

(Q1)


(Q2)


(Q3)

Figure 5.1 A bi-plot displaying various dispositions of perseverance amongst

students and how they relate to specific gender-proficiency level

student groupings on the two principal dimensions

Figure 5.2 A bi-plot displaying various dispositions of openness for

problem-solving amongst students and how they relate to specific

gender-proficiency level student groupings on the two principal

dimensions

Figure 5.3 A bi-plot displaying various dispositions of ICT use at home for

school-related tasks amongst students and how they relate to

specific gender-proficiency level student groupings on the two

principal dimensions

Figure 5.4 A bi-plot displaying various learning characteristics amongst

students and how they relate to the phenomenon of grade repetition

on the two principal dimensions

List of Appendices

Appendix 1 Coding guide of the example MP3 PLAYER test unit

- 10 -

Executive Summary

Commissioned by DSEJ of Macao Government, University of Macau undertook the PISA 2012

Study during 2010-2013. There were two components of this study, i.e. assessment of print and

digital literacy of 15-year-old secondary students. The results of the print component were

released in December 2013, indicating that Macao’s basic education was largely very healthy

and promising. After a decade of PISA assessment since 2003, Macao emerges to become one

of the eight educational systems in the world which is both high in educational quality and

equity. Macao is also one of the eleven places in the world demonstrating growth in terms of

print mathematical, scientific and reading literacy.

The results of the digital component of PISA 2012 were released in April 2014. The main

purpose of this report is to complement the PISA 2012 Digital Assessment International Report,

entitled “PISA 2012 results: Creative problem solving – Student’s skills in tacking real-life

problems (Volume V)” (OECD, 2014). It is hoped that this report can serve as a good starting

point for any systematic enquiry that makes use of PISA 2012 digital assessment data in the

examination of quality and equity of basic education in Macao. The following is an executive

summary of this Macao PISA 2012 digital assessment report.

1. Approximately 85,000 students coming from 44 countries/economies were randomly

sampled to participate in the PISA 2012 digital assessment. Of these 44

countries/economies 28 were OECD member countries and 16 were partner

countries/economies. For Macao, a total of 3,147 15-year-old secondary students (1,570

males, 1,577 females) coming from 45 schools were tested. Each sampled student was

randomly assigned one of the 24 test forms containing clusters of problem-solving,

mathematics and/or reading test units delivered on computer-based platforms which

take approximately 40 minutes to complete.

2. From a comparative education perspective Macao’s 15-year-olds performed very well in

digital problem-solving and quite good in digital mathematics literacy. Performance on

digital reading is also very satisfactory. Generally speaking, Macao females outperform

males in digital reading, whereas it is the other way round for the males in digital

mathematics and problem-solving literacy. All standard deviations are low, showing that

student performance on the three digital literacy scales are quite homogeneous as

compared with the average of the OECD countries participating in the PISA 2012

digital assessment.

- 11 -

3. Male and female students’ digital problem-solving and mathematics proficiency levels

are mainly concentrated at levels 3 and 4. For digital reading, the concentration is

mainly at level 2, 3 and 4. Percentages of students with proficiency below level 2 for the

three digital literacy scales remain at very low levels (<8%), showing that the number of

low-performing students who cannot function productively in society is small.

Unfortunately, the numbers of high-performing students with proficiency level 6 in

digital problem-solving and mathematics literacy respectively are not high compared

with the top-performing countries in PISA 2012 digital assessment. In the case of digital

reading the situation is also not that favorable as Macao has only 5.1% of its students

assessed at proficiency level 5 and 6.

4. In PISA 2012 Macao ranks fourth in digital problem-solving amongst the 44

participating economies. Though lacking behind Singapore, Korea and Japan Macao

adolescents’ digital problem-solving performance is comparable in level with that of

its other three Chinese-speaking counterparts, i.e. Hong Kong, Shanghai and Chinese

Taipei. After considering the sampling and measurement errors, Macao ranks 4th

to 6th

in PISA 2012 digital problem-solving assessment. In the last decade when the

assessment mode changes from print to digital, Macao’s 15-year-olds still maintain

their superiority in problem-solving ability, which is an important 21st century skills

needed inculcation in the school curriculum. Macao is one of the four ICT-developed

places in the world whose students are close to meet the goal to use basic

problem-solving tools to meet unknown everyday challenges.

5. In PISA 2012 Macao ranks fifth in digital mathematical literacy amongst the 32

participating economies. Though lacking behind Singapore and Shanghai, Macao

adolescents’ digital mathematical literacy performance is comparable to that of Korea,

Hong Kong, Japan, and Chinese Taipei. After considering the sampling and

measurement errors, Macao ranks 3rd

to 7th

in PISA 2012 digital mathematics

assessment. Comparing the results between print and digital assessment, Macao’s

15-year-olds not only have maintained but also increased their superiority in

mathematical literacy when the assessment mode changed from print to the digital.

Noteworthy is that there is a drop in the percentage of high-performing students when

the mode of PISA assessment is shifted from print to digital and concomitantly there is

a drop in the percentage of low-performing (i.e. below level 2) students.

6. In PISA 2012 Macao ranks eleventh in digital reading literacy amongst the 32

participating economies. Though lacking behind its Asian Pacific counterparts like

- 12 -

Singapore, Korea, Japan, Hong Kong and Shanghai, and western counterparts like

Canada, Estonia and Australia, Macao adolescents’ digital reading performance is

comparable in level with that of Chinese Taipei, Ireland, United States and France.

After considering the sampling and measurement errors, Macao ranks 9th

to 13th

in

PISA 2012 digital reading assessment. Comparing the results between print and digital

assessment, Macao’s 15-year-olds not only have maintained but also greatly increased

their superiority in reading literacy when the assessment mode changed from print to

the digital. Noteworthy is that there is a drop in the percentage of high-performing

students (i.e. levels 5 and 6) when the mode of PISA assessment is shifted from print

to digital, and concomitantly there is a drop in the percentage of low-performing

students (i.e. < level 2).

7. Same as previous cycles of PISA assessment, the slope of the literacy-ESCS

relationship is gentle and the percentage of digital literacy performance variance

explained by economic, social and cultural status (ESCS) is the lowest of the

participating countries/economies. Therefore, Macao’s basic education system continues

to provide equitable schooling opportunities for the student body it served. In spite of

this favorable comment, it is noteworthy that there is some degree of educational

inequity observed in Macao’s basic education system. This is because although school

digital literacy performance basically is not related to school ESCS at the low end of the

ESCS continuum, however at the higher end the literacy performance of the schools are

very favorable (i.e. above the OECD average).

8. Correspondence analyses of the associations between categories of variables such as

gender of student, international grade studied, frequency of grade repetition in

primary/secondary school, perseverance, openness for problem-solving, ICT use at

home for school-related tasks, and digital problem-solving proficiency level have been

undertaken. The results suggest that teachers and schools should render assistance to the

grade repeaters, particularly those who have repeated two times or more, in accordance

with students’ zone of proximal development. Grade repeaters should be provided with

ample opportunities to use ICT for school-related tasks at home. Through proper

guidance and counselling by teachers, parents and peers, they are initiated and modeled

not to give up easily when confronted with problems. Furthermore, they should learn

the many sound pedagogical ways to links facts together and their minds become more

receptive for solving the problems assigned to them or encountered in daily life.

- 13 -

Chapter 1

Conduct of Enquiry

Abstract

This chapter recapitulates the conduct of enquiry of the PISA 2012 digital assessment

undertaken in Macao from 21 April to 31 May 2012. It comprises five sections: (1) Introduction;

(2) Sample design; (3) Digital problem-solving assessment framework; (4) An example of a set

of digital problem-solving assessment tasks; (5) Description of proficiency levels of the digital

problem-solving, mathematics and reading scales.

1.1 Introduction

Print and digital literacy assessments are the two principal components of PISA 2012 Main

Survey. Details of the conduct of enquiry and findings of the print assessment are documented

in the Macao-PISA 2012 Report (Cheung, Sit, Mak & Ieong, 2013). This report supplements

the earlier Macao-PISA 2012 Report, focusing entirely on the assessment results of digital

problem-solving, mathematics and reading literacy.

In PISA 2012, each participating economy of both print and digital assessment not only is able

to compare its digital mathematics and reading literacy performance results with that of the

OECD average, but also with its own print literacy performance results (see Chapter 2). In

addition, in this report, Macao’s strengths and weaknesses are revealed when its students’

performance on the released digital problem-solving and mathematics test items are compared

with that of some selected high-performing economies in PISA 2012 (see Chapter 4). As in the

previous Macao-PISA 2012 Report, statistics pertaining to Macao’s educational equity (see

Chapter 3) and some quality education indicator variables (i.e. perseverance, openness for

problem-solving, ICT use at home for school-related tasks) affecting Macao student digital

problem-solving literacy performance across gender are also presented in this report (see

Chapter 5).

Not all countries/economies participated in the PISA 2012 print assessment arranged their

- 14 -

students to take the additional digital assessment. Approximately 85,000 students coming from

44 countries/economies were randomly sampled to participate in the PISA 2012 digital

assessment. Of these 44 countries/economies 28 were OECD member countries and 16 were

partner countries/economies.

1.2 Sample design

Table 1.1 presents characteristics of the schools and the 15-year-olds sampled and tested in the

PISA 2012 digital assessment, broken down by school type, study program, and language of

instruction.

Table 1.1

Characteristics of schools in Macao’s PISA 2012 digital assessment sample

Stratifying Variable Number of

schools

sampled

Number of

schools

tested

Number of

students

sampled

Number of

students

tested

School Type

Government 4 4 168 165

Private-In-Net 32 32 2338 2311

Private 9 9 683 671

Study Program

Grammar-International 40 40 3009 2970

Technical-Prevocational 5 5 180 177

Language of Instruction

Chinese 32 32 2073 2052

English 7 7 579 572

Portuguese 1 1 43 41

Chinese & English 4 4 417 406

Chinese & Portuguese 1 1 77 76

Total 45 45 3189 3147 Note 1: All sampled schools offered basic education courses to 15-year-olds. Two schools were excluded from the

designed school sample; one offered senior secondary vocational education only to a few students and the

other was a school offering special education at the secondary level.

Note 2: Sampled students were all 15-year-olds born in 1996.

Table 1.1 presents the number of students (males/females) sampled and tested in Macao’s PISA

2012 digital assessment. The response rate is very satisfactory (98.7%), showing that the

achieved sample is highly representative of the Macao’s 15-year-old student population.

- 15 -

Table 1.2

Number of 15-year-olds sampled and tested in Macao’s PISA 2012 digital assessment

Number of 15-year-olds

Sampled 3,189

(1,590 Males, 1,599 Females)

Tested 3,147

(1570 Males, 1577 Females)

Response rate (%) 98.7

1.3 Digital problem-solving assessment framework

PISA 2012 defines problem-solving competence as “an individual’s capacity to engage in

cognitive processing to understand and resolve problem situations where a method of solution

is not immediately obvious. It includes the willingness to engage with such situations in order

to achieve one’s potential as a constructive and reflective citizen” (OECD, 2014, p.30).

The PISA 2012 digital problem-solving assessment framework comprises three core elements:

(1) Problem context; (2) Nature of problem situation, and (3) Problem-solving processes (see

Figure 1.1).

圖 1﹕PISA 2012問題解決能力評核框架

Figure 1.1

Main features of the PISA 2012 digital problem-solving assessment framework

Problem-

solving

processes

Problem context

Nature of problem

situation

Involves a technological device?

Relates to personal or social setting?

Interactive Static

Exploring &

understanding

Representing &

formulating

Planning &

executing

Monitoring &

reflecting

- 16 -

The problem context of the digital assessment is embedded in student’s everyday life. As an

example, the problem scenario may involve a technological device or relate to the personal (e.g.

student or family) or social environment (e.g. community or society) of the student. Regarding

the nature of the problem situation, there are two kinds of tasks depending on whether the

information needed to solve the problem is disclosed at the outset. For interactive tasks, not all

information is disclosed, and some information has to be uncovered by exploring the problem

situation. For static tasks, all relevant information for solving the problem is disclosed at the

outset. Regarding the problem-solving processes, there are four pertinent cognitive processes

needed examination: (1) Exploring and understanding the information provided with the

problem; (2) Constructing graphical, tabular, symbolic or verbal representations of the problem

situation and formulating hypotheses about the relevant factors and relationships between them;

(3) Devising a plan by setting goals and sub-goals, and executing the sequential steps identified

in the plan; (4) Monitoring progress, reacting to feedback, and reflecting on the solution, the

information provided with the problem, or the strategy adopted (OECD, 2014, p.31).

Compared with the PISA 2003 problem-solving assessment framework, the main

characteristics of the PISA 2012 problem-solving assessment are that more attention are paid

on the usage of technological devices and the tasks are more interactive than static in nature, so

as to keep in pace with the advances and potentials of the development of the ICT in the 21st

century.

- 17 -

1.4 An example of a test unit of digital problem-solving assessment tasks

Source: OECD (2014) and PISA released items website (http://erasq.acer.edu.au/)

Figure 1.2

An example of a released PISA 2012 digital problem-solving test unit (MP3 PLAYER)

The problem context of the above illustrative test unit MP3 PLAYER concerns with personal

usage of a technological device. The nature of the problem situation is that it entails

- 18 -

manipulation of the MP3 menus in an interactive manner. The whole test unit comprises 4

assessment tasks. Question 1 entails the problem solving process exploring and understanding,

requesting examinee to explore the MP3 menus system to select the correct type of music and

set the volume of the music. The problem goal is to understand a menus system of a

technological device. Question 2 pertains to the planning and executing of the problem-solving

process. Examinee is required to set the MP3 player to the Rock music, as well as the volume

and bass of the piece of music. Specifically, student needs to plan how to control the menu

system sequentially so as to achieve a given outcome. The problem-solving process of question

3 is representing and formulating. Examinee needs to demonstrate that a correct mental

representation of the menu system has been formed. In this regard, he/she needs to analyze the

various scenarios of the MP3 menu system, analyze the data so as to arrive at hypotheses, judge

which representation is not functioning properly, and finally choose the menu that should be the

correct answer. Last, Question 4 loads on monitoring and reflecting, examinee is asked to

suggest a modification to the design of the MP3 menu system. Examinee has to base on their

practical experiences regarding the usage of the technological device in order to reflect and

propose new ways to change the operation of the MP3. The coding guide of this test unit is

shown in Appendix 1 of this report.

1.5 Description of the proficiency levels of the digital problem-solving,

mathematics and reading scale

Test items (assessment tasks) of the same test unit are organized under the same stimulus of the

whole test unit (see the illustrative example shown in the previous section), and there are

several kinds of item response formats (e.g. multiple-choice, short-answer/extended constructed

response) in the construction of the test items. In multiple-choice item response format,

students are required to select the best or most correct answer amongst several options, and in

the constructed response item response format students provide their answers as requested. In

PISA 2012 digital assessment, each student was randomly assigned one of the 24 test forms

containing clusters of problem-solving, mathematics and/or reading test units delivered on

computer-based platforms which take approximately 40 minutes to complete.

Table 1.4 to 1.6 present descriptions of the proficiency levels of the literacy scales for digital

problem-solving, mathematics and reading respectively (OECD, 2014). All PISA literacy

- 19 -

proficiency scales have been calibrated on the sample responses sampled from the OECD

countries. The scales in PISA 2012 digital assessment were adjusted based on the responses to

the link items common to the previous PISA cycles of assessment, as well as that of the PISA

2012 print assessment. In the benchmarking assessments (i.e. PISA 2003 for mathematics,

PISA 2006 for science, PISA 2000 and PISA 2009 for reading) the mean scale score was

initially set at 500 and standard deviation at 100.

- 20 -

Table 1.3

Proficiency level descriptions of the digital problem-solving scale

Level What students can typically do at each level?

6

Students can develop complete, coherent mental models of diverse problem

scenarios, enabling them to solve complex problems efficiently. They can

explore a scenario in a highly strategic manner to understand all information

pertaining to the problem. The information may be presented in different

formats, requiring interpretation and integration of related parts. When

confronted with very complex devices, such as home appliances that work in an

unusual or an unexpected manner, they quickly learn how to control the devices

to achieve a goal in an optimal way. Level 6 problem-solvers can set up several

hypotheses about a system and thoroughly test them. They can follow premise

through to a logical conclusion or recognize when there is not enough

information available to reach one. In order to reach a solution, these highly

proficient problem-solvers can create complex, flexible, multi-step plans that

they continually monitor during execution. Where necessary, they modify their

strategies, taking all constraints into account, both explicit and implicit.

5

Students can systematically explore a complex problem scenario to gain an

understanding of how relevant information is structured. When faced with

unfamiliar, moderately complex devices, such as vending machines or home

appliances, they respond quickly to feedback in order to control the device. In

order to reach a solution, Level 5 problem-solvers think ahead to find the best

strategies that addresses all the given constraints. They can immediately adjust

their plans or backtrack when they detect unexpected difficulties or when they

make mistakes that take them off course.

4

Students can explore a moderately complex problem scenario in a focused way.

They grasp the links amongst the components of the scenario that are required

to solve the problem. They can control moderately complex digital devices,

such as unfamiliar vending machines or home appliances, but they don’t always

do so efficiently. These students can plan a few steps ahead and monitor the

progress of their plans. They are usually able to adjust these plans or

reformulate a goal in light of feedback. They can systematically try out

different possibilities and check whether multiple conditions have been

satisfied. They can form a hypothesis about why a system is malfunctioning

and describe how to test it.

- 21 -

Note: There is an additional “below 1” level for those students who cannot attain at the lowest level (i.e. level 1)

in the digital problem-solving competence scale.

3

Students can handle information presented in several different formats. They

can explore a problem scenario and infer simple relationships among its

components. They can control simple digital devices, but have trouble with

more complex devices. Problem-solvers at level 3 can fully deal with one

condition, for example, by generating several solutions and checking to see

whether these satisfy the condition. When there are multiple conditions and

inter-related features, they can hold one variable constant to see the effect of

change on the other variables. They can devise and execute tests to confirm or

refute a given hypothesis. They understand the need to plan ahead and monitor

progress, and are able to try a different option if necessary.

2

Students can explore an unfamiliar problem scenario and understand a small

part of it. They try, but only partially succeed, to understand and control digital

devices with unfamiliar controls, such as home appliances and vending

machines. Level 2 problem-solvers can test a simple hypothesis that is given to

them and can solve a problem that has a single, specific constraint. They can

plan and carry out one step at a time to achieve a sub-goal, and have some

capacity to monitor overall progress towards a solution.

1

Students can explore a problem scenario only in a limited way, but tend to do so

only when they have encountered very similar situations before. Based on their

observations of familiar scenarios, these students are able only to partially

describe the behavior of a simple, everyday device. In general, students at level

1 can solve straightforward problems provided there is a simple condition to be

satisfied and there are only one or two steps to be performed to reach the goal.

Level 1 student tends not to be able to plan ahead or set sub-goals.

- 22 -

Table 1.4

Proficiency level descriptions of the digital mathematics scale


6

Students can conceptualize, generalize, and utilize information based on

their investigations and modeling of complex problem situations, and can

use their knowledge in relatively non-standard contexts. They can link

different information sources and representations and flexibly translate

among them. Students at this level are capable of advanced mathematical

thinking and reasoning. These students can apply this insight and

understandings along with a mastery of symbolic and formal mathematical

operations and relationships to develop new approaches and strategies for

attacking novel situations. Student at this level can reflect on their actions,

and can formulate and precisely communicate their actions and reflections

regarding their findings, interpretations, arguments, and the

appropriateness of these to the original situation.

5

Students can develop and work with models for complex situations,

identifying constraints and specifying assumptions. They can select,

compare, and evaluate appropriate problem solving strategies for dealing

with complex problems related to these models. Students at this level can

work strategically using broad, well-developed thinking and reasoning

skills, appropriate linked representations, symbolic and formal

characterizations, and insight pertaining to these situations. They begin to

reflect on their work and can formulate and communicate their

interpretations and reasoning.

4

Students can work effectively with explicit models for complex concrete

situations that may involve constraints or call for making assumptions.

They can select and integrate different representations, including

symbolic, linking them directly to aspects of real-world situations.

Students at this level can utilize their limited range of skills and can

reason with some insight, in straightforward contexts. They can construct

and communicate explanations and arguments based on their

interpretations, arguments, and actions.

3

Students can execute clearly described procedures, including those that

require sequential decisions. Their interpretations are sufficiently sound to

be a base for building a simple model or for selecting and applying simple

problem solving strategies. Students at this level can interpret and use

representations based on different information sources and reason directly

from them. They typically show some ability to handle percentages,

fractions and decimal numbers, and to work with proportional

relationships. Their solutions reflect that they have engaged in basic

interpretation and reasoning.

- 23 -

Note: There is an additional “below 1” level for those students who cannot attain at the lowest level (i.e.

level 1) in the digital mathematical literacy scale.

2

Students can interpret and recognize situations in contexts that require no

more than direct inference. They can extract relevant information from a

single source and make use of a single representational mode. Students at

this level can employ basic algorithms, formulae, procedures, or

conventions to solve problems involving whole numbers. They are

capable of making literal interpretations of the results.

1

Students can answer questions involving familiar contexts where all

relevant information is present and the questions are clearly defined.

They are able to identify information and to carry out routine procedures

according to direct instructions in explicit situations. They can perform

actions that are almost always obvious and follow immediately from the

given stimuli.

- 24 -

Table 1.5

Proficiency level descriptions of the digital reading scale

Note: There is an additional “below 2” level for those students who cannot attain at the lowest level (i.e.

level 2) in the digital reading literacy scale.


Above

4

Tasks at this level typically require the reader to locate, analyze and

critically evaluate information, related to an unfamiliar context, in the

presence of ambiguity. They require the generation of criteria to evaluate

the text. Tasks may require navigation across multiple sites without

explicit direction, and detailed interrogation of texts in a variety of

formats.

4

Tasks at this level may require the reader to evaluate information from

several sources, navigating across several sites comprising texts in a

variety of formats, and generating criteria for evaluation in relation to a

familiar, personal or practical context. Other tasks at this level demand that

the reader construe complex information according to well-defined criteria

in a scientific or technical context.

3

Tasks at this level require that the reader integrate information, either by

navigating across several sites to find well-defined target information, or

by generating simple categories when the task is not explicitly stated.

Where evaluation is called for, only the information that is most directly

accessible or only part of the available information is required.

2

Tasks at this level typically require the reader to locate and interpret

information that is well-defined, usually relating to familiar contexts. They

may require navigation across a limited number of sites and the application

of web-based tools such as dropdown menus, where explicit directions are

provided or only low-level inference is called for. Tasks may require

integrating information presented in different formats, recognizing

examples that fit clearly defined categories.

- 25 -

Chapter 2

A Profile of Digital Literacy Performance for

15-year-olds in Macao

Abstract

This chapter recapitulates the key results, particularly those pertaining to Macao, reported in

the PISA 2012 Study International Report (OECD, 2013, 2014). It details the profiles of student

performance in digital problem-solving, mathematics and reading literacy broken down by

gender. It presents the relationships between Macao student performance in print and digital

assessment on mathematics and reading so as to allow the readers to have a glimpse of the

differences between the two modes of assessment (i.e. print vs. digital) in PISA 2012. From an

international comparison perspective, this chapter highlights a number of countries/economies

that may serve as exemplary models for Macao’s educational improvement and curriculum

reform.

2.1 Macao 15-year-olds’ digital literacy performance

Table 2.1 presents Macao 15-year-olds’ performance results in the three domains of digital

assessment, i.e. problem-solving, mathematics and reading literacy, broken down by gender.

Table 2.1

Macao’s 15-year-olds’ digital literacy performance results

Descriptive

Statistics

Digital Literacy

Problem-solving Mathematics Reading

Total = 3147

Mean 540.5 542.9 515.3

SD 79.2 82.9 70.5

Males = 1570

Mean 545.6 549.1 506.3

SD 80.8 86.6 73.1

Females = 1577

Mean 535.1 536.4 524.7

SD 77.1 78.2 66.3 Note: In all the PISA literacy scales, the items have been calibrated against the participating OECD

countries, with mean score set at 500 and standard deviation at 100.

As seen in Table 2.1, from a comparative education perspective Macao’s 15-year-olds

- 26 -

performed very well in digital problem-solving and quite good in digital mathematics literacy

(mean = 540.5 and 542.9 respectively). Performance on digital reading is also very satisfactory

(mean = 515.3). Generally speaking, Macao females outperform males in digital reading (524.7

vs. 506.3), whereas it is the other way round for the males in digital mathematics (536.4 vs.

549.1) and problem-solving (535.1 vs. 545.6) literacy. All standard deviations (SD) are low

(ranges from 66.3 to 86.6), showing that student performance on the three digital literacy scales

are quite homogeneous as compared with the average of the OECD countries participating in

the PISA 2012 digital assessment.

Each sampled student in the PISA 2012 digital assessment is assigned to the highest proficiency

level for which she/he would be expected to answer correctly the majority of the assessment

items. From the PISA perspective, students classified as “Below level 2” in the digital literacy

assessment are unable to demonstrate competency in situations required by the easiest digital

literacy items, and therefore they are regarded as at a disadvantage for full participation in the

knowledge society in the information age. Table 2.2 presents the frequency distribution of

Macao’s 15-year-old student proficiency levels of the digital problem-solving, mathematics and

reading literacy scales, broken down by gender.

- 27 -

Table 2.2

Distribution of Macao 15-year-olds’ proficiency levels on the digital literacy scales

Proficiency

Level

% of Students

Problem-solving Mathematics Reading

Total =3147

6 2.8 5.6 5.1

5 13.8 16.6 4 28.9 28.5 25.3 3 29.5 26.4 39.8 2 17.5 15.3 22.8 1 6.0 5.9

7.0 Below 1 1.6 1.7

Males=1570

6 3.5 7.3 4.5

5 15.6 18.7 4 29.2 28.1 22.5 3 27.9 24.3 38.1 2 16.7 13.9 25.5 1 5.6 5.9

9.4 Below 1 1.5 1.8

Females=1577

6 2.0 3.8 5.8

5 12.0 14.4 4 28.6 29.0 28.3 3 31.1 28.6 41.5 2 18.4 16.8 20.0 1 6.4 5.9

4.4 Below 1 1.6 1.5

As seen in Table 2.2, both male and female students’ digital problem-solving and mathematics

proficiency levels are mainly concentrated at levels 3 and 4, totaling 58.4% and 54.9% of the

sampled students respectively. For digital reading, the concentration is mainly at level 2, 3 and

4, totaling 87.9% of the sampled students. Percentages of students with proficiency below level

2 for the three digital literacy scales remain at very low levels (<8%), showing that the number

of low-performing students who cannot function productively in society is small. Unfortunately,

the numbers of high-performing students with proficiency level 6 in digital problem-solving

and mathematics literacy respectively are not high compared with the top-performing countries

in PISA 2012 digital assessment (see Figure 2.4 & 2.7). In the case of digital reading the

situation is also not that favorable as Macao has only 5.1% of its students assessed at

proficiency level 5 and 6 (see Figure 2.6).

- 28 -

Figure 2.1 and Figure 2.2 show further the percentages of 15-year-olds at different digital

problem-solving proficiency levels across grades in the Macao sample. In 2012, similar to that

of the print assessment there is a clear relationship between grade level and digital literacy

performance in the sample of 15-year-old students in Macao.

Figure 2.1

Percentage of 15-year-olds at different digital problem-solving proficiency levels across grades

in the Macao sample

Figure 2.2

Percentage of 15-year-olds at different grade levels across digital problem-solving proficiency

levels in the Macao sample

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

Grade 7 Grade 8 Grade 9 Grade 10 Grade 11

below level 1

level 1

level 2

level 3

level 4

level 5

level 6

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

40.0

below

level 1

level 1 level 2 level 3 level 4 level 5 level 6

Grade 7

Grade 8

Grade 9

Grade 10

Grade 11

- 29 -

2.2 An international comparison of performance in the three digital literacy

domains

One special design feature of the PISA 2012 international assessment is that across cycles of

PISA assessment one can compare student performance between print problem-solving in PISA

2003 and digital problem-solving in 2012. Another design feature of no less importance is that

in PISA 2012 student performance in both print and digital mathematics and reading are

assessed at the same occasion. The findings pave way for the move to change from print to

digital assessment in PISA 2015 and beyond.

2.2.1 A comparison between print versus digital problem-solving performance of 15-years-olds

in PISA 2003 and PISA 2012

In PISA 2012 Macao ranks fourth in digital problem-solving amongst the 44 participating

economies. Though lacking behind Singapore, Korea and Japan the ICT education of which

are relatively well-developed, Macao adolescents’ digital problem-solving performance

(mean=540.5) is comparable in level with that of its other three Chinese-speaking

counterparts, i.e. Hong Kong, Shanghai and Chinese Taipei (see Table 2.3). Noteworthy is that

in PISA 2003 when print problem-solving was the minor domain of assessment, Macao

ranked sixth amongst the 41 participating countries/economies. After considering the

sampling and measurement errors, Macao ranks 5th

to 10th

in PISA 2003 print

problem-solving and 4th

to 6th

in PISA 2012 digital problem-solving assessment. Therefore in

the last decade when the assessment mode changes from print to digital, Macao’s 15-year-olds

still maintain their superiority in problem-solving ability, which is an important 21st century

skills needed inculcation in the school curriculum.

- 30 -

Table 2.3 Problem-solving performance of 15-years-olds in PISA 2003 and PISA 2012

PISA 2003 PISA 2012

Rank Print score Rank Digital score

1. Korea 550.4 1. Singapore 562.4

2. Hong Kong-China 547.9 2. Korea 561.1

3. Finland 547.6 3. Japan 552.2

4. Japan 547.3 4. Macao-China 540.5

5. New Zealand 532.8 5. Hong Kong-China 539.6

6. Macao-China 532.4 6. Shanghai-China 536.4

7. Australia 529.9 7. Chinese Taipei 534.4

8. Liechtenstein 529.5 OECD Average 500.1

9. Canada 529.3

10 Belgium 525.3

OECD Average 500.0

(41 participating economies) (44 participating economies)

Note: After considering the sampling and measurement errors, Macao ranks 5-10 in PISA 2003 print

problem-solving and 4-6 in PISA 2012 digital problem-solving assessment.

2.2.1 A comparison between print versus digital mathematics literacy performance of

15-years-olds in PISA 2012

Macao ranks fifth in digital mathematical literacy amongst the 32 participating economies in

PISA 2012 digital assessment. Though lacking behind Singapore and Shanghai, Macao

adolescents’ digital mathematical literacy performance (mean = 542.9) is comparable to that

of Korea, Hong Kong, Japan, and Chinese Taipei (see Table 2.4). Noteworthy is that in the

print assessment of PISA 2012 when mathematical literacy was the major domain of

assessment, Macao ranked sixth amongst the 65 participating countries/economies. After

considering the sampling and measurement errors, Macao ranks 6th

to 8th

in print mathematics

and 3rd

to 7th

in digital mathematics of the PISA 2012 international assessment. Therefore, the

finding is that Macao’s 15-year-olds not only have maintained but also increased their

superiority in mathematical literacy when the assessment mode changed from print to the

digital.

- 31 -

Table 2.4

Mathematics literacy performance of 15-years-olds in PISA 2012


1. Shanghai-China 612.7 1. Singapore 566.0

2. Singapore 573.5 2. Shanghai-China 562.3


4. Chinese Taipei 559.8 4. Hong Kong-China 549.6

5. Korea 553.8 5. Macao-China 542.9

6. Macao-China 538.1 6. Japan 539.0

7. Japan 536.4 7. Chinese Taipei 537.3

8. Liechtenstein 535.0 OECD Average 497.2

OECD Average 494.0

(65 participating economies) (32 participating economies)

Note: After considering the sampling and measurement errors, Macao ranks 6-8 in print and 3-7 in digital

mathematics literacy in PISA 2012.

2.2.2 A comparison between print versus digital reading literacy performance of 15-years-olds

in PISA 2012

Macao ranks eleventh in digital reading literacy amongst the 32 participating economies in

PISA 2012 digital assessment. Though lacking behind its Asian Pacific counterparts like

Singapore, Korea, Japan, Hong Kong and Shanghai, and western counterparts like Canada,

Estonia and Australia, Macao adolescents’ digital reading performance is comparable in level

(mean= 515.3) with that of Chinese Taipei, Ireland, United States and France (see Table 2.5).

Noteworthy is that in the print assessment of PISA 2012 when reading literacy was one minor

domain of assessment, Macao ranked 18th

amongst the 65 participating countries/economies.

After considering the sampling and measurement errors, Macao ranks 12th

to 22nd

in print

reading and 9th

to 13th

in digital reading of the PISA 2012 international assessment. Therefore,

the finding is that Macao’s 15-year-olds not only have maintained but also greatly increased

their superiority in reading literacy when the assessment mode changed from print to the

digital.

- 32 -

Table 2.5

Reading literacy performance of 15-years-olds in PISA 2012


1. Shanghai-China 569.6 1. Singapore 567.0


3. Singapore 542.2 3. Hong Kong-China 549.8

4. Japan 538.1 4. Japan 544.8

5. Korea 535.8 5. Canada 532.3

6. Finland 524.0 6. Shanghai-China 531.3

7. Ireland 523.2 7. Estonia 522.8

8. Chinese Taipei 523.1 8. Australia 520.6

9. Canada 523.1 9. Ireland 520.2

10. Poland 518.2 10. Chinese Taipei 519.5

11. Estonia 516.3 11. Macao-China 515.3

12. Liechtenstein 515.5 12. United States 511.2

13. New Zealand 512.2 13. France 510.9

14. Australia 511.8 OECD Average 496.9

15. Netherlands 511.2

16. Belgium 509.1

17. Switzerland 509.0

18. Macao-China 508.9

19. Vietnam 508.2

20. Germany 507.7

21. France 505.5

22. Norway 503.9

OECD Average 496.5

(65 participating economies) (32 participating economies) Note: After considering the sampling and measurement errors, Macao ranks 12-22 in print and 9-13 in digital

reading literacy in PISA 2012.

2.3 Implications for Macao’s stake-holders when mode of PISA assessment

changes from print to the digital

Due to the full-fledged implementation of digital assessment of pertinent literacies in PISA

2015 and beyond, there are three implications of the findings for the various stakeholders of

Macao’s basic education to take note of.

First, the digital mathematical literacy performance of the high-performing (i.e. >level 4)

students is less well than their counterparts on print mathematical literacy performance (see

Figure 2.3). The reverse is true for those students with proficiency levels below 4. There is a

drop in the percentage of high-performing students from 24.4% to 22.2% when the mode of

- 33 -

PISA assessment is shifted from print to digital. Concomitantly, there is a drop in the

percentage of low-performing (i.e. below level 2) students from 10.8% to 7.6% (see Table

2.6). It is reckoned that the three high-performing countries, i.e. Singapore, Korea and Japan

also suffer and benefit from this shift in mode of PISA assessment (see Figure 2.4).

Figure 2.3

QQ Plot of print versus digital mathematical literacy in PISA 2012

- 34 -

Table 2.6 Distribution of mathematical literacy proficiency levels of

Macao’s 15-year-olds in PISA 2012

Proficiency Level % of Students

Digital Print

Total 3,147 5,335

6 5.6 7.6

5 16.6 16.8

4 28.5 24.4

3 26.4 24.0

2 15.3 16.4

1 5.9 7.6

Below 1 1.7 3.2

Males 1,570 2,731

6 7.3 8.2

5 18.7 17.9

4 28.1 23.5

3 24.3 23.4

2 13.9 15.5

1 5.9 7.8

Below 1 1.8 3.7

Females 1,577 2,604

6 3.8 6.9

5 14.4 15.6

4 29.0 25.5

3 28.6 24.6

2 16.8 17.5

1 5.9 7.3

Below 1 1.5 2.7

- 35 -

Figure 2.4

Percentage of 15-year-olds for Macao and selected economies at each proficiency level of

mathematical literacy in PISA 2012 (Digital vs. Print)

- 36 -

Second, the digital reading literacy performance of the medium to high performing students

(i.e.>level 3) performed less well than their counterparts on print reading literacy performance

(see Figure 2.5). The reverse is true for those students with proficiency levels below 3. There

is a drop in the percentage of high-performing students (i.e. levels 5 and 6) from 7.0% to 5.1%

when the mode of PISA assessment is shifted from print to digital. Concomitantly, there is a

drop in the percentage of low-performing students (i.e. < level 2) from 11.4% to 7.0% (see

Table 2.7). It is reckoned that the three high-performing economies, i.e. Singapore, Korea,

Japan and Hong Kong their high-performing students do not suffer from this shift in mode of

PISA assessment (see Figure 2.6).

Figure 2.5

QQ Plot of print versus digital reading literacy in PISA 2012

- 37 -

Table 2.7

Distribution of reading literacy proficiency levels of Macao’s 15-year-olds in PISA 2012

Proficiency Level % of Students

Digital Print

Total 3,147 5,335

6 0.2 0.6

5 4.9 6.4

4 25.3 24.0

3 39.8 34.3

2 22.8 23.3

1a 6.1 9.0

1b 0.8 2.1

Below 1b 0.1 0.3

Males 1,570 2,731

6 0.2 0.3

5 4.3 4.4

4 22.5 19.6

3 38.1 33.0

2 25.5 26.2

1a 8.1 12.5

1b 1.2 3.4

Below 1b 0.1 0.6

Females 1,577 2,604

6 0.2 0.8

5 5.6 8.6

4 28.3 28.7

3 41.5 35.6

2 20.0 20.2

1a 3.9 5.3

1b 0.5 0.8

Below 1b 0.0 0.0

- 38 -

Figure 2.6


reading literacy in PISA 2012 (Digital vs. Print)

- 39 -

Third, In PISA 2012, there are altogether 6 proficiency levels in the digital problem-solving

scale. Students who cannot reach the lowest level (i.e. level 1) are regarded as disadvantaged,

and they run the risks of being unable to function productively in the life-long learning society

in the 21st Century. Only a low level of 1.6% of the students is thus seriously at risk. Students

performing below level 2 are regarded as low-performers, about 7.6% of Macao’s

15-year-olds performed at this low level. Students who can reach the top two levels (i.e. level

5 and 6) are crowned as high-performers. They are cherished as valuable talents who are

much needed in nowadays knowledge society. In Macao, close to 17% of the adolescents are

high performers in digital problem-solving (see Table 2.8). Compared with our other three

Chinese-speaking counterparts (i.e. Shanghai, Hong Kong and Chinese Taipei), Macao’s

advantage is to have a slightly lower percentage of students with low digital problem-solving

ability, though its disadvantage is to have a slightly lower percentage of students with high

digital problem-solving ability (see Figure 2.7). According to the PISA International Digital

Problem-solving Report (OECD, 2014), Macao is one of the four ICT-developed places in the

world whose students are close to meet the goal to use basic problem-solving tools to meet

unknown everyday challenges. Problem-based learning in Macao schools should therefore be

quite promising.

- 40 -

Table 2.8

Distribution of digital problem-solving proficiency levels of Macao’s 15-year-olds

in PISA 2012

Proficiency Level % of students

Total = 3,147

6 2.8

5 13.8

4 28.9

3 29.5

2 17.5

1 6.0

Below 1 1.6

Males =1,570

6 3.5

5 15.6

4 29.2

3 27.9

2 16.7

1 5.6

Below 1 1.5

Females = 1,577

6 2.0

5 12.0

4 28.6

3 31.1

2 18.4

1 6.4

Below 1 1.6

- 41 -

Figure 2.7


digital problem solving in PISA 2012

2.0 2.1 1.8 1.6 3.3 3.1 3.4 6.5 8.2 6.0 4.8 5.3 6.0 7.1 7.5 8.2

14.1 13.2 13.8 12.9 14.6 17.5 16.3 17.5 17.8

25.5 22.0 21.9 23.7 26.9

29.5 27.4 27.4 26.3

28.1 25.6 27.0 28.8

29.2

28.9 26.5 26.2 25.9

18.4 19.6 19.7 20.0

16.9 13.8

14.2 14.1 14.6

6.2 8.9

9.6 7.6 5.3 2.8 5.1 4.1 3.8 1.2 2.5

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Singapore Korea Japan Macao-

China

Hong

Kong-

China

Shanghai-

China

Chinese

Taipei

Portugal OECD

average

level 6

level 5

level 4

level 3

level 2

level 1

below level 1

- 42 -

Chapter 3

Relationships between Digital Literacy Performance and ESCS for

Macao Schools

Abstract

This chapter analyzes the intricate relationships between digital literacy performance and ESCS

(i.e. PISA index of economic, social and cultural status of the home) so as to throw lights on the

equity dimension of educational provision for 15-year-olds in Macao.

3.1 Plots of digital literacy performance with ESCS in the Macao sample

As seen in Figure 3.1, there are almost linear relationships in the Macao sample between

student’s digital problem-solving, mathematics and reading literacy performance with

economic, social and cultural status (ESCS) of the home. Generally speaking, higher ESCS is

associated with higher digital problem-solving, mathematical and reading performance.

Figure 3.1

Plots of digital literacy performance with ESCS

350

400

450

500

550

600

650

700

-2.50 -2.00 -1.50 -1.00 -0.50 0.00 0.50 1.00

Digital problem solving ability Digital mathematics literacy

Digital reading literacy

Index of Economic, Social and Cultural Status (ESCS)

- 43 -

Although the impact of ESCS on digital problem-solving, mathematics and reading literacy is

not pronounced by international standard (OECD, 2013), elevating homes of low ESCS to the

higher levels are always desirable. In the long run, this will bring about educational

opportunities for the students and at the same time increase their digital problem-solving,

mathematics and reading literacy levels.

3.2 Relationships of school digital literacy performance with school ESCS

Table 3.1 presents the school mean of the digital problem-solving, mathematics and reading

literacy performance score of the sampled students in Macao, as well as the mean of the ESCS

of each of the 45 participating schools.

Table 3.1

Digital literacy performance and ESCS of participating schools

School ID Problem Solving Mathematics Reading ESCS

1 490.13 496.47 459.38 -1.337

2 472.17 491.35 479.93 -1.342

3 523.86 524.47 541.51 0.018

4 588.27 571.70 497.40 -0.728

5 554.91 549.41 540.38 -0.722

6 538.61 518.95 534.04 -1.124

7 564.56 606.80 557.95 -0.866

8 454.20 441.56 439.74 -0.850

9 561.28 570.43 520.71 -1.088

10 557.11 565.45 544.25 -1.074

11 556.13 538.31 512.08 -1.207

12 602.72 613.08 562.47 -1.110

13 545.31 559.22 532.58 -1.299

14 520.60 523.30 509.82 -1.262

15 573.90 558.54 531.64 -0.796

16 554.18 535.15 517.97 -1.310

17 536.27 505.58 488.22 -1.366

18 568.98 579.12 550.38 -0.071

19 457.56 488.47 469.91 -1.189

20 556.01 575.32 526.82 -0.895

21 507.88 479.79 454.99 -0.743

22 544.15 541.39 538.09 -1.350

- 44 -

23 471.30 453.85 451.71 -1.026

24 498.15 488.78 496.34 -1.196

25 497.03 470.03 459.66 -1.135

26 508.86 513.41 499.41 -1.222

27 541.42 551.31 526.34 -0.983

28 563.56 562.91 545.40 -0.940

29 461.33 461.39 440.09 -1.437

30 527.59 542.37 443.44 -0.597

31 489.05 459.79 487.88 -0.175

32 557.59 581.73 502.85 -0.530

33 465.59 429.52 441.20 -1.028

34 573.21 559.11 571.53 -0.344

35 496.23 499.37 475.18 -1.462

36 428.92 393.90 457.83 -0.781

37 585.30 603.31 575.85 0.098

38 506.47 527.95 495.48 -0.760

39 560.72 557.81 523.23 -0.395

40 554.18 550.40 509.85 -0.089

41 551.26 562.15 527.30 0.218

42 564.30 561.32 551.98 0.123

43 535.31 591.19 591.35 0.162

44 518.99 528.41 530.00 0.937

45 526.70 532.44 504.06 -0.859

Macao Mean 540.46 542.90 515.26 -0.886

Based on the data shown in Table 3.1, the school performance-ESCS relationship for each of

the three digital literacy measures may be plotted (see Figure 3.2, 3.3 and 3.4). It is observed

that school digital literacy performance is not related to school ESCS at the low end of the

ESCS continuum (say ESCS <-0.50). However, at the higher end (say ESCS > 0.00), the

literacy performance of the schools are very favorable (i.e. above the OECD average).

Regarding this, although Macao is famed worldwide over the years for its very high level of

educational equity there is very slight sign of educational inequity present in Macao’s basic

education system. How to help students of disadvantaged homes for better learning

opportunities and educational provision deserves the attention of the researchers and policy

makers.

- 45 -

Figure 3.2

Plot of school digital problem-solving performance with school ESCS

Figure 3.3

Plot of school digital mathematical literacy performance with school ESCS

- 46 -

Figure 3.4

Plot of school digital reading performance with school ESCS

Same as the previous three cycles of PISA assessment for the print medium, the slope of the

digital literacy performance and ESCS relationship is gentle and the percentage of total digital

literacy performance variance explained by the PISA index of economic, social and cultural

status (ESCS) of the home is amongst the lowest of the 44 participating economies (OECD,

2014). Therefore, Macao’s basic education system replicates the findings of previous cycles of

assessment in succeeding to provide equitable schooling opportunities for the student body it

served.

- 47 -

Chapter 4

The 15-year-olds’ Performance on the

Released Digital Assessment Tasks

Abstract

This chapter presents 15-year-old students’ performance on a number of released digital

problem-solving and mathematics assessment tasks. A number of high-performing economies

in PISA 2012 are also included for purposes of international comparison. Readers of this report

are able to see how the released assessment tasks (4 digital problem-solving and 3 digital

mathematics test units) relate to the PISA 2012 assessment framework, as well as examine the

strengths and weaknesses of the performance of Macao’s students in comparison with her

high-performing counterparts.

As an instance, CLIMATE CONTROL is an example of a digital problem-solving task

generally done well by Macao students. In this interactive task, students need to find out which

of the three controls of an air-conditioner influence temperature and humidity by experimenting

with the various control levels of the sliders. Macao students perform less well than their

counterparts from Singapore, Korea and Japan, but a little bit better than students from Hong

Kong, Shanghai and Chinese Taipei (see Table 4.2).

TICKETS is an example of a digital problem-solving task poorly done by Macao students. In

this interactive task, students need to buy a train ticket on an automated ticketing machine.

Macao students lack this kind of daily life experience, and only a meagre 26 percent of students

can correctly buy a full fare country trip ticket with two individual trips (see Table 4.4). On the

contrary, some eighty percent of the Japanese students are able to do these kinds of interactive

tasks successfully.

- 48 -

4.1 Digital problem solving tasks

There are 4 test units released after the PISA 2012 Digital Assessment, namely: (1) CLIMATE

CONTROL, (2) TICKETS, (3) ROBOT CLEANER, and (4) TRAFFIC. The blueprint used in

the design of digital problem-solving assessment tasks is presented in Table 4.1 below.

Table 4.1

Blueprint used in the design of digital problem-solving assessment tasks

Nature and Context of

the Problem Situations

Problem-solving Processes

Exploring &

Understanding

Representing &

Formulating

Planning &

Executing

Monitoring &

Reflecting

Interactive

Technology

CLIMATE

CONTROL

CP025Q01#

CLIMATE

CONTROL

CP025Q02#

TICKETS

CP038Q01*

TICKETS

CP038Q02*

TICKETS

CP038Q03*

Non-technology

Static

Technology

Non-technology ROBOT

CLEANER

CP002Q08*

ROBOT

CLEANER

CP002Q06*

TRAFFIC

CP007Q01*

TRAFFIC

CP007Q03*

ROBOT

CLEANER

CP002Q07*

TRAFFIC

CP007Q02*

Note: Tasks with intent on “Exploring & Understanding” and “Representing & Formulating” are known

as “Knowledge-acquisition” tasks, whereas those on “Planning & Execution” are termed as

“Knowledge-utilization” tasks

# Personal context;

* Social context.

The four released test units are exemplary of the main kinds of problem-solving tasks (i.e.

“interactive technological” versus “static non-technological”) in the PISA digital assessment.

Interested readers can access the following website to examine the contents of the digital test

units: http://www.oecd.org/pisa/test/

- 49 -

4.1.1 CLIMATE CONTROL (2 Questions)

Figure 4.1

PISA released digital problem solving task – CLIMATE CONTROL (Q1)

Table 4.2


CP025Q01

% of Response

0

(no credit)

1

(partial credit)

2

(full credit)

Missing Not

reached

Singapore 15.26 8.63 75.45 0.66 0.00

Korea 16.07 6.07 77.40 0.46 0.00

Japan 23.16 12.22 62.00 2.52 0.10

Macao-China 16.59 11.32 71.90 0.20 0.00

Hong Kong-China 20.62 13.23 65.02 1.13 0.00

Shanghai-China 20.19 14.90 64.67 0.24 0.00

Chinese Taipei 20.59 12.21 66.21 0.99 0.00

Canada 26.22 11.11 60.88 1.67 0.13

Australia 26.50 11.35 60.09 1.92 0.13

Finland 27.90 15.54 54.43 2.13 0.00

OECD average 31.83 11.80 53.24 2.94 0.20

- 50 -

Figure 4.2


Table 4.3


CP025Q02

% of Response

0

(no credit)

1

(partial Credit)

2

(full credit)

Missing Not

reached

Singapore 47.24 23.68 24.91 4.17 0.00

Korea 42.94 26.76 25.85 4.44 0.00

Japan 40.11 21.85 29.91 7.81 0.31

Macao-China 48.95 20.66 26.52 3.87 0.00

Hong Kong-China 39.28 24.29 28.07 8.36 0.00

Shanghai-China 55.45 22.58 19.38 2.59 0.00

Chinese Taipei 52.39 17.00 26.07 4.54 0.00

Canada 46.86 22.79 22.43 7.80 0.13

Australia 43.57 24.03 22.13 10.12 0.15

Finland 43.96 24.00 24.72 7.20 0.11

OECD average 49.33 21.13 17.28 11.98 0.28

- 51 -

4.1.2 TICKETS (3 Questions)

Figure 4.3

PISA released digital problem solving task – TICKETS (Q1)

Table 4.4


CP038Q02 % of Response

0 (no credit) 1 (full credit) Missing Not reached

Singapore 27.39 72.35 0.26 0.00

Korea 37.56 62.02 0.42 0.00

Japan 19.24 80.37 0.39 0.00

Macao-China 73.77 25.85 0.38 0.00

Hong Kong-China 53.17 44.95 1.88 0.00

Shanghai-China 58.23 40.26 1.52 0.00

Chinese Taipei 54.05 45.02 0.93 0.00

Canada 36.05 63.01 0.81 0.13

Australia 29.41 69.26 1.20 0.13

Finland 40.11 59.24 0.65 0.00

OECD average 40.22 57.96 1.69 0.13

- 52 -

Figure 4.4


Table 4.5


CP038Q01

% of Response

0

(no credit)

1

(partial Credit)

2

(full credit)

Missing Not

reached

Singapore 15.97 37.00 46.56 0.47 0.00

Korea 13.18 51.18 34.75 0.89 0.00

Japan 17.64 36.92 44.57 0.77 0.10

Macao-China 19.83 45.95 33.65 0.57 0.00

Hong Kong-China 12.95 37.52 48.09 1.45 0.00

Shanghai-China 20.96 63.60 14.13 1.31 0.00

Chinese Taipei 20.69 41.67 36.81 0.83 0.00

Canada 19.85 47.01 31.56 1.44 0.13

Australia 19.63 42.01 35.45 2.78 0.13

Finland 20.33 43.44 34.44 1.79 0.00

OECD average 24.07 46.20 26.94 2.62 0.18

- 53 -

Figure 4.5





Singapore 39.61 59.82 0.57 0.00

Korea 46.30 53.50 0.21 0.00

Japan 29.44 69.90 0.56 0.10

Macao-China 51.56 48.25 0.19 0.00

Hong Kong-China 46.94 51.79 1.27 0.00

Shanghai-China 61.72 37.28 1.01 0.00

Chinese Taipei 46.34 52.83 0.83 0.00

Canada 47.62 51.04 1.21 0.13

Australia 44.91 53.14 1.81 0.14

Finland 47.25 51.80 0.95 0.00

OECD average 55.19 42.81 1.80 0.20

- 54 -

4.1.3 ROBOT CLEANER (3 Questions)

Figure 4.6

PISA released digital problem solving task – ROBOT CLEANER (Q1)

Table 4.7


CP002Q08

% of Response

A

(no credit)

B

(no credit)

C

(full credit)

D

(no credit)

Missing Not

reached

Singapore 2.35 7.57 82.56 7.13 0.39 0.00

Korea 0.98 8.95 81.45 8.63 0.00 0.00

Japan 2.77 13.30 71.52 12.15 0.26 0.00

Macao-China 2.47 9.01 78.66 9.86 0.00 0.00

Hong Kong-China 2.28 8.50 79.46 9.22 0.52 0.00

Shanghai-China 2.40 14.91 75.39 6.84 0.48 0.00

Chinese Taipei 2.18 10.30 76.91 10.23 0.38 0.00

Canada 2.33 15.07 66.77 15.36 0.47 0.00

Australia 3.10 13.99 69.19 13.23 0.48 0.01

Finland 3.44 13.16 68.18 14.86 0.36 0.00

OECD average 3.83 19.35 63.06 12.86 0.81 0.10

- 55 -

Figure 4.7


Table 4.8


CP002Q07

% of Response

A

(no credit)

B

(full credit)

C

(no credit)

D

(no credit)

Missing Not

reached

Singapore 1.72 57.31 32.62 8.35 0.00 0.00

Korea 3.63 61.38 27.91 6.85 0.23 0.00

Japan 4.10 55.69 29.98 9.76 0.47 0.00

Macao-China 3.99 58.68 27.79 9.35 0.19 0.00

Hong Kong-China 4.54 60.24 26.76 8.35 0.12 0.00

Shanghai-China 4.13 57.03 27.93 10.43 0.48 0.00

Chinese Taipei 4.97 57.58 31.33 5.95 0.16 0.00

Canada 3.05 45.87 36.88 13.67 0.53 0.00

Australia 4.16 43.98 36.81 14.12 0.91 0.01

Finland 7.49 46.04 29.63 16.24 0.60 0.00

OECD average 6.13 46.80 31.52 14.36 1.10 0.09

- 56 -

Figure 4.8


Table 4.9


CP002Q06

% of Response

0

(no credit)

1

(partial credit)

2

(full credit)

Missing Not

reached

Singapore 6.92 64.98 25.45 2.65 0.00

Korea 13.57 62.40 22.98 1.05 0.00

Japan 20.33 51.29 22.81 5.57 0.00

Macao-China 13.74 58.98 21.71 5.56 0.00

Hong Kong-China 7.13 60.13 24.50 8.23 0.00

Shanghai-China 5.49 59.03 32.74 2.74 0.00

Chinese Taipei 6.29 57.16 32.83 3.72 0.00

Canada 12.50 64.98 21.16 1.37 0.00

Australia 6.00 66.89 24.97 2.11 0.03

Finland 13.59 65.98 17.20 3.23 0.00

OECD average 16.24 64.58 14.89 4.19 0.10

- 57 -

4.1.4 TRAFFIC (3 Questions)

Table 4.9

PISA released digital problem solving task – TRAFFIC (Q1)

Table 4.10


CP007Q01

% of Response

A

(full credit)

B

(no credit)

C

(no credit)

D

(no credit)

Missing Not

reached

Singapore 93.51 4.36 0.97 0.40 0.76 0.00

Korea 91.58 5.99 1.40 0.67 0.36 0.00

Japan 90.15 6.08 2.20 0.83 0.73 0.00

Macao-China 92.27 4.14 1.32 1.34 0.93 0.00

Hong Kong-China 89.41 7.48 1.80 0.92 0.39 0.00

Shanghai-China 89.79 6.80 1.71 0.96 0.74 0.00

Chinese Taipei 91.85 5.22 1.46 0.96 0.51 0.00

Canada 89.48 6.44 1.69 1.36 1.02 0.00

Australia 87.87 7.66 2.27 1.14 1.05 0.01

Finland 86.98 8.19 3.00 0.97 0.86 0.00

OECD average 86.42 7.85 3.09 1.64 0.98 0.03

- 58 -

Figure 4.10


Table 4.11




Singapore 16.69 81.03 2.28 0.00

Korea 13.43 84.99 1.59 0.00

Japan 13.00 83.07 3.92 0.00

Macao-China 17.44 80.87 1.69 0.00

Hong Kong-China 20.31 77.10 2.59 0.00

Shanghai-China 20.20 77.21 2.58 0.00

Chinese Taipei 19.36 78.72 1.92 0.00

Canada 20.21 76.62 3.17 0.00

Australia 19.64 75.59 4.67 0.09

Finland 17.37 78.56 4.07 0.00

OECD average 23.59 70.40 5.98 0.04

- 59 -

Figure 4.11


Table 4.12




Singapore 15.13 83.20 1.67 0.00

Korea 11.92 85.89 1.98 0.21

Japan 9.55 88.27 2.18 0.00

Macao-China 17.73 79.46 2.81 0.00

Hong Kong-China 17.32 80.73 1.95 0.00

Shanghai-China 20.81 77.12 2.07 0.00

Chinese Taipei 19.60 78.62 1.78 0.00

Canada 13.16 84.26 2.57 0.00

Australia 15.25 80.81 3.84 0.09

Finland 10.36 82.70 6.94 0.00

OECD average 16.59 78.12 5.21 0.08

- 60 -

4.2 Digital mathematics tasks

There are three mathematics test units released after the PISA 2012 digital assessment, namely:

(1) CD PRODUCTION, (2) STAR POINTS, and (3) BODY MASS INDEX. The blueprint used

in the design of digital mathematical literacy assessment tasks is presented in Table 7 below.

Table 4.13

Blueprint used in the design of digital mathematics assessment tasks Process Category

Formulate Employ Interpret

Content

Category

Quantity

MC

CD

PRODUCTION

CM015Q01@

CR

Space & Shape

MC

STAR POINTS

CM020Q02*

CM020Q04*

CR

STAR POINTS

CM020Q01*

CM020Q03*

Change &

Relationships

MC

CR CD

PRODUCTION

CM015Q02@

CD

PRODUCTION

CM015Q03@

Uncertainty &

Data

MC

BODY MASS

INDEX

CM038Q03#

CR

BODY MASS

INDEX

CM038Q05#

CM038Q06#

MC= Multiple Choices; CR = Constructed Response (Manual/Expert)

# Societal context;

@ Occupational context;

* Scientific context

- 61 -

The three released test units are exemplary of the various content and process categories in the

digital mathematical literacy assessment. Interested readers can access the website

(http://erasq.acer.edu.au/) to examine the contents of the digital mathematics test units.

http://erasq.acer.edu.au/)

- 62 -

4.2.1 CD PRODUCTION (3 Questions)

Figure 4.12

PISA released digital mathematics task – CD PROCUCTION (Q1)

Table 4.14


CM015Q01

% of Response

A

(no credit)

B

(no credit)

C

(full credit)

D

(no credit)

Missing Not

reached

Singapore 11.30 8.24 77.73 2.73 0.00 0.00

Shanghai-China 12.28 10.07 72.69 3.78 1.18 0.00

Korea 15.23 9.02 70.44 4.82 0.48 0.00

Hong Kong-China 6.54 13.77 74.86 4.47 0.35 0.00

Macao-China 12.41 10.55 69.91 6.74 0.38 0.00

Japan 4.96 15.54 72.59 5.87 0.93 0.11

Chinese Taipei 9.42 14.16 69.79 6.08 0.54 0.00

Canada 17.40 9.98 65.49 5.40 1.34 0.39

Australia 19.01 15.22 59.40 5.66 0.50 0.21

Finland -- -- -- -- -- --

OECD average 17.24 14.97 58.94 7.57 1.15 0.13

- 63 -

Figure 4.13


Table 4.15


CM015Q02

% of Response

0

(no credit)

1

(partial Credit)

2

(full credit)

Missing Not

reached

Singapore 50.64 6.02 22.04 21.29 0.00

Shanghai-China 44.20 7.46 30.53 17.81 0.00

Korea 54.67 4.30 16.50 24.53 0.00

Hong Kong-China 62.51 3.76 12.64 21.09 0.00

Macao-China 64.95 4.05 12.98 18.02 0.00

Japan 46.09 5.52 11.85 36.44 0.11

Chinese Taipei 50.78 3.55 18.74 26.92 0.00

Canada 55.48 7.78 13.94 22.41 0.39

Australia 58.72 3.81 8.64 28.62 0.21

Finland -- -- -- -- --

OECD average 52.39 3.25 6.76 37.48 0.13

- 64 -

Figure 4.14


Table 4.16


CM015Q03

% of Response

0

(no credit)

1

(partial credit)

2

(full credit)

Missing Not

reached

Singapore 21.74 33.03 38.16 6.59 0.48

Shanghai-China 37.20 20.86 30.91 11.03 0.00

Korea 31.55 25.45 30.68 12.31 0.00

Hong Kong-China 34.57 25.10 23.42 16.92 0.00

Macao-China 38.51 29.84 22.35 9.31 0.00

Japan 21.85 37.30 21.68 18.92 0.25

Chinese Taipei 37.56 22.12 23.74 16.35 0.23

Canada 29.31 32.91 27.03 10.34 0.41

Australia 41.87 26.17 20.08 11.62 0.27

Finland -- -- -- -- --

OECD average 41.81 23.45 17.19 17.17 0.39

- 65 -

4.2.2 STAR POINTS (4 Questions)

Figure 4.15

PISA released digital mathematics task – STAR POINTS (Q1)

Table 4.17


CM020Q01

% of Response

0

(no credit)

1

(partial credit)

2

(full credit)

Missing Not

reached

Singapore 33.68 44.73 20.14 0.96 0.49

Shanghai-China 37.24 32.80 28.41 1.55 0.00

Korea 33.99 40.45 24.01 1.54 0.00

Hong Kong-China 40.61 43.36 13.50 2.54 0.00

Macao-China 39.43 44.42 14.61 1.36 0.19

Japan 31.68 30.72 31.17 6.29 0.14

Chinese Taipei 44.06 38.64 14.04 3.26 0.00

Canada 50.18 33.01 12.97 3.16 0.68

Australia 53.50 32.64 9.69 3.90 0.27

Finland -- -- -- -- --

OECD average 48.35 30.96 13.99 6.32 0.39

- 66 -

Figure 4.16


Table 4.18


CM020Q02

% of Response

A

(no credit)

B

(full credit)

C

(no credit)

D

(no credit)

Missing Not

reached

Singapore 9.26 57.37 12.77 18.13 1.98 0.49

Shanghai-China 12.50 50.50 14.11 22.61 0.29 0.00

Korea 11.29 52.13 9.38 26.75 0.45 0.00

Hong Kong-China 10.58 49.19 12.45 24.43 3.36 0.00

Macao-China 10.29 46.60 13.06 28.07 1.97 0.00

Japan 8.76 64.38 11.42 11.70 3.59 0.14

Chinese Taipei 7.64 53.59 13.83 21.90 3.04 0.00

Canada 8.86 52.26 11.55 22.74 4.19 0.40

Australia 9.26 48.92 12.02 24.43 5.04 0.34

Finland -- -- -- -- -- --

OECD average 10.85 47.22 12.32 24.84 4.34 0.44

- 67 -

Figure 4.17


Table 4.19


CM020Q03 % of Response


Singapore 57.46 37.57 4.48 0.49

Shanghai-China 65.41 32.70 1.88 0.00

Korea 65.68 30.91 3.42 0.00

Hong Kong-China 61.86 34.00 4.13 0.00

Macao-China 68.25 28.78 2.97 0.00

Japan 53.18 39.23 7.44 0.14

Chinese Taipei 66.21 28.13 5.67 0.00

Canada 62.65 30.66 6.28 0.41

Australia 66.12 25.33 8.10 0.45

Finland -- -- -- --

OECD average 63.15 26.78 9.56 0.51

- 68 -

Figure 4.18


Table 4.20


CM020Q04

% of Response

A

(no credit)

B

(no credit)

C

(full credit)

D

(no credit)

Missing Not

reached

Singapore 14.48 18.70 44.63 19.57 2.12 0.50

Shanghai-China 10.04 20.30 52.73 16.19 0.75 0.00

Korea 8.94 20.43 52.97 16.16 1.50 0.00

Hong Kong-China 9.00 19.67 48.67 20.17 2.49 0.00

Macao-China 13.20 24.00 42.65 18.17 1.98 0.00

Japan 7.94 17.93 56.75 13.55 3.69 0.14

Chinese Taipei 11.74 24.67 45.91 15.57 2.11 0.00

Canada 11.90 20.62 47.82 15.36 3.84 0.46

Australia 10.52 21.32 48.48 14.25 4.82 0.60

Finland -- -- -- -- -- --

OECD average 12.63 22.01 43.88 16.48 4.41 0.59

- 69 -

4.2.3 BODY MASS INDEX (3 Questions)

Figure 4.19

PISA released digital mathematics task – BODY MASS INDEX (Q1)

Table 4.21




Singapore 25.10 73.48 0.92 0.50

Shanghai-China 29.03 70.37 0.60 0.00

Korea 22.94 75.86 1.21 0.00

Hong Kong-China 30.80 68.37 0.51 0.32

Macao-China 34.99 63.63 1.38 0.00

Japan 22.03 76.93 0.89 0.14

Chinese Taipei 36.18 62.84 0.97 0.00

Canada 30.86 67.30 1.37 0.47

Australia 29.08 69.65 0.54 0.73

Finland -- -- -- --

OECD average 31.20 66.68 1.45 0.67

- 70 -

Figure 4.20


Table 4.22




Singapore 39.21 53.82 6.47 0.50

Shanghai-China 72.64 20.81 6.55 0.00

Korea 46.16 43.29 10.03 0.52

Hong Kong-China 37.89 50.60 11.19 0.33

Macao-China 68.37 22.69 8.93 0.00

Japan 60.10 24.80 14.95 0.14

Chinese Taipei 80.81 11.23 7.69 0.28

Canada 50.64 39.76 9.11 0.49

Australia 54.10 35.16 9.92 0.81

Finland -- -- -- --

OECD average 54.02 27.51 17.56 0.91

- 71 -

Figure 4.21


Table 4.23




Singapore 43.28 44.72 9.46 2.55

Shanghai-China 53.69 35.92 8.35 2.04

Korea 50.82 37.31 8.40 3.46

Hong Kong-China 50.91 32.17 12.78 4.14

Macao-China 66.27 18.63 12.15 2.95

Japan 57.15 24.55 13.72 4.59

Chinese Taipei 66.36 15.46 15.43 2.76

Canada 58.26 29.01 10.19 2.54

Australia 57.33 27.76 9.91 5.00

Finland -- -- -- --

OECD average 55.87 21.64 15.24 7.25

- 72 -

Chapter 5

Three Quality Education Indicators for

Improving Digital Problem Solving Ability of grade repeaters in

Macao Schools

Abstract

This chapter examines the association of three quality education indicators in PISA 2012 digital

assessment, namely, perseverance, openness for problem-solving, and ICT use at home for

school-related tasks with differing levels of proficiency of digital problem-solving and gender

of student, so as to provide empirical evidences to schools to help grade repeaters of both sexes

to advance in digital problem solving literacy.

5.1 Assessment of Perseverance, Openness for problem solving, and ICT use at

home for school-related tasks

In PISA 2012 digital assessment, two constructs pertaining to student problem-solving

experiences (i.e., perseverance and openness for problem-solving) are developed to relate to

student digital problem-solving performance. There is another construct, ICT use at home for

school-related tasks, developed for the same purpose. For Macao, the Pearson correlations of

these three constructs with digital problem-solving performance are 0.134, 0.248 and 0.159

respectively. In total, 7.7% of performance variance can be explained by the three constructs

taken together.

Evidences of student perseverance include: (1) When confronted with problems one gives up

easily; (2) One puts off difficult problems; (3) One remains interested in the tasks that one

starts; (4) One continues working on tasks until everything is perfect; and (5) When confronted

with a problem one does more than what is expected of him/her.

Evidences of student openness for problem-solving include: (1) One can easily link facts

together; (2) One is quick to understand things; (3) One seeks explanations for things; (4) One

can handle a lot of information; and (5) One likes to solve complex problems.

- 73 -

Evidences of student ICT use at home for school-related tasks include: (1) Browsing the

Internet for schoolwork; (2) Using email for communication with other students about

schoolwork; (3) Using email for communication with teachers and submission of homework or

other schoolwork; (4) Downloading, uploading or browsing materials from the school’s website;

(5) Checking the school’s website for announcements; (6) Doing homework on the computer;

and (7) Sharing school-related materials with other students.

In section 5.2, for purpose of illustration, “When confronted with problems one gives up

easily”, “One can easily link facts together” and “Browsing the Internet for schoolwork” will

be subjected to correspondence analyses so as to find out in greater depth how specific aspects

of perseverance, openness for problem-solving, and ICT use at home for school-related tasks

are associated with differing proficiency levels (i.e. high, middle, low) of digital

problem-solving ability (called DPS levels: 3-2-1) across gender of student (F=female,

M=male). Other aspects of perseverance, openness for problem-solving and ICT use at home

for school-related tasks are analyzed in a similar manner.

Students are regarded as top-performing in digital problem-solving (i.e. DPS level =3) when

the digital problem-solving proficiency level is 5 or 6, and bottom-performing (i.e. DPS level=1)

when the proficiency level is below the baseline (i.e. below level 2) (see Chapter 1.5 for the

description of the proficiency levels). Students are regarded as medium-performing in digital

problem-solving (i.e. DPS level =2) when the digital problem-solving proficiency level is 2, 3

or 4. Thus, F1 and M1 respectively are two disadvantaged groupings of females and males

whose DPS are at the bottom level (=1). On the contrary, F3 and M3 respectively are two

advantaged groupings of females and males whose DPS is at the top level (=3). F2 and M2 are

the rest of the six groupings with DPS in-between.

5.2 Correspondence analysis of “Perseverance – Give up easily”

Correspondence analysis is conducted on the two-way (6 x 5) contingency table, so that the 6

rows refer to the six Gender x DPS level grouping of students, and the 5 columns refer to the 5

categories of the Likert response scale used to answer the perseverance scale item (i.e. Give up

easily). The 5-point Likert response scale ranges from “Very much like me” to “Not at all like

me”.

- 74 -

The row profiles in Table 5.1 show that across the 6 groupings of Macao sampled students (i.e.

F1, F2, F3, M1, M2 and M3), “Very much like me” is more likely to correspond with F1 or M1

(compare 0.094 and 0.078 with 0.035). “Mostly like me” is also in a similar situation as that of

“Very much like me”. In a similar vein, “Not at all like me” is more likely to correspond with

F3 or M3. “Not much like me” is also in a similar situation as that of “Not at all like me”. Last,

“Somewhat like me” is more likely to correspond with F1 or F2.

The column profiles in Table 5.1 show that across the 5 groupings of Macao’s sampled students

with different degrees of perseverance rated on the 5-point Likert scale, F1 is more likely to

correspond with “Very much like me” (compare 0.081 with 0.030). F2 is more likely to

correspond with “Mostly like me” or “Somewhat like me”. F3 is more likely to correspond with

“Not much like me” or “Not at all like me”. M1, like F1, is more likely to correspond with

“Very much like me”. M2, a bit strange in response behavior, is more likely to correspond with

“Very much like me” or “Not at all like me”. Last M3, like F3 is more likely to correspond with

“Not much like me” or “Not at all like me”.

Table 5.1

Correspondence analysis of “Perseverance – Give up easily”: Row & column profiles Row Profiles

Gender x

DPS Level

(3 levels)

ST93Q01: When confronted with problems, I give up easily

Very much

like me

Mostly like

me

Somewhat

like me

Not much

like me

Not at all

like me

Active

Margin

F1 .094 .122 .472 .264 .047 1.000

F2 .027 .109 .426 .370 .069 1.000

F3 .022 .063 .316 .466 .133 1.000

M1 .078 .156 .306 .343 .117 1.000

M2 .040 .082 .360 .402 .116 1.000

M3 .021 .054 .256 .502 .167 1.000

Mass .035 .092 .375 .397 .101

Column Profiles

Gender x

DPS Level ST93Q01: When confronted with problems, I give up easily

Very much

like me

Mostly like

me

Somewhat

like me

Not much

like me

Not at all

like me Mass

F1 .081 .040 .038 .020 .014 .030

F2 .298 .458 .439 .360 .262 .387

F3 .040 .043 .053 .073 .082 .063

M1 .064 .049 .024 .025 .033 .029

M2 .461 .356 .383 .403 .456 .399

M3 .056 .055 .064 .118 .153 .093

Active Margin 1.000 1.000 1.000 1.000 1.000

- 75 -

Correspondence analysis shows that two dimensions (out of four) are sufficient to explain a

substantial proportion of the total inertia (=0.036) of the contingency table. The two dimensions

taken together are able to account for 96.0% (Dimension 1=76.1%; Dimension 2=19.9%) of the

total inertia. Figure 5.1 shows a bi-plot displaying the various degrees of perseverance (the 5

red dots) of the 5-point Likert scale and how they relate to specific gender-proficiency level

student groupings (6 green dots) on the plane spanned by the two extracted principal

dimensions. Specifically, compared across the six gender-proficiency level groupings, it can be

seen that F1 and M1 both have a higher tendency to correspond with “Very much like me”,

whereas F3, M3 and M2 all have higher chances to be associated with “Not much like me” or

“Not at all like me”. F2 has a higher tendency to be associated with “Somewhat like me”, or

“Mostly like me”.

Figure 5.1

A bi-plot displaying various dispositions of perseverance amongst students and how they relate

to specific gender-proficiency level student groupings

5.3 Correspondence analysis of “Openness for problem solving – Easily link

facts”



- 76 -

categories of the Likert response scale used to answer the openness for problem solving scale

item (i.e. Easily link facts). The 5-point Likert scale ranges from Very much like me to Not at all

like me.


F1, F2, F3, M1, M2 and M3), “Very much like me” is likely to correspond with F3 or M3

(compare 0.186 and 0.197 with 0.125). “Mostly like me” is also in a similar situation as that of

“Very much like me”. In a similar vein, “Not at all like me” is more likely to correspond with

F1 and M1. “Not much like me” is also in a similar situation as that of “Not at all like me”.

Last, “Somewhat like me” is more likely to correspond with F2 or M1.

The column profiles in Table 5.2 show that across the 5 groupings of Macao sampled students

with different degrees of openness dispositions rated on the 5-point Likert scale, F1 is more

likely to correspond with “Not much like me” (compare 0.044 with 0.030). F2 is more likely to

correspond with “Somewhat like me” or “Not much like me”. F3 is more likely to correspond

with “Very much like me” or “Mostly like me”. M1, unlike that of F1, is more likely to

correspond with “Not at all like me”. M2, a bit strange in response behavior, is only not that

likely to correspond with “Not much like me”. Last M3, is more likely to correspond with

“Very much like me” or “Mostly like me”.

- 77 -

Table 5.2

Correspondence analysis of “Openness for problem solving – Easily link facts”: Row & column

profiles

Row Profiles

Gender x

DPS Level

(3 levels)

ST94Q10: I can easily link facts together

Very much

like me

Mostly like

me

Somewhat

like me

Not much

like me

Not at all

like me

Active

Margin

F1 .104 .122 .395 .294 .085 1.000

F2 .098 .238 .414 .230 .019 1.000

F3 .186 .350 .357 .108 .000 1.000

M1 .088 .167 .431 .236 .078 1.000

M2 .130 .259 .398 .190 .023 1.000

M3 .197 .324 .339 .130 .009 1.000

Mass .125 .256 .397 .199 .022

Column Profiles

Gender x

DPS Level

(3 levels)

ST94Q10: I can easily link facts together

Very much

like me

Mostly like

me

Somewhat

like me

Not much

like me

Not at all

like me Mass

F1 .025 .014 .030 .044 .114 .030

F2 .302 .360 .403 .447 .329 .386

F3 .093 .086 .056 .034 .000 .063

M1 .020 .019 .031 .034 .102 .029

M2 .413 .403 .400 .380 .418 .399

M3 .146 .118 .080 .061 .038 .093

Active Margin 1.000 1.000 1.000 1.000 1.000


substantial proportion of the total inertia (=0.037) of the contingency table. The two principal

dimensions taken together are able to account for 98.6% (Dimension 1=80.7%; Dimension

2=17.8%) of the total inertia. Figure 5.2 shows a bi-plot displaying various degrees of openness

for problem-solving (5 red dots) of the 5-point Likert scale amongst students and how they

relate to specific gender-proficiency level student groupings (6 green dots) on the plane

spanned by the two principal dimensions. Specifically, compared across the six

gender-proficiency level groupings, it can be seen that F1 and M1 both have a higher tendency

to correspond with “Not at all like me”, whereas F3 and M3 both have higher chances to be

associated with “Very much like me” or “Mostly like me”. F2 and M2 both have a higher

tendency to be associated with “Somewhat like me”, though their chances associated with “Not

much like me” or “Mostly like me” are also not low.

- 78 -

Figure 5.2

A bi-plot displaying various dispositions of openness for problem-solving amongst students and

how they relate to specific gender-proficiency level student groupings

5.4 Correspondence analysis of “ICT use at home for school-related tasks –

Internet for schoolwork”



categories of the Likert response scale used to answer the ICT use at home for school-related

tasks scale item (i.e Internet for schoolwork). The 5-point Likert scale ranges from “Never or

hardly ever” to “Everyday”.


F1, F2, F3, M1, M2 and M3), “Never or hardly ever” is likely to correspond with F1 or M1

(compare 0.270 and 0.389 with 0.120). “Once or twice a month” is more likely to correspond

with F1 or M2. “Once or twice a week” and “Everyday” are more likely to correspond with F3

or M3. Last, “Almost every day” is more likely to correspond with F2 and F3.

The column profiles in Table 5.3 show that across the 5 groupings of Macao sampled students

- 79 -

with different degrees of ICT usage rated on a 5-point Likert scale, F1 is more likely to

correspond with “Never or hardly ever” (compare 0.064 and with 0.028). F2 is more likely to

correspond with “Once or twice a week” or “Almost every day”. F3 is more likely to

correspond with “Once or twice a week”, “Almost every day” or ‘Every day”. M1 is more

likely to correspond with “Never or hardly ever”. M2 is more likely to correspond with “Never

or hardly ever” or “Once or twice a month”. Last M3 is more likely to correspond with “Every

day”.

Table 5.3

Correspondence analysis of “ICT use at home for school-related tasks – Internet for

schoolwork”: Row & column profiles

Row Profiles

Gender x

DPS Level

(3 levels)

IC09Q01: Browsing the Internet for schoolwork (e.g. for preparing an

essay or presentation)

Never or

hardly ever

Once or

twice a

month

Once or

twice a

week

Almost

every day Every day

Active

Margin

F1 .270 .469 .187 .061 .013 1.000

F2 .077 .433 .370 .096 .025 1.000

F3 .016 .353 .473 .116 .042 1.000

M1 .389 .360 .162 .055 .034 1.000

M2 .160 .464 .292 .056 .029 1.000

M3 .074 .405 .401 .078 .042 1.000

Mass .120 .437 .337 .077 .029

Column Profiles

Gender x

DPS Level

(3 levels)

IC09Q01: Browsing the Internet for schoolwork (e.g. for preparing an

essay or presentation)

Never or

hardly ever

Once or

twice a

month

Once or

twice a

week

Almost

every day Every day Mass

F1 .064 .030 .016 .022 .013 .028

F2 .256 .397 .439 .495 .348 .401

F3 .008 .048 .083 .089 .086 .059

M1 .091 .023 .014 .020 .033 .028

M2 .526 .418 .341 .283 .388 .394

M3 .055 .084 .107 .091 .131 .090

Active Margin 1.000 1.000 1.000 1.000 1.000


substantial proportion of the total inertia (=0.062) of the contingency table. The two dimensions

taken together are able to account for 96.8% (Dimension 1=90.8%; Dimension 2=5.9%) of the

total inertia. Table 5.3 shows a bi-plot displaying various dispositions of ICT use at home for

school-related tasks (5 red dots) of the 5-point Likert scale amongst students and how they

- 80 -

relate to specific gender-proficiency level student groupings (6 green dots) on the plane

spanned by the two extracted principal dimensions in the correspondence analysis. Specifically,

compared across the six gender-proficiency level groupings, it can be seen that F1 and M1 both

have a higher tendency to correspond with “Never or hardly ever”, whereas F3, M3 and F2 all

have higher chances to be associated with “Everyday”, “Almost every day” or “Once or twice a

week”. M2 has a higher tendency to be associated with “Once or twice a month”.

Figure 5.3

A bi-plot displaying various dispositions of ICT use at home for school-related tasks amongst

students and how they relate to specific gender-proficiency level student groupings

5.5 Implications for the grade repeaters in Macao schools

PISA 2012 digital problem-solving assessment results, like results obtained in other cycles of

PISA assessment, are of paramount importance for informed policy decision making. For

example, more analyses can be conducted to throw lights on the associations amongst variables

like gender, perseverance, openess in problem-solving, ICT usage for school-related tasks, and

proficiency level of digital problem-solving of the 15-year-olds in Macao. Given that there are

inverse relationships between student grade level of study with digital problem-solving

- 81 -

performance (see Figure 2.1 and 2.2) and that frequency of grade repetition is associated

negatively with perseverance, openess for problem-solving, and ICT use at home for

school-related tasks (r = -0.106, -0.111, -0.205 respectively), it is fruitful if the associations and

relationships uncovered in section 5.4 of this chapter can be clarified through the use of

multiple correspondence analysis so as to arrive at a number of implications for the grade

repeaters in Macao schools.

Specifically, the associations between categories of variables gender of student, international

grade studied, frequency of grade repetition in primary/secondary school, perseverance,

openness for problem-solving, ICT use of school-related tasks, and DPS level have been

explored simultaneously. The bi-plot of the multiple correspondence analyses reveals the

following interesting results for the 15-year-old grade repeaters or non-repeaters in Macao (see

Figure 5.4):

1. Those students who have not repeated grades at all should be studying at international

grade level 10 or 11 (i.e. Form 4 or 5 in Macao’s secondary school system). Their

digital problem-solving proficiency level, whether for males or females, are also the

highest (i.e. reaching level 5 or 6 of the proficiency scale) in the sampled students. Their

ICT uses at home for school related task (e.g. browsing the Internet for school work for

preparing an essay or presentation) are also very frequent, amounting to almost every

day. They generally will not give up easily when confronted with problems, and their

minds are more open in problem solving.

2. Those students who have repeated grades two times or more should be studying at

international grade level 7 or 8 (i.e. Form 1 or 2 in Macao’s secondary school system).

Their digital problem-solving proficiency level, whether for males or females, are also

the lowest (i.e. below the baseline level 2 of the proficiency scale) in the sampled

students. They never or hardly ever use ICT at home for school related task (e.g.

browsing the Internet for school work for preparing an essay or presentation). They

generally will give up easily when confronted with problems, and their minds are not

that open in problem solving.

3. Those students who have repeated grades one time should be studying at international

grade level 9 (i.e. Form 3 in Macao’s secondary school system). Their digital

- 82 -

problem-solving proficiency level, whether for males or females, are mediocre (i.e.

level 2, 3 and 4 of the proficiency scale) in the sampled students. Their ICT uses at

home for school-related task (e.g. browsing the Internet for schoolwork for preparing an

essay or presentation) are not very frequent, amounting to once or twice a month. They

may not give up easily when confronted with problems, and they are quite open in

problem solving by easily linking facts together.

The implication is clear: Teachers and schools should render assistance to the grade

repeaters, particularly those who have repeated two times or more, in accordance with

students’ zone of proximal development. Grade repeaters should be provided with ample

opportunities to use ICT for school-related tasks at home. Through proper guidance and

counselling by teachers, parents and peers, they are initiated and modeled not to give up

easily when confronted with problems. Furthermore, they learn the many sound

pedagogical ways to links facts together and their minds become more receptive for solving

the problems assigned to them or encountered in daily life.

- 83 -

Figure 5.4

A bi-plot displaying various learning characteristics amongst students and how they relate to the phenomenon of grade repetition

- 84 -

References

Cheung, K.C., Sit, P.S., Mak, S.K., & Ieong, M.K. (2013). Macao-PISA 2012 Report:

Assessment of Mathematical, Scientific and Reading Literacy Performance of

15-year-old Students from an International Comparison Perspective. Macao:

Educational Testing and Assessment Research Centre, University of Macau.

OECD (2013). PISA 2012 Results: What Students Know and Can Do: Student

Performance in Mathematics, Reading and Science (Volume 1), OECD

Publishing.

OECD (2014). PISA 2012 Results: Creative Problem Solving: Students’ Skills in

Tackling Real Life Problems (Volume V), OECD Publishing.

- 85 -

Appendix 1: Coding guide of the example MP3 PLAYER test unit

Question 1: MP3 PLAYER

The bottom row of the MP3 player shows the settings that you have chosen. Decide whether each of the following statements about the MP3 player is true or false. Select “True” or “False” for each statement to show your answer.

Statement True or False?

You need to use the middle button ( ) to change the type of music. True False

You have to set the volume before you can set the bass level. True False

Once you have increased the volume, you can only decrease it if you change the type of music you are listening to.

True False

MP3 PLAYER SCORING 1

QUESTION INTENT:

Description: Understand a menu system of a device

Nature of Problem Situation: Interactive

Problem Solving Process: Exploring and Understanding

Context: Technology, Personal

Full Credit

Code 1: All three correct: True, False, False in that order

No Credit

Code 0: Other responses

Code 9: Missing

- 86 -


Set the MP3 player to Rock, Volume 4, Bass 2.

Do this using as few clicks as possible. There is no RESET button.


QUESTION INTENT:

Description: Control a system to achieve a given outcome


Problem Solving Process: Planning and executing


Full Credit

Code 2: MP3 player is set in 13 clicks or less so that the bottom row reads: Rock, 4, 2

Partial Credit

Code 1: MP3 player is set in more than 13 clicks so that the bottom row reads: Rock, 4, 2

No Credit


Code 9: Missing

- 87 -


Shown below are four pictures of the MP3 player’s screen. Three of the screens cannot happen if the MP3 player is working properly. The remaining screen shows the MP3 player when it is working properly.

Which screen shows the MP3 player working properly?

MP3 PLAYERSCORING 3

QUESTION INTENT:

Description: Demonstrate that a correct mental representation of a system has been formed


Problem Solving Process: Representing and formulating


Full Credit

Code 1: B

No Credit


Code 9: Missing

- 88 -


Describe how you could change the way the MP3 player works so that there is no need to have the bottom button ( ). You must still be able to change the type of music, and increase or decrease the volume and the bass level.

..................................................................................................................................................

..................................................................................................................................................

..................................................................................................................................................


QUESTION INTENT:

Description: Suggest a modification to the design of a menu system of a device


Problem Solving Process: Monitoring and reflecting


Full Credit

Code 1: Gives an answer that describes how the MP3 player could still operate with only one arrow button Change the way the top button works so that once you reach the right side of the display, one more

click takes you back to the left of the display Using one arrow, each line cycles around e.g. Music-Volume-Bass-Music The right arrow could just take you back to the far left of the screen if you reach the rightmost

entry – for example, once you are on “bass”, pushing the right arrow button could take you back to “Music”

The volume is set at 3 by default. If you want to change it to two or one, it could be set up

so that when you click the middle button to set the volume, it defaults to one (the lowest

setting). Then you can use the right arrow to change it to whatever you want.

When you want to change a property and you move to the line it is on, it should default to the lowest setting for that property.

Use the one arrow to go all the way round (in a circle). [Minimal.]

No Credit

Code 0: Other responses It would work without that button. You could change it so it didn’t need that button. [No explanation.] The middle button could move you to the left. [Insufficient explanation.]

Code 9: Missing

Documents

PISA - University of Macau