Changing issues, changing questions, changing approaches

An overview of research in the Alternative Admissions Research Project

The late 1980s to about 2001

The main questions were about:

• Access• Finding ways to identify talented students

from poor schools (focus on the DET)

(extract: Placement Test in English for Education Purposes construct)

• aim to predict the performance of candidates in future situations in which language will be an important, but not sole, variable (language-as-vehicle rather than language-as-target);

• acknowledge the effects on cognitive functioning of the quantity and quality of prior mediated learning opportunities experienced by an individual (i.e. take seriously the impact of educational disadvantage by building in mediated learning opportunities).

• be based on a notion of knowing and learning which views learners as actively involved as individuals and in collaboration with others, in creating and negotiating meaning in a wide variety of settings. This process of conceptual development is seen as highly dependent on specific areas of expertise involving knowledge and information, and on the connections between these.

• be based on a componential model of language ability, which comprises topical knowledge and language knowledge, mediated by strategic competence (metacognitive strategy use) and affective schemata.

The main assessment related challenges were:

• producing a greater range of scores (spread), producing a greater range of scores (spread), so that capable students could be more so that capable students could be more clearly differentiated from weaker studentsclearly differentiated from weaker students

• improving (raising) the level of stronger improving (raising) the level of stronger students’ scoresstudents’ scores

• increasing the predictive validity of the test increasing the predictive validity of the test (that is, did the test correctly distinguish (that is, did the test correctly distinguish between weaker and stronger students)between weaker and stronger students)

Towards ‘dynamic’ testing:the old ELPT and the new PTEEP

•The same three major writing tasks (summary, description and contrast/comparison)

•The same additional text types (graph and diagram)

The PTEEP included structured and sequenced tasks designed to act as mediation for the writing assignments. The ELPT did not include these ‘scaffolding’ tasks.

•The same prose texts

DifferencesSimilarities

ASSESSING THE IMPACT OF THE SCAFFOLDING APPROACH

Questionnaire evidence from pilot groups suggests that candidates found the new test to be more ‘user-friendly’.

Correlations between first-year academic (UCT) performance and the selection tests strengthened.

Stronger candidates use preparatory tasks whereas weaker candidates do not see the connections between tasks (such tasks therefore help to widen the gap).

Based on scores from the same tasks (summary, short description and one page essay), the range of scores increased.

Candidate perceptions

Predictive validity

Task preparation

study

Range of scores

ELPT and PTEEP Scores

0

100

200

300

400

500

600

No

. o

f C

an

did

ate

s

10 20 30 40 50 60 70 80 90 100

Test Results (%)

ELPT

PTEEP

Quantitative approaches used at this stage and in early 2000s

• Problems with correlational analysis• A move to ‘survival analysis’• Classical item analysis• Psychometrically ‘naïve’ tests relying on

ranking – limitations for score use• Reliance on group performance for

‘standards’

The early 2000s

• Expansion of “AARP” testing• Introduction of Quantitative Literacy test• Major innovations in the area of test

delivery (use of innovative technology etc)• Pressure on tests reduced ‘potential’

aspect• Development of Health Science

Consortium group of tests

2005 →the main questions …

• Concerns largely shifted from access to throughput

• Placement function emphasised – implications for curriculum (focus mainly on extended progs)

• Concerns about the new school curriculum and qualification – target now all applicants (mass test)

• Increasing tensions with Department of Education (with stronger FET curriculum links)

Issues and challenges

• Academic literacy/ies construct widened to include greater focus on quantitative literacy (and reading not writing the focus)

• For the first time direct testing in the school curriculum (Mathematics & Maths Literacy)

• Radically different development trajectories, research traditions for the new NBTs (achievement tests) and TAPs (re-designing the old PTEEPs)

Research approaches:quantitative

• From Classical Test Theory to Item Response Theory

• Psychometric test development approaches

CTT vs IRT (Hambleton and Jones, Comparison of CTT and IRT and their applications to test

development)

Item and Test Characteristic Curves; Test Information Functions

and Equating

0

2

4

6

8

10

12

14

16

18

20

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3

Information

Theta

Test Information Functions for Forms QL1 and QL2

Form QL1 TIF

Form QL2 TIF

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

-3.0 -2.5 -2.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 2.0 2.5 3.0

P(theta)

Theta

Test Characteristic Curves for Forms QL1 and QL2

Form QL2 TCC

Form QL1 TCC

0.00

0.20

0.40

0.60

0.80

1.00

1.20

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3

P(U=1|THETA)

THETA

ICC Plots for 3-PL Form QL1 Series2

Series3

Series6

Series19

Series20

Series23

Series24

Series26

Series28

Series50

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

-3 -2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5 3

SUM of P(u=1 | THETA)

THETA

Test Characteristic Curve for 3-PL Form QL1

TCC form QL1

THE NATIONAL BENCHMARK

TESTS PROJECT

Why did HESA commission the NBTP?

2. Demonstrable inefficiencies in Higher Education itself (low throughput etc.)

• Difficulties in identifying students’ educational needs• Lack of appropriate curriculum flexibility at entry to meet these needs

2. Concerns about how to interpret the new NSC

In summary ….

• The NBTP is about higher education getting its own act in order – it is not about pointing fingers at the school system

• It sets out to do this by providing information about the competence of entering students in terms of 3 core domains of knowledge / skills

• it is important to note that higher education’s ‘take’ on what these core sets are, and at what level they should be mastered, will in all probability differ somewhat from those deemed most salient by the school-leaving system.

What do the NBTs aim to do?

– Provide additional information about performance in core, underlying areas (additional to NSC information)

– The core (domain) areas are:

• Academic literacy one 3-hr test • Quantitative literacy• Mathematics one 3-hr test

Performance in domain areas suggests that academic performance will not be adversely affected.

If admitted, students should be placed on regular programmes of study.

Challenges in domain areas identified such that it is predicted that academic progress will be affected.

If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision).

Serious learning challenges identified: it is predicted that students will not cope with degree level study without extensive & long-term support, perhaps best provided through bridging programmes or FET. Institutions registering students performing at this level would need to provide such support.

0%

100%

Proficient

Basic

Intermediate

Example - Competency Specification Proportions NBT Quantitative Literacy

Competence area

KnowingApplying routine

procedures in familiar contexts

Applying multistep procedures in a

variety of contexts

Reasoning and

reflecting

1+ 1- 2+ 2- 3+ 3- 4+ 4- n %

Comprehending: identifying or locating

Vocabulary5 0 7 7 2 4 1 5 31

19.0

15-20

Representations of numbers and operations

1 0 1 2 1 1 1 0 7 4.3 5-10

Conventions for visual representations

4 0 10 6 1 5 1 5 3219.6

20-25

Acting, interpreting, communicating

Using representations of data

2 0 6 3 1 4 1 4 2112.9

20-25

Computing (and estimating?)

0 0 4 10 2 4 0 2 2213.5

15-20

Conjecturing 0 0 0 0 0 0 0 1 1 0.6 0-5

Interpreting5 0 10 5 3 5 2 5 35

21.5

10-15

Reasoning 0 0 1 0 0 0 1 4 6 3.7 5-10

Representing quantitative information

0 0 3 1 0 0 2 0 6 3.7 5-10

Describing quantitative relationships

0 0 1 0 1 0 0 0 2 1.2 0-5

n 17 0 43 34 11 23 9 26 163

% 10.4 0.0 26.4 20.9 6.7 14.1 5.5 16.0

% 10.4 47.2 20.9 21.5

Mathematical and statistical ideas

Quantity, number and operations (LO1) 1 0 6 9 3 4 1 2 26

31.0

25-30

Shape, dimension and space (LO3) 3 0 4 3 0 2 1 1 14

16.7

10-15

Relationships, pattern, permutation (LO2a) 0 0 5 2 1 0 2 1 11

13.1

10-15

Change and rates (LO2b)0 0 4 2 1 0 1 1 9

10.7

10-15

Data representation and analysis (LO4a) 2 0 5 3 1 4 0 6 21

25.0

20-25

Chance and uncertainty (LO4b) 0 0 1 0 0 2 0 0 3 3.6 5-10

n

6 0 25 19 6 12 5 11 84

%

7.1 0.0 29.8 22.6 7.1 14.3 6.013.1

% 7.1 52.4 21.4 19.0

15-20 25-30 25-30 25-30

How are the benchmark (cut-off points) derived?

• The process is fundamentally different to the examination paper design procedures, and the norm referenced standardising and resulting processes of the NSC

• All items need to have been through rigorous review (fairness, content etc) and be statistically robust

• All items need to have been piloted

• The benchmark setting process is NOT about whether students can pass an item or not - the process is based on a set of probability assessments made by first year lecturers, with the core questions being:

“if a student can’t pass this item / do this, will s/he experience academic difficulties – and if so, how severe?”

NBT information

INDIVIDUAL LEVEL• Benchmark level (Basic, Intermediate, Proficient)• Description of what this means for each domain (ie what does being in the

‘Basic’ category mean a student knows and can do in Mathematics)• Clear recommendations about the type and extent of support needed

GROUP LEVEL• At the level of a faculty, or qualification, or institution ….• Give clear indication of the needs and strengths of entering cohorts, either

before entry, or at registration: useful for placement into existing courses, and/or with course design or modification.

DATA BASED ON

FEB 2009 PILOTS

ACADEMIC LITERACY (overall)

N = 12,202

Participating institutions: Mangosuthu, Rhodes, Stellenbosch, UCT, UKZN, UWC, Wits.

Serious learning challenges – long term, pre-tertiary intervention needed.

Challenges identified such that it is predicted that academic progress will be adversely affected. If admitted, students’ educational needs should be met in a way deemed appropriate by the institution (eg extended or augmented programmes, special skills provision)

Performance such that academic performance will not be affected.

If admitted, students should be placed on regular programmes of study.

851

55715780

0

1000

2000

3000

4000

5000

6000

7000

Total

ACADEMIC LITERACYNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient

ACADEMIC LITERACY by Faculty

250

63

289

22141

31

317

1256

119

1067

187

668

210

1179

2289

35

1393

340

575

142

1007

0

500

1000

1500

2000

2500

Commerce Education Engineering Health Humanities Law Science

ACADEMIC LITERACYNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient

Quantitative LiteracyN = 12,202

3055

6125

3022

0

1000

2000

3000

4000

5000

6000

7000

Total

QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient

Quantitative Literacy by Faculty

596

136

563

78

590

142

599

1807

77

1265

292

673

212

1235

1392

4

921

179121

29

669

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Commerce Education Engineering Health Humanities Law Science

QUANTITATIVE LITERACYNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient

Mathematics (overall)

1644

7390

737

0

1000

2000

3000

4000

5000

6000

7000

8000

Total

MATHEMATICSNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient

Mathematics overall[Intermediate ‘Top’ and ‘Bottom’]

5233

2157

0

1000

2000

3000

4000

5000

6000

Total

MATHEMATICSNBT Intermediate Benchmark Level, February 2009

Intermediate Bottom

Intermediate Top

Mathematics by Faculty

910

23

289

43 50 30

455

2765

58

1956

411

68 80

1658

372 387

821

189

0

500

1000

1500

2000

2500

3000

Commerce Education Engineering Health Humanities Law Science

MATHEMATICSNBT Benchmark Levels, February 2009

Basic

Intermediate

Proficient