Interpreting Feedback from Baseline Tests – Predictive Data Course: CEM Information Systems for Beginners and New Users Day 1 Session 3 Wednesday 17 th

Interpreting Feedback from Baseline Tests – Predictive Data

Course: CEM Information Systems for Beginners and New Users

Day 1 Session 3

Wednesday 17th October 2012

Peter Hendry: CEM Consultant

[email protected]

The word ‘PREDICTION’:

Quite probably the most contentious term that is used!! Concerns include:

• A prediction for GCSE at start of year 7?• What has the baseline test got to do with my

subject?• I know my pupils!• The predictions are too low: not valid!!!• And what about my professional judgement?

BASELINE SCORE

G

RA

DE

G

RA

DE

* ** *** *******************

* ** *** ********************************

* ** *** ********************************* *** ** *

* ** *** ************************************ *** ** *

* ** *** ******************************** **** ** *

* ** ***************************** *** ** *

* ** *** ******************* ** * *

* ** *********** ** *

How is a ‘prediction’ generated?

C

A*

A

B

50 100 150

3 key points are:

• The higher the baseline score the higher the final grade

• Any one grade is achievable from a range of baseline scores

• From any baseline score, a range of grades are possible

BASELINE SCORE

G

RA

DE

G

RA

DE

* ** *** *******************

* ** *** ********************************

* ** *** ********************************* *** ** *

* ** *** ************************************ *** ** *

* ** *** ******************************** **** ** *

* ** ***************************** *** ** *

* ** *** ******************* ** * *

* ** *********** ** *

Subject National trend line (regression line)

How is a ‘prediction’ generated?50% on or above the trend line

50% on or below the trend line

‘PREDICTION’ (expected grade)

C

A*

A

B

2008 GCSE Maths (representative sample)

0

1

2

3

4

5

6

7

8

9

0 20 40 60 80 100

baseline score

Re

su

lt

‘Predictions’…...are based on

Average performance by similar pupils in past examinations

The problem with the word ‘prediction’ is…?

An alternative is ‘expected’ grade

Predictions

4 points = D

6.6 points = A/B Trend line

50 60 70 80 90 100 110 120 130 140 1501

2

3

4

5

6

7

8

MidYIS Year 7 to GCSE 2011 Value Added Analysis Art & Design

Biology

English

French

History

Mathematics

Science

MidYIS Test Score

GC

SE

Po

ints

Sc

ore

Comments?

The graph below shows the middle 2/3 of some subject trend lines

Some Subjects are More Equal than Others….

E

D

C

B

A

C

Gra

de

B A A*

Average GCSE

Physics

Maths

Psychology

Sociology

Latin

Photography

English Lit

>1 grade

FACTORS THAT WILL INFLUENCE RELIABILITY OF PREDICTIONS:

• Knowledge of student • Parental support/home life• Peer influences/social life• Student attitude, interest, language• Expectations of staff• Department/institution ethos • Resources • Quality of teaching and learning: pace of lessons• Understanding how children learn………• And the reliability of the predictions......

12

Correlation = 1

Res

ult

Correlation = 0

Correlation = 0.7

subjectMIDYIS( Yr 7)

to GCSEART 0.49BIOL 0.56ENG LANG 0.7FRENCH 0.64HISTORY 0.63MATHS 0.74MUSIC 0.57

CORRELATIONS

SUBJECTGCSE to A2

ART and DESIGN 0.54BIOLOGY 0.69ECONOMICS 0.65ENG LIT 0.71FRENCH 0.6HISTORY 0.69MATHS 0.6MUSIC 0.65PHYS EDUC 0.65PSYCHOLOGY 0.62

stud

ent n

o.

Sex

For

m

Mid

YIS

Sco

re

Mid

YIS

Ban

d

Art

& D

esig

n

Bio

logy

Eng

lish

Fre

nch

His

tory

Mat

hem

atic

s

Sci

ence

1 M 7E 131 A 6.8 7.2 7.1 6.8 7.2 7.5 7.0

2 F 7E 120 A 6.3 6.7 6.4 6.1 6.4 6.5 6.3

3 F 7C 110 B 5.8 6.2 5.7 5.4 5.5 5.6 5.5

4 F 7J 101 B 5.4 5.8 5.1 4.8 4.9 4.9 4.9

5 M 7D 82 D 4.5 4.9 3.8 3.5 3.4 3.2 3.5

1 M 7E 131 A A A A A A A*/A A

2 F 7E 120 A A/B A/B A/B B A/B A/B A/B

3 F 7C 110 B B B B/C B/C B/C B/C B/C

4 F 7J 101 B B/C B C C C C C

5 M 7D 82 D C/D C D D/E D/E E D/E

Point and grade ‘predictions’ to GCSE• Compare the

predictions for English and Mathematics.

What pattern do you notice?

• Look at Art and Design, Biology and French. Comments?

50 60 70 80 90 100 110 120 130 140 1501

2

3

4

5

6

7

8

MidYIS Year 7 to GCSE 2011 Value Added Analysis Art & Design

Biology

English

French

History

Mathematics

Science

MidYIS Test Score

GC

SE

Po

ints

Sc

ore

Comments?

The graph below shows the middle 2/3 of some subject trend lines

0 0 1

5

19

39

27

9

10

5

10

15

20

25

30

35

40

45

U G F E D C B A A*

Per

cen

t

Grade

Individual Chances Graph for Student 4- GCSE EnglishMidYIS Score 101 MidYIS Band B

Teacher's Adjustment : 0 grades / levels / points

Prediction/expected grade: 5.1 grade C

Most likely grade

Predictions & Chances Graphs to GCSE subjects

based on Year 9 nationally standardised scores

Please refer to the Information Sheet for details of

the method used to make these predictions

0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0

Student Surname Forename Sex Form CEM Id Cust Mid

YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d S

cien

ce

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.1

2 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6

3 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4

4 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7

Tea

cher

's

Adj

ustm

ent

(Gra

des)

Predictions based on 2011 Regression Analysis

MidYIS Year 9 Test Academic Year 2011/2012

GCSE Point PredictionsPoints To Grades Adjust

CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile of Schools

Prior Value added

Generate File for MIS Import

to place school at 75th percentile of VAD





0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2 0.2


YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d S

cien

ce

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

1 F 9140005J 105 B 4.6 5.5 6.0 6.1 5.3 6.1 5.5 5.4 5.6 5.7 5.2 5.3

2 F 9140005K 111 A 4.9 6.0 6.3 6.5 5.8 6.5 5.9 5.8 6.0 6.1 5.6 5.8

3 F 91400042 108 B 4.8 5.8 6.2 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6

4 F 91400043 101 B 4.4 5.2 5.8 5.9 5.0 5.9 5.2 5.2 5.3 5.5 4.9 4.9

Tea

cher

's

Adj

ustm

ent

(Gra

des)




CEM Centre Scale

QCA Scale

Display Legend

Reset Adjustments

Prior Value Prior Value addedadded






0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0


YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d S

cien

ce

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.3 5.8 5.9 5.1 5.9 5.3 5.2 5.4 5.5 5.0 5.1

2 F 9140005K 111 A 4.7 5.8 6.1 6.3 5.6 6.3 5.7 5.6 5.8 5.9 5.4 5.6

3 F 91400042 108 B 4.6 5.6 6.0 6.1 5.4 6.1 5.5 5.4 5.6 5.7 5.2 5.4

4 F 91400043 101 B 4.2 5.0 5.6 5.7 4.8 5.7 5.0 5.0 5.1 5.3 4.7 4.7

Tea

cher

's

Adj

ustm

ent

(Gra

des)




CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile of Schools

Prior Value added


Or insert own valuesand click adjust





0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4


YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d S

cien

ce

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

1 F 9140005J 105 B 4.4 5.6 7.2 6.6 5.1 6.6 5.3 6.1 6.7 6.3 5.7 6.5

2 F 9140005K 111 A 4.7 6.1 7.5 7.0 5.6 7.0 5.7 6.5 7.1 6.7 6.1 7.0

3 F 91400042 108 B 4.6 5.9 7.4 6.8 5.4 6.8 5.5 6.3 6.9 6.5 5.9 6.8

4 F 91400043 101 B 4.2 5.3 7.0 6.4 4.8 6.4 5.0 5.9 6.4 6.1 5.4 6.1

Tea

cher

's

Adj

ustm

ent

(Gra

des)




CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile Upper Quartile of Schoolsof Schools

Reset Adjustments






0.0 0.3 1.4 0.7 0.0 0.7 0.0 0.9 1.3 0.8 0.7 1.4


YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al A

pplie

d S

cien

ce

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

1 F 9140005J 105 B C/D B/C A A/B C A/B B/C B A/B A/B B/C A/B

2 F 9140005K 111 A C/D B A*/A A B/C A B/C A/B A A/B B A

3 F 91400042 108 B C/D B A*/A A B/C A B/C A/B A A/B B A

4 F 91400043 101 B D B/C A A/B C A/B C B A/B B B/C B

Tea

cher

's

Adj

ustm

ent

(Gra

des)



GCSE Grade PredictionsGrades To Points Adjust

CEM Centre Scale

QCA Scale

Display Legend

Upper Quartile Upper Quartile of Schoolsof Schools

Reset Adjustments


Prediction/expected grade: 6.4 grade A/B

Most likely grade


based on Year 9 Independent Sector standardised scores




YIS

Sco

re

Mid

YIS

Ban

d

Add

ition

al S

cien

ce

Art

& D

esig

n

Bio

logy

Bus

ines

s S

tudi

es

Che

mis

try

Cla

ssic

al C

ivilis

atio

n

Des

ign

& T

echn

olog

y

Dra

ma

Eng

lish

Eng

lish

Lite

ratu

re

Fre

nch

Geo

grap

hy

Ger

man

1 F 9140005J 87 D 5.8 6.6 6.4 5.7 6.4 5.8 6.1 6.1 6.1 6.2 5.7 6.1 5.8

2 F 9140005K 94 C 6.1 6.8 6.7 6.0 6.7 6.1 6.4 6.4 6.5 6.4 6.1 6.5 6.2

3 F 91400042 90 C 6.0 6.7 6.6 5.9 6.6 6.0 6.3 6.3 6.3 6.3 5.9 6.3 6.0

4 F 91400043 81 D 5.5 6.5 6.1 5.4 6.1 5.6 5.8 5.9 5.9 5.9 5.4 5.8 5.6

Predictions based on 2011 Independent Sector Regression Analysis


GCSE Point PredictionsPoints To Grades

CEM Centre Scale

QCA Scale

Display Legend


Prediction/expected grade: 5.9 grade C

Most likely grade

Independent Sector

Not a label for life...just another piece of information

• The Chances graphs show that, from almost any baseline score, students come up with almost any grade - - -there are just different probabilities for each grade depending on the baseline score.

• In working with students these graphs are more useful than a single predicted or target grade

• Chances graphs show what can be achieved: – By students of similar ability– By students with lower baseline scores

Yellis predictive data: baseline score 103 (55%)

SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)French 5.4 (B/C)Geography 5.6 (B/C)Mathematics 5.7 (B/C)Physical Education 5.7 (B/C)Science: GCSE 5.6 (B/C)Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.3 (B/C)

Weighted Average 5.6 (B/C)

Yellis Average

Predicted SubjectBusiness Studies 5.6 (B/C)*English 5.9 (B)*French 5.6 (B/C)*Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.0 (B)*Science: GCSE 5.8 (B)*Science: GCSE Additional 5.8 (B)*SC Religious Studies 5.6 (B/C)*

Weighted Average 5.8 (B)

Yellis Average

Predicted

SubjectBusiness Studies 5.4 (B/C)English 5.7 (B/C)*French 5.4 (B/C)Geography 5.8 (B)*Mathematics 5.9 (B)*Physical Education 6.7 (A/B)*Science: GCSE 6.3 (A/B)*Science: GCSE Additional 5.6 (B/C)SC Religious Studies 5.7 (B/C)*

Weighted Average 5.8 (B)

Yellis Average

Predicted

Polly Bolton

AUTUMN SPRING SUMMER

Subject Teach

curr

ent

grad

e

targ

et

gra

de

is:

effo

rt

conc

ern

curr

ent

grad

e

targ

et

gra

de

is:

effo

rt

conc

ern

year

10

exam

targ

et

gra

de

is:

effo

rt

conc

ern

English CB C B LIKELY 5 ORG B B SECURE 5

English Literature CB B B SECURE 5 ORG C B LIKELY 5

Maths MC E B UNLIKELY 3 HW D B POSSIBLE 4

Science CPa E C UNLIKELY 4 ORG D C POSSIBLE 4 ORG

Science Additional CPa D C LIKELY 4 ORG B C LIKELY 4 ORG

French CK C B LIKELY 4 C B LIKELY 5

History KM A A SECURE 5 B A SECURE 5

RS CG D B POSSIBLE 4 C B POSSIBLE 4

KEY - target is: SECURE LIKELY POSSIBLE UNLIKELY

effort: 5 excellent - 4 good - 3 satisfactory - 2 poor - 1 very poor

concern: WW working well - ATT attitude - BEH behaviour TEN attendance - PUN punctuality - HW homework CON confidence - ORG organisation - EAL language

Student 1

Student 2

Why is the ‘predicted’ grade not always equal to the highest bar ?

Most likely grade

Predicted (‘expected’) grade

Predicted (‘expected’) grade

AT WHICH POINT WILL THE SEE-SAW BE BALANCED?

i.e. the lower grades ‘pull’ the prediction to the left

Student 3

Student 4

Student 4 - IPR

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 330

5

10

15

20

25

30

Vocab Item Responses

Question

Res

po

nse

Tim

e (s

)

Correct

Wrong

Basing Targets on Prior VA – One Methodology from an Alis School

• Discuss previous value added data with each HoD

• Start with an agreed REALISTIC representative figure based, if available on previous (3 years ideally) of value added data

• add to each pupil prediction, and convert to grade (i.e. in-built value added)

• Discuss with students, using professional judgment and the chances graphs, adjust target grade

• calculate the department’s target grades from the addition of individual pupil’s targets

DEPARTMENT: ATarget Setting

yearno. of pupils av. GCSE av. TDA raw resid.

Std. Resid

3yr. Av. Std resid

2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65

From and including 2002, a raw residual of 20.0 is equivalent to one grade

SUGGESTED TARGETS FOR 2007, based on ALIS pred and dept's value added historyThe target grade has an in-built value added of 15 points (one grade is 20 points)

target grade

dept adj target

the total target grades are as follows: A 1 0B 2 3C 6 5D 1 1E 0 0

Surname Forename AveGCSE TDA Prediction TARGETtarget grade

Teacher adj target RESULT

4.7 28 49.3 64.3 D D D5.8 30 73.2 88.2 C C C6.9 48 96.4 111.4 A B B6.2 61 80.8 95.8 B C C5.1 39 57.8 72.8 C B B5.5 30 66.3 81.3 C C D5.4 54 63.4 78.4 C C B5.2 33 59.9 74.9 C C C6.1 53 79.1 94.1 B B B

AVERAGE 5.7 41.8 69.6 84.6 C



Std. Resid

3yr. Av. Std resid

2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65



target grade

dept adj target





AVERAGE 5.7 41.8 69.6 84.6 C



Std. Resid

3yr. Av. Std resid

2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65



target grade

dept adj target





AVERAGE 5.7 41.8 69.6 84.6 C



Std. Resid

3yr. Av. Std resid

2002 2 6.8 49.0 24.5 1.2 0.72003 7 7.1 49 13.3 0.6 0.82004 6 6.6 51 18.2 0.7 0.82005 12 6.17±0.22 44.50±3.84 12.82±4.05 0.60±0.29 0.65



target grade

dept adj target





AVERAGE 5.7 41.8 69.6 84.6 C

Key Questions for Target Setting

• What type of valid and reliable predictive data should be used to set the targets?

• Should students be involved as part of the process (ownership, empowerment etc.)?

• Should parents be informed of the process and outcome?

Key points to consider might include:

• Where has the data come from?• What (reliable and relevant) data should we use?• Enabling colleagues to trust the data: Training (staff)• Communication with parents and students• Challenging, NOT Demoralising, students…….• Storage and retrieval of data• Consistency of understanding what the data means

and does not mean

Documents

Interpreting Feedback from Baseline Tests – Predictive Data Course: CEM Information Systems for Beginners and New Users Day 1 Session 3 Wednesday 17 th