Evaluation of Schools for Quality I mprovement

Evaluation of Schools for Quality Improvement

Vitor AlaizUNIVERSIDADE CATÓLICA PORTUGUESA

FACULDADE DE EDUCAÇÃO E PSICOLOGIA

School Development in Central Asia Torino, 7-8 MAY 2009

Portuguese Educational system

• 1950 - 1974

Dual system

Portuguese Educational System

• General education schools (Liceu)• VET schools

Dual system

VET

Professional

Courses

Vocational

Courses

Education and

Training Courses (CEF)

Apprenticeship

courses

State

General+Professional Second

ary school

s

Private

Professional courses Professional

schools

Pluralistic ways to quality assurance

Quality

Accreditation

Certification

Systemic

evaluation

Audit

Schools make …No

difference

All the difference

Difference

SchoolQuality

School effectiveness

School improvement

Quality assurance

SCHOOLQUALITY

EXTERNALEVALUATION

INTERNALEVALUATION

EFQMISO 9000

Contingencial Models

ExternalTests, surveys

Team visits

INDICATORSSYSTEMS

InternationalNational

Local

Programa AVESAvaliação de Escolas com Ensino Secundário10

Programa AVESAvaliação de Escolas com Ensino Secundário

AVES Program: what is it?

MODELExternal evaluation

that supports internal evaluation processes

Multidimensional evaluation: Qualitative + quantitative dimensions

Network of schools, considering the context of each one

CIPP modelCONTEXT

INPUTPROCESS

PRODUCT

Stufflebeam

Programa AVESAvaliação de Escolas com Ensino Secundário14

MultidimensionalityLEVELs DIMENSIONs

Input Initial academic performance (Math, Port, Hist, Nat Sc)Context Sociocultural context (1,2,3,4 levels)Processes School + classroom processes

Results

STUDENTS1. Final academic performance (Math, Port, Hist, Nat Sc)

2 – Values + atitudes3 – Learning strategies 4 – Cognitive skils 5 – Opinion about the school

PARENTSOpinion about the school

TEACHERSSchool climate

Programa AVESAvaliação de Escolas com Ensino Secundário15

Programa AVESAvaliação de Escolas com Ensino Secundário

TOOLS

ACADEMIC TESTS

each year > 200 000

80000 tests120000 questionaires

QUESTIONAIRES

Programa AVESAvaliação de Escolas com Ensino Secundário16

Questionaires

Tests

Data (statistical treatment)

Report 1 Port, Math, Hist, Nat Sc

Report 3Value Added

Report 2 Values, Attitudes,

Cogn Skils …

“Fluxogram” of Program AVES

Report 3School Climate

Programa AVESAvaliação de Escolas com Ensino Secundário17

Reports: academic performance

0

10

20

30

40

50

60

70

80

90

100Es

cala

0-10

0

Factores

Matemática 7º ano

Escola 32 42 33 47 37 41 48 35 38Contexto 1 44 32 53 40 44 49 39 41Total 36 26 41 35 37 43 32 34

Números/Cálculo

Proporcion. Estatística Geometria Nota Aquisição Compreen

sãoResolução

de Problema

Programa AVESAvaliação de Escolas com Ensino Secundário18

0,00%

5,00%

10,00%

15,00%

20,00%

25,00%

30,00%

35,00%

Esca

la 0-

100

CONTEXTO SOCIOCULTURAL

Distribuição dos alunos pelo Contexto

Escola 6 24,20% 19,20% 32,40% 24,20%Nível 4 Nível 3 Nível 2 Nível 1

Reports: graphic with the context

Programa AVESAvaliação de Escolas com Ensino Secundário19

Reports: OCV (values, attitudes and cognitive skils)

0102030405060708090

100

Esca

la 0-

100

Factores

OPINIÃO SOBRE A ESCOLA 7º ano

Escola 1 55,8 59,9 43,1 66,2 52,2 75,3 67,7Contexto 2 66,2 64,9 62,2 69,6 58,8 71,6 73,5Total 58,9 66,3 49,1 64,1 51,3 72,4 68,5

Ordem Professores Actividades Director de

TurmaClassificaç

ões Colegas Satisfação

Programa AVESAvaliação de Escolas com Ensino Secundário20

1.00 6.00 7.00 10.00 13.00

escola

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

39

41153207

154

320

71

154

41

inputoutput

3º Ciclo

Média de versus resultados à entrada

resultados à saída

1.00 6.00 7.00 10.00 13.00

escola

10.00

20.00

30.00

40.00

50.00

60.00

70.00

80.00

39

41153207

154

320

71

154

41

inputoutput

Report: value added

Programa AVESAvaliação de Escolas com Ensino Secundário21

Reports: school climate

Figura 1: VA do 3º Ciclo do Ensino Básico

Why school climate?“School’s climate, as a variable, is far more directly responsible for pupils’ success than their social, economic and racial background.“

(Brookover et al. (1979) cf. Lima, 2008: 113)

Why school climate?“Culture shapes school’s capacity for change and improvement.”

(Sarason, 1996; Stoll, 1999b; Stoll and Earl, 2003: 495)

“Grosin’s research (1995) in Sweeden identified positive links between school climate and student’s mathematic progress, reading comprehension and behaviour” (Hopkins, 2003: 518)

School climate Definition:

the way as organizational

culture is perceived by the

members of one organization (Gonzalez Galan, 2004).

Methodology Questionnaire

Items: 40Scale: Likert scale, 5 leves (totaly desagree, …)

Subjects: 2500 teachers

Schools: 27 general schools (HS)+ 11 VET schools

Data:

Statistical treatement : averages of each item in: each school + each school of each type + all schools

Statistical tests: significance of average difference

Questionnaire: matrix 1

Questionnaire: matrix 2

Items aggregated in 8

constructs

ResourcesLeadersh

ip Internal discipline (behaviour)

Social systemSatisfaction level

Job climatePerceived

effectiveness

Personal relationships

Questionnaire: items (example)

The school facilities encourage teaching and learning

The school has a well defined educational direction

The level of discipline at the school allows for the good functioning of lessons

Teachers feel that their work is appreciated by pupils

Teachers are unhappy with colleague interaction within the school

Teachers participate with professionalism in school activity

Teachers are confident with the results of their work

Teachers have autonomy in their teaching role

Disciplina + aulas P3: O nível de disciplina existente na escola permite o bom funcionamento das aulas.

Cumprim RI P16: As normas do regulamento interno são cumpridas por todos, sem excepção.

Cumprim RE P26: A direcção da escola procura que as normas e orientações do Ministério da Educação sejam sempre cumpridas.

3 - DISCIPLINA INTERNA

1,0

1,5

2,0

2,5

3,0

3,5

4,0

4,5

5,0

ESCOLA 62 4,3 3,5 4,3

TOTAL Escolas Profissionais 4,0 3,5 4,2

TOTAL EP+ER 3,9 3,3 4,3

p03 p16 p26

Legenda:

Report: example

Pfs aprec p alun P4: Os professores sentem que o seu trabalho é apreciado pelos alunos.Pfs motiv trab P17: Os professores estão motivados para o seu trabalho lectivo.Pfs reconh pais P27: Os professores vêem o seu trabalho reconhecido pelos pais e encarregados de educação.Pfs reconh Dir P31: Os professores vêem o seu trabalho reconhecido pela direcção da escola.Pfs aprec Dir P32: Os professores apreciam positivamente o trabalho da direcção da escola.Pfs mot trab nlect P37: Os professores estão motivados para o seu trabalho não lectivo.

4 - RELACIONAMENTO PESSOAL

1,01,5

2,02,53,0

3,54,0

4,55,0

ESCOLA 62 3,8 3,5 3,0 3,3 3,7 3,4

TOTAL Escolas Profissionais 3,9 3,9 3,7 3,9 3,9 3,5

TOTAL EP+ER 3,7 3,5 3,5 3,6 3,6 3,0

p04 p17 p27 p31 p32 p37

Legenda:

Report: example

Study:Comparison

General schools VET schools

School Climate

ConclusionSchool climate of a set of VET

schools may have been

better than the one

of a lot general schools

Colect information

CommunicateEvaluation

UTILITY

PlanEvaluation

Treatinformation

ACCURACY PROPRIETY

FEASIBILITY

Good Quality Evaluation

THE JOINT COMMITTEE ON STANDARDSFOR EDUCATIONAL EVALUATION

Thank you!Vitor Alaiz

UNIVERSIDADE CATÓLICA PORTUGUESAFACULDADE DE EDUCAÇÃO E PSICOLOGIA

School Development in Central Asia Torino, 7-8 MAY 2009

vitoralaiz@gmail.com