Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 5th QualiFLY Meeting Hamburg, 20. 02. 07 Evaluation of a Family Literacy Project in

Evaluation Family Literacy

Dr. Peter May. LIQ-1 QualiFly 19./20.02.07

5th QualiFLY MeetingHamburg, 20. 02. 07

Evaluation of a Family Literacy Project in Hamburg

Peter May

State Institute for teacher training and school developementHamburg, Germany



Overview

1 Internal and external evaluation

2 Phases of evaluation

3 Applied methods

4 First Findings

5 Problems



FörMig Förderung von Kindern und Jugendlichen mit Migrationshintergrund

external evaluation internal evaluationUniversity of Hamburg State Institute, Hamburg

Sub-project: HAVAS Sub-project: Family Literacydiagnosis-based aid parental support

Target group: children parents and children

Form of evaluation:summative formative

Data collection instruments:intelligence test participant observationlanguage testing interviewsquestionnaire about children questionnaire parents

questionnaire educationalist



Phases of evaluation

1. Pre-Investigation of circumstances in the fieldschool year 03/04 • Interests (parents, educationalists,

headmasters of schools and kindergardens)• Circumstances in possible locations

2. Implementation of the project in preschool classesschool year 04/05 • Analysis of the concept for preschool classes

• Description of target group • Evaluation of the educational concept

3. Consolidation and extension of the concept to first grade classesschool year 05/06 • Analysis of the concept extension into first

grade classes• Circumstances in new locations

4. Networking and sustainability School year 06/07 • Building up a network with other projects

• Transitions from preschool to first grade• Learning effects of the children




1. Pre-Investigation of circumstances in the fieldschool year 03/04 • interests (parents, educationalists,

headmasters) • circumstances in possible locations

Methods: guideline-based interviews • with parents from different ethnical and social groups: e.g. families from Turkey, Africa, Russia, German underclass families• with educationalists in possible locations (schools, day-care centres)• with headmasters from schools and day-care centres




2. Implementation of preschool classes

school year 04/05 • analysis of the concept for preschool classes • description of target group • evaluation of the educational concept

Methods: • participating observation - in introductory events and evening classes for parents - in groups of parents and children in the morning - in advanced training courses for disseminators (teachers, educators)

• questionnaires, predominantly with open answers - parents at the beginning: motives, circumstances, interests - disseminators: Implementation of the concept at locations

• questionnaires, predominantly with given answers - parents at the end of the courses: changes (behaviour, attitudes),

satisfaction with the courses - teachers: assess the learning development




3. Consolidation and extension of the concept to first grade classesschool year 05/06 • analysis of the extension of the concept in first

grade classes• circumstances in new locations

Methods: • participating observation - in advanced training courses

- in courses for parents in first grade classes

• questionnaires, predominantly with given answers - parents at the beginning: interests, characteristics of the family - parents at the end of the courses: changes, satisfaction

- disseminators: - design of the courses- estimation of success

• testing of the children: - non-verbal intelligence - language competence at the beginning of the course




4. Networking and sustainabilityschool year 06/07 • building up a network with other projects

• transitions from preschool to first grade • learning effects of the children

Methods: • guideline-based interviews with new partners: - other aids in preschools and primary schools - Volkshochschule (language classes for parents)

• questionnaires: interviews with parents before and after

• systematic coverage of performances of children: - sequel testing (pre-, post- and follow-up) in preschools, first and

second grade (language, reading, writing) - teacher’s assessment of educational development



Methods of evaluation

(1) participating observation - courses for parents- events with parents and children- meetings of disseminators

advantages: - direct impressions („atmosphere“)- opportunities to get in contact with parents and educationalists- informal conversations with many details

disadvantages: - high costly- influence of subjective impressions




(2) Oral interview (guideline-based interviews)• with headmasters of schools and kindergarten• with single disseminators • with special experts• with single parents (occationally with translators)

advantages:- private atmosphere- possibility to adjust the questions spontaneously- offers new, sometimes surprising insights

disadvantages: - costly- can be subjectively influenced- difficulty to generalise




(3) questionnaires (a) with open answers

- disseminators: narration of concepts and procedures collection of suggestions

- parents: survey of perceptionscollection of wishes

required: translations (Turkish, Russian, Farsi, English)

advantages: - less costly than interviews- often offers new insights

disadvantages: - no systematic coverage of relevant characteristics - partially costly analysis, requires analysis of contents



Methods of evaluation(3) questionnaires

(b) with given answers

- disseminators: systematic enquiry of circumstances in locationspersonal backgrounds and motivesestimation and assessment

- parents familial backgroundspersonal characteristics of children

changes in behaviour and attitudes (before - after)

required: translations

advantages: - very economic- systematic coverage of relevant characteristics- statistical analysis possible

disadvantages:- reduced answers- sometimes difficult to interpret




(4) Testing- intelligence test- language test: language competence at the beginning

sequel testing (post, follow-up in first grade)- reading and writing in second grade

advantages:- no subjective influence- very economic (group testing)- provides computable data

disadvantages:- high input to establish objective circumstances for testing- „psychological barriers“ for parents and educationalists



Outcomes of questioning of parents

Number of questionnaires for parents:(8 locations)School year 2004/05: approx. 70; 2005/06: approx. 100; 2006/07: approx. 130overall: approx. 300 parents

Contents of questioning: • personal und familial background of parents• motives und wishes for participation in courses• changes in the families (reading habits, support for children)• rate the courses (feed back)



First language of parentsAnzahl Prozent

Trkisch 55 28,5Deutsch 53 27,5Polnisch 13 6,7Russisch 13 6,7Dari 10 5,2Afrikanische Sprachen 8 4,1Persisch 7 3,6Hindi 6 3,1Arabisch 5 2,6Bosnisch 4 2,1Afghanisch 2 1,0Bulgarisch 2 1,0Chinesisch 2 1,0Georgisch 2 1,0Griechisch 2 1,0Portugiesisch 2 1,0Thai 2 1,0urdu 2 1,0vietnamesisch 2 1,0Mazedonisch 1 0,5

193 100,0

born in Germany:

33 %



sex

18 % male82 % female

professions

professional educated: 59 %employed: 40 %

wide broad of professions (from non educated workers to university graduates)



parents pre-questionnaire: reading and writing at home

9

39

2

14

47

78

28

13 13

49

1

7

0

10

20

30

40

50

60

70

80

90

I read books/magazines.

I write something (e.g. letters, post cards).

I buy books/reading materials for my children.

pe

rce

nt

never once/twice a month once/twice a week daily



parents pre-questionnaire: reading and writing with the children

3

13

6

13

26

15

38

35

41

46

26

38

0

5

10

15

20

25

30

35

40

45

50

We read out at home. We look at picture books athome.

We write at home.

perc

ent

never once/twice a month once/twice a week daily



Expectations of participants to course attendance

82 % Interest in new ideas and continuation of own education

56 % Support for the own child

42 % Exchanging experiences and to establish contacts

10 % cooperation with teachers



parents post-questionnaire: reading and writing at home

12

22

6

61

66

58

27

13

36

0

10

20

30

40

50

60

70

I read books/magazines.

I write something (e.g. letters, post cards).

I buy books/reading materials for my children.

perc

ent

less unchanged more



parents post-questionnaire: reading and writing with children

912

18

76

67

52

15

21

27

0

10

20

30

40

50

60

70

80

We read out at home. We look at picture books athome.

We write at home.

perc

ent

less unchanged more



parents post-questionnaire : benefit of the courses for parents

16 17

11

7

14

55

43

3739

57

29

40

5254

29

0

10

20

30

40

50

60

I got helpfulsuggestions to

help my childrenmore successfullywith learning at

home.

I got morecertainty inlearning and

playing with mychild.

Thecommunication

with my child hasimproved.

The cooperationwith the school

got better.

I establishedcontact to other

families.

perc

ent

none/few some many



parents post-questionnaire : benefit of the courses for the children

18

10

29

48

54

41

0

10

20

30

40

50

60

The language skills of my child have improved.

My child got more enthusiastic about learning.

perc

ent

not at all slightly much



Evaluation of the courses by the parents

27 % very good

72 % good

1 % satisfying

0 % dissatisfying



Questionnaire for the multiplicators

Guiding objectives fort the project:

advancement of families concerning education and equity of opportunities

Improvement of the communication between the families

Improvement of the relation school-parents



Distinction to conventional preschool education

conscious orientation towards reading and writing

inclusion of the parents as active and equal partners

Disseminators questionnaire:



Success with the work in the parentcourses and the parent-childcourses refers to:

the attitude of the parents towards Reading and Writing;

the realisation of the course ideas at home;

the communication and support within the groups;

the reduction of fear concerning school;

Improvement of the social climate at school




Difficulties in the realization of the project:

Continuity in participation of the parents,

reached target group,

lacking German language skills of the mothers,

structural conditions of the school/kindergarten,

financial and material capabilities.





Discussion points:

How can an obligatory participation be secured?

Is the target group reached? What about the ćsocial weakŅ families? How can they be more included?

How could the courses combined with systematic language education of the mothers;

Further qualification of the teachers for family literacy;

How can given structural and financial conditions be improved?



Summary of evaluation results so far:

• highly motivated disseminators (teachers and educators)

• high satisfaction of the parents

• few changes of parents literacy behaviour

• more advancement in helping children reading at home

• benefit for children’s learning in parental view

• open question : children‘s benefit in objective view (tests)



Problems with the evaluation

• serious fluctuations in participation

• lack of control groups

• test conditions hard to control

• analysis of effect conditions complicated, due to complex mass of arrangements.

Documents

Evaluation Family Literacy Dr. Peter May. LIQ-1QualiFly 19./20.02.07 5th QualiFLY Meeting Hamburg, 20. 02. 07 Evaluation of a Family Literacy Project in