Inspire External Evaluation August18 08 09

Inspire Summer School, Salzburg, 18-20 August 2009

External evaluation of the INSPIRE project:

Innovative Science Pedagogy in Research and Education

By Yves Beernaert & Magda Kirsch, Educonsult

Comenius Transnational projectINSPIRE

Contract number: 2007-3430 / 001-001Project number – 133867-LLP-1-2007-1-BE-COMENIUS-CMP


Why the INSPIRE project ?

• It supports the Lisbon agenda 2000– 2010: EU is to become the most competitive and dynamic knowledge-based economy in the world,

capable of sustainable economic growth with more and better jobs and greater social cohesion

• It contributes to the benchmarks 2010– Halve the rate of early school leavers, in order to achieve an EU-average rate of

10%or less.– Halve the level of gender imbalance among graduates in MST & secure an

significant increase of the total number of MST graduates,– Average % of 25-64 yrs in the EU with at least upper sec. education reaches 80%or

more– Halve % of low-achieving 15 year olds in reading, mathematical and scientific

literacy– Increase EU-average level of participation in LLL to at least 15% of working adult


Lisbon agenda: Detailed workprogramme:

3 strategic objectives

1. Improving the quality and effectiveness of education and training systems in Europe

2. Facilitating access of all to education and training systems

3. Opening Up education and training systems to the wider world


Strategic objective 1: Improving the quality and effectiveness of education and training systems in

Europe

• 1.1. Improving education and training for teachers and trainers

• 1.2.Developing skills for the knowledge society

• 1.3. Ensuring access to ICT for everyone• 1.4. Increasing the recruitment to scientific

and technical studies


Strategic objective 2: Facilitating access of all to education and training systems

• 2.1. Open learning environments• 2.2. Making learning more attractive• 2.3. Supporting active citizenship, equal

opportunities and social cohesion


Strategic objective 3: Opening Up education and training systems to the wider world

• 3.1. Strengthening the links with working life and research, and society at large

• 3.2. Developing the spirit of enterprise• 3.3. Improving foreign language learning• 3.4. Increasing mobility and exchange• 3.5. Strengthening European co-operation


External evaluation of INSPIRE: AIMS

To assess:• the quality & relevance of activities, processes implemented• the quality and relevance of the ‘educational ‘ or pedagogical’

outputs or the results• the quality and relevance of the dissemination / exploitation /

valorisation actions • the sustainability of the project activities • the impact on various stakeholdersTo identify key success and failure factors of the overall projectTo propose measures that will improve effective project delivery in

future partnerships


Elements of evaluation

• Relevance: contribution to specific needs at local, nat. level versus objectives of Comenius

• Efficiency: relationship input versus output• Effectiveness: match between results and Comenius objectives• Impact / Sustainability on teachers, decision-makers, education system


Two parts of the INSPIRE External evaluation

• a) Quality of the project, consortium and activities / outcomes, its organisation

• b) Quality of the educational activities / outcomes and their (possible) impact on science education


External evaluation tools of b) pedagogical activities

• Protocol of experimentation: data collection– Questionnaires for:

– Teachers– Pupils– National coordinators– School coordinators

– Informal interviews with stakeholders

• Analysis of the data• Participation in meetings• Draft report plus feedback from stakeholders


Results of the external evaluation• An external evaluation report• Parts of the report

– Description of aims, methodology– Evaluation of functioning, organisation of

partnership– Evaluation of the pedagogical activities

• European Report• 5 Country reports


Learning objects in MST teaching and learning

Presentation of the European evaluation report

of the pedagogical activities


Part 1:Participation in the project


Types of schools involved


Characteristics of the schools


Characteristics of the schools


Teachers participating in the survey


Experience of teachers in experiments


Pupils participating in the survey


Age of respondent pupils


Questions

• Still confusion about number of Austrian schools. How many exactly?

• Nature of VET-schools: secondary or adult education?

• Reasons for drop-out of German and Austrian schools?– Please contact Magda or Yves during summer

academy!


Part 2: Organisation of project activities


Part 2: Organisation of project activities

• Overall thorough preparation /information• Coordinating school teachers informed about

the project most refer to one (training) meeting or several meetings.

• The national coordinators advised the teachers as to the LO to be used through e-mail, meetings, oral and written guidelines

• Some NC indicated which LO could be used at the different levels of the schools.


Training of teachers and pupils


Training of teachers and pupils

• trainings of pupils lasted between 15 minutes and 10 hours

• trainings for pupils given by a number of people involved such as ICT teachers, the school coordinator, the physics teacher etc.

• training lasted between two hours and twenty hours.

• trainings were given by national coordinator or the school coordinator.


Organisational facilitators


Evaluation tools teachers


Evaluation tools pupils


Coordination of evaluation activities: key =coordinator


Strong appreciation of School Coordinator


Strong appreciation of National Coordinator


Evaluation of monitoring


Any remarks ?

• Concerning preparation or information?• Concerning training?• Concerning monitoring and support?

– KEY role of national coordinators– KEY role of school coordinators


Part III: Implementation

• Selection criteria of LO• Use of LO per category and per subject• Use of LO for extracurricular activities


Selection criteria of LO


Use of LO per subcategory


Most popular LO per subcategory

• Biology: – B2-5-26 , “Habitat of animals” used by 26

teachers, B3-5-27, the “Virtual frog builder”, 23 teachers

• Chemistry– C3-5-15, “Periodic table tetris” used 23 times and

LO C4-5-16, “Characteristics of materials, used 17 times


Most popular LO per subcategory

• Mathematics– LO M2-5-38, “Maya numbers” used 23 times and LO

M2-13-42 “Trigonometry” used 21 times.

• Physics– LO P4-5-4, “Conservation of energy” used 22 times LO

P2-17-10, “Forces of the pendulum” 21 times.

• IT– T3-13-55, “Computer parts hangman” 33 times and LO

T4-13-56 “Click and learn computer parts” selected by 30 teachers


Disciplines where LO were used


Extracurricular activities where LO were used


How did pupils work?


Average number of LO used per pupil


How were LO used?

• Average groups of around 20 pupils• Average time spent on LO : 1.5 – 2 hours

– Ranging from 15 minutes (languages) to 14 hours

• Mostly Inquiry-Based Science Education (IBSE) approach

• 88% of the teachers worked with special equipment– Usually computer room (sometimes with Internet)– Sometimes beamer


Questions

• How is it possible to use LO without special IT equipment?

• What is meant by working for the LO 2 weeks, 3 months? Repeated use of same LO?


Part IV: Impact• On teachers

– Expected impact on teaching before use of LO – Impact on teaching after use of LO– Impact on pupils’ competences– Impact on pupils’ attitudes

• On pupils– Impact of MST before use of LO– Impact of MST after use of LO

• On schools• On education system


Teachers: Expected impact of LO


Teachers: Impact after use of LO


Teachers: Impact on teaching before and after


Teachers: Impact of LO on pupils’ appreciation of MST


Teachers: Impact on pupils’ attitudes


Pupils: Impact of MST before use of LO


Pupils’ appreciation of MST - gender


Pupils: Impact of MST after use of LO


Pupils: Impact of MST after use of LO – gender


Pupils: Impact of MST after use of LO – gender and age


Pupils: Impact of MST after use of LO – gender and age


Pupils: Impact of MST after use of LO youngest pupils– gender


Impact of LO: teachers vs. pupils


Impact of Inspire project on schools


Impact of Inspire project on education system


Impact of project: dissemination






Questions?

• What affected the impact most?• How could the impact be increased?


Part V: Success factors and facilitators


Differences

Some results from the country chapters


Expectations




Conclusions

• The Inspire project has been a success thanks to:– Motivation and hard work of all the teachers involved– Motivation and hard work of NC– Motivation and hard work of SC– Motivation of the pupils involved– The organisational structure (NC, SC etc.) set up by

European Schoolnet– The involvement of several ministries (official status

of project)


Conclusions• The use of LO definitely enhances the quality of teaching

and learning of MST • The use of LO enhances the acquisition of LLL skills• The impact of the LO is overall higher on boys than on

girls (exception LT)• The younger the pupils are the higher the impact• The number of LO used does not affect the impact• The impact of the project is not only experienced on the

schools involved but also on the education systems of the countries involved.


Recommendations

• Use LO much more widely across the curriculum– Selection of LO should not only be per

subcategory but also per age group– Use LO as much as possible with younger age

groups as they seem to have most impact with them

• Disseminate outcomes of Inspire project to the members of the MST Cluster of DG EAC


Thank you for your attention!

Any more questions or remarks?Any supplementary information is welcome!