Validation Service for running school pilotsPresenters: Dorothy Cassells, Jim Ayre, Anja Balanskat

http://lsl.eun.org

Living Schools Lab

http://lsl.eun.org #lsl_eu

3

Introduction Co-ordinated by European

Schoolnet (EUN)

15 partners, network of schools and teachers across 12 countries

Supported by the European Commission’s FP7

A two-year Project: October 2012 to September 2014

4

Living Schools Lab project•Creating a network of primary and secondary schools to showcase and demonstrate innovative use of ICT;•Gathering evidence of innovative practice in classrooms across Europe;•Providing new professional development opportunities for teachers;•Mainstreaming innovative practice via pan-European and regional collaboration;•Building a network of schools for validation pilots and a turnkey validation service.

Validation Service for running school pilotsPresenters: Jim Ayre, Anja Balanskat

LIVING SCHOOL LAB PROJECT: OBJECTIVES

1. Showcase a diverse range of innovative pedagogical practice involving ICT.

2. Strategies to embed innovative use of ICT across the whole school.

3. A growing network of schools interested in change management.

4. Provide a teacher community and CPD to help schools change their pedagogical practice.

11

5. Develop a sustainable network of schools offering a validation service for future projects and research.

LSL RATIONALE – VALIDATION SERVICE?• By the end of the project the Commission hopes

to have a network of schools interested in:• Testing and providing feedback on the results from

EC-funded projects.• Running school / classroom pilots with ICT vendors,

content providers and other stakeholders.•What will motivate schools and teachers to want to do this?

www.europeanschoolnet.org - www.eun.org

A flexible learning space with state-of-the-art technology to experiment with different styles of learning

A space for policy makers, companies and teachers to rethink future learning strategies

FUTURE CLASSROOM LAB

THE FCL IN SCHOOLS ACROSS EUROPE

Ancona, Italy

Crema, Italy

Ghent, Belgium

Setubal, Portugal

Zagreb, Croatia

…and soon near you? - Tel Aviv, Israel- Turnhout, Belgium- Tallinn, Estonia- Bejing, China- Kuala Lumpur, Malaysia- …

SUCCESSFUL EXAMPLES

Free online professional development courses for teachers.• Innovative Practices for Engaging STEM Teaching: #STEMcourse• Future Classroom Scenarios: #FCScourse• www.EuropeanSchoolnetAcademy.org #EUNAcademy

www.europeanschoolnet.org - www.eun.org

iTEC http://itec.eun.org CPDLab http://cpdlab.eun.org

Living Schools Lab http://lsl.eun.org

Creative Classrooms Lab http://creative.eun.org

Synergy of FCL with EUN projects

TOOLKITS TO SUPPORT INNOVATION IN LEARNING AND TEACHING

Scenario Development Learning Activity Design

Pilot and Evaluation

Future Classroom Toolkit

Scenarios inspiring innovation and guiding change management.

Innovative learning designs using technology.

Evidence of Impact influencing policy, delivery and supply.

European Schoolnet Academy

Future Classroom Ambassadors

Toolkits for school leaders and teachers

Teacher Communities

Training and CPD

11

So, before we offer a FCL validation service, the first step is to develop a network of Future Classroom Ambassadors

TEACHER MOTIVATION FOR JOINING LSL

17

TEACHER VALIDATION REWARDS / INCENTIVES

17

R ewa rd schools fina nc ia l ly for t he wor k involved.

I nvolve t he sc hool in a n a mba ssa dor prog r amme.

R ec og nise t he schools by promoting t heir wor k (e . g . via ca se st udies, or videos).

D ona t e t he ir t echnolog y a nd pr ovide a fr ee l ic ense t o use t he ir soft wa re/cont ent a ft er t he end of t he pi lot .

6

8

7

10

4

3

5

7

Extremely important Very important

VALIDATION PROCESS FOR SCHOOLS• Training is important

• Synchronous/asynchronous, some face-to-face• Role of CoP, help line…..

• Simple data recording• Time-efficient & flexibility• Online meetings, short & simple surveys, videos of practice• Dependent on validation technology

• Timeframe• At least six months

• Timing• October to May, excluding December & January

EC PROJECTS• Discussions with 9 projects

•educational games, artificial/intelligent tutors, learning agents.....

• Possibility that they can provide:• professional development workshops • or community of practice

• Unlikely to provide hardware/software• Many developing research prototypes• Hard to work with running projects (fixed budget)

INDUSTRY PARTNERS IN THE FUTURE CLASSROOM LAB

ICT COMPANIES

• Are interested in a validation service• Need faster research than in EC projects• Possibility to provide some hardware, software, content, services (e.g. Creative Classrooms Lab project on tablets)

• Even SMEs can provide a free licence for software

LESSONS LEARNED• Material benefits for schools is preferred over personal benefits for teachers

• Hardware donations are a key incentive and training could be an important incentive

• Even SMEs can provide some training and possibly software

• But software may not be an incentive if it requires teachers to invest a lot of time (e.g. a new learning platform)

• Many ICT vendors have unrealistic expectations about what is needed to conduct an impact study

VALIDATION SPECTRUM

‘Heavier’ Classroom

Pilots €€€€

‘Lighter’ teacher testing

€

Example:- Weeks or months- Sustained use of technology with students in

the classroom- Classroom observations- National focus group- FCL workshops in Brussels- Teacher summer school

Example:- Hours or days- Quick feedback on a web portal,

content, course…- Online questionnaires

…….TEACHER / SCHOOL INCENTIVES & REWARDS…….

European Schoolnet Academy

FCL ambassador teachers

VALIDATION SERVICE

• A validation service encompasses all the actions and processes that need to be undertaken to carry out the pilot and the evaluation.

This includes: • identifying clear objectives for the validation• defining specific research questions• developing an evaluation methodology and instruments for data

collection;• documenting how to set up and run a school/classroom pilot that

includes criteria for selecting schools/teachers and training/support mechanisms.

To set up a school pilot

Coordination

Selection

Validation

Incentives

EUN

Industry

EC

Schools

National

Regional policy

Main actors involved in the validation process

Evaluation Design

Timeframe

Scope

Budget

Purpose of the

evaluation

Product/Intervention

variable

Testbed/Schools

Factors influencing the evaluation design

IN THE FCL VALIDATION SERVICE WE USE THE TERM VALIDATION…

• …in its wider sense and mean the scientific EVALUATION of the (pilot) intervention, which aims to:

• test/validate a specific ICT product, tool, content or service

• in an educational context • against a set of objectives and research

questions • jointly agreed upon by the initiator of the

validation and the evaluator.

WHAT IS EVALUATION?

Evaluation (in teaching and learning) refers to a systematic process aimed at judging the effectiveness of any teaching and learning programme.

• Formative evaluation: Evaluation that is used to modify or improve products, programs, or activities and is based on feedback obtained during their planning and development.

• Summative evaluation: Evaluation at the conclusion of an activity or plan to determine its effectiveness.

ACTION RESEARCH?

• Action Research’ is a recognized form of applied research that focuses on the effects of the researcher's direct actions of practice within a participatory community with the goal of improving the performance quality of the community or an area of concern (Dick, 2002, et.al.)

• Help professionals to improve practice by involving them actively in the research

• Research is driven by the practitioner (e.g. an identified problem)

• A way of engaging and motivating teachers

• Does not dictate the use of any specific research tools

• Compatible with short-term investigation into the changes in teaching and learning imposed by rapid technological change

FCL EVALUATION APPROACH AND METHODS

Quantitative

Simple test and control groups

Before and after approach

Statistical Matching

Surveys

Randomised experimentation

Qualitative

Case studies

Story telling and narrative inquiry

Surveys

Action Research

ComplexityReliabilityExpertise

THE EVALUATION DESIGN

1. What is the purpose of the investigation?2. Formulating a research question 3. Carry out desk research 4. Define the sample 5. Decide on the instruments required

Research question:• What, what if, why, who, how,

where, when?• Predict• Understand• Explore• Testing• Explanation• Description• Relations• Comparisons• Correlations

• Action research• Qualitative,

quantitative, mixed method

• Online surveys• Focus groups• Randomised control

trials• Blog entries• Case studies• Semi- structured

interviews• Classroom observations

• Target population (country, education level, grade, gender, subjects, school)

• Random sampling/representivity

SCENARIO EXAMPLE 1: SMALL SCALE PILOT

• Purpose: Observing T&L practice when using X ?• Question: How does a platform/application work in different

national contexts?• Size : e.g. 10 to 30 teachers, 10 teachers per country, 1

to 3 schools per county, 1-3 countries• Duration: 1-3 months• Methods: Focus groups, interviews, story telling• Support: Online webinar and workshop in Brussels

SCENARIO EXAMPLE 2: MEDIUM SCALE PILOT

• Purpose: Testing a change in teaching and learning practice when using X

• Question: Evaluate the effectiveness of a new education specific tablet

• Size : e.g. 50 teachers per country, 5-10 schools per country, 5 countries

• Duration: 6 months, minimum one school term• Methods: Before and after approach, case studies, observation

visits• Support: Webinars, online platform, community of practice,

workshop in BXL

SCENARIO EXAMPLE 3: LARGE SCALE PILOT

• Purpose: Analysing the impact of X on teaching and learning practice

• Question: What is the impact of a new tablet device on educational outcomes?

• Size : e.g. e.g. 3000 teachers from 10 countries, 500 to 1000 teachers per country, 20-30 schools per country, 10 -15 countries

• Duration: 1 to 3 years• Methods: e.g. Randomised control trial/ large scale quantitative

survey• Support: Workshop in Brussels, online training , national

coordinators, online testing platform

CONTEXTUAL EVALUATION FACTORS What you put in will affect your outcomes…

TechnologyServiceMethod

Means for evaluation

Pilot support

Pilot Coordination

ReadinessIncentives

NEXT STEPS1. Ministries of Education are being asked to support Lead

Future Classroom Ambassadors in each country2. 12 Ministries have already appointed a Lead Ambassador3. Lead Ambassadors will help coordinate volunteer Future

Classroom Ambassadors at national level4. 6-7 November - Training workshop for Lead Ambassadors in

the Future Classroom Lab in Brussels5. January 2015 - information for teachers on how to become

an ambassador and participate in the validation service

VALIDATION MANUAL – NOVEMBER 2014A step by step guide on:1. Different evaluation

approaches in school pilots

2. How to design and run your own school pilots

3. Templates and tools to help you do this

Thank youweb http://lsl.eun.org and http://fcl.eun.orgemail lsl-info@eun.org

The work presented in this presentation is partially supported by the European Commission’s 7th Framework Programme – project Living Schools Lab (Grant agreement Nº 317587). The content of this presentation is the sole responsibility of the consortium members and it does not represent the opinion of the European Commission and the Commission is not responsible for any use that might be made of information contained herein.