Validation Service for running school pilotsPresenters: Dorothy Cassells, Jim Ayre, Anja Balanskat
http://lsl.eun.org
Living Schools Lab
http://lsl.eun.org #lsl_eu
3
Introduction Co-ordinated by European
Schoolnet (EUN)
15 partners, network of schools and teachers across 12 countries
Supported by the European Commission’s FP7
A two-year Project: October 2012 to September 2014
4
Living Schools Lab project•Creating a network of primary and secondary schools to showcase and demonstrate innovative use of ICT;•Gathering evidence of innovative practice in classrooms across Europe;•Providing new professional development opportunities for teachers;•Mainstreaming innovative practice via pan-European and regional collaboration;•Building a network of schools for validation pilots and a turnkey validation service.
Validation Service for running school pilotsPresenters: Jim Ayre, Anja Balanskat
LIVING SCHOOL LAB PROJECT: OBJECTIVES
1. Showcase a diverse range of innovative pedagogical practice involving ICT.
2. Strategies to embed innovative use of ICT across the whole school.
3. A growing network of schools interested in change management.
4. Provide a teacher community and CPD to help schools change their pedagogical practice.
11
5. Develop a sustainable network of schools offering a validation service for future projects and research.
LSL RATIONALE – VALIDATION SERVICE?• By the end of the project the Commission hopes
to have a network of schools interested in:• Testing and providing feedback on the results from
EC-funded projects.• Running school / classroom pilots with ICT vendors,
content providers and other stakeholders.•What will motivate schools and teachers to want to do this?
www.europeanschoolnet.org - www.eun.org
A flexible learning space with state-of-the-art technology to experiment with different styles of learning
A space for policy makers, companies and teachers to rethink future learning strategies
FUTURE CLASSROOM LAB
THE FCL IN SCHOOLS ACROSS EUROPE
Ancona, Italy
Crema, Italy
Ghent, Belgium
Setubal, Portugal
Zagreb, Croatia
…and soon near you? - Tel Aviv, Israel- Turnhout, Belgium- Tallinn, Estonia- Bejing, China- Kuala Lumpur, Malaysia- …
SUCCESSFUL EXAMPLES
Free online professional development courses for teachers.• Innovative Practices for Engaging STEM Teaching: #STEMcourse• Future Classroom Scenarios: #FCScourse• www.EuropeanSchoolnetAcademy.org #EUNAcademy
www.europeanschoolnet.org - www.eun.org
iTEC http://itec.eun.org CPDLab http://cpdlab.eun.org
Living Schools Lab http://lsl.eun.org
Creative Classrooms Lab http://creative.eun.org
Synergy of FCL with EUN projects
TOOLKITS TO SUPPORT INNOVATION IN LEARNING AND TEACHING
Scenario Development Learning Activity Design
Pilot and Evaluation
Future Classroom Toolkit
Scenarios inspiring innovation and guiding change management.
Innovative learning designs using technology.
Evidence of Impact influencing policy, delivery and supply.
European Schoolnet Academy
Future Classroom Ambassadors
Toolkits for school leaders and teachers
Teacher Communities
Training and CPD
11
So, before we offer a FCL validation service, the first step is to develop a network of Future Classroom Ambassadors
TEACHER MOTIVATION FOR JOINING LSL
17
TEACHER VALIDATION REWARDS / INCENTIVES
17
R ewa rd schools fina nc ia l ly for t he wor k involved.
I nvolve t he sc hool in a n a mba ssa dor prog r amme.
R ec og nise t he schools by promoting t heir wor k (e . g . via ca se st udies, or videos).
D ona t e t he ir t echnolog y a nd pr ovide a fr ee l ic ense t o use t he ir soft wa re/cont ent a ft er t he end of t he pi lot .
6
8
7
10
4
3
5
7
Extremely important Very important
VALIDATION PROCESS FOR SCHOOLS• Training is important
• Synchronous/asynchronous, some face-to-face• Role of CoP, help line…..
• Simple data recording• Time-efficient & flexibility• Online meetings, short & simple surveys, videos of practice• Dependent on validation technology
• Timeframe• At least six months
• Timing• October to May, excluding December & January
EC PROJECTS• Discussions with 9 projects
•educational games, artificial/intelligent tutors, learning agents.....
• Possibility that they can provide:• professional development workshops • or community of practice
• Unlikely to provide hardware/software• Many developing research prototypes• Hard to work with running projects (fixed budget)
INDUSTRY PARTNERS IN THE FUTURE CLASSROOM LAB
ICT COMPANIES
• Are interested in a validation service• Need faster research than in EC projects• Possibility to provide some hardware, software, content, services (e.g. Creative Classrooms Lab project on tablets)
• Even SMEs can provide a free licence for software
LESSONS LEARNED• Material benefits for schools is preferred over personal benefits for teachers
• Hardware donations are a key incentive and training could be an important incentive
• Even SMEs can provide some training and possibly software
• But software may not be an incentive if it requires teachers to invest a lot of time (e.g. a new learning platform)
• Many ICT vendors have unrealistic expectations about what is needed to conduct an impact study
VALIDATION SPECTRUM
‘Heavier’ Classroom
Pilots €€€€
‘Lighter’ teacher testing
€
Example:- Weeks or months- Sustained use of technology with students in
the classroom- Classroom observations- National focus group- FCL workshops in Brussels- Teacher summer school
Example:- Hours or days- Quick feedback on a web portal,
content, course…- Online questionnaires
…….TEACHER / SCHOOL INCENTIVES & REWARDS…….
European Schoolnet Academy
FCL ambassador teachers
VALIDATION SERVICE
• A validation service encompasses all the actions and processes that need to be undertaken to carry out the pilot and the evaluation.
This includes: • identifying clear objectives for the validation• defining specific research questions• developing an evaluation methodology and instruments for data
collection;• documenting how to set up and run a school/classroom pilot that
includes criteria for selecting schools/teachers and training/support mechanisms.
To set up a school pilot
Coordination
Selection
Validation
Incentives
EUN
Industry
EC
Schools
National
Regional policy
Main actors involved in the validation process
Evaluation Design
Timeframe
Scope
Budget
Purpose of the
evaluation
Product/Intervention
variable
Testbed/Schools
Factors influencing the evaluation design
IN THE FCL VALIDATION SERVICE WE USE THE TERM VALIDATION…
• …in its wider sense and mean the scientific EVALUATION of the (pilot) intervention, which aims to:
• test/validate a specific ICT product, tool, content or service
• in an educational context • against a set of objectives and research
questions • jointly agreed upon by the initiator of the
validation and the evaluator.
WHAT IS EVALUATION?
Evaluation (in teaching and learning) refers to a systematic process aimed at judging the effectiveness of any teaching and learning programme.
• Formative evaluation: Evaluation that is used to modify or improve products, programs, or activities and is based on feedback obtained during their planning and development.
• Summative evaluation: Evaluation at the conclusion of an activity or plan to determine its effectiveness.
ACTION RESEARCH?
• Action Research’ is a recognized form of applied research that focuses on the effects of the researcher's direct actions of practice within a participatory community with the goal of improving the performance quality of the community or an area of concern (Dick, 2002, et.al.)
• Help professionals to improve practice by involving them actively in the research
• Research is driven by the practitioner (e.g. an identified problem)
• A way of engaging and motivating teachers
• Does not dictate the use of any specific research tools
• Compatible with short-term investigation into the changes in teaching and learning imposed by rapid technological change
FCL EVALUATION APPROACH AND METHODS
Quantitative
Simple test and control groups
Before and after approach
Statistical Matching
Surveys
Randomised experimentation
Qualitative
Case studies
Story telling and narrative inquiry
Surveys
Action Research
ComplexityReliabilityExpertise
THE EVALUATION DESIGN
1. What is the purpose of the investigation?2. Formulating a research question 3. Carry out desk research 4. Define the sample 5. Decide on the instruments required
Research question:• What, what if, why, who, how,
where, when?• Predict• Understand• Explore• Testing• Explanation• Description• Relations• Comparisons• Correlations
• Action research• Qualitative,
quantitative, mixed method
• Online surveys• Focus groups• Randomised control
trials• Blog entries• Case studies• Semi- structured
interviews• Classroom observations
• Target population (country, education level, grade, gender, subjects, school)
• Random sampling/representivity
SCENARIO EXAMPLE 1: SMALL SCALE PILOT
• Purpose: Observing T&L practice when using X ?• Question: How does a platform/application work in different
national contexts?• Size : e.g. 10 to 30 teachers, 10 teachers per country, 1
to 3 schools per county, 1-3 countries• Duration: 1-3 months• Methods: Focus groups, interviews, story telling• Support: Online webinar and workshop in Brussels
SCENARIO EXAMPLE 2: MEDIUM SCALE PILOT
• Purpose: Testing a change in teaching and learning practice when using X
• Question: Evaluate the effectiveness of a new education specific tablet
• Size : e.g. 50 teachers per country, 5-10 schools per country, 5 countries
• Duration: 6 months, minimum one school term• Methods: Before and after approach, case studies, observation
visits• Support: Webinars, online platform, community of practice,
workshop in BXL
SCENARIO EXAMPLE 3: LARGE SCALE PILOT
• Purpose: Analysing the impact of X on teaching and learning practice
• Question: What is the impact of a new tablet device on educational outcomes?
• Size : e.g. e.g. 3000 teachers from 10 countries, 500 to 1000 teachers per country, 20-30 schools per country, 10 -15 countries
• Duration: 1 to 3 years• Methods: e.g. Randomised control trial/ large scale quantitative
survey• Support: Workshop in Brussels, online training , national
coordinators, online testing platform
CONTEXTUAL EVALUATION FACTORS What you put in will affect your outcomes…
TechnologyServiceMethod
Means for evaluation
Pilot support
Pilot Coordination
ReadinessIncentives
NEXT STEPS1. Ministries of Education are being asked to support Lead
Future Classroom Ambassadors in each country2. 12 Ministries have already appointed a Lead Ambassador3. Lead Ambassadors will help coordinate volunteer Future
Classroom Ambassadors at national level4. 6-7 November - Training workshop for Lead Ambassadors in
the Future Classroom Lab in Brussels5. January 2015 - information for teachers on how to become
an ambassador and participate in the validation service
VALIDATION MANUAL – NOVEMBER 2014A step by step guide on:1. Different evaluation
approaches in school pilots
2. How to design and run your own school pilots
3. Templates and tools to help you do this
Thank youweb http://lsl.eun.org and http://fcl.eun.orgemail [email protected]
The work presented in this presentation is partially supported by the European Commission’s 7th Framework Programme – project Living Schools Lab (Grant agreement Nº 317587). The content of this presentation is the sole responsibility of the consortium members and it does not represent the opinion of the European Commission and the Commission is not responsible for any use that might be made of information contained herein.