Introduction

Katy Owen Senior Policy Advisor

Cabinet Office

City Year

Projects we’re funding

Evidence • Educational Endowment Fund (EEF) Toolkit

• Strong theories of change / logic models

• Promising evidence

• Independent evaluations commissioned

Evidence

“Meta-analyses indicate that pupils [receiving 1-to-1 tuition] might make about 4 or 5 months progress during an intensive programme.”

Evidence “City Year provides enthusiastic and appropriately trained young volunteers to act as mentors and role models for the pupils. Together with teachers, they help establish and maintain very good relationships between pupils and staff.”

Evidence

Percentage of School Leavers Attaining a University Place in 2012

Building the evidence base • Impact evaluations • Process evaluations

The Marketplace

Michael Stevenson Education Consultant and Nesta

Associate

Code Club

• Besieged by suppliers unwanted emails and calls

• Suppliers failing to target their offers

• Suppliers not able to articulate educational outcomes

• Most evidence-based research is difficult to use and not context specific

Schools struggle to make informed and objective decisions when purchasing interventions

Key findings

9

• Highly fragmented market and difficult to know exactly what is out there to choose from

• Unclear credentials

Uncertainty about what services exist

Shot-gun approach to selling

Lack of evidence on quality outcomes

• Buying frameworks are in place

• Purchaser rarely consults program specific research

• Purchasing decision usually based on recommendations from other teachers

Purchasing is recommendation-led

1

3

4

5

• Schools liked the idea of an online directory and teacher recommendation portal

• Schools loved the idea that this could include independent third-party impact evaluation

A ‘TripAdvisor’, digital-based solution is attractive

6

• Significant variation in the service quality experienced

• Misdistribution of high quality services, limited outside of London and especially in rural areas

Variable quality and poor distribution of services

2

Other market participants advocate a broader approach to securing an effectively functioning market

Key findings

10

Digital platform

• What should students know and be able to do?

• How should we measure success?

• Clear set of maintained standards

• Industry-wide training

• Support for a healthy and efficient market of high quality interventions

Key solution components already exist and can be built upon

Proposed solution

11

Provider (product name)

Simple rating system

Enables user

feedback

Program specific impact

data

Provides evidence

based reviews

Cost to use

Usability

Note

Education Endowment Foundation (Toolkit)

Social Research Unit, Dartington (Investing in Children)

Institute for Effective Education, University of York (Evidence4Impact)

Notes: (*) Online version in pilot. All other products shown here are accessible online. Phase 2 (earliest 2015) will incorporate predictive capability to assess needs and use third-party to review sourced impact data

FREE

FREE

FREE

Increasing credibility

Click to

buy

• Analyses categories of service Easy

Capability

Partial capability

No capability

Easy

Easy • Interventions are rated only if

evidence is available

Key component

Database / framework

Database / framework

Database / framework

Teachers Development Trust (Good CPD Guide)

Edukit* £70 per

child

FREE

Easy • Supports student diagnosis and

provides a matching service

Proto-platform

Proto-platform

The digital platform should identify student needs, match needs to programs and provide comprehensive assessment of quality

Proposed solution

12

Functionality Minimum viable product Ideal product

Identify student needs

• Identify and prioritize student’s needs, using teacher and student data

• Identify and prioritize student’s needs, using teacher, student and school data

• Introduce online tests to help students identify their needs

• Student MOT at 11

Match student’s needs to supplier’s service

• Search database • Identify and prioritize relevant programs • Show location and price of program • Deploy behavioral economic techniques

• Search results to show independent impact analysis, where available

• Search results to show characteristics of schools or students who’ve used program

Assessment of service impact

• Show ratings and awards • Show supplier’s analysis of impact on learning

outcomes • Show user references • Show user complaints • Track service uptake, via online purchasing data

• Show tracking analysis of teacher prediction at 11 versus actual outcome (post program)

Assuming the pilot is successful, the sector (schools, suppliers, organisations) should work closely with government to steer the development of the market as proposed

Proposed actions

13

1. Clarify and introduce assessment of what students should know and be able to do at 16

2. Design a long-term approach to measuring the impact of individual services and programs

3. Design and establish effective regulation: platform and services

4. Continue to strengthen the profession’s competence in the use of data to commission services and programs

5. Support suppliers to strengthen the demonstrable impact of their programs and services on education outcomes