26 June 2013

South West Forum Group

How impact measurement relates to

and supports quality improvement

Paul Fletcher Senior HMI

How impact measurement relates to and supports quality improvement

The Context

The Common Inspection Framework as an improvement tool

Tools used on Prepared for Inspection toPromote quality improvement

Local Accountability Survey – March 13 - Findings

Overall, the government’s policy to deregulate the curriculum for the learning and skills sector has been well received […], but systems to measure the impact of these changes are insufficiently robust.

One year after the policy was introduced, [providers] did not have sufficient evidence to determine the extent of the impact of any changes they had made to their provision on reducing youth unemployment and supporting local economic growth. Most [..]had not monitored these changes rigorously enough.

Local Accountability Survey – March 13 - Findings

Destination data were generally not sufficiently complete for managers to evaluate fully the impact of these programmes on supporting learners into work.

Managers and governors did not fully evaluate their work with, and its impact on, local communities and employers. A lack of coherent, consistent sector-wide measures of all learners’ destinations, and systems to track their progression over time, hampered the [..] ability to demonstrate how effectively they developed learners’ wider skills and supported their progression to sustainable employment.

The lack of consistent, sector-wide measures of learners’ progression, including into sustainable employment, has hindered [the] ability to evaluate the wider impact of their [..] work in the local area.

The Common Inspection Framework (CIF) The evaluation schedule is provider and context

specific

Forms the basis of self-assessment

Explicit Impact Measures - data Implicit Impact Measures – soft measures Outcomes Teaching, learning and assessment Leadership and management

Outcomes for Learners

All learners achieve and make progress relative to their starting points and learning goals

Achievement gaps are narrowing between different groups of learners

Learners develop personal, social, and employability skills

Learners progress to courses leading to higher-level qualifications and into jobs that meet local and national needs

Defining Outcomes –What Ofsted means by outcomesAchievement   

How well learners do overall taking into account their success rates, progress made, skill acquisition and development, standard of work and progression.

Progress   

The rate of learning and acquisition of knowledge and skills leading to the achievement of challenging targets, such as the level or grade. 

Starting points   

Learners’ prior attainment in terms of previous qualifications or skill level when they begin their course or programme of study.

Learning goals   

Learners’ main qualification, group of qualifications or agreed learning aim(s), if not on an accredited programme, including the level at which these are to be achieved.

Achievement gap   

The percentage point gap between the success rate of one group and another, such as age, gender, ethnicity and disability.

Defining Outcomes –What Ofsted means by outcomesPersonal and social skills   

The development of learners’ understanding of rights and responsibilities, their moral and cultural awareness and understanding of utilising resources in a sustainable way.

Employability skills   

The development of English, mathematics, functional and broader skills, such as leadership, teamwork and problem solving, to enable learners to progress to higher-level courses, and to employment and further training.

Defining Outcomes –What Ofsted means by outcomesRARPA  

Recognising and recording progress and achievement for non-accredited learning for adults such as in ACL provision, eg upholstery

Internal progression  

When learners progress from one level to the next within the same provider.

Destinations   

Where learners go when they leave the provider.

Job outcome payments  

Relates to courses for the unemployed where part of the payment a provider receives relates to a learner gaining a job even if they leave before achieving the learning aim.

Using learner-level data forquality improvement – the learner’s journey

Applications and recruitment Initial assessment Attendance Progress Assessment At-risk learners Retention Achievement Progression

Using learner-level data forquality improvement

Symptom? Cause? Effect?

Using learner-level data forquality improvement

Area of provision

for investigat

ion:

Compare the:

with the:Quality Improvement Questions:

Interpretation should be indicative to your provision

Progress 1

rate of progress of individual learners

expected rate of progress of individual learners by subject area (or sub-area)

What can the comparison tell you about:•your success in closing the ‘achievement gap’?•the thoroughness and accuracy of your initial assessment procedures?•the effectiveness of your review system for determining whether or not learners are progressing at the rate set out in their individual learning plans?•the sufficiency of the detail in your individual learning plans to be able to show a clear attainment plan?•your learners’ awareness of their relative progress rate?•the adequacy and relevance of the learner support you provide?•the support your learners are receiving from their employers?What is your progress-rate trigger figure for determining whether or not a learner is falling into an ‘at-risk’ category?What changes could you make to improve your provision?

Project no. 4 - Learners’ Progress

Using learner-level data forquality improvement

These data projects are available at www.excellencegateway.org.uk > Good Practice Database > Discussion Tools. You are welcome to use this information and adapt it for use within your organisation. Please do not use this information outside your organisation and always acknowledge the source.

Ends