Jason@thinklearndo.org Evaluating Capabilities: Building consensus around capability measures and...

jason@thinklearndo.org

Evaluating Capabilities: Building consensus around

capability measures and indicators

Jason PollardThink:Learn:Do

jason@thinklearndo.org

Background

• Integrative Development Model (Kia-Keating, 2009)

• Young Foundation Report

• Adoption by commissioners

• Challenge for service deliverers

• Opportunity for the sector

jason@thinklearndo.org

The challenges of evaluation

• Bringing capabilities into a single tool

• Developing a set of (valid) indictors

• Scoring a capability measure

• Getting the right data out of the service

jason@thinklearndo.org

Meeting the Challenge

• Local evaluation network

• Feasibility study –

– Asked can operationally useful information be collected to support operational decision making

jason@thinklearndo.org

Evaluation Network

• Recruited six youth providers through VAL

• Jointly developed a set of indicators

• Pilot evaluation and pre-test of data collection tool

• Full evaluation to ‘road test’ tools

jason@thinklearndo.org

Developing the Indicator Set

Definitions

Measures

Indicators

jason@thinklearndo.org

Lewisham – Capability Measures

1. Being Creative2. Communication3. Relationships and Leadership4. Resilience5. Confidence and Agency6. Managing Feelings7. Planning and Problem Solving8. Strengthening Citizenship9. Making the most of London

jason@thinklearndo.org

Delphi – Agreeing Definitions

Example – Being Creative

Content Category What we mean by....

Learning about how to express their feelings. in expressing emotional

life Using Creativity to express how I

feel

A process and a way of channelling emotions, confidence, insecurities and confidence

Skills [and] tools to illustrate emotions

Music and poetry as a tool to deliver a message to communicate with

others Discussing friendships

Development of communication

jason@thinklearndo.org

Delphi – Agreeing Measures

How to measure Being Creative

What we mean by ... How we will measure it ...

Using creativity to express how I feel

The young person has the ability to express themselves creatively

Exploring learning through creative activities

The young person has the ability to use his or her creativity to set and pursue learning goals

Creative thinking and practical problem solving

The young person can test and apply learning to new and existing problems they encounter

jason@thinklearndo.org

Delphi – Agreeing Indicators

What would we expect to see?

Creativity

The young person can express themselves in a range of activities

The young person can articulate what they mean, and use collaborative activities to communicate this to others

The young person can identify and set personal learning goals

The young person is able to challenge their view of the world, and look for new solutions to problems they face

The young person takes the lead in finding solutions to problems

The young person is able to adapt their plans on taking advice

jason@thinklearndo.org

The Indicator Set

• 27 indicators

• Equally split over 9 outcomes

• What we would expect to see

• Conceptually linked to outcomes

jason@thinklearndo.org

Developing the Data Collection Tool

• A ‘composite’ tool – brings together several capability scales

• Valid and reliable (in pre-test)

• Soft Data – questions relating to the development of the young person and their engagement with their project

• Hard Data – age-specific questions directly relating to the capabilities

jason@thinklearndo.org

Next Steps

• Pilot - Scale-up: 10+ organisations to collect common data

• Explore Thematic indicators (talent; financial capability, etc)

• Establish evaluation communities of interest (Co-ops; CiCs)

jason@thinklearndo.org

End