SCC2013 - The challenges of measuring informal science learning - Steph Sinclair

Informal Science Learning

Stephanie Sinclair

Today’s Talk

• Reminder what the Review set out to achieve

• Highlight key findings from the Informal Learning Review

• Examine what this means for the sector

Aims of the Review

Practical outcomes of the study:

• A better understanding of the scope of informal learning, its theoretical base and the types of change it can bring about in people’s understanding, behaviour and attitudes to science

• Better understanding of how to evaluate the impact of informal science learning

• Best practice in reaching deprived learners schools and families

• Best practice in linking informal and formal learning.

2 Reports Commissioned

• From UK-based GHK Consulting – report examines who is doing what, who are they reaching and how are they evaluating.

• From Stanford - Oregon State University- examining the science education ‘ecosystem’ and the role of informal learning within it.

• Plus a commentary piece from John Holman and Clare Matterson

Fieldwork

Autumn 2011 – Spring 2012

Interviews: 60 stakeholders of informal learning

Surveys: online surveys of informal learning providers

Case studies: 10 exemplar organisations

Family studies: family choices for their leisure time

Literature review: academic and ‘grey’ literature

Ecological mapping: to describe the science learning eco-system

Findings – the Informal Learning Landscape

• Diversity: ‘…. As far as we can tell, such diversity is not matched elsewhere’

• Strong sense of mission

• Complex interactions…

Quantity of interactions among UK science education sectors – greatest interactivity in the middle

Stanford-Oregon: Analysing the UK Science Education Community: The contribution of informal providers: November2012: Wellcome Trust

Findings – Evaluation

Evaluation activity is widespread but unsophisticated:

- user surveys are most common methodology used

- mostly done by providers themselves, not by external evaluators

‘Overall, this is a community eager to find out what its users think of its activities, but less inclined to measure long-term impact’

What best describes your evaluation activity?

Statement   Responses Percentage

Formative evaluation   160 91%

Summative evaluation   29 15%

Evaluate when required by funders   69 39%

Planned rolling evaluation programme in place   37

21%

Evaluate if concerns identified   32 18%

Evaluation undertaken on an ad hoc basis   24 14%

Barriers to evaluation

Findings – Research, practice and profession

How well known is the literature?

• Modal value for how many individuals had read most cited publications = 0

• Modal value for how many recognised but had not read each publication = 2

• Most read article read by < 50%

So what do they read?

- Policy documents, evaluations and online resources (eg > 50% read the ASDC newsletter)

So what do these findings mean for the sector?

- How we evaluate

- How we increase research base and research capacity

- Better linking research and practice

Moving towards a more sophisticated evaluation model

• greater sharing of results

• identification of a common set of related proximal and distal indicators of impact

• working with other sectors to test alternative methods

Linking research and practice

What Wellcome Trust are doing

•Exploring potential programme of research grants with National Science Foundation

What we can all do

-Ensure we are using research to underpin our own practice