Creating a culture - NEON · Creating a culture. Creating buy-in/raising awareness •Unpicking...

Creating and Embedding an Evaluation Culture in WP Project Work

Lydia Redican – Widening Participation Project Officer University of Liverpool lydia.redican@liverpool.ac.uk

Creating a culture

Creating buy-in/raising awareness

• Unpicking previous evaluation work that had been done

• Understanding who is a resource or a source of support

• Guidance from OfS should now give a focus with the new self assessment toolkit

• Number of new Evaluation Officer posts has grown and has created

Initial research

• Use the OfS Guidance, Standards of Evidence and Standards of Evaluation (Crawford et al 2017).

• Other sources of material

• Has anything else been written about your programmes, for example Programme Reviews, Focus Groups, Strategic Plans etc.

Resources and capacity

• Once the framework is set it impacts less on capacity

• Divide up areas of focus between teams

• Time for Evaluation is set in PDR

Embedding Evaluation

Logic Modelling

The ‘Liverpool Model’: methodology

Activity mapping

Practitioner intuition

Professional perspectives

Milestone data

Creating evidenced attribution

Survey, focus groups, etc.

Insight; moving to strategic inquiry

Dialogue, critical reflection, reporting, evaluation refinement, hypothesis, plausible scenarios

Moving to second phase and ‘2nd tier rigour’Case studies, control-based comparison studies, refined focus, extended inquiry etc.

Creating tentative attribution

First phase and ‘1st tier rigour’

Decide what is your core and enhanced evaluation• We decided what was our core evaluation

that could be measured consistently over an extended period of time

• Core• UG Surveys

• First Year Success

• Set framework meetings

• Enhanced: • Focus groups, academic interviews, practitioner

insight, student surveys.

Regular meetings with key stakeholders to develop and discuss the focus

• Who are your key stakeholders? • Admissions, Strategic Planning, Academics

with education/evaluation research experience, Student Recruitment, WP Leads in faculties, Student support/Careers and Employability, your students, parents and teachers.

• What is your focus? • Reviewing your core evaluation and

deciding how to tackle your enhanced

Map out your evaluation framework

• Identify your stakeholders and set your meetings and activities, have it planned at least a year in advance.

• Set your mile stones for evaluation activity, surveys, focus groups, data analysis.

• Where does the evaluation go?

• What do you do with the evaluation? Your programmedesign!

Liverpool Scholars Evaluation Framework

• Core Evaluation • Focus Groups

• Academic Interviews

• End of Programme Survey

• Undergraduate Survey for qualitive Impact

• Undergraduate data analysis for quantativeimpact

• Monitoring

• Enhanced – events, academic skills, experience, UCAS, BAME, pastoral support.

Focus Groups – Liverpool Scholars Example

• Every year we identify an area of focus to discuss with our students

• How we delivered the focus groups

• Our Expected Assumptions from the questions

• The Emerging Assumptions from the feedback

• Paper discussing both sets of assumptions and finish with an Action Plan

lydia.redican@liverpool.ac.uk