Curtis Cartwright Programme Evaluation Jan2010

www.curtiscartwright.co.uk CC382D006 – 1.0 - 1

Evaluation

Dr Geoff CurtisDr Rob Hawtin

29 January 2010

Institutional Innovation Programme

www.curtiscartwright.co.uk CC382D006 – 1.0 - 2

Overview

• Institutional Innovation Programme (IIP)

• Evaluation questions

• Headlines from evaluation of Round 1 projects

• Initial findings

• Lessons identified

• Evaluation of Round 2 projects

• Approach

• Your contribution

• Self-evaluation guidance

www.curtiscartwright.co.uk CC382D006 – 1.0 - 3

Institutional Innovation Programme

• Issues with JISC approach to ICT in education and research

• Often applies to specific problems of department / institution

– Not the sector

• Solution often technology- rather than business-led

• Not necessarily sustainable

www.curtiscartwright.co.uk CC382D006 – 1.0 - 4

IIP concept

• Institutional approach

• Build on advances from JISC innovation activities

– Proven practices, technologies, standards and services

– Often substantial internal business change for institution

– Supported by significant senior management involvement

• Fund range of substantial ICT deployments/business change activities

• Use as exemplars to demonstrate innovation and good practice

• Extend benefits to sector

– Benefits realisation activities

• Problem areas identified

– professional bodies

– other initiatives

www.curtiscartwright.co.uk CC382D006 – 1.0 - 5

Top-level evaluation questions

• 1: Is IIP concept useful as general approach for institution-wide

problems?

• 2: Is IIP delivering desired outcomes at institutional level for key

areas of institutional ICT concern?• Improved community engagement• Improved and sustained JISC leadership to the sector• Improved capacity, knowledge and skills and diversity of background;• Sustainable technological solutions• Development of good practice approaches• Evidence base for good practice and dissemination• Overall sustainability of outcomes

• 3: Is IIP being managed well?

www.curtiscartwright.co.uk CC382D006 – 1.0 - 6

Evaluation

• Assessment of evaluation questions

• Lessons identified?

• Improvements?

• What else?

• Evaluating programme not the projects

www.curtiscartwright.co.uk CC382D006 – 1.0 - 7

Initial findings –IIP concept

• Assessment

• Useful general approach for creating sustainable change

• More general applicability

• Lesson identified

• Need to select/manage projects to meet IIP characteristics

• Areas for improvement

• Benefits realisation activities

• More stringent selection of areas of institutional concern and projects

• Training on concept and implementation

• Review approach to planning/funding of benefits realisation activities

www.curtiscartwright.co.uk CC382D006 – 1.0 - 8

Initial findings – IIP delivery

• Assessment

• Too early for definitive assessment

– SSBR had only just started

– Likely to mature post project completion

– Aspirational nature of outcomes

• Continue to focus on desired outcomes

www.curtiscartwright.co.uk CC382D006 – 1.0 - 9

Initial findings – IIP management

• The programme is generally well managed

• Lack of programme management resource and overall scale of Round 2

projects might be a risk to programme

• Focus role of SSBR

www.curtiscartwright.co.uk CC382D006 – 1.0 - 10

Initial findings – summary

Question After Round 1 Potential

1 IIP concept

2 IIP delivery

3 IIP management

Overall

www.curtiscartwright.co.uk CC382D006 – 1.0 - 11

Evaluation of IIP – phase 2

• Approach

• Phase 1 was by interviews against detailed questions

– Many more Round 2 projects

• Evaluation based on

– Questionnaires– Detailed evaluation questions as before– Quantitative and qualitative– Stakeholders– Project teams

– Follow-up interviews, where necessary

– Other available information– Programme meetings– Final and completion reports

– Self-evaluation

www.curtiscartwright.co.uk CC382D006 – 1.0 - 12

Evaluation of Round 2 projects

• Your contribution

• Please find time to respond to questionnaires

– End February

• Please do your final and completion reports

– Good things

– Lessons identified

www.curtiscartwright.co.uk CC382D006 – 1.0 - 13

Self-evaluation

• JISC evaluation handbook

• Based on your evaluation plans

• Tips on self-evaluation

• Outcomes

www.curtiscartwright.co.uk CC382D006 – 1.0 - 14

Contact us

• geoff.curtis@curtiscartwright.co.uk 01483 685023

• rob.hawtin@curtiscartwright.co.uk 01483 685024