Telling the Difference between Snake Oil and Good Oil; Evaluation Capacity Building in the public sector

Bronte Koop, Principal Consultant, Strategy and Evaluation

John Collins, Manager, Program Evaluation

COAG Reform Agenda: Human Capital Development

• National Partnerships

• Literacy and Numeracy

• Teacher Quality

• Low SES communities

• Not getting what you expected from an evaluation

What do we mean by Snake Oil

• Staff as passive recipients

• Evaluation questions not addressed

• Use lots of words

• Recommendations to continue the program and evaluate it again

• Lack of evidence based recommendations for improvement

• No discussion of data or design limitations

• No discussion of alternative explanations for program effect

• Evaluators reluctant to address client concerns about clarity or focus

• Useful information to guide decision making

So what’s the Good Oil

• Staff critical participants

• Provides answers to specific questions

• Balances qualitative with quantitative data

• Reports are just sufficient in length

• Conclusions follow from the evidence

• Has logical recommendations

• Data or design limitations are discussed

• Alternative explanations for program effect are discussed

How do we get there?

teamwork

commitment

learning proactivity

What are we trying to change?

Attitudes and values Knowledge and skills

Organisational behaviour

Overcoming your fears is the first step

Principles for evidence based program management

• Demand evidence

• Examine the logic

• Treat the program as an unfinished prototype

• Embrace an attitude of wisdom

Jeffrey Pfeffer and Robert Sutton on evidence based management

1. Executive leadership2. Consulting Services3. Resources4. Professional development5. Policy and operations6. Networks7. Strategic Oversight

Where should be put our efforts?

CREATING

CHANGE

• Locate near the top

• Recruit the knowledge, skills and motivation

• Build credibility

• Provide an ‘evaluation perspective’

Consulting Services

Resources

If you build it, they will come

What have others done?

Accessible Resources

Practical Resources

Practical Resources

Provide a starting point

Project Management

Stage 1Initiation

Stage 2Planning

Stage 3Execution

Stage 4Transition & Closure

· Project Proposal

· Appoint project sponsor and board

· Appoint project manager· Investigate similar projects· Discuss scope and goals· Discuss evaluative activities· Gain approval to plan

Project Management Templates

Program Evaluation Templates

Program and Project Management Tasks

· Logic Model· Project Plan· Budget· Aboriginal Impact· Gateway· Evaluation plan

· Status reports· Progress monitoring· Change of scope· Transition plan

· Project closure report· Evaluation report

· Evaluation plan· Good practice checklist· Logic model and performance

measures· Focus on expected changes· Stakeholder perspectives· Resource assessment· Developing questions· Choosing an evaluation design· Data collection and sampling

· Participant consent· Engaging an external evaluator· Evaluation brief· Evaluation work plan

· Final report· Quality assurance checklist· Reflecting on management

effectiveness

Guide to project and program management stages

· Define goals and outcomes· Identify evaluation questions· Plan activities· Schedule activities· Plan evaluation· Gain approval to implement· Identify sources of evidence· Scope data requirements

· Conduct team meetings· Monitor implementation· Liaise with governance groups· Conduct implementation

evaluation· Report on progress· Modify scope where needed· Gain approval for change

· Implement transition· Conduct post project review· Respond to evaluation findings· Disseminate evaluation findings· Finalize activities

Evaluation & planning

Professional Development

Time to stretch your thinking

Induction

Customized training to needs

Procurement:Expression of Interest

Two part contracts: plan + do

Executive summariesInterviewsHard questions

Policy and Operations

Think outside the square

Network, network, network

Strategic objective 2011

“We will foster a culture of evaluation and review to

better direct our services.”

Strategic objective 2011

“We will foster a culture of evaluation and review to

better direct our services.”

Strategic Plan 2012-2016

• We take seriously our responsibility to invest public funds effectively and efficiently.

• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . .

Strategic Plan 2012-2016

• We take seriously our responsibility to invest public funds effectively and efficiently.

• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.

Strategic Plan 2012-2016

• We take seriously our responsibility to invest public funds effectively and efficiently.

• Our funding decisions will be guided by the potential of our policies and programs to improve learning outcomes . . . current investments will be evaluated and we will reinvest where appropriate.

• Represents a major cultural change.

• Represents a major cultural change

• From an ‘accountability’ model to one centred on the corporate goal.

• Represents a major cultural change

• From an ‘accountability’ model to one centred on the corporate goal.

How?

A marketingapproach

A marketing approach

• Product life-cycle

A marketing approach

• Product life-cycle

A marketing approach

• Product life-cycle

• Customer

A marketing approach

• Product life-cycle

• Customer– Needs and wants

A marketing approach

• Product life-cycle

• Customer– Needs and wants

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

• Brands and brand equity

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

• Brands and brand equity

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

• Brands and brand equity

• Competition

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

• Brands and brand equity

• Competition

A marketing approach

• Product life-cycle

• Customer– Needs and wants

• The grudge purchase

• Brands and brand equity

• Competition

• 4 Ps – Profit Product Place Promotion

Priorities?

• To continue development & refinement of our ‘system’

• To address changing thefocus of our evaluation culture

• To move from ‘good oil’ to ‘best value’ oil

Disclaimer

The views and interpretationsexpressed in this presentation are those of the authors anddo not necessarily reflect those of the Department of Education and Children's Services or the Government of South Australia

Contact

• Bronte.Koop@sa.gov.au

• John.Collins2@sa.gov.au