View
2
Download
0
Category
Preview:
Citation preview
Evaluation Resource Booklet2016–21
© Commonwealth of Australia, 2016 ISBN 978-1-925401-44-8 June 2016/INFRA2828
Ownership of intellectual property rights in this publication Unless otherwise noted, copyright (and any other intellectual property rights, if any) in this publication is owned by the Commonwealth of Australia (referred to below as the Commonwealth).
Disclaimer The material contained in this publication is made available on the understanding that the Commonwealth is not providing professional advice, and that users exercise their own skill and care with respect to its use, and seek independent advice if necessary.
The Commonwealth makes no representations or warranties as to the contents or accuracy of the information contained in this publication. To the extent permitted by law, the Commonwealth disclaims liability to any person or organisation in respect of anything done, or omitted to be done, in reliance upon information contained in this publication.
Creative Commons licence With the exception of (a) the Coat of Arms; (b) the Department of Infrastructure’s photos and graphics; copyright in this publication is licensed under a Creative Commons Attribution 3.0 Australia Licence.
Creative Commons Attribution 3.0 Australia Licence is a standard form licence agreement that allows you to copy, communicate and adapt this publication provided that you attribute the work to the Commonwealth and abide by the other licence terms.
A summary of the licence terms is available from http://creativecommons.org/licenses/by/3.0/au/deed.en. The full licence terms are available from http://creativecommons.org/licenses/by/3.0/au/legalcode. This publication should be attributed in the following way: © Commonwealth of Australia 2016, Evaluation Resource Booklet 2016–21.
Use of the Coat of Arms The Department of the Prime Minister and Cabinet sets the terms under which the Coat of Arms is used. Please refer to the Department’s Commonwealth Coat of Arms and Government Branding web page http://www.dpmc.gov.au/guidelines/index.cfm#brand and in particular, the Guidelines on the use of the Commonwealth Coat of Arms publication.
Contact us This publication is available in hard copy or PDF format. All other rights are reserved, including in relation to any Departmental logos or trademarks, which may exist. For enquiries regarding the licence and any use of this publication, please contact:
Director — Publishing and Communications Section Department of Infrastructure and Regional Development GPO Box 594, Canberra ACT 2601 Australia
Email: publishing@infrastructure.gov.au Website: www.infrastructure.gov.au
Evaluation Resource Booklet2016–21
An effective intervention, but not necessarily the most efficient option
It runs really well – but is it taking me where
I want to go?
An efficient intervention, but not necessarily effective
An effective intervention, but not necessarily the most efficient option
It takes me to where I want to go, but is it the best vehicle for the job?
2
Evaluation Resource Booklet 2016–21
3
Contents
Purpose of this Booklet 4
Evaluation-related activities in the Department 4
Training, workshops, seminars 4Developing a policy/programme/regulatory logic 5Planning for short, medium and long-term outcomes 6Developing a M&E framework 6Undertaking an evaluation or regulatory review 7Management response 10Lessons learned 10
Templates 10
Logic diagram template 11M&E framework and data matrix template 12Evaluation plan template 13Evaluation report template 14Management response template 15Lessons learned template 16
Examples of past evaluation-related activities 17
Reference list 19
Department of Infrastructure and Regional Development
4
Purpose of this BookletThis Booklet accompanies the Department’s Evaluation Strategy 2016–21 and is designed to give practical information and guidance to staff before undertaking an evaluation-related activity:
• an outline of key concepts;
• details of the support available;
• templates for use (and modification as required) when developing a policy/programme/regulatory logic; monitoring and evaluation (M&E) framework; evaluation plan; evaluation report; management response; and a summary of the lessons that were learned from the evaluation or regulatory review;
• examples of evaluation-related activities that the Department has undertaken from 2011–12 to 2015–16;
• a summary of the governance and oversight of the Department’s evaluation-related activities, including relevant committees; and
• a reference list for staff who may want more in-depth information about evaluation and related concepts.
Evaluation-related activities in the DepartmentTraining, workshops, seminarsA range of training, workshops and seminars will be undertaken in the Department, including:
• small-group mentoring sessions with evaluation specialists, primarily in response to demand from line areas;
• mentoring to assist line areas in developing draft policy/programme/regulatory logics; M&E strategies; evaluation plans or draft reports;
• regular training courses that are run in-house, e.g. on how to develop a logic and an M&E framework, and how to manage and conduct an evaluation or regulatory review; and
• occasional all-staff seminars on evaluation-related topics.
The Department’s intranet/ENTR site (under ‘evaluation’) lists upcoming events and contact details for the Governance Unit.
Evaluation Resource Booklet 2016–21
5
Developing a policy/programme/regulatory logicA policy/programme/regulatory logic diagram explains:
• what an activity intends to achieve;
• how and why activities will achieve the intended results;
• the rationale for the causal links between inputs, outputs and outcomes; and
• the assumed outcomes of actions.
It is often represented as a diagram on a page, showing:
• The Portfolio Budget Statement outcome – for the policy/programme/regulation (if available), demonstrating a link to how the activity will contribute to achieving the Department’s purpose;
• The outcomes and contributory outcomes – the changes that will be brought about, and may be expressed as short, medium or long-term outcomes; the overall outcome will normally provide the basis for the objectives;
• The outputs – the tangible product of the inputs and projects, which are fundamental to the outcomes being achieved, e.g. plans, reports, grants and contracts;
• The activities – the main activities (or projects) that turn the inputs into outputs; e.g. processing grant applications;
• The inputs – normally the funds, people and other resources required for the policy/programme/regulation to operate; and
• Context and assumptions – key assumptions that the policy/programme/regulation has been developed on; the context in which it is operating; and activities that may be outside the Department’s direct responsibility.
It is best to work on creating a logic diagram with team members and other relevant areas within the Department. This way, the logic process gathers input from the departmental officers who are involved in the work and ensures you are all on the same page.
A logic diagram identifies the outcomes and subsequent success measures which provide the basis for monitoring, reporting and evaluating progress in meeting objectives over the life of the policy, programme or regulation.
Policy logics ask additional questions, about: the drivers that have prompted government intervention or action, the problem the action was intended to solve, the consequences of failing to address the problem, the barriers that prevent the problem from being resolved, why is it necessary to intervene, what else is underway to help address the problem, do we need to work in partnership with others, what data is available to inform an evaluation.
Logic diagrams are a useful communication tool, which help explain the policy/programme/regulation to stakeholders and clients.
Department of Infrastructure and Regional Development
6
Planning for short, medium and long-term outcomesMost managers report on operational measures: contracts signed, hits on websites, units delivered. But is the activity achieving its intended outcomes?
By talking about long and short term measures of success, managers and policy developers can have a conversation about the intention of policy and how to ensure it hits the mark. It‘s often useful to work backwards:
• Success in the long term
What would ‘success‘ look like if the original policy problem was ‘fixed‘? What attitudes and behaviours would need to have changed? Which stakeholders or influencers would have ‘come on board‘? Is there a point where your agency could wind up the program or hand responsibility to others?
• Success in the mid term
Is the activity getting traction? If you are on track to achieve the long term goals, you should see some symptoms of this in the mid-term. For example, there may be a change in attitudes or in the way the issue is reported in the media. Information requests may rise. Key stakeholders may begin taking responsibility for the problem, allocating resources or developing initiatives of their own.
• Success in the first year
So — if you are going to achieve the outcomes you want in the medium and long term, what do you need to do in the first year? Yes, you need to have worked through all the logistics: resources, timelines, infrastructure, contracts, communication and deliverables. But is that enough? To achieve some of the outcomes you need in the mid-term, you may need to show that you have also laid early foundations for important partnerships with stakeholders and considered the linkages with other activities and initiatives.
Developing a M&E frameworkAn M&E framework (or strategy) allows the measurement of the success of a policy/programme/regulatory activity, to know if it is on track to meet its objectives.
The main features of an M&E framework are the:
• Logic diagram – e.g. what are the inputs, outputs and outcomes?
• Measures of success – e.g. what would indicate that you are achieving the outcomes?
• Data sources – e.g. what data and information needs to be collected, and when, to form an evidence base?
• Evaluation strategy – e.g. the approach that will be used to evaluate the policy/programme/regulation over the course of its life, along with possible evaluation questions.
Evaluation Resource Booklet 2016–21
7
Undertaking an evaluation or regulatory reviewEvaluation is the systematic and objective assessment of a policy, programme, or regulatory activity. It normally addresses aspects of efficiency, effectiveness and appropriateness. This leads to well-informed judgements about the impact, achievements and/or merit of that activity.
Evaluation (or regulatory review) findings inform decision-making. Disseminating findings widely allows them to be used to deliver improved infrastructure, transport and regional services; effectively implement government policies, programmes and regulations; and develop investment strategies.
By using evaluation processes consistently in our work, we ensure that:
• current activities are robust, effective and efficient; and are responding to the relevant key issues;
• practical policy options are identified to address emerging issues and challenges;
• skills and capabilities are actively built to achieve the Department’s vision and outcomes; and
• risks are proactively managed at all relevant levels; for the program, Department and government.
The Department’s evaluation activities have strategic oversight by an executive Steering Committee (SC), which endorses the evaluation plans and draft reports. The Department’s Finance, Reporting and Programme Committee (FRPC) monitors the implementation of the management response.
Department of Infrastructure and Regional Development
8
Figure 1 outlines the policy cycle. Figure 2 outlines how evaluation can support policy activities.
Figure 1: The policy cycle
Consultation
Coordination
Decision
Implementation
Evaluation
Identifyingissues
Policy analysis
Policy instruments
The familiar diagram, from The Australian Policy Handbook by Caroline Althaus et al (2007), presents a picture of policy activity as an orderly progression arising from analysis and consultation, moving smoothly through decision into implementation and evaluation.
However, most policy and program workers experience a more unpredictable progression. Policy is often developed under pressure, sometimes in response to political events. Political considerations may impact on a policy’s progression to approval. Policy may be developed without the opportunity for consultation. Implementation may be rushed and there may be no opportunity for piloting.
Figure 2: Evaluation and policy
Demonstrate and provide evidence on the benefits to the policy• Public/stakeholder communication –
demonstrate value • Obtain further commitment and ownership
for intervention
Test and analyse current interventions and their performance• Formative evaluations • Annual and/or mid-term• Ex-post evaluation • Provide evidence of achievements to the
public and key stakeholders
Identify and analyse policy intervention options• What is the role of the Australian
Government/Department?• What do we already know (draw on existing
knowledge and expertise)?• What would success look like and how will
we get there? • Who else might need to be involved and how?• Analyse pros and cons for options as to best
solution to the problem• Evidence to support policy options
Identify and be clear on current/emerging problems• A structured way at looking at
problems/issues• Who and how will the problem impact?
Evaluation Resource Booklet 2016–21
9
The flowchart in Figure 3 outlines the Department’s evaluation (or regulatory review) process.
Figure 3: The Department’s evaluation process
How did the evaluation eventuate? Identified on the Departmental Evaluation Schedule
Line area drafts an Evaluation Plan
Line area provides an ED-cleared draft report and management response, to Governance Unit
Governance Unit arranges SC meeting, and prepares a cover paper, seeking consideration/endorsement of the report and the management response
Governance Unit provides the findings and approved management response, to Audit Committee members, for their information
Governance Unit uploads the SC and FRPC cleared report, management response and lessons learned, to the intranet/ENTR site
Governance Unit, if agreed by FRPC, uploads the report and approved management response on the Department’s website
Line area implements the recommendations in the approved management response
Line area conducts and manages the Evaluation: • Data collection • Regular progress reports • Draft report • Proposed management response • Lessons learned
Governance Unit arranges a Steering Committee meeting, and prepares a cover paper, seeking SC endorsement of the Executive Director-cleared Evaluation Plan
Governance Unit provides the report and management response to the Finance, Reporting and Programme Committee, for consideration/approval,
and, seeks agreement for public release
Governance unit provides a quarterly report to FRPC on the progress with implementation of the management response,
until full implementation/completion
Determined via consultation with Divisions and endorsed by executive
Line area may request use of a mentor (if/as required)
Governance Unit reviews/makes suggestions on the draft
Governance Unit manages the contract with the mentor and reviews
the progress reports
Governance Unit reviews/makes suggestions on the draft report, final report, draft management response
and draft lessons learned
Line area and mentor can approach the Governance Unit if any issues arise
throughout the Evaluation
Department of Infrastructure and Regional Development
10
Management responseThe management response is prepared by the line area that undertook the evaluation or regulatory review, and it is cleared by their Executive Director. It lists what it is that will be done, by whom, and by when, to address the recommendations or findings. The Department’s FRPC receives a quarterly update on the implementation progress of the management response, coordinated by the Governance Unit, with input from the relevant line area.
Lessons learnedThe ‘lessons learned’ is also prepared by the line area that undertook the evaluation or regulatory review. It highlights key points or issues arising from the findings and recommendations that staff can apply to their work; advice worth disseminating to other areas in the Department (in terms of how we design and implement policies/programmes/projects/regulations); and comments about the evaluation process itself (what went well and what was challenging).
TemplatesBasic templates1 are provided for:
• Logic diagram
• M&E framework and data matrix
• Evaluation plan
• Evaluation report
• Management response, and
• Lessons learned summary.
The intranet/ENTR site (under ‘evaluation’) includes electronic copies of these templates as a guide for staff to utilise and modify as required.
The detail included in templates will normally be covered and addressed by the in-house training provided by the Department.
Mentors can also provide guidance to line areas, for example, assistance in determining how to present information in a plan or report, or advice on how to select data sources and/or data collection methods.
Templates for regulatory reviews and best practice regulation can be found on the Office of Best Practice Regulation’s website (at dpmc.gov.au under ‘OBPR’).
1 Colour is used in these templates for presentation/training purposes only.
Evaluation Resource Booklet 2016–21
11
Logic diagram template
Inputs
Context and assumptions:
Activities/projects
The overall outcome:
Outcomes
Outputs
Department of Infrastructure and Regional Development
12
M&E framework and data matrix template
M&E framework template
1. Introduction and purpose of the M&E
Provide a brief description
2. Logic
Insert a copy of the policy/programme/regulatory logic diagram
3. Measures of success and data sources
List the success measures: which would indicate an outcome is being achieved.
List (ideally presented in a matrix) the data sources: where data and information will come from, how and when it will be collected, for example:
Source/Method A Source/Method B Source/Method C
Terms of Reference 1
Q1
Q2
Q3
Terms of Reference 2
Q1
Q2
Q3
4. Evaluation strategy
Outlines the approach to evaluation over the life of the policy/programme/regulation
Evaluation Resource Booklet 2016–21
13
Evaluation plan template
Evaluation plan template
1. Introduction and purpose of the evaluation plan
Provide a brief description about why the evaluation is being undertaken and what it will be used for
2. Terms of reference (TORs)
List the TORs
3. Key evaluation questions
List the key evaluation questions for what the evaluation will ‘ask’
4. Scope
Describes the focus of the evaluation, identifying what is in-scope as well as what is out-of-scope and the reason(s) ‘why’
5. Methodology
Indicates the methodology that will be used when undertaking the evaluation (referring to the M&E framework data sources, collection methods and matrix)
6. Timeframes and deliverables
Indicatesthekeystepsthatwillbeundertakenandwhen,includingwhenthedraftandfinalreportswillbe completed
7. Resourcing and management of the evaluation
Brieflydescribethecompositionoftheevaluationteamandrolesandresponsibilities;useofamentor; also providing a table of the budget that will be allocated for key tasks/phases such as data collection, consultation, analysis, reporting etc
Department of Infrastructure and Regional Development
14
Evaluation report template
Evaluation report template
1. Executive Summary
Provideaconciseoutlineofthefindingsandrecommendations
2. Introduction
Provide a brief description
3. Background
To the policy/programme/regulation
4. Methodology
Outline the key evaluation questions as well as the data and information that was collected and analysed (including both qualitative and quantitative means)
5. Key findings
Summarisethemainresultsandfindings
6. Recommendations
List the recommendations from the evaluation or review
7. Attachments
Evaluation Resource Booklet 2016–21
15
Management response template
Management response template
Recommendations or findings
Management response
(e.g. agreed/disagreed; and what actions will be
undertaken)
Responsibility
(e.g. Division/Branch)
Implementation timeframe
(e.g. by 30 June)
1.
2.
3.
4.
5.
Department of Infrastructure and Regional Development
16
Lessons learned template
Lessons learned template
1. Did the evaluation meet its purpose and were the TORs answered?
2. What main evaluation findings and recommendations are going to be relevant to the team or branch’s work in the future?
3. After doing the evaluation, what do you think would be useful for other areas of the Department to know about if they were to plan an evaluation?
4. Is there any other advice you would give, if the Department were to design and manage a new policy/programme/regulation in the future?
Evaluation Resource Booklet 2016–21
17
Examples of past evaluation-related activitiesThe following tables lists the evaluation-related activities undertaken in the Department over 2011–12 to 2015–16. The intranet/ENTR site (under ‘evaluation’) also provides electronic examples of these activities, and will be updated annually.
Evaluation activities undertaken from 2011–12 to 2015–16 (excludes training and workshops)
Priority Evaluations
Title Year Division
Heavy Vehicle Safety and Productivity Program (HVSPP) 2011–12 II
Regional Aviation Access Program (RAAP) 2011–12 AA
Keys 2 Drive 2012–13 STP
Strengthening Aviation Security Initiative (SASI) 2012–13 OTS
Roads to Recovery (R2R) 2013–14 II
Liveable Cities 2013–14 II
WA Service Delivery to Indian Ocean Territories (IOT) 2014–15 LGT
Asset Management 2015–16 Corporate & LGT
Indicative Evaluations
Title Year Division
Airport Tripartite Deeds 2014–15 AA
Regional Passenger Screening Programme (RPSP) 2014–15 OTS
Aviation and Maritime Security Identification Card Schemes (ASIC-MSIC) 2015–16 OTS
Last Ports of Call and International and Regional Capacity Building (LPOC) 2015–16 OTS
Review of the Departmental Evaluation Strategy 2015–16 Corporate
Regulatory Reviews
Title Year Division
Motor Vehicle Standards Act 1989 (MVSA) options discussion paper 2014–15 STP
Airport Curfew Administration Arrangements 2015–16 A&A
Continued ...
Department of Infrastructure and Regional Development
18
Evaluation activities undertaken from 2011–12 to 2015–16 (excludes training and workshops) continued ...
Logics and Monitoring and Evaluation Strategies (M&Es)
Title Year Division
National Stronger Regions Fund (NSRF) 2013–14 II
Bridges Renewal Program 2014–15 II
Community Development Grants 2014–15 II
Jervis Bay Territory Reform 2014–15 LGT
Regional Aviation Access Programme (RAAP) 2014–15 A&A
WA Service Delivery to Indian Ocean Territories 2014–15 LGT
Northern Australia Roads Programmes (Beef Roads and Nth Australia Roads) 2015–16 II
Drought Communities Programme 2015–16 II
Stronger Communities Programme 2015–16 II
Indian Ocean Territories (IOT) 2015–16 LGT
Norfolk Island Reform 2015–16 LGT
STP logics, M&Es and risk management; of each main activity 2015–16 STP
Evaluation Resource Booklet 2016–21
19
Reference listYear Title ISBN/Reference Author
2016 NSW Government, Program Evaluation Guidelines 978-0-7313-5499-3 NSW Government Premier & Cabinet
2015 Australian Government, Department of the Environment, Evaluation Policy 2015–2020
Creative Commons 4.0 Australia
Commonwealth of Australia
2015 Australian Government, Department of Industry and Science, Office of the Chief Economist, Evaluation Strategy 2015–2019
978-1-925092-67-7 Commonwealth of Australia
2015 Learning from Failure: why large government policy initiatives have gone so badly wrong in the past and how the chances of success in the future can be improved
978-0-9943750-0-1 Commonwealth of Australia,
Professor Peter Shergold AC
2015 Australian Government, Department of Finance, Resource Management Guide No. 131, Developing Good Performance Information
978-1-925205-17-6 Commonwealth of Australia
2014 Queensland Government, Queensland Government Program Evaluation Guidelines
Creative Commons
3.0 Australia
QLD Treasury and Trade and QLD Government Statistician’s Office
2013 Australian Government Department of the Prime Minister and Cabinet Cabinet Implementation Unit Toolkit 5: Monitoring, review and evaluation
- Australian Government Department of the Prime Minister and Cabinet, CIU
2013 Australasian Evaluation Society Evaluators’ Professional Learning Competency Framework
- Australasian Evaluation Society Professional Learning Committee
2013 United Nations Evaluation Group, Impact Evaluation in UN Agency Evaluation Systems: Guidance on Selection, Planning and Management
- United Nations Evaluation Group Impact Evaluation Task Force
2012 United Kingdom, Quality in qualitative evaluation: a framework for assessing research evidence (supplementary Magenta Book guidance)
978-1-84532-999-0 UK HM Treasury
2012 United Kingdom, Quality in policy impact evaluation: understanding the effects of policy from other influences(supplementaryMagentaBookguidance)
978-1-909096-49-3 UK HM Treasury; Department of Energy and Climate Change; and Department for Environment, Food and Rural Affairs
2011 United Kingdom, The Magenta Book, Guidance for Evaluation
978-1-84532-879-5 UK HM Treasury
Continued ...
Department of Infrastructure and Regional Development
20
Year Title ISBN/Reference Author
2011 United Kingdom, Major Projects Authority – AssuranceofbenefitsrealisationinMajorProjects– Supplementary guidance
UK Cabinet Office
2010 United Nations Evaluation Group Quality Checklist for Evaluation Terms of Reference and Inception Reports
- United Nations Evaluation Group
2010 United Nations Evaluation Group Quality Checklist for Evaluation Reports
- United Nations Evaluation Group
2010 United Nations Evaluation Group Good Practice Guidelines for Follow-up to Evaluations
- United Nations Evaluation Group
2010 The Canadian Evaluation Society, Competencies for Canadian Evaluation Practice
- Canadian Evaluation Society
2009 Australian Government, Department of Finance, Outcome Statements Policy and Approval Process
- Commonwealth of Australia
2005 United Nations Evaluation Group, Norms for Evaluation in the UN System
- United Nations Evaluation Group
2003 United Kingdom, The Green Book, Appraisal and Evaluation in Central Government
- UK HM Treasury
Websites: www.betterevaluation.org, www.evaluationtoolbox.net.au, www.aes.asn.au
Reference list continued ...
Recommended