Strategies for making evaluations more influential in supporting program management and informing decision-making

Australasian Evaluation Society2011 Conference

Anita DavisExecutive Director

Policy Innovation, Research & Evaluation UnitDepartment of Immigration and Citizenship

1 September 2011

Strategies for making evaluations more influential in supporting program management and informing decision-making

2

The strategic importance of research

• Managing migration in a connected world• The role of DIAC in managing migration • Understanding migration through research and evaluation

Enhancing the maturity of the evaluation function

• Investing in research and evaluation• An integrated approach to research and evaluation• Enhancing the maturity of the evaluation function

Managing migration in a connected world

3

• Migration has always played a critical role in Australia but the nature of migration is changing.

• The movement of people globally is increasing and becoming increasingly complex.

• International trends intersect with domestic issues to create dilemmas that place migration at or near the top of the policy agenda.

• The environment for migration policy in Australia is highly contested.

The role of DIAC in managing migration

4

• DIAC is at the front line in managing migration.

• DIAC plays a crucial role in delivering the government’s economic, social and security agendas.

• DIAC is one of the few agencies with an integrated responsibility for policy development, program management, service delivery, investigation and enforcement functions.

• DIAC is a large global organisation and its policies and programs need to work in a wide variety of settings for a large range of stakeholders and clients.

Understanding migration through research and evaluation

5

• Understanding migration and its impacts through research and evaluation is vital to the delivery of effective policies and programs.

• The use of evidence base in policy and program development helps gain public confidence in the migration program.

• Yesterday’s approaches are not necessarily those appropriate for current dilemmas.

• Research and evaluation are key sources of insight about policy, program and service delivery challenges and opportunities.

• ‘… policies are not made in a vacuum… they emerge from a maelstrom of political energy, vested interests and lobbying.’

- Gary Banks, Productivity Commission

Investing in research and evaluation

6

• The Policy Innovation, Research and Evaluation Unit (PIREU) and the Economic Analysis Unit (EAU) shape policy and program development through insights generated from research, evaluation and analysis.

• The key to strengthening the department’s research and evaluation capability is an integrated and strategic research and evaluation program.

An integrated approach to research and evaluation

7

• The example of settlement outcomes of humanitarian entrants:

From broad level•Forecasting net overseas migration

•Research into the economic, social and civic contributions of first and second generation humanitarian entrants

•Research into the longer term inter-generational contributions of the family and skills streams

•New landmark longitudinal survey of refugees

•Survey into settlement outcomes of new arrivals

•Evaluation of Australia’s Cultural Orientation Program

•Evaluation of the Settlement Grants

To program level

Enhancing the maturity of the evaluation function

8

• DIAC is taking a strategic approach to building a normative culture in which research and evaluation are essential components of policy and program development and decision-making.

• DIAC has adopted a three-year strategy for enhancing the maturity of the evaluation function.

Evaluation Strategy

9

• The evaluation strategy articulates the department’s vision for an effective and sustainable evaluation system and culture.

• Five broad, interrelated areas of focus:

• organisation

• culture

• capability

• support

• quality

Evaluation Strategy - Organisation

10

• Key outcome - improving the coordination and management of evaluation in DIAC.

• This outcome will be achieved through the implementation of the evaluation policy, which

• provides a common evaluation vocabulary and frame of reference

• establishes an institutional framework for evaluation

• communicates the expectation that evaluations are a normal part of good policy or program management

• ensures roles and responsibilities for evaluation are clear

• provides a clear line of sight to executive committee of evaluation activity

• 2011-12 surge capacity

Evaluation Strategy - Culture

11

• Key outcome - embedding a strong and sustainable evaluation culture in DIAC.

• Key actions include:

• expanding the DIAC evaluation network and developing an evaluation community of practice

• improving communication about evaluation and its benefits

• identifying and working with evaluation champions

• promoting organisational learning from evaluation

• acknowledging evaluation achievements in the department

Evaluation Strategy - Capability

12

• Key outcome – building the evaluation capabilities of DIAC staff so they are ‘informed consumers of evaluation and its findings’.

• Key actions include:

• identifying current levels of evaluation skills and knowledge

• designing and delivering targeted evaluation learning and development courses including an online e-learning course

• facilitating on-the-job evaluation learning and development opportunities in the workplace

Evaluation Strategy - Support

13

• Key outcome – ensuring staff have access to the resources and support they need to enable them to conduct or manage evaluations and use the findings.

• Key activities:

• providing advice and assistance to staff undertaking evaluation activities

• developing evaluation resources for staff to use

• ensuring appropriate financial and human resources are allocated to evaluations

Evaluation Strategy - Quality

14

• Key outcome – enhancing the quality of policy development, program management and decision making through effective use of evaluation.

• Key activities include working with business areas to ensure that evaluation are of high quality and issues are addressed:

• how sensitive programs could be evaluated• managing risk and stakeholder relationships• understanding and capitalising on the diversity in the field of evaluation• being mindful of the political context within which the evaluation is conducted• mitigating or managing pressure points and bias• ensuring credibility and meeting the standards and guidelines set out in the

evaluation policy• taking account of fiscal constraints and cost effectiveness• conforming to DIAC and APS values and meeting ethical standards

Evaluation Strategy - Quality

15

Measuring success

• We are reviewing ways of systematically measuring the impact of our investment in research and evaluation including undertaking periodic reviews to monitor evaluation quality and the use of their findings.

Conclusion

16

“Yes, it’s quite a noise – but are we having any impact?”

Australasian Evaluation Society - 2011 Conference

17

Anita DavisExecutive Director

Policy Innovation, Research & Evaluation UnitDepartment of Immigration and Citizenship

Contact: Anita.Davis@immi.gov.auEvaluation@immi.gov.au