1

Sean K. Long Professor Reifenberg International Development in Practice 13 November 2013

From Evidence to Influence: 3ieImpact as a Case Study for Policy Influence @arneduncan, the Twitter account for United States Secretary of Education Arne

Duncan, has 115 thousand followers. Priya Manickchand, Guyana’s Minister of Education,

sent her first tweet on October 19th and has 25 followers. For a group attempting to influence

them or other high-level government officials, a fundamental question emerges: What are the

best ways to shape the path for policymakers towards a desired policy outcome, and how

does that differ both across and within nations? This is the dilemma of the Inter-American

Development Bank’s (IDB) Education Division, which hopes to effectively communicate a

complex 43-page document on its education sector priorities to policymakers, taking into

account the vast disparities both between and within Latin American countries—from Brazil

to Haiti. In this paper, I examine how the International Initiative for Impact Evaluation

(3ieImpact) serves as a useful case study as IDB seeks ‘best practices’ to communicate its

message to key policymakers region-wide. To accomplish this, I feature 3ieImpact’s “Policy

Influence Plan” template as a theory of change to translate evidence into policy action.i I

conclude with the most salient lessons for how IDB can learn from 3ieImpact to effect policy

change and measure its success in the process.

Given 3ie’s objective to fund “impact evaluations and systemic reviews that generate

high quality evidence on what works in development,” this case is relevant not in its concrete

work, but rather in its theory of change framework to influence policy outcomes.ii The

“Policy Influence Plan,” a prerequisite for 3ie grantees, is a post-research blueprint to “to

engage policymakers and policy influencers and increase the likelihood that the research will

inform policies.”iii The theory of change underlying a Policy Influence Plan (PIP), which

2

systemically maps assumptions for how policy change happens, has five overarching goals:

states the policy question, identifies the needs of policy influencers and entry points in the

national debate, defines a short and long-term vision of success, locates the factors that

influence change, and pursues engagement strategies.iv Written at the outset of a project, the

plan outlines the step-by-step process towards policy action given a region’s distinct realities,

shown here:v

1.) Context: What is the political, economic and social environment that could affect the transition from research to policy influence?

2.) Risks: Are there potential dangers associated with the dissemination and policy influence of the evaluation (e.g. limited freedom of the press, high political turnover)

3.) Targeting of Key Stakeholders: What individuals or groups are the focal points to influence policy? 4.) Stakeholder Engagement Plan: A detailed plan for stakeholder engagement at each stage in the

process (e.g. design, presentation of findings, dissemination), and indicators to measure influence (e.g. number of people reached by workshops, number of meetings with target stakeholders, website clicks)

5.) Budget: How many resources—human or financial—will it take to carry out the stakeholder engagement plan?

6.) Evaluation and Learning: How will you measure the influence of your evaluation?

Given IDB’s Education Division communication objective to position itself as [in

Spanish] “la institución de referencia para el conocimiento de políticas y programas

educativos y programas educativos de América Latina y el Caribe,” IDB has an important

interest in determining ‘what works’ to inform and engage policymakers.vi To illustrate this

evidence-based approach to policy influence, consider the policy brief, a short memorandum

turning the ‘what’—key findings from an impact evaluation—into ‘how’—policy

implications and next steps—for dissemination to policymakers.vii 3ie released a report in

August 2012 entitled “Measuring the Impact and Influence of the Policy Brief,” in which it

used a randomized control trial (RCT) with over 800 respondents to measure the effect of the

policy brief on policymaker engagement.viii

The lessons? A brief is most effective at changing the opinions of those with no prior

opinion on the topic and, while the brief’s impact was independent of its form (page length or

whether it included opinion pieces), those who read briefs with an opinion-piece from an

3

authoritative source were more likely to share it. Most interestingly, a gender difference

revealed that brief design matters; women were “significantly less likely than men to find the

brief ‘convincing,’” suggesting that brief format and design might make women “less

inclined to take action.”ix Therefore, not only does this shed light on a new body of research

into ‘what works’ (and does not) in making interventions ‘stick’ with policymakers, but it

also implies that internal factors such as style and presentation matter. This also reveals a

larger theme about the evaluation and learning process in policy influence, where indicators

such as the number of citations from stakeholders, web metrics data, and “knowledge-

sharing” events can help measure whether a communication strategy has indeed led to

substantive influence.x

Mozambique presents one site of 3ieImpact’s efforts to turn research into policy.

With funding from 3ie, Save the Children and the World Bank conducted a study on early

childhood education in 2007 and found that children who attended preschool programs were

“24 percent more likely to enroll in primary school” than those not in the program.xi It was

welcome news, but the next step was to convince Mozambican policymakers that early

education was worth the investment. From the study’s design phase in 2007 through 2011, a

coalition of groups took concrete action—from “keeping partners informed at every stage”

by presenting its project design to Ministry officials in September 2007 to convening a

national-level meeting with over 70 government representatives in June 2011.xii The

outcome: the inclusion of early education in Mozambique’s 2012-2016 national education

plan and the creation of a national Early Childhood Development commission.xiii Most

importantly, a metacognitive reflection stage took place at the end to critically examine what

could have been improved; for instance, one proposed solution to initial skepticism from key

policymakers was that “we should have taken some policymakers on a field visit before

4

presenting our baseline findings.”xiv To conclude, an important lesson from the post-

intervention, reflection stage is that something as simple as taking policymakers to witness

the results for themselves could have catalyzed action at an earlier stage.

The Mozambique case represents just one 3ie story of change. 3ie released a

working paper in August 2012 with lessons for how the evaluations of a cash transfer

program in Argentina affected policy change.xv For example, as noted in the PIP, context

matters; Argentina was in the midst of an economic crisis after 2001, leading to high social

unrest and inequality. According to the report, a policy window opened as policymakers

became more “open to the evidence from research […] that could provide solutions to the

policy issues to be addressed.”xvi

In conclusion, 3ieImpact provides three salient lessons for IDB. First, given IDB’s

interest in its “Mapa de audiencias clave” to target “Minesterios de educación de la region,”

it can learn from 3ie’s Policy Influence Plan to develop a concrete and measurable strategy

that identifies the changes in attitudes and key policymakers to target.xvii Second, with one

example being how to make a policy brief ‘stick,’ 3ie is on the cutting-edge of an expanding

field of research into ‘what works’ in policy influence and IDB can learn from these

developments. Finally, from the stories of policy for preschoolers in Mozambique to cash

transfer programs in Argentina, IDB can listen for specific effective practices from stories of

change in 3ie-funded projects, such as taking policymakers to ground level through field

visits or understanding when a political or economic practice might open a window for

outside influence. Together, these lessons shed light on a communication framework for

turning research documents into policy outcomes.

5

Works Cited

3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. Web. 08 Nov. 2013.

Agosto, Gabriela. "Working Paper 17: From Impact Evaluations to Paradigm Shift. A Case Study of the Buenos Aires Ciudadanía Porteña Conditional Cash Transfer Programme." 3ie:International Initiative for Impact Evaluation, Aug. 2012. Web. 10 Nov. 2013.

Chapoy, Christelle. "What Makes a Policy Brief Stick Lessons from a Pioneer Experiment Comments." Research to Action. 29 Aug. 2012. Web. 10 Nov. 2013.

Houndolo, Deo-Gracias, and Stuti Tripathi. "Impact Evaluations and Policy Influence." International Initiative for Impact Evaluation. 18 Jan. 2013. Web. 08 Nov. 2013.

Hruskovec, Katherina. "Estrategia De Comunicación 2013-2014." Banco Interamericano De Desarrollo (2013). Web. 10 Oct. 2013.

"The Impact of Mother Literacy and Participation Programs on Children." International Initiative for Impact Evaluation. Web. 9 Nov. 2013.

Kelly, Melissa. "Happy Endings for Mozambican Preschoolers." Blog-3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. 15 Feb. 2012. Web. 9 Nov. 2013.

"Measuring the Impact and Influence of the Policy Brief." International Initiative for Impact Evaluations (3ie), Norad, Institute of Development Studies, Aug. 2012. Web. 7 Nov. 2013.

"Policy Briefs." 3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives, 2012. Web. 9 Nov. 2013.

"Policy Influence Plans." 3ie: International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. 2012. Web. 08 Nov. 2013.

"Tutorial Introduction." Policy Impact Toolkit. International Initiative for Impact Evaluations (3ie) and the Overseas Development Institute (ODI). Web. 08 Nov. 2013.

i "Policy Influence Plans." 3ie: International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. 2012. Web. 08 Nov. 2013. ii 3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. Web. 08 Nov. 2013. iii "Tutorial Introduction." Policy Impact Toolkit. International Initiative for Impact Evaluations (3ie) and the Overseas Development Institute (ODI). Web. 08 Nov. 2013. iv Houndolo, Deo-Gracias, and Stuti Tripathi. "Impact Evaluations and Policy Influence."International Initiative for Impact Evaluation. 18 Jan. 2013. Web. 08 Nov. 2013. v "The Impact of Mother Literacy and Participation Programs on Children." International Initiative for Impact Evaluation. Web. 9 Nov. 2013. vi Hruskovec, Katherina. "Estrategia De Comunicación 2013-2014." Banco Interamericano De Desarrollo (2013). Web. 10 Oct. 2013. vii "Policy Briefs." 3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives, 2012. Web. 9 Nov. 2013. viii "Measuring the Impact and Influence of the Policy Brief." International Initiative for Impact Evaluations (3ie), Norad, Institute of Development Studies, Aug. 2012. Web. 7 Nov. 2013. ; Chapoy, Christelle. "What Makes a Policy Brief Stick: Lessons from a Pioneer Experiment Comments." Research to Action. 29 Aug. 2012. Web. 10 Nov. 2013. ix ix "Measuring the Impact and Influence of the Policy Brief." International Initiative for Impact Evaluations (3ie), Norad, Institute of Development Studies, Aug. 2012. Web. 7 Nov. 2013. x "The Impact of Mother Literacy and Participation Programs on Children." International Initiative for Impact Evaluation. Web. 9 Nov. 2013. xi Kelly, Melissa. "Happy Endings for Mozambican Preschoolers." Blog-3ie:International Initiative for Impact Evaluation | Evaluating Impact, Informing Policy, Improving Lives. 15 Feb. 2012. Web. 9 Nov. 2013. xiiIbid. xiiiIbid. xivIbid. xv Agosto, Gabriela. "Working Paper 17: From Impact Evaluations to Paradigm Shift. A Case Study of the Buenos Aires Ciudadanía Porteña Conditional Cash Transfer Programme." 3ie:International Initiative for Impact Evaluation, Aug. 2012. Web. 10 Nov. 2013. xvi Agosto, Gabriela. "Working Paper 17: From Impact Evaluations to Paradigm Shift. A Case Study of the Buenos Aires Ciudadanía Porteña Conditional Cash Transfer Programme." 3ie:International Initiative for Impact Evaluation, Aug. 2012. Web. 10 Nov. 2013. xvii "Tutorial Introduction." Policy Impact Toolkit. International Initiative for Impact Evaluations (3ie) and the Overseas Development Institute (ODI). Web. 08 Nov. 2013. ; Hruskovec, Katherina. "Estrategia De Comunicación 2013-2014." Banco Interamericano De Desarrollo (2013). Web. 10 Oct. 2013.