Measuring Outreach Effectiveness

Web4Dev – November 2006Alex McKenzie - amckenzie@worldbank.orgKnowledge & Evaluation Capacity DevelopmentIndependent Evaluation Group – World Bank

Key Points:

► Share IEGWB’s approach for aligning web/online practice with mandate and business strategy

► Share our experience running several outreach campaigns, discuss metrics

► Measuring effectiveness and impact► Would appreciate hearing about your

own experience and/or feedback

Business Strategy

► IEG is an independent unit that assesses the effectiveness of World Bank Group’s interventions

► Well regarded for its evaluation capabilities and reports

► Combines Accountability & Learning ► “To be effective, IEG want’s to be an influential and

reliable source of knowledge”► New Outreach and Communications Strategy:

– Target key audiences– Embed outreach throughout the evaluation process– Accelerate learning, encourage utilization– Measure results

► Decided on a campaign-based approach, 10 to 12 major evaluation products per year.

► Multi-pronged outreach campaigns: media, print, online, in-person, translations, etc.

► Piloted many new ideas, mainstreamed what worked

Key Audiences

► Systematic effort to identify audiences has led to two sets: “core” & campaign-specific (specialists)• Identify decision-makers & influencers• Identify key network nodes

► Evolving audience criteria: from “static” classifications, to others based on opportunity or time-sensitive demand

► Customize products to audience whenever economically feasible (e.g. conference CDROMs, translations, etc.)

Embed in Evaluation Process

Web support for:

Design Phase Research Phase Outreach Phase

Communications - Release approach paper

- Interim data and research- Audience identification

- Outreach campaign*- Campaign metrics

Process support - Virtual team workspace- Research surveys

- Feedback, discussion, comments

Value Created>>

Transparency, Inclusion

Transparency, Efficiency

Key messages for relevant audiences

* Initial role for web

Encourage Utilization

Encourage Utilization

Measuring Results

► Results-Based Management causal chain

Inputs (Activities) Outputs

Intermediate Outcomes

Outcomes

Area of Control

Area of Influence

Measuring Results – Causal Chain

Source: “Strengthening EAP’s Results Orientation”, April 2004

Measuring Results - OutputsWeb Outreach Campaign Comparison

0

500

1000

1500

2000

2500

3000

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

Campaign Day (normalized)

Page

Vie

ws

Dev.Effectiveness

Global Program

Trade

Email campaign targetting about 15,000 people.

Sustained interest through Google advertising

Online campaigns focus on key audiences, and generate follow -up engagements or requests from relevant parties (numbers are for entire campaign):Global: 4644 requests;Dev.Effect.: 983 requests; Trade: 662 requests.

Measuring Results - Outputs

Web Outreach Campaign Patterns

0

500

1000

1500

2000

2500

3000

3500

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61

Campaign Day (normalized)

Pag

e V

iew

s

ARDE 2004

Community-based Dev.

Dev.Effectiveness

HIPC

HIV/AIDS

Fragile States

Natural Disasters

Pension Reform

Primary Education

Trade

All campaigns exhibit similar trends: Initial traffic peaks converge quickly to a lower, but stable levels.

Measuring Results - Outcomes► Intermediate Outcomes: perception survey*

of IEG evaluation products, covering:– Readership– Quality– Influence on Bank processes and policies (similar for

external clients)– Utilization

► Annual Review of Operations Evaluation (AROE) 2006, to be released in November/December.

*Source: IEG’s Annual Review of Operations Evaluation (AROE). See http://www.worldbank.org/ieg/aroe

Measuring Results - Outcomes► Survey targeted individuals defined as intended

audience: 4285 internal (WBG) and 2759 external contacts. Results are indicative for respondents (22%) but are not generalizable to population

Readership  Internal External

Direct awareness: 42% 55%

Overall awareness: 56% 76%

heard via email 64% 69%

visited web site 12% 28%

received hard-copy 20% 15%

Perception of Quality – Staff/EDs (E4)

4.88

4.84

4.74

4.02

4.76

4.78

4.29

4.71

4.47

4.243.55

3.58

3.69

3.73

3.76

3.87

4.00

4.13

4.21

4.28

0 1 2 3 4 5 6

Relevance to your work

Ease of understanding

Concise presentation of conclusions

Timeliness

Usefulness of recommendations

Unbiased and objective analysis

Transparency and clarity of the methodology

Strong link between conclusions and evidence

Depth of analysis

Incorporation of all available information

ED Staff

Perception of Quality – External Clients (E13)

4.75

4.73

4.66

4.51

4.49

4.45

4.34

4.34

4.32

4.15

1 2 3 4 5 6

Ease of understanding

Concise presentation of conclusions

Relevance to your work

Usefulness of recommendations

Transparency and clarity of the methodology

Timeliness

Unbiased and objective analysis

Strong link between conclusions and evidence

Depth of analysis

Incorporation of all available information

Influence – Staff (E8)

3.80

3.62

3.51

3.30

3.29

3.27

3.23

3.22

3.21

1 2 3 4

Helped you gain a clear view of essential lessons learned.

Influenced your understanding of the subject area.

Helped you gain a clear view of best practices.

Influenced the Bank's sector strategy.

Influenced the Bank's lending and non-lending services.

Influenced the Bank's country strategy.

Influenced how outcomes are linked to outputs (ResultsFramework) in projects.

Influenced how outcomes are linked to outputs (ResultsFramework) in the sector.

Influenced how outcomes are linked to outputs (ResultsFramework) at the country-level within the Bank.

Influence – External Clients (E15)

4.51 4.45 4.31 4.26 4.23 4.12

1

2

3

4

5

6

Helped youbetter

understand theBank's

activities withina sector.

Helped yougain a clear

view ofessentiallessonslearned.

Provided youwith practicalinformation to

help you makedecision on

strategies andprograms.

Influencedyour

understandingof the subject

area.

Helped yougain a clearview of bestpractices.

Helped youbetter

understand theBank's

activities withina country.

Staff Use of Evaluations (E9)

3.60

3.57

3.57

3.22

3.21

3.05

3.02

2.68

1 2 3 4 5 6

Commenting on or making inputs into the workof others

Making the case for a particular course of action

Providing advice to your clients

Modifying policies and/or strategies

Designing/Modifying your Results Framework

Designing new lending operations

Designing new non-lending operations

Modifying on-going operations

External Clients Use of Evaluations (E16)

3.653.813.954.00

57%*62%*70%*69%*

1

2

3

4

5

6

Research Making the case fora particular course

of action

Refocusing on-going strategies or

programs

Education

*Percentages show how many respondents rated 4 or higher

Measuring Results – Survey Implications

• IEG provides good services to its main client, the Board. External audiences are also highly satisfied with the quality of IEG evaluations and considered them to be influential.

• However external audiences would like IEG to broaden its external consultation during evaluations.

• Bank staff see IEG products useful with deepening their understanding of a given subject.

• However they don’t see them as useful in day-to-day work. IEG needs to focus on influencing ongoing and future operations.

Measuring Results – Survey Implications

► All audiences suggest:• High value placed on timeliness to provide

information when decisions need to be made• Deepening analysis through more consultation

and attention to context• Focus on operational value of findings

► A call for action for IEG’s outreach program• 4-page briefs• Customized “findings” brief, upstream, via email• More use of non-print media• Balance “lecturer” approach with conversations

Thoughts Going Forward

► "What information consumes is rather obvious; it consumes the attention of its recipients. Hence, a wealth of information creates a poverty of attention."

Herbert Simon, Nobel laureate

Your Experience/Feedback

► What has worked in your organization? (audience analysis; relevance/timeliness; measuring outputs, outcomes, impacts, etc.)

► Choices related to channels? High quality content vs. open/collaborative models

► Outreach models used? Worthwhile consultants & advice?

► What challenges do you see ahead?