1

Chesapeake Bay Funders Network

Program Evaluation

Training Workshop

OMG Center for Collaborative Learning

January 9-10, 2008

Evaluating Advocacy

2

Building an Evaluation Plan

I. Theory of Change

II. Outcomes

III. Indicators

IV. Methods

V. Putting Evaluation to Work for You

Same Building Blocks for Evaluating Advocacy

3

What is advocacy?

For our purposes, advocacy represents…

Strategies devised Actions taken Solutions proposed

…to inform or influence local, state, or federal decision making.

4

Evaluating Advocacy: How is it Different?

• Advocacy has always been considered hard to measure - less tangible

• Is evaluating advocacy different than evaluating traditional programs?

Yes and No....

5

Evaluating advocacy is different from evaluating traditional program work because of the nature of advocacy itself.

1. Advocacy strategies evolve over time; activities and outcomes can shift quickly

2. The policy process has many unpredicted and uncontrollable variables (windows of opportunity)

3. So many groups focus on an issue – hard to isolate attribution (Hence, contribution – is it plausible that your activities contributed to the end result?)

Evaluating Advocacy: How is it Different?

6

Evaluating Advocacy: How is it the same?

The same principles and approaches that we talked about earlier can be applied to evaluation of advocacy

1. You can develop a Theory of Change for advocacy

2. You need to determine outcomes and the indicators that will tell you whether the outcome has happened

3. You can use qualitative or quantitative approaches to collecting the data

7

Evaluating Advocacy: How is it the same?

4. Purpose is the same: Accountability – reporting on your results

Promotion – leveraging further interest in your work (other funders, volunteers, partners, etc..)

Inform Constituents/Stakeholders – results of their actions, learning

Decision-making – inform your work as you go, and reflection – what worked, what didn’t, what to do differently

8

What is the primary advocacy or policy change goal?

Public Awareness: Aims to increase public knowledge or salience of a particular problem or policy solution

Public Will: Aims to increase public willingness to act onbehalf of a particular policy issue

Political Will: Aims to increase policymaker willingness toact on behalf of a particular policy issue

*Slides 8-17 draw from “Using the Advocacy and Policy Change Composite Logic Model toArticulate an Advocacy Strategy or Theory of Change”, Julia Coffman, Harvard Family Research Project

Creating a Theory of Change for Advocacy

9

What contextual factors might affect your strategy’s success?

• Political Climate

• Economic Climate

• Social Climate

• Prior Experience with Advocacy

• Issue Competition

• Potential Partners/Competitors/Opponents

Creating a Theory of Change for Advocacy

10

Determining outcomes for advocacy-– TIMING MATTERS!

A Theory of Change for advocacy must have a realistic timeline– This type of work typically takes a long time

to yield outcomes– Short term, intermediate, long term

Creating a Theory of Change for Advocacy

11

Timing Matters: Short-term/Intermediate Outcomes

• Creating Partnerships or Alliances

• Collaboration and Alignment (include messaging)

• New Advocates

• New Champions (including policymakers)

• Media Coverage

Creating a Theory of Change for Advocacy

12

Timing Matters: Short-term/Intermediate Outcomes

• Issue Reframing

• Awareness

• Salience

• Attitudes or Beliefs

• Public Will

• Constituency or Support Base Growth

Creating a Theory of Change for Advocacy

13

Timing Matters: Longer Term Outcomes

• Policy Development

• Placement on the Policy Agenda

• Policy Adoption (passes through ordinance, ballot measure, legislation or legal agreement)

Creating a Theory of Change for Advocacy

14

Timing Matters: Longer Term Outcomes

• Policy Implementation (funding, resources, q.a.)

• Policy Monitoring and Evaluation

• Policy Maintenance

• Policy Blocking (opposition to a policy proposal that would go against your goals)

Creating a Theory of Change for Advocacy

15

Who is the audience?

• Most strategies will target multiple audiencesFor example: Elected officials, Voters, Media

Other potential audiences: Public administrators, Business, Community leaders, Courts, Specific constituencies

• Think about who needs to be doing the advocating and who needs to be hearing it to achieve the goal (levels of influence)

Creating a Theory of Change for Advocacy

16

What will it take to convince or move the audience?Activities needed to achieve the outcomes (Sequence)

Creating a Theory of Change for Advocacy

Policymaker/Candidate Education

New Champions Placement on the Policy Agenda

Demonstration Projects or Pilots

Awareness Policy Adoption

Constituency/Support Base Growth

17

I will know there is… When I see…

Advocacy: Outcomes and Indicators

New Champions High profile individuals speaking out on the issue

Placement on the Policy Agenda

Issue/idea appears on schedule for hearings

Awareness Target audience recognizes the issue/is familiar with the terms

Constituency/Support Growth

Increased number of people participate in advocacy or action on the issue

Policy Adoption Successful passing of the policy proposal through an ordinance, ballot measure, legislation or legal agreement

18

1. Focus Groups/Interviews/Surveys • Used to measure Awareness, Salience, Attitudes/Beliefs,

Public Will, Political Will, Constituency or Support Base Growth

• Specific techniques:1. “Bell weather Assessment”: Is the issue on people’s radar

screens? Semi-structured interviews with people who know what is going on in the relevant political realm

2. Rating Scales: Is there political salience/will for an issue? Interview top advocates working on an issue to a) Rate the legislators on receptivity/likelihood of voting in favor/against; b) Rate influence within the legislature; and c) Rate confidence in their own rating

Advocacy: Methods

19

2. Media Analysis – level of coverage, type of language used, increase in media hits following various actions (Lexis-Nexus on-line search engine for media mentions; services)

3. Growth in Partnerships/Alliances over time – see OMG Framework to assess partnership strength

4. Record Review to Assess Constituency/Support Base Growth; #s attending certain events, #s signing petitions, #s participating in list-serves, e-networks.

Advocacy: Methods

EXAMPLE: Fullojunk Watershed Alliance Advocacy

Strategic FocusImprove the environmental quality of the Fullojunk Watershed through local advocacy.

Long-term Outcomes

Passage of local ordinances to support comprehensive land use planning and reduction in storm water runoff

Short-term Outcomes

New champions and advocates (including Mayor)

Increased public awareness

Organizational visibility and recognition

Alliance Advocacy ActivitiesConduct issue/policy analysis research

Build relationships with decision makers and new administration staff

Lead grassroots organizing and mobilization

Assumptions

Assumptions

Assumptions

Contextual Analysis

There is a new mayor who is pro-environment

The Alliance has very little advocacy experience

Because there is a new mayor, a lot of advocacy organizations are clamoring to have their message heard, leading to issue competition

21

Exercise (Whole Group)

Who has an advocacy goal that they are trying to reach?

- What is the goal?- What are your activities?- Short term outcomes? Long term?- Assumptions along the way?

Group Input