NCCMT Spotlight Webinar: Guide to Policy-Influence Evaluation

Follow us @nccmt Suivez-nous @ccnmo

Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The

views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..

Guide to Policy-Influence Evaluation

Presenters:

Marla Steinberg, PhD CE

Anima Anand, PhD

Michele Hopkins, RSW, MSW

Shannon Bradley Dexter, MSc

January 25, 2017 1:00 – 2:30 PM ET

Follow us @nccmt Suivez-nous @ccnmo2

Housekeeping

Use Chat to post comments and/or

questions during the webinar

• ‘Send’ questions to All (not

privately to ‘Host’)

Connection issues

• Recommend using a wired

Internet connection (vs.

wireless),

• WebEx 24/7 help line

• 1-866-229-3239

Participant Side

Panel in WebEx

Chat

Follow us @nccmt Suivez-nous @ccnmo3

After Today

The PowerPoint presentation (in English and French)

and English audio recording will be made available.

These resources are available at:

PowerPoint: http://www.slideshare.net/NCCMT/

Audio Recording:

https://www.youtube.com/user/nccmt/videos

Follow us @nccmt Suivez-nous @ccnmo4

How many people are watching

today’s session with you?

Poll Question #1

A. Just me

B. 1-3

C. 4-5

D. 6-10

E. >10

Follow us @nccmt Suivez-nous @ccnmo

Your profession? Put a √ on your answer (or RSVP via email)

/

Epidemiologist Management (director,

supervisor, etc.)

Allied health

professionals (nurse,

dietician, dental

hygenist, etc.)

Librarian Physician / Dentist Other

5

Follow us @nccmt Suivez-nous @ccnmo

Guide to Policy-Influence

Evaluation

http://www.nccmt.ca/resources/search/241

Episode 30

6

NCC

Infectious

DiseasesWinnipeg, MB

NCC

Methods

and ToolsHamilton, ON

NCC Healthy

Public PolicyMontreal, QC

NCC

Determinants

of HealthAntigonish, NS

NCC

Aboriginal

HealthPrince George, BC

NCC

Environmental

HealthVancouver, BC

7

Registry of Methods and Tools

Online Learning

Opportunities

WorkshopsMultimedia

Public Health+

Networking and

Outreach

NCCMT Products and Services

8

Follow us @nccmt Suivez-nous @ccnmo9

Poll Question #2

How familiar are you with the

method or tool we are discussing

today?

A. I am not familiar with the method or tool

B. I have heard of the method or tool

C. I have used the method or tool

Follow us @nccmt Suivez-nous @ccnmo10

Presenter

Shannon Bradley Dexter, MSc

Senior Policy Analyst,

Innovation Strategy, Public

Health Agency of Canada

Follow us @nccmt Suivez-nous @ccnmo11

Presenter

Marla Steinberg, PhD CE

Evaluation Consultant

Follow us @nccmt Suivez-nous @ccnmo12

Presenter

Anima Anand, PhD

Project Lead, Healthy

Weights for Children, The

Bridge Youth & Family

Services

Follow us @nccmt Suivez-nous @ccnmo13

Presenter

Michele Hopkins, RSW, MSW

Project Coordinator, Healthy

Weights for Children, The

Bridge Youth & Family

Services

Evaluating Policy-Influence

NCCMT Webinar

January 25, 2017

Marla Steinberg, Ph.D. CEEvaluation Consultant

Anima Anand, Ph.D. and Michele Hopkins M.S.W.Healthy Weights for Children Project

Shannon Bradley DexterInnovation StrategyPublic Health Agency of Canada

Welcome

By the end of this webinar, you should be able to:1. List the complexities of evaluating policy-influence.2. Assess how the Guide to Policy-Influence Evaluation can support your work.

Establishing our Learning CommunityPoll: What is your interest in policy-influence?

A. I am a practitioner who does policy-influence work

B. I am a public health manager or decision-maker with a mandate for policy-influence work

C. I am a funder who funds policy-influence work

D. I am an evaluator who evaluates policy-influence work

E. I am a student interested in policy-influence work

F. Other

G. I don’t really know what policy-influence work is……………..

Poll Question #3

Have you ever…..

• Wondered how you get senior decision makers and politicians to take an interest in your program or issue?

• Wondered how best to support the scale-up of an intervention?

• Wondered how to evaluate your influence on policy?

• Wondered what should be considered a realistic achievement for policy-influence work?

The Guide to Policy-Influence Evaluation may be just what you need….

Needs, definitions, and the complexity of policy-influence work and evaluation

Setting the context for the guide

Origins of the Guide: The Public Health Agency of Canada’s Innovation Strategy (IS)

• Funds population health intervention research in diverse communities across Canada to build evidence about "what works" "for who" and "in what context" to reduce health inequities.

• Promotes and shares intervention results to inform future policy, program design, and other actions that will improve population health.

• Funding provided in three phases.

Current focus areas: Mental Health Promotion and Achieving Healthier Weights

IS Evaluation Requirements

• IS projects are required to undertake a comprehensive evaluation including assessing the uptake of evidence or knowledge generated through the project.

• Uptake is often achieved through the “influence on policy.”

THE ISSUE: Projects needed guidance, tools, and resources to effectively report on how IS projects were influencing policy. They

wanted to be able to tell their policy-influence stories!

Exploring the world of policy-influence work

Definitions

What is policy?

• Formal and wordy definition:

Policy is a means of governing action with the aim of attenuating or promoting particular phenomena occurring in the population.

Policies can outline rules, provide principles that guide action, set roles and responsibilities, reflect values and principles, as well as state intentions.

Policies can be enacted by all levels of government (federal, provincial, regional and municipal), community organizations, businesses, and schools.

Polices can guide programs, practice or education.

• Adapted from the National Collaborating Centre for Healthy Public Policy (2010) Accessed at: http://www.ncchpp.ca/docs/MethodPP_EN.pdf

• Endorsed by 95% of IS project respondents in the policy evaluation needs assessment conducted in the fall of 2013.

What is policy-influence work?

Policy-influence work supports the uptake or spread of evidence-based

interventions.

An example

“With partner organizations, we developed a food security questionnaire that went to all party leaders during the recent provincial election to see where they stand on food security issues. The next step will be holding them to some promises. We have also been working with the municipality to develop a food charter and food strategy. Prior to this we prepared a fact sheet on food security as a municipal issue, which was distributed to supporters and encouraged people to write in support of including food security in the Regional Plan”

Policy maker education

Policy development

Public awareness campaign

Advocacy

Other terms……….

What is unique about policy-influence work and its evaluation?

Policy-Influence is Complex

“Policy-influence is a highly complex process shaped by a multitude of

interacting forces and actors. Outright success, in terms of

achieving specific, hoped-for changes in policy, is rare, and the work that

does influence policy is often unique and rarely repeated or replicated,

with many incentives working against the sharing of ‘good practice’”

(Jones, 2011, p. 1)

A lot of policy-influence is about being there when the

window opens and being nimble to jump on board. A

lot of policy work is stand and wait and then jump when you need to act”

Bonnie Leadbeater – WITS (Mental Health IS project)

Policy-making and policy-influence are processes

All roads lead to Rome…..

There are a wide variety of activities that can influence policy

There are a variety of policy impacts:

(1) Conceptual – changing the thinking of key stakeholders

(2) Instrumental – changing actions

Carol Weiss

Jones (2011) offers further elaborations on therange of policy impacts:1. Framing debates and getting issues on to the political

agenda; drawing attention to new issues and affecting the awareness, attitudes or perceptions of key stakeholders

2. Encouraging discursive commitments; affecting language and rhetoric to promote the recognition of specific groups or endorsements of policy recommendations

3. Securing procedural change; changes in the process whereby policy decisions are made, such as opening new spaces for policy dialogue

4. Affecting policy content5. Influencing behaviour change in key actors: policy

change requires changes in behavior and implementation at various levels in order to be meaningful and sustainable

http://tclocal.org/food/Attributing Change is Difficult

Theories of change are useful (required!)

There are a variety of methods for evaluating policy-influence

Working Group of IS

Projects Created

Needs Assessment Completed

Guide Developed

Case Studies Added

Final Production

and Translation

The development of the guide

Recommends a limited number of useful (and free) resources that are organized around a four-step evaluation planning process

Four steps are aligned with other evaluation planning frameworks

CDC Framework for Evaluation in Public

Health

Includes three case studies:

WITS - anti-bullying program for primary schools from the mental health stream

Healthy Weights Connections -System change intervention to improve public health services for aboriginal children and families

Our Food - Creating a food strategy for Halifax

Includes completed evaluation plans for the cases developed through using the resources

User Experience Insights

Set of “raw” ingredients

http://www.otisfundraisingideas.com/product/our-ingredients

To be selected, prepared, adapted, and consumed as needed to suit your context.

What it is not……

A step-by-step recipe to be rigidly followed

What it is not……

A step-by-step recipe to be rigidly followed

A resource on how to evaluate the implementation, effectiveness or impact

Who is the audience

Funders

Projects or organizations involved in policy-influence work

Evaluators

Assumes a basic understanding of evaluation

Uses for the guide

Read it to understand the complexities of and options for policy-influence work and its evaluation

Uses for the guide

Read it to understand the complexities of policy-influence evaluation

Use some of the frameworks to develop evaluation requirements for funded projects

Uses for the guide

Read it to understand the complexities of policy-influence evaluation

Use some of the frameworks to develop project-specific evaluation requirements

Use the tools to work with projects to plan their policy-influence work and evaluation

Healthy Weights for Children Project

created an innovative family education model

‘Healthy Together’© (HT)

• Integrated healthy eating and physical activity within group learning to bring families together to learn to make healthier choices and build healthy relationships.

• Established a collaborative national network to guide program integration within core practice and policy.

• 21 sites in 8 Provinces and Territories (BC, AB, SK, MB, ON, NL, NB, NWT)

• 150+ trained professionals across Canada• 1,000+ children, youth and caregivers • Community context:

• Northern, rural, urban, Aboriginal, multi-cultural• Low income, immigrant/refugee, foster families

Snapshot of HT

Program effectiveness

• HT demonstrated that inclusion of healthy cooking/eating and physical activity within group learning was key to participant engagement, leading to positive behaviour change.

• Reduced consumption of sugar sweetened beverages• Increased physical activity• Reduced screen time

• Relationships that developed during HT sessions led to building social support for families, particularly in underserved communities.

HT ‘policy influence’ needs

• An effective policy development framework • Concerted effort by multi-sectorial partners to promote

healthier weight practices and policy• Tools to measure the impact of policy influence work

Complexity of HT policy influence

• In order to be effective, HT needed to

– Be responsive

– Be adaptable

– Build capacity

– Enhance knowledge uptake

HT policy influence - process

Policy Development

Placement on Policy Agenda

Policy Adoption

Policy Implementation

Policy Maintenance

Tool for HT Policy Influence Work –Goals & Activities

Practice/ Policy Goal Activities/Tactics Timeline

What is an example of a practice/ policy related to achieving healthier weights?

How do you envision Healthy Together influencing practice/policy discussions?

What practice/policy goal(s) do you think is possible within your sphere of influence?

What activities/tactics will you engage in, to achieve the above goals, and the estimated timeline?

Comments

Thank you for your participation!

Office/Agency

Organization/Network

Community

Province

Other

HT policy influence activities

• Program uptake/practice change by sites

• Establishing relationships with key decision makers

• Policy makers’ education/ capacity building

• Policy development/maintenance

• Policy implementation/evaluation

• Demonstration projects/pilots

• Electronic outreach through social media

• Briefings/presentations

• Adoption of HT approach into core practice• Communities/systems engaged in ‘buy-in’

• Stakeholders leverage partnerships to support uptake

Ultimate outcome:

‘Vulnerable children will experience greater equality of health outcomes and achieve healthier weights. ‘

HT policy outcomes

HT theory of change

• If families at-risk of developing unhealthy weights come together through programming that enables healthy eating, increases enjoyment of physical activity and strengthens the family bond, then members will improve their overall health and overall quality of life with skills and habits that will last.

How do I get the guide?

NCCMT:

http://www.nccmt.ca/uploads/media/media/0001/01/c3374b8be4b35e1340385f1b593d3bb9f50f6a38.pdf

the guide?

Wrap Up – Key Messages

• Policy-influence in order to support scale-up of evidence-based programs is a concern to many funders and organizations

• There is a burgeoning literature on how to do and evaluate policy-influence work

• The Guide to Policy-Influence Evaluation will help you navigate this literature with a select number of high quality useful resources

• The Guide can help to:• Develop evaluation requirements for policy-influence work

• Support projects and organization to do and evaluate policy-influence work

Thank You!Questions/Comments/Contact

Marlasteinberg@telus.net

Anima.anand@thebridgeservices.ca

Michele.hopkins@thebridgeservices.ca

shannon.bradleydexter@canada.ca

www.healthy-together.ca

The Guide: http://www.nccmt.ca/resources/search/241

Follow us @nccmt Suivez-nous @ccnmo66

Your Comments/Questions

• Use Chat to post comments

and/or questions

• ‘Send’ questions to All (not

privately to ‘Host’)

Chat

Participant Side

Panel in WebEx

Follow us @nccmt Suivez-nous @ccnmo67

Poll Question #4

Could this method or tool be useful

in practice?

A. Very useful

B. Somewhat useful

C. Not at all useful

D. Don’t know

Follow us @nccmt Suivez-nous @ccnmo68

Your Feedback is Important

Please take a few minutes to share your thoughts

on today’s webinar.

Your comments and suggestions help to improve

the resources we offer and plan future webinars.

The short survey is available at: https://nccmt.co1.qualtrics.com/SE/?SID=SV_8ib8Yz9pRao

GLBz

Follow us @nccmt Suivez-nous @ccnmo69

Poll Question #5

Would you be willing to receive an

email from the NCCMT with a

follow-up survey in 2-3 months?

A. Yes

B. No

Follow us @nccmt Suivez-nous @ccnmo70

Join us for our next webinarNCCMT Spotlight on Methods & Tools: CDC Clear Communication Index

Date: Wednesday, February 1, 2017

Time: 1:00 – 2:30pm EST

Interested in evidence-based criteria for clear communication

of public health material? Do you develop and review public

health communication materials? Are you interested in

enhancing clarity and ease of understanding of these

materials? Join us for a webinar on how the Clear

Communication Index can help you.

Register at: https://health-evidence.webex.com/health-

evidence/onstage/g.php?MTID=eb180a59b9994f1d4da0729d

c3d31345d

Follow us @nccmt Suivez-nous @ccnmo71

Poll Question #6

What are your next steps? (Check all

that apply)

A. Access the method/tool referenced in the presentation

B. Read the NCCMT summary about the method/tool described today

C. Consider using the method/tool in practiceD. Tell a colleague about the method/tool

Follow us @nccmt Suivez-nous @ccnmo

Funded by the Public Health Agency of Canada | Affiliated with McMaster UniversityProduction of this presentation has been made possible through a financial contribution from the Public Health Agency of Canada. The

views expressed here do not necessarily reflect the views of the Public Health Agency of Canada..

For more information about the National Collaborating Centre for Methods and Tools:NCCMT website www.nccmt.ca

Contact: nccmt@mcmaster.ca