1

Knowledge Brief August 2018 info@threadstl.org

EVALUATION LEARNING COHORT

www.threadstl.org

2

Introduction

Purpose This document serves two core purposes:

1) To document the knowledge and resources generated and shared by participants in Thread STL’s Evaluation Learning Cohort (ELC); and

2) To create a living document which other Thread STL members can learn from, contribute to, and build off as they continue to explore pertinent knowledge, skills, and resources for successful evaluation of community partnerships.

Guiding Question In what concrete ways can St. Louis community partnership and coalition leaders assess and evaluate complex collaborative efforts specific to the issues they are addressing in the St. Louis region?

Methodology This report was completed by Casey Knotts, a practicum student working to build Thread STL's capacity and document the learning and knowledge-sharing of the Learning Cohorts from April 2018 through August 2018. Information was collected through informational interviews with Thread STL stakeholders, discussions with Thread STL professional staff, and observations of the last three Evaluation Learning cohort meetings.

www.threadstl.org

3

Context

In the context of this Knowledge Brief, leaders and managers of community partnerships are the core staff, or volunteers, guiding a coalition of change-makers toward achieving their shared goals and desired outcomes that ultimately add up to population-level change. One central task in this work is coalition evaluation.

Evaluating the impact of a coalition is significantly more difficult than programmatic or organizational evaluation as the goals of a coalition are not as linear nor tangible. Consequently, it can be more difficult to determine if the goals of the coalition are being met. Furthermore, people have difficulty implementing theory behind evaluation into practice. Specifically, many members of the ELC stated having trouble applying evaluation theory to their unique practice setting. Finally, it can be challenging to communicate the value of evaluation to certain stakeholders or organizational leadership, particularly in a resource-constrained environment. Nevertheless, assessment and evaluation are vital to establishing the attribution of a community partnership or coalition’s work to any change in a particular issue. When coalition leaders and supporters feel comfortable and confident in implementing even the simplest of evaluations, community partnerships are better situated to determine areas of improvement and make a case for their contribution to desired changes in their respective communities.

This group of coalition leaders and supporters convened to discuss frameworks specific to coalition evaluation, share tools and instruments, and problem-solve around particular challenges in coalition evaluation. The ELC began meeting in April 2018 with an initial group of nine people, but other Thread STL members have attended one or more meetings as they have been able to.

Becky Reinhart, Director of Community Health Initiatives – DeSales Community Development

Lindsay Noblot, Project Director – SouthSide Early Childhood Center

Matthew Moore, Community Engagement Specialist – North Newstead Association

Joe Monahan, Manager - Systems Change Strategies – Ready by 21

Nataly Garzon, Specialist - Systems Change Strategies – Ready by 21

Casey Locey, Early Childhood Resource Manager – Collective Impact Network

Terry Bates, Information Resource Manager – Collective Impact Network

www.threadstl.org

4

Hopes and Priorities

Upon the ELC’s initial meeting, participants took time to identify their aspirations for the group. The ensuing list served as a target for the group’s discussion and activities:

Build confidence around evaluation skills and knowledge.

Develop further understanding of basic frameworks in evaluation.

Build concrete skills grounded in a framework that is applicable to the context of community partnerships and collaborations.

Meeting Summaries

Please see the resources page for more information on the activities discussed below.

Meeting Activities

April 10th, 2018

The primary goal of this meeting was to develop learning objectives for the cohort, articulate hopes and fears, and establish group norms. The activities used during this meeting included “Connection Circle” and “Hopes and Fears” to help break the ice and invite participants to share their motivations for signing up for the cohort and what they wanted to get out of the program.

May 15th,

2018

The second meeting of the ELC focused on previous members' evaluation experience. The group also discussed what attributes they value during an evaluation process through a small group activity (Attribute Activity).

June 19th,

2018

Paul Evenson of Social Innovation STL joined the ELC for the third meeting to discuss some of his work with the Falmouth Prevention Partnership in Massachusetts and what steps coalitions should take when implementing an evaluation process.

July 17th,

2018

The final meeting of the 2018 ELC included a presentation from Sarah Kennedy at Generate Health STL about Results-Based Accountability. The group also reflected on the ELC experience and discussed the potential for continuing the group.

www.threadstl.org

5

Key Insights

Common Language

One of the main difficulties that ELC participants identified was the lack of a universal language pertaining to coalition and partnership evaluation. Coalitions are frequently comprised of a variety of disciplines so terminology can mean something different across professions. Similarly, professionals from different disciplines may prioritize certain characteristics or attributes of an evaluation, such as "Does no harm to the community" or "practical". This can lead to conflict amongst stakeholders and can impact the strength and validity of the evaluation's final product. Prior to undergoing an evaluation process, it is important to convene all necessary stakeholders and establish a shared lexicon of language and attributes that will continuously inform the evaluation process.

Document Document Document

According to Paul Evensen in his June 2018 presentation to the ELC, all coalitions are ready and should engage in some type of process evaluation. Whether that is through a brief sign-in sheet at the beginning of every coalition meeting, or a more formal software system that tracks specific strategies and their affiliated indicators and outcomes, coalitions who document their work are better able to link their work to impact than those that do not. Documentation can be implemented freely or cheaply, and will improve coordination, provide accountability for funders or community stakeholders, set up member organizations to share progress, and allows for coalitions to celebrate any victories.

Just Do It!

Evaluation at the coalition level is difficult; there are few concrete examples to draw from and organizations that already struggle to fund their various strategies and programs are hard pressed to channel funds into evaluation. This combination of factors can make evaluation seem like a risk, and fear of failure can be a strong deterrent for many in the social impact sector. However, it is better to try something, potentially fail, and learn from those mistakes than to not try at all. A good mantra shared by Paul Evensen is “If it’s not written down then it didn’t happen;” further, those coalitions that do not engage in any kind of evaluation have more difficulty demonstrating their attribution to any impact, so just do it!

www.threadstl.org

6

Next Steps

More Specialized Topics

The present iteration of the Evaluation Learning Cohort focused on the basic building blocks and frameworks of coalition evaluation since evaluation at the regional and systems levels is still novel. The next round of the Evaluation Learning Cohort could focus on the process of developing specialized evaluation methodology tailored to a specific community partnership or coalition. Similarly, future Learning Cohorts could focus on more specialized topics within evaluation.

Subject Matter Experts

One of the largest themes in the feedback received from participants was the desire to hear from more subject matter experts (SMEs) who have engaged in evaluation work at the coalition/community partnership level. Paul Evensen's presentation on his previous work in coalition-level evaluation was identified as particularly useful by ELC participants as it provided them with concrete steps they could use in their own coalitions. Bringing in other evaluation experts, either at the organizational level or community partnership level, can provide future ELC participants with guideposts for exploring their own evaluation efforts.

www.threadstl.org

7

Meeting Resources

Meeting Resources

April 10th, 2018

Connection Circle - Use this detailed script to identify connections between a seemingly unconnected group of topics or to help the group prioritize and select their shared objectives.

Hopes and Fears -- This activity is designed to help break the ice in the group and to familiarize members with each other’s motivations for joining.

May 15th,

2018

Attributes Activity --This activity provides space for discussion to determine what “attributes” or qualities are most vital to a partnership’s evaluation, and how to ensure that these attributes are incorporated into the structure and philosophy of the evaluation.

June 19th, 2018

Evaluation 101 -- Use this resource to learn about the basic building blocks of process, outcome, and impact evaluation.

Falmouth, MA Case Study -- This presentation by Paul Evensen provides an overview of his work with the Falmouth Prevention Partnership and their efforts to evaluate the impact of their coalition’s work in reducing teen alcohol consumption. This resource also provides step-by-step instructions for implementing a coalition-level evaluation.

July 17th, 2018

Results-Based Accountability -- This website provides an overview to Results-Based Accountability, an evaluation framework that can be employed in the evaluation of coalitions and partnerships.

Performance Accountability -- This is a form that can be used to guide stakeholders through the development of a Results-Based Accountability evaluation process by asking a series of questions including baseline information, key partners, and identifying measures of success.

Population Accountability -- Similar to the Performance Accountability form, this form walks stakeholders through a series of questions that will help frame what outcomes are desired and achievable at the population level.

www.threadstl.org

8

Outside Resources

Introductory Resources Coalition Evaluation 101 (Social Innovation Readings)

Introduction to Evaluation (KU Community Toolbox)

Fundamentals of Evaluating Partnerships (CDC)

A Practical Approach to Evaluating Partnerships (Tom Wolf @ tomwolff.com)

FLOURISH Evaluation Plan (Sarah Kennedy)

Scriptapedia -- an introduction to group model building that uses structured small group exercises called scripts. (Social System Design Lab @ the Brown School)

Advanced Resources

Guide to Evaluating Collective Impact (Foundation Strategies Group)

A Developmental Evaluation Primer (The J.W. McConnell Foundation)

Evaluating Community Initiatives for Health and Development (World Health Organization)