Measuring Results That Matter: Evaluating CED Impacts

Measuring Results That Matter: Evaluating CED Impacts

The Community Economic Development Technical Assistance Program (CEDTAP)

Presenters:

David DriscollChair, CEDTAP Advisory Panel

Edward JacksonDirector

Gail ZbochNational Coordinator

Ray FunkPrairies Regional Coordinator

Colleen KastingBC Regional Coordinator

3

4


Overview of session:

1:30-4:00

CEDTAP at-a-glance CED knowledge clusters Results to date Evaluating CED impacts: assessing the CEDTAP

portfolio Group discussion Sharing preliminary results of impact surveys Next steps

5

Goal, Objectives and Methods

Goal Demonstrate the effectiveness and legitimacy of community

economic development as a strategy for economic and social change

Objectives Strengthen the capacity in CED of community-based

organizations Strengthen the capacity of the CED sector as a whole Methods Grant-making to CEDOs to access technical assistance Building of a pool of TA Providers Grant-making to sector organizations Knowledge production and dissemination Participation in research and policy alliances and consortia

6

CEDTAP At-a-glance

3CI

CEDTAPNational Secretariat

Donors:J.W. McConnell Family Fdn Bell Canada The Ontario Trillium FdnWED, DEC-Québec Other Corporations

Advisory Panel

CED Organizations

(CEDOs)

Technical Assistance Providers

CEDTAP Regional Coordinators

7

Program Phases, 1997-2008

Phase I pilot phase (1997-2000) $3M Some 90 grants, 30 TA Providers, three national conferences Funder: McConnell Foundation Phase II expansion phase (2001-2005) $7M Some 330 grants, 800 TA Providers, three national

conferences, policy roundtables Funders: McConnell Foundation, Bell Canada, The Ontario

Trillium Foundation, DEC Québec, WED-BC, WED-Prairies, RBC, Power Corp., Tides Canada

Phase III transition phase (2006-2008) $3M+ Grant-making, knowledge mobilization, corporate

engagement, research alliances Funders: Bell Canada, The Ontario Trillium Foundation,

McConnell Foundation, WED-BC, WED-Prairies, DEC-Québec, SSHRC, Power Corp., RBC, others.

8

20 CED Knowledge Cluster Areas

• Agriculture & Fishing• Arts & Culture• Disability and Mental Health-

Consumer Survivor Businesses

• Community Land Trusts• Community Economic

Renewal• Community Tourism• Employment Strategies• Environmental Management

& Enterprises• E-Strategies

• Financial Equity• Food Processing & Food

Security• Health & Nutrition• Homelessness & Poverty• Individual Development

Accounts• Industrial Re-conversion• Forestry• Sustainable Housing• Women Entrepreneurs• Youth Entrepreneurship

9

Community Strengthening Projects Categorized by CED Cluster (2001-2005)

0

5

10

15

20

25

30

35

40

Comm

unity

Econo

mic

Renew

al

Comm

unity

Land

Tru

sts

Comm

unity

Loan

Fun

ds

Comm

unity

Tour

ism

Disabi

ltiy &

Men

tal H

ealth

Cons

umer /

Sur

vivor

Bus

iness

es

E-stra

tegie

s

Emplo

ymen

t Stra

tegie

s

Enviro

nmen

tal M

anag

ement

& E

nter

prise

s

Financia

l Equit

y

Food

Proce

ssing

/ M

arke

ting

(inclu

ding

food

ban

ks)

Fores

try

Health

& N

utrit

ion

Homel

essn

ess &

Pov

erty

Indi

vidua

l Dev

elop

men

t Acc

ount

s

Indu

stria

l Rec

onve

rsio

n

Susta

inab

le H

ousin

g

Wom

en E

ntre

pene

urs

Youth

Ent

repr

enur

ship

Servic

es

10

Results: British Columbia, 2001-05

Top CED clusters: Community economic

renewal Employment strategies Environmental

management Food

processing/marketing

Current activities: Bell/McConnell WED support for regional

coordination Partnership with Vancity

Community Foundation

0

5

10

15

20

2001 2002 2003 2004 2005

$564,000 granted to 49 projects

11

Potluck Café Society Inc.

Potluck's Flash Frozen Gourmet Meals Project

12

Results: Prairies, 2001-05

Top CED clusters: Agriculture & fisheries Community economic

renewal Disability and mental

health-consumer survivor businesses

Current activities: Bell / McConnell WED support for regional

coordination

0

5

10

15

20

2001 2002 2003 2004 2005


13

Stardale Women’s Group Inc. Foundation

Stardale Artistic Co-operative

14

Results: Ontario, 2001-05


renewal Disability and mental

health-consumer survivor businesses

Employment strategies Women entrepreneurs

Current activities: Bell/McConnell The Ontario Trillium

Foundation

05

10152025

2001 2002 2003 2004 2005

$ 762,000 granted to 76 projects

15

Good Day Workshop Inc.

Participants at the Good Day Workshop,

refinishing old furniture

16

Results: Quebec, 2001-05


renewal Employment strategies Environmental

management Youth entrepreneurship

Current activities: Bell / McConnell DEC Québec

0

5

10

15

20

25

30

2001 2002 2003 2004 2005

$ 625,500 granted to 74 projects

17

L’atelier De Formation Socioprofessionnelle De La Petite-nation

Des participant(e)s de l’Atelier FSPN

18

Results: Atlantic Region, 2001-05

Top CED clusters: Arts & culture Community loan funds Community tourism

Current activities: Bell / McConnell

0

1

2

3

4

5

6

7

8

9

2001 2002 2003 2004 2005


19

Central Development Association

Promotion of social and economic development in the Northern Peninsula Central Region of Newfoundland

20

Evaluating CED Impacts:Assessing the CEDTAP Portfolio

Why evaluate CED impacts? New policy actors (eg, new federal government) are

unfamiliar with CED Increased emphasis on accountability, value-for-money

and results Heightened competition for scarce public and private

funds Need to evaluate program effectiveness and impact Opportunity to contribute to the body of knowledge on

CED policy and practice Opportunity to enable CED sector to lever bigger impacts

in high growth contexts (winter Olympics in BC; energy and resources on prairies)

21

How to Evaluate CED Impacts?

Test against program results chain for community initiatives

Utilize best-practice methods and tools from the field as a whole and the portfolio, in particular

“Slice and dice” the portfolio by knowledge cluster and policy area

Be clear about units of analysis: individual/household, group or enterprise, program/fund, CEDO, community, region

Assess progress on qualitative and quantitative indicators Survey CEDOs Survey Technical Assistance Providers (TAPs) Sample a small number of organizations and projects for

detailed case study analysis (eg, 15)

22

Results Chains

From: CEDTAP Program Summary, 2002

Results Chain for Community Initiatives

TA Initiative Outputs Organizational Outcomes Community ImpactsCoaching and Mentoring

TrainingPlanning process

AdviceTools

Evaluation Processes

New Knowledge AcquiredStronger Leadership and Vision

Business / Project PlanningManagement / SystemsEconomic Sustainability

Social Viability / LegitimacySectoral Expertise Gained

EmploymentBetter Quality of Life

Income / AssetsSocial Capital

Credibility of CEDOther Household Measures

Effective Partnerships in and Between Communities

Results Chain for Sector Initiatives

Sector Initiative Outputs Sector Outcomes Sector ImpactsTools

ResearchPolicy

DisseminationWorkshopsConferences

LearningConnectivityPartnerships

Resource UsePolicy Work

Credibility of CEDPro-CED Policy

Increased Public & Private Spending on CEDIncreased # of CED professionals

23

Evaluation: “The 21st Cluster”

Unit of analysis organization/project

Region Western Valley Development Authority, NS Evaluation Toolkit of Local Capacity

Community CIEL, Community Vitality Index

CEDO OISE/UT, Expanded Added Value Statement*PEACH, Evaluation Framework

Program Saint John Human Development Council /Tamarack institute, Small Business Program

Evaluation Framework

Enterprise Social Capital Partners, Social Return on Investment*

Households/ Eko Nomos/WEDC, Sustainable LivelihoodsIndividuals Approach*

Flavour Budzzz, Wisconsin Quality of Life Index

*Not CEDTAP project

24

Expanded Value Added Statement

Measures the economic and social value added to raw materials, products and services by non-profits and cooperatives’ use of labour and capital

A social accounting model created by Mook (see Quarter, Mook and Richmond, What Counts, 2003)

Estimates a comparative market value for non-monetized social contributions

Primary outputs refer to the value realized by the organization’s direct provision of services to advance its mission (eg. Revenues from fees for service, social or volunteer labour, services donated to the community)

25

Expanded Value Added Statement (Cont)

Secondary outputs refer to value to an organization’s members or customers realized through indirect outputs (eg. Skills training, learning through membership meetings, board and committee governance meetings)

Tertiary outputs refer to value realized by those other than the organization’s members or customers (eg. Consultations with other organizations or distribution to stakeholders)

In practice, EVAS builds credibility through the use of conservative assumptions in calculating comparative market value of social contributions

26

Key Themes in Building a Long List of Potential Case Study Cedos

Urban youth/immigrant employment Urban homelessness/poverty/redevelopment Youth entrepreneurship Alternative energy (urban) Community loan funds (urban) Women’s entrepreneurship (urban) Disability groups (urban) Mental health consumers/survivors (urban) Restaurant/catering (urban) Wood products (rural) Community tourism (rural) Community futures/development corporations (rural)

27

Exercise: Group Discussion

At your table, discuss two questions:

2. How does your organization assess CED impacts?

3. What is the biggest challenge your organization faces in assessing CED impacts?

Please record your group’s answers. You have 10 minutes for this exercise!

28

Findings in Process: Surveys of CEDTAP Grantees and TA Providers

1. Significant impacts of CEDTAP projects in which you have been involved?

2. Significant impacts of non-CEDTAP projects in which you have been involved?

3. Important indicators of: economic, social and environmental impact?

4. Key success factors that enable CED organizations to generate significant impacts?

5. Methods, tools and indicators for assessing impact that are most useful in your work?

6. Key barriers or obstacles that limit the impacts produced by CED organizations?

29

Generating Impacts: Top 10 Success Factors

Funding: start-up, core, promotional

Realistic and achievable plan

Input from stakeholders

Experienced and strong volunteer and board commitment

Commitment and participation of of management and staff

Time to plan and develop

Government, business and community support

Community partnerships

Public relations and marketing strategy

Ability to manage financial and social goals simultaneously

30

Economic Indicators

Number of jobs created Assisting people with training or starting a business Creation of social enterprises Increase in wealth – individual and community Increased spending on purchase of local supplies Decreased expenses through increased productivity,

sharing resources, eliminating wasteful practices Change of attitude and behaviour in evaluating business

from a financial measure to include the blended bottom line

Degree to which the venture can or will contribute to the overall society

31

Social Indicators

Ability to support oneself and one’s family Increase in self-esteem and self-image Community ownership of initiative Development of local leadership Improvement in community quality of life issues: safety,

housing, health, less hospitalization, recreation, transportation, childcare

Increase in the participation levels of the client group both as participants in the venture and recipients of the service

Legacy of work supported through long-term planning with government, business and not-for-profit sector

32

Notable Methods & Tools:

Formative and summative surveys

Good benchmark data socio-economic indicators that measure lifestyle & healthy community improvements

Identify targets and projections coupled with goals and objectives, all in keeping with the mission

Pre and post evaluations using qualitative and quantitative measures

Working with an outside facilitator (arm’s-length)

33

Notable Methods & Tools (Cont):

Feedback from focus groups

Communicate often and openly with stakeholders

Sharing the learning with colleagues and community: ongoing email, coffees, lunches

“….Organizations have asked for a copy of our CEDTAP-funded strategic plan”

Tamarack teleconferencing and initiating our our teleconferencing for specific projects

34

Generating Impacts: Barriers/obstacles

Lack of enough resources: financial & human

Time spent on writing grants and looking for money

Lack of interest by mainstream social service agencies to take a CED approach

Lack of long-term vision and plan for sustainability

Lack of leadership

Reliance on volunteers

35

Barriers/obstacles (Cont)

Projects too short in length Inability to engage community and stakeholders Lack of understanding by government agencies

and sponsors of the complexities of new industry development as well as “soft” changes that measure success

Success requires long-term commitment, for which some funders may not have patience

36


Next steps: Additional analysis of survey data Further analysis by clusters of the portfolio Identification of case study organizations Joint research with CEDOs, technical

assistance providers, academics and funders Dissemination of findings

Thank you!

Documents

Measuring Results That Matter: Evaluating CED Impacts