1

Evaluation and Effectiveness

of Social Enterprises

2

Outline

• Basics of outcomes measurement

• Fads and buzzwords• Effectiveness

3

Are We Doing a Good Job?

• Stakeholders demand accountability.

• Many enterprises invest in evaluation…

• …to measure outcomes.• This informs stakeholders about

the enterprise’s effectiveness.

4

Groups to Which Social Enterprises are Accountable

• Government• The nonprofit sector

(peers)• Direct constituents

– Clients– Donors– Trustees– Staff and volunteers

• General public

Mechanisms and standards differ between groups

Ref. Brody 2002

5

Accountability Mechanisms

• Institutional mechanisms– Disinterested BoD– Nondistribution constraint

• New models– Disclosure and reporting

• Government regulation (e.g. IRS, AGs)• Private standards and accreditation (e.g. BBBs, NPO

councils, FASB)– Formal evaluation

• Private standards and accreditation• Donors (e.g. UW regulations)

– Markets• Donors• Member-elected BoDs• Staff willingness to work• Tax payer-dictated laws

6

The Role of Formal Evaluation

• Effective enterprises subject themselves to evaluation– Regular, outside criticism– Benchmarking– Regular reassessment of objectives– Regular assessment of progress in

meeting objectives

7

Evaluation Metrics

OUTCOMESOUTCOMES

INPUTSINPUTS

ACTIVITIESACTIVITIESACTIVITIES

OUTPUTSOUTPUTS

IMPACTS

Social value

8

Outcomes Measurement

• Outcomes– Benefits for participants during or after

their involvement with a program– May relate to knowledge, skills, attitudes,

values, behavior

• Outcome measures– Indicators of the actual impact or effect

upon a stated condition– Typically expressed as a percentage, rate

or ratio.

9

Why Measure Outcomes?

• External “pull”– Increasing

accountability– Competition– Funding

•Internal “push”– Adherence to

mission– Measure results– Strategic

planning– Budgeting

10

Outcome Levels

• Initial outcomes– Most directly related to programs– Example: Children’s nutrition program raises

awareness among parents• Intermediate outcomes

– Link immediate program effects to ultimately-desired outcomes

– Example: Parents feed their kids differently• Long-term outcomes

– The ultimate goals of a program– Example: Kids are healthier

• Outcome measured depends on timing

11

Data for Outcome Measurement

• Existing data– Proprietary– Secondary—group comparisons for

benchmarking• Observations• Generated data

– Surveys•Quantitative•Qualitative

12

Typical Data Problems

• Low sample sizes– Random distortion

• Certain areas undersampled– Nonresponse

• Improper survey design– Results biased

• Unrepresentative observation environment

13

Measurement Tools

14

Outline

• Basics of outcomes measurement

• Fads and buzzwords• Effectiveness

15

Fads and Buzzwords

• Double bottom line• Social return on investment• Financial vulnerability• Spending nothing on fundraising

16

The Double Bottom Line

Net revenues

Mission adherence

Rmax

Mmax

R*M=0, R=0Net revenues

Mission adherence

Rmax

Mmax

R*M=0, R=0

17

Social Return on Investment

Enterprise valueFinancial return on investment

Social purpose valueCosts and savings from serving social mission

Blended valueEconomic + socioeconomic value of enterprise

18

Calculating Social Return on Investment

Enterprise value Value of sales- Cost of good and services sold- Operating expenses

 Social purpose value

+ Grants and gifts- Fundraising and grantwriting costs+ Social cost savings- Social operating costs+ Increases in Taxes

 Debt - Debt carried by social enterprise___________________________________________= Blended value

19

Financial Vulnerability Measures

High equity balance(Assets-liabilities)/Total revenues

Low revenue concentrations1

2+s22+….=RC (s1+s2+…=1)

Low administrative costsAdmin costs/Total costs

High operating margin(Revenues-Costs)/Revenues

20

The right level of fundraising to demonstrate

• NPOs always minimize fundraising

• Maximizing revenues to spend on mission means fundraising to the point that the last dollar spent earns a dollar back

• Do NPOs spend too much…or too little?

21

The evidence

0

0.5

1

1.5

2

2.5

3

3.5

All nonprofits

Arts &

culture

Education

Health

Social w

elfare

Other nonprofits

Ret

urn

on

th

e la

st d

olla

r sp

ent

on

fu

nd

rais

ing

22

Outline

• Basics of outcomes measurement

• Fads and buzzwords• Effectiveness

23

Effectiveness Must Be Understood

Multidimensionally

Goal attainmentAre we adhering to

our mission?(Do we knowour mission?)

Resource acquisitionDo we have

adequate funding?

EfficiencyAre we producing our desired

output at minimum cost?

Client satisfactionAre our constituents

satisfied?

ENVIRONMENTALADAPTATION AND RESPONSIVENESS

24

Effective Nonprofits:External Characteristics

NP experts• Collaborate with

outside org’ns• Earn income• Diversify funding• Measure

outcomes

NP executives• Collaborate with

outside org’ns• Measure

outcomes • Diversify funding • Know clients• Earn income

Light 2002

25

Effective Nonprofits:Internal Characteristics

NP experts• Exploit technology• Give staff authority• Work in teams• Stay “flat”

NP executives• Push authority

downward• Exploit technology• Work in teams• Save for crises• Stay “flat”• Maintain a diverse

staff

Light 2002

26

Effective Nonprofits:Leadership Characteristics

NP experts• Open

communications• Motivate employees• Focus on

fundraising• Clear BoD-staff

relations• Seek input

NP executives• Open

communications• Motivate employees• Clear BoD-staff

relations• Focus on

fundraising• Give staff freedom

to take risks

Light 2002

27

Effective Nonprofits: Management System Characteristics

NP experts• Use BoD• Clear

responsibilities• Use data in

decisionmaking• Invest in training

NP executives• Plan• Clear

responsibilities• Use BoD• Track funds

Light 2002

28

What Does an Effective NPO Look Like to Donors?

• Has an impact• Follows through on promises in

service• Employees “care” about society

and supporters

29

Performance Measures:Client Satisfaction

• A legal status and statutory base that meets its needs.

• A mission statement that is current, applied, and helps to guide action

• Responsiveness to beneficiary needs

• Accountability to the community via the governing Board

• Involvement of beneficiaries and other stakeholders in program decisions

• Dissemination and communication of results and its needs

• Building of coalitions, partnerships, and networks

• Excellent relationships with principal government departments

• Excellent relationships with principal sources of funds

• A public image of integrity, cooperation, and capability

30

Performance Measures:Resource Acquisition

• Diversified financial resources, and resource development programs

• Human resources to deliver service, attract resources, and promote mission.

• Leadership resources who provide vision and strategic direction

• Diversity in gender, age, location, and ethnicity consistent with local setting

• Adequate physical resources and infrastructure in HQ, branches, sites

• Intellectual capital: knowledge, judgment, unique skills and/or individuals

• Purchasing and procurement skills to optimize the use of financial resources

• Technology and infrastructure resources: IT, logistics, telecom, MIS

• Ability to exploit interactions between resources

31

Performance Measures:Efficiency

• Facilitates decision-making, team development, and conflict resolution

• Has effective internal communications

• Has logistical capability appropriate to serve the vulnerable.

• Works to develop staff, volunteer, management, and leadership resources.

• Has and uses processes for organizational learning and development.

• Has control and reporting via budgeting, planning, reporting, and audit

• The Board provides guidance and review

• Management is accountable via strategic planning and evaluation of service

• Is efficient, with growing productivity and with appropriate capacity.

• Uses its resources in an ethical and just manner.

32

Performance Measures:Goal Attainment

• Uses long-term, strategic planning to achieve the mission

• Achieves its long-term, strategic goals

• Has a program development strategy to respond to vulnerability

• Achieves its program goals

• Makes administrative plans (operations, structure, budget, resource use)

• Achieves its administrative goals

• Has operational planning to deliver services according to strategy

• Achieves its operational goals

• Evaluates outcomes against goals at strategic, program, administrative, operational levels

33

Performance Measures:Adaptation and Responsiveness• To constituency

change (vulnerable, political / social environments)

• To resource changes, e.g. economic change

• To changes in managerial practice

• Adapts goals to changing circumstances

• Changes actions in response to evaluations

• Learns, implementing new knowledge and innovation from these scans