Embed Size (px)
DESCRIPTION
As a solution, the BER model uses a two-dimensional matrix to aid the evaluation of complex multi-unit programs, with quadrants to identify over and underperforming units. The BER model was inspired by portfolio management approaches from the Boston Consulting Group and the General Electric Grid, as well as quadrant analysis by Andreasen (1995). However, its core principles are based on the concept of social return on investment, where output is always compared to input. It provides a relative perspective on performance that allows evaluators to account for impact based on the resources invested in an initiative.
Citation preview
BASIC EFFICIENCY RESOURCE:A framework for measuring the relative performance of multi-unit programsBrian Cugelman, PhDManaging Director, AlterSpark
4 October 2010
Canadian Evaluation Society's Annual ConferenceOptimizing the Practice of EvaluationToronto, Canada
WHERE BER STARTEDEvaluation of Oxfam
GB’s Global Climate Change Campaign
Solution to evaluation challenges
Lot’s of interest
Leitmotiv and AlterSpark joint publication on BER
2
EVALUATION CHALLENGES
Simplifying complex multi-unit programs
Many evaluations are about ROI, but ROI is difficult to assess in social contexts
Nothing is good or bad, except in comparison to something else
3
THE BER SOLUTION
Provide a simple framework for evaluating complex multi-component programs, campaigns, or activities
Build on the basic concepts of SROI to evaluate unit's impact compared to their resources
Offers a relative perspective on performance where units of analysis are judged in comparison to their peer units, operating under similar conditions
4
THEORETICAL FOUNDATIONS: MATRIX ANALYSIS FRAMEWORKS• Boston Consulting Group
• General Electric Grid
• Customer satisfaction quadrant analysis by Andreasen
• Bloc modeling techniques used by social network analysts
• Multi dimensional scaling
5
THEORETICAL FOUNDATIONS: SROI
SROI: efficiency is output relative to input
• Input constitutes a program’s resources which may be measured by their budget, number of staff, pool of talent, social capital, or any measure of capacity, concrete or abstract
• Output measures a program’s impact, and will vary according to a program's purpose
6
THEORETICAL FOUNDATIONS: SROI
7
Input
High Below Average Efficiency
Average Efficiency
Low Average Efficiency
Above Average
EfficiencyLow High
Output
CASE STUDY: OXFAM GB’S GLOBAL CLIMATE CHANGE CAMPAIGN
8
3.53.02.52.0
Investment
4.5
4.0
3.5
3.0
Impact
Visual stunts & media relations
Effective com at policy events
Using celebrities
Global Oxfam affiliates
Research papers
Rapid news dissemination
Public campaigns/mobilization
Policy analysis
Partnerships with others
Oxfam internal program links
Online campaigning
Media engagement
Lobbying and advocacy
Staff in UNFCCC delegations
Climate hearings
Adopt a negotiator
Perc
eive
d im
pact
Perceived resourcing
High
HighLow Low
CONDUCTING A BER ANALYSISThe example in following section is fictional and for illustrative purposes.
9
1. SELECTING UNITS OF ANALYSIS
10
1. Units of Analysis
3. Perceived Input (Budgets)
2. Perceived Output (impact)
Online engagement
•Most•Average•Least•I don't know
•Most•Average•Least•I don't know
Research papers •Most•Average•Least•I don't know
•Most•Average•Least•I don't know
Lobbying and advocacy
•Most•Average•Least•I don't know
•Most•Average•Least•I don't know
2. MEASUREMENT TOOLS
11
Input Output Program budgets (perceived and
real) Number of staff Number and level of staff
How often a lobbying keyword appeared in policy
Number of widgets produced Number of people engaged Perceptions of impact
Completely
Disagree1
2 3 4 5
Completely
Agree6
I can't say
Online engagementResearch papersLobbying and advocacy
3. DATA TYPES
• Quantitative input data may include budgets, number of staff, or combined multi-dimensional resource measure
• Quantitative output data may include process evaluation measures such as the number of people engaged by a campaign or media hits
• Qualitative measures can include perceived program investments and perceived output achieved
12
4. VISUALIZATION APPROACHES
13
High
Low
Low High
4. VISUALIZATION APPROACHES
14
Input
High
Media relations Public relations
Lobbying and advocacy
Coalition/partnership building
Low
Intra organisational coordination
Online engagement
Research papersPublic mobilization
Low HighOutput
4. VISUALIZATION APPROACHES
15
5. INTERPRETATION
Use BER is as a starting point for deeper discussions into the performance of units, their challenges, opportunities, and operating environment
Understand the units of analysis and the informants who shared their perceptions
Not all units within a program operate under the same conditions
Some units contribute indirect effects, by empowering other units
16
LIMITATIONS AND RISKS
It is easy to draw conclusions from the simple visualizations that would never stand in the face of a deeper understanding of the reality behind charts
Kotler et al. (2005) noted, reliance on matrix approaches prompted a number of companies to sell off strategic assets and plunge into businesses that they lacked the experience to manage
17
FUTURE WORK AND BER DEVELOPMENT
Download a copy of the BER whitepaper: • www.alterspark.com/insights/publications.
html
Send comments or examples of BER analyses:• Dr. Brian Cugelman, AlterSpark• Eva Otero, Leitmotiv
Share your feedback on Facebook:• http://www.facebook.com/group.php?gid=143779348990230 18
THANK YOU
Brian Cugelman, PhDManaging Director, AlterSpark
Phone: +1 (416) [email protected]
www.AlterSpark.com@AlterSpark alterspark alterspark alterspark
19
