Fix it or Flee it?Proven approaches for dealing with

failing, flagging and floundering association programs

Greg Melia, CAE @gmeliacae

What We’ll Do Today

• Identify 5 steps essential to program assessment

• Use real world examples to highlight pitfalls to avoid and key points to remember

• Talk about real world issues and challenges

Photo credit: hebedesign on flickr

Program Assessment is a lot of GRIEF

Program Assessment is a lot of GRIEF

• Goals

• Research

• Impact

• Efficiency

• Finances Credit: Angie Torres on Flickr

Goals

Credit: dkwonsh on flickr

Defining the Goal of the Review

–Work with key stakeholders and decision-makers

–Solicit their key questions:• What do they want to know?• What information will help their decision-

making?• What do they think needs to be

evaluated?

–Refine questions to be SMART

Original Program Aspects

Investigate the Key “Whys”:• Why was the program originally established?• Why has it continued to be offered?• Why has it change over time?

Review the Original Intended Approach:• Who, what, when, where, why and how?• What was implemented? • What was not?

Lessons Learned

• Verbalizing goals of the review are an important part of preparing for change

• Seek superordinate goals

• Some agreements are easier reached upfront

Research3 kinds of data:

Corpus Inscriptionum

Opinions and Descriptions

Imponderabilia of behavior

Bronislaw Malinowski

Credit: Photographie et Ethnologie: Les photographes français au XXème siècle

devant d’autres formes de cultures.

Lessons Learned

• Historical minutes and documents can be VERY informative

• Seek to understand program mutations

• Good decisions based on bad data usually give bad results

• Remember they may have been involved

Impact Measure

• Outputs = Immediate

I attended

• Outcomes = Short-term

I gained knowledge

• Impacts = Long-termI served more members as a

result

Lessons Learned

• Associations tend to over focus on outputs and outcomes.

• Hard to argue “happy & full”, but sometimes necessary.

• Is the program impact large and broad enough to displace alternatives?

Efficiency

Good, Better, BestNever let it rest

Until your Good is Better

And your Better is Best!

Credit: eatwell.in on Flickr

Efficiency

Staff and Volunteer Time

Complaints and Comments

Marketing/Communication

Registration/Orders

Technology Photo credit: mansikka on Flickr

Lessons Learned

• How do people feel about it? Do they have suggestions to increase efficiency?

• What are the trends in terms of level of effort?

• What is cost/benefit of doing it the same way versus trying a new approach?

• Can improved efficiency save it?

Finances

• Direct fixed expenses– Incurred regardless of how many people

participate, buy, or are served

• Variable expenses– Incurred per each additional unit or

person that is serviced

• Mixed expenses– Incurred per each additional set of units

or persons serviced

Types of Costs

• Direct fixed expenses– Staff, Rent– Purchased equipment– Per event contracts

• Variable expenses– Per person contracts– Attendee gifts, handouts, etc.

• Mixed expenses– Temporary help– Office supplies

Key Calculations

• Revenue – Expense = Margin

– Margin per attendee/unit

– Margin per staff hour

– Margin per impact

Allocating Overhead Costs

• Equal allotment

• Proportional allotment – To budget– To number served– To staff hours

Example

Annual Giving Campaign

Revenue:$12,000

Expenses: $ 2,000

# Solicited: 1,000

Staff Hours: 200

# of contributors: 100

Example

• Revenue – Expenses = Margin$12,000 - $2,000 =

$10,000• Margin per unit (solicitation):

$10,000/1,000 = $10

• Margin per staff hour:$10,000/200 =

$50• Margin per impact (contributor):

$10,000/100 = $100

Lessons Learned

• One full flight is more profitable than two half full ones.

• Most do not realize the true full cost.

• Understanding costs helps determine price.

• Price increases (and closing programs) take time.

Tips on Implementation

• Internal Evaluators:– Program knowledge &

skills– May create staff buy-in– Less $$$ outlay

• External Evaluators– Usually seen as more

objective– Assessment expertise– Comparative ideas– More $$$ outlay

Photo credit: Matthew Burpee on Flickr

Implementation

• Who to involve–Stakeholders–Staff–Non-users

• Who to interview–Former volunteers

–Former staff

–Vendors

Photo credit: kierenmccarthy.co.uk

Communication of Results

• Review purpose of review

• Review what was done

• Arrange key findings in logical order, highlighting interpretations where appropriate

• Close with recommendations or issues to be addressed

Credit: EE.UTD.Events on Flickr

Lessons Learned

• Equivocal recommendations get equivocal results

• Decision-makers need synthesized data, not the opportunity for data analysis

• Survey data is important, but the proof is in historical performance

• Plan and communicate transitions

Questions?

Thank You!Greg Melia, CAE

gmelia@asaecenter.org@gmeliacae