Embed Size (px)
Citation preview
Articulating the Value of Consumer Engagement2 – 3:30 pm ET July 25, 2013
Context
• CE Sustainability RFP: 8 projects• Technical assistance available
o Webinar serieso Individual calls
• Contribution to the field• Other Sustainability TA underway
Articulating the Value of
Consumer Engagement
Carla Zema, PhDZema Consulting
July 18, 2013
Outline• Importance• Recap of ROI from Sustainability Workshop• Articulating Value – Understanding Different
Methods• Review of Key Elements of Analysis• Determining Costs and Benefits• Valuing Benefits• Conclusion and Next Steps
Importance
One of the great mistakes is to judge policies and programs by their
intentions rather than their results
-- Milton Friedman
More EffectiveLess
Effective
More Costly
LessCostly
ADOPT
DO NOT ADOPT Tradeoff to
consider
Tradeoff to consider
Weighing Effectiveness and Cost
Use of results• Justify implementation to internal and/or
external stakeholders
• Interest potential partners, such as funders, community partners
• Justify use of resources
• Complement other evaluations
ROI TemplateElement Contents
Description Scope of the activity or program
Audience/point of view Stakeholders/funder(s) who will obtain ROI from participation
Financial benefits Hard dollar categories, $ for each
Strategic benefits Description with $ if possible
Intangible benefits Description, qualitative value
Costs Hard dollar costs; other costs a stakeholder might want to include
Additional components e.g., time, probability of success
ROI Calculation and/or qualitative evaluation
AF4Q Sustainability Workshop, 2013
A rose by any other name…
• ROI
• Cost-benefit analysis (CBA)
• Cost-effectiveness analysis (CEA)
Comparison of Methods
CEA CBA ROI
Basic calculation
Cost-effectiveness ratio = Total costs--------------------------
Measure of effectiveness
Net benefit = Total benefit-Total costs Ratio = Total benefits/Total costs
ROI =
Net benefit-------------------------------
Total cost
Result Cost per measure of effectiveness (or incremental/marginal cost per unit measure of effectiveness)
Net benefit valued in dollars
Ratio results in number greater than or less than 1
Percentage
Comparison of Methods
CEA CBA ROI
Pros • Easy to compare programs with similar outcomes
• Useful if outcomes are difficult to value
• Results are in dollars making interpretation easy
• Useful when considering only one program and/or when programs have different outcomes
• Easy interpretation
• “ROI” more widely known
Cons • Measure of effectiveness can be awkward to interpret
• Difficult to use if more than one benefit
• Can be difficult to value benefits
• Indirect or intangible benefits are often not included
Steps to Conduct an Analysis
• Define the framework for the analysis• Determine the perspective of the analysis• Decide on the timeframe for the analysis• Define the population• Identify and measure the costs and
benefits• Value the benefits (for a CBA)• Discount (depending on timeframe)• Calculate results• Conduct a sensitivity analysis• Use results for decision-making
Define the Framework
• What is the overall objective?• Which options are being considered?• How will effectiveness be measured?• What is the status quo? (What if no action
is taken?)• Will the analysis be prospective,
retrospective, or concurrent?• Will a CEA or CBA be conducted?
Determine the Perspective of the Analysis
• Critical to defining the costs and benefits • Must be maintained throughout the
analysis• May or may not be the primary audience
for the results• Examples:
oFundersoPayersoPracticeoEmployersoSocietal
Decide on Timeframe
• Must be clearly defined• Considerations:
o Does program have a specific timeframe for implementation?
o When would results be expected?o What timeframe is meaningful to the primary
audience for the results?
Define the Population
• Considerations:o Particular subgroup of patients or practices?o Certain geography (e.g. state, county)?o Types of practices?o Multiple systems?o Types of payers?o All practices or just those with a certain number of
patients/provider?
Identify and Measure the Costs and Benefits
• Categories:o Directo Indirecto Intangible
• Include only actual or real costs and benefits that are beyond the status quo
• Do not include transfers• Handling missing data
Value the Benefits for a CBA or ROI
• For a CBA or ROI, benefits must be valued• Can be “gains” or “costs avoided”• Options:
o Market value of benefito Downstream financial impact of the benefit
• Make sure the valuations are realistic and meaningful to stakeholders
Discounting• Costs and benefits should be discounted if
greater than one year
• No consensus on standard discount rate
• Range of 3%-5% with 4% commonly used for social program
Calculate Results• Spreadsheet software such as Microsoft
Excel make calculations simple and transparent
• Keep each costs and benefit as well as quantities and rates as separate cells to increase transparency and make changes easier
• Results cells are then simply functions that calculate the results
Conduct a Sensitivity Analysis
• Opportunity to explore the variation in the results if inputs are modified
• Helpful given that some inputs have some uncertainty or a range of possible values
• Two types:o Partialo Extreme case
Partial Sensitivity Analysis
• One input variable is varied at a time while holding all other values constant
• Compare new results with base case results• Consider varying variables:
o representing assumptions that were made for the analysis
o that have a range of possible valueso that were estimated due to missing or
unavailable datao that have uncertain values
Tornado Diagram Example
Payer mix
Reimbursement rate
P4P incentive
Increase in improvement
Number of providers
Number of practice staff
Practice volume
Cost of training
Number of patient partners
Extreme Case Sensitivity Analysis
• Similar to partial sensitivity analysis in terms of variables to vary in the analysis
• Instead of one variable at a time, all variables that are selected are varying simultaneouslyo Once in a “best-case” scenarioo Again in a “worst-case” scenario
Summary• CEAs/CBAs are not a silver bullet and do
not provide a definitive answer
• Offer important information to inform decision-making and complement other evaluation information on effectiveness
• Critical component of any program or project evaluation
Next Steps• Webinar series will walk you through the
process• Homework:
o Finalize frameworko Begin to identify costs and benefits
• Individual technical assistance is available• Next webinar will be opportunity to get
group feedback
