Embed Size (px)
Citation preview
Less Pain, More Gain:An Evidence-Based Approach to Long-term
Deficit Reduction
Jon BaronCoalition for Evidence-Based Policy
March 2013
Coalition for Evidence-Based Policy
• A nonprofit, nonpartisan organization.
• Mission: To increase government effectiveness through rigorous evidence about “what works.”
• Coalition has no affiliation with any programs or program models – thus serves as an objective, independent resource on evidence-based programs.
• Funded independently, by national philanthropic foundations (e.g., MacArthur, William T. Grant).
Program reforms shown to produce major savings without loss of quality or benefit:
Successful welfare-to-work programs (Riverside, L.A., Portland)
• Evaluated in large, convincing randomized trials
• Increased employment and earnings by 20-50%
• Produced net taxpayer savings of $1700-$6000/person
• Had important policy impact (1996 welfare reform act)
Cost-saving program reforms, continued:
Transitional Care Model (a nurse-led hospital discharge and home follow-up program)
• Evaluated in two well-conducted randomized trials
• Reduced rehospitalizations 30-50%
• Produced net savings of $4000/patient
• Successful national replication could save $10 billion/year
Cost-saving program reforms, continued:
Subsidized Guardianship as alternative to foster care
• Evaluated in large randomized trial in Illinois
• Increased permanent placements by 8%
• Reduced days in foster care by 16%
• Saved Illinois $2300/child ($90 million over 10 years)
• Expanded nationally (2008), CBO scored $800M savings
Too few examples currently exist, because:
• Rigorous evaluations are still relatively uncommon in most program areas.
• Promising ideas, when rigorously tested, too often are found not to work.
Promising ideas, when rigorously tested, too often are found not to work
Medicare Coordinated Care Demonstration (MCCD)
• Preliminary studies had suggested large cost savings (e.g., 25%)
• However, the more definitive, randomized MCCD found a net cost increase of 11%.
Other examples:
• In medicine: 50-80% of interventions found promising in phase II (nonrandomized studies or small efficacy trials) are found ineffective in phase III (sizable RCTs).
• In business: Google and Microsoft have conducted ~13,000 RCTs of new products/strategies, 80-90% of which found no significant effect (Manzi).
• In K-12 education: Of 120 interventions evaluated in
large, convincing RCTs funded by Institute of Education Sciences 2003-2012, ~90% had weak or no effects.
Proposed strategy for long-term deficit reduction:
“If you want to increase your success, double your failure rate.” - Thomas Watson, IBM
• Accelerate development of promising, innovative program reforms with potential for cost savings.
• Greatly increase the number that are rigorously evaluated.
How to do this?
• Low-cost rigorous evaluations make it possible to test hundreds of program models/strategies each year, rather than a select few (as is currently done).
• Federal “waivers” from law/regulation can be used to stimulate state and local innovation and evidence-building aimed at entitlement savings.
• New high-visibility “prize competition” seeks to jump-start low-cost trials and testing of cost-saving reforms.
Examples:
Medicare Health Support Pilot (disease management)
• Preliminary studies suggested a high ROI (3:1 to 7:1)
• However, large randomized MHS found a net cost increase of 4-11%.
OMB & CBO Budget Scoring Creates New Incentive for Development/Expansion of
Evidence-Based Programs
• OMB will credit entitlement savings a program generates, that have been demonstrated in “rigorous experimental research.” (2010 guidance)
• CBO stated that it “gives greater weight to demonstrations and experiments that use random assignment …” when scoring the cost of legislation. (2011)
Comprehensive Child Development Program: Impact on Mothers’ Employment Rate
Impact on Family Welfare Receipt
Impact on Percent of Children “At Risk” in Cognitive Development & Behavior
Standardized Achievement Scores
<85%
Mental Processing Scores <85%
Behavioral Scores in Clinical Range
0%
10%
20%
30%
40%
50%
60% Age 3
Age 3
Age 2
Age 5
Age 5
Age 5
Treatment
Impact on Percent of Children “At Risk” in Cognitive Development & Behavior
Standardized Achievement Scores
<85%
Mental Processing Scores <85%
Behavioral Scores in Clinical Range
0%
10%
20%
30%
40%
50%
60% Age 3
Age 3
Age 2
Age 5
Age 5
Age 5
Age 3
Age 3
Age 2 Age 5
Age 5
Age 5
Treatment Control
Career Academy Students
Non-randomized Comparison Group*
*The comparison group consists of similar students in similar schools nationwide. Their estimated rates of postsecondary completion are statistically adjusted to control for observed differences between their background characteristics and those of the Career Academy group.
Randomized Trial Results
Comparison Group Study Results*
Impact of Career Academies on Completion of a Postsecondary Credential
Source: Data provided by James Kemple, MDRC Inc.
Career Academy Students
RandomizedControl Group
Websites that identify evidence-based programs
• Top Tier Evidence www.toptierevidence.org
• Social Programs that Work www.evidencebasedprograms.org
• Blueprints for Violence Prevention www.colorado.edu/cspv/blueprints/index.html
• What Works Clearinghouse (K-12 Education) www.ies.ed.gov/ncee/wwc
• Best Evidence Encyclopedia (K-12 Education) www.bestevidence.org
