Benefit-Cost Analysis: ATP Experience Jeanne Powell Economic Consultant Technology Program Evaluation: Methodologies from the Advanced

Benefit-Cost Analysis:Benefit-Cost Analysis:ATP ExperienceATP Experience

Jeanne PowellEconomic [email protected]

Technology Program Evaluation: Methodologies from the Advanced Technology Program March 18, 2008 National Science Foundation

Jeanne Powell, Economic Consultant

2

Benefit-Cost Analysis: OutlineBenefit-Cost Analysis: Outline

Benefit-Cost Analysis of Federally-funded Technology:

What is it? What is it good for? What metrics are computed? Whose return on investment? How does it work? ATP Studies Advantages Disadvantages Summary


3

Benefit-Cost AnalysisBenefit-Cost AnalysisWhat is it? What is it?

Evaluation tool for:– Estimating net impacts of a federal technology

development project or group of related projects according to program:

Mission Logic Model Timeline

– Comparing total estimated costs with total estimated benefits over a project’s useful life

Can be Prospective (reflect projected future effects), or Retrospective (reflect past effects), or a mix of both


4

What is it good for?What is it good for?

Quantitative case study Economic estimation Generally most suited to long-term evaluation

– Risks and uncertainties of R&D are large in early phases; therefore, the probability distributions of the economic estimates are broad until technical and business uncertainties are reduced over the longer term

Generally most suited to applied R&D, close to market


5

What does it measure?What does it measure?

Return on Investment:– Net benefits– Benefit-to-cost ratio– Internal rate of return

What investment?– Depends on whose return is being

measured


6

Whose Return?Whose Return?Social Return AnalysisSocial Return Analysis

Private return– return to project participants on their own investment/cost

share Private profits to technology innovator

Spillover return– return to others beyond the project participants, including

broad societal benefits, on combined sources of investment Benefits to technology users (intermediate and final) Broader societal benefits (e.g., public goods)

Social return– return to project participants and nation broadly on

combined public program and private investments needed to generate return


7

Whose Return?Whose Return?Public Return AnalysisPublic Return Analysis

Public return– Increased social return enabled by

public investment relative to that public investment

– Additionality effect– If multiple public sources contributed

to the benefit, need to allocate benefit


8

How Does It Work? (1)How Does It Work? (1)

Usually is based on monetary valuation of multiple types of benefits expressed as cash-flow analysis


9

Cash Flows for Project

Time

Investments

Cash FlowBenefits

Offsets


10

How Does It Work? (2)How Does It Work? (2)

Cash flows are time adjusted– To constant dollars– For opportunity cost of money

OMB Circular A-94 mandates use of a 7% real rate. As a result, ATP has used a 7% real rate in nearly all benefit-cost studies

Discount rate consistency will be important for comparing results of different investments

Cash flows are most commonly discounted back to the start of the project/investment being analyzed

Cash flows are adjusted for risks and uncertainties using expected value analysis

Metrics are computed—typically with Excel

ATP’s Studies To DateATP’s Studies To DateShort Title (# of Projects) Pub

NumberContractor Pub

Date

Green Technologies Cluster (2) GCR 06-897 Delta Research 2007DNA Diagnostics Cluster (2) GCR 06-898 RTI

International2007

Photonics Cluster (2) GCR 05-879 Delta Research 2005Composites Cluster (2) GCR 04-863 Delta Research 20042mm Project—Retrospective (1) GCR 03-856 MIT 2004HDTV Joint Venture (1) GCR 03-859 RTI

International2004

A-Si Detectors for Digital Mammography (1)

GCR 03-844 Delta Research 2003

Component-Based Software (8) GCR 02-834 RTI International

2002

Closed-Cycle Refrigeration (1) GCR 01-819 Delta Research 2001Digital Data Storage (2) GCR 00-790 RFF 2000Flow-Control Machining in Auto Industry (1)

NISTIR 6373

NIST-BFRL 1999

Tissue Engineering (7) GCR 97-737 RTI International

1998

2mm—Early Assessment (1) GCR 97-709 CONSAD 1997Printed Wiring Board (1) GCR 97-722 Albert N. Link 1997


12

Advantages Advantages (Cash-flow analysis approach)(Cash-flow analysis approach)

Provides quantitative estimates of impact that complement qualitative analysis

Based on standard financial analysis used by private businesses. (The public finance literature has extended them to the analysis of public and societal benefits.)

Models are intuitive and relatively easy for policy makers and others to understand

Can be applied systematically and relatively uniformly

Can be extended to analyze multiple projects


13

DisadvantagesDisadvantages(cash-flow analysis approach)(cash-flow analysis approach)

Modeling of spillover benefits is challenging and expensive as with other tools

Many important spillover benefits may be difficult to express in monetary terms

Private benefits are often confidentialResults are sensitive to uncertainties

about technical success, market adoption, and inter-industry diffusion


14

Advantages and DisadvantagesAdvantages and DisadvantagesSummarySummary

Benefit-cost analysis is a practical choice for long-term evaluation of program impacts

Utility increases as empirical basis increases:

Technical accomplishment Product development Entrepreneurial activity (including financing) Market adoption Economic diffusion


15

Send e-mail to: [email protected]

Visit ATP/TIP website: www.atp.nist.gov

View ATP/TIP publications: www.atp.nist.gov/eao/eao_pubs.htm

For further information:For further information:

Documents

Benefit-Cost Analysis: ATP Experience Jeanne Powell Economic Consultant Technology Program Evaluation: Methodologies from the Advanced