Improving and Integrating Evaluation into Program Management

Panel Presentation:Henry M. Doan, Ph.D.

Suzanne Le Menestrel, Ph.D.

CSREESSteve Loring, Ph.D.

New Mexico State UniversityCheryl J. Oros, Ph.D., Discussant

NIH

AEA 2006Portland, Oregon

The Portfolio Review Expert Panel(PREP) Process:

Planning and Accountability Office, CSREES Use and Perspective

Henry M. Doan, Ph.D.

Planning and AccountabillityCSREES

AEA 2006Portland, Oregon

CSREES Mission and Function

MissionTo advance knowledge for agriculture, the environment, human health and well-being, and communities

Functions• Program leadership to identify, develop, and manage programs to sponsor university-based and other institutional research, education, and extension.• Fair, effective, and efficient administration of federal assistance in implementing education, research, and extension awards and agreement.

Management Cycle

Planning•Identification of needs/problems, solutions•Conceptualization•Formulation of evaluation questions & designs

ImplementationData collection & analysis

Feedback•Sharing findings with program managers•Refining programs•Budget decisions

• Goal 1: Enhance Economic Opportunities for Agricultural Producers• Goal 2: Support Increased Economic Opportunitiesand Improved Quality of Life in America• Goal 3: Enhance Protection and Safety of the Nation’sAgriculture and Food Supply• Goal 4: Improve the Nation’s Nutrition and Health• Goal 5: Protect and Enhance the Nation’s NaturalResource Base and Environment

CSREES STRATEGIC GOALS(2004-2009)

Cascading Alignment

Strategic Goal

Strategic Objective

Portfolio

Knowledge Area Code

Projects

Agency Mission

**

**** May cross-cut objectivesand portfolios.

Alignment ExampleGoal 3: Enhance Protection and Safety of the Nation’s

Agriculture and Food Supply

Strategic Objective 4.1: Reduce the Incidence of FoodborneIllnesses and Contaminants through Science-Based Knowledge

and Education

Food Safety Portfolio

Knowledge Areas

• (KA 711) Ensure Food Products Free of Harmful Chemicals, Including Residues from Agricultural and Other Sources• (KA 712) Protect Food from Contamination by Pathogenic Microorganisms, Parasites, and Naturally Occurring Toxins

OMB Program Assessment Rating Tool (PART)

PART:• Program Purpose & Design• Strategic Planning• Program Management• Program Results

CSREES Portfolio Reviews:2004: Goal 1

2005: Goals 3 & 52006: Goals 2 & 4

This completed the first cycle of 14 portfolio reviews, covering 14 strategic objectives and all five CSREES 2004-2009 strategic goals

A Portfolio Approach toEvaluating Research, Education, and

Extension Efforts

• OMB PART/BPI led to development of new portfolio assessment tool and measures

• Portfolio analysis (meta-analysis) used to assess progress toward goals; guide announcements for grants

• Uses OMB R&D criteria (relevance, quality, performance)

Portfolio Review Expert Panel (PREP) Process

• Focus on outcomes rather than processes• Level of analysis is a portfolio identified via Knowledge Area codes in databases• Expert Panels score portfolio progress & provides recommendations for CSREES

PORTFOLIO: A New Concept

• Portfolio as a unit of analysis is a new concept• Portfolios are not included in funding lines, programs, and organization of CSREES work units• Use of Knowledge Areas codes for all work classification is new• Portfolio concept allows complex interrelated programs and funding lines to be described as they address CSREES strategic goals and objectives

PREP Unique Features

• Expert panelists are asked to systematically assess distinct dimensions of the 3 OMB R&D criteria (Relevance, Quality, and Performance)• Scoring process is standardized across portfolios, transparent, & scientifically based• Therefore, PREP can provide quantitative performance assessment of portfolios of research work

PREP Process

1. Identify/Select Expert Panels2. Develop Self-Study Report3. Compile Evidentiary Materials4. Self Score Prior To Panel Meeting5. Convene Expert Panels

1. Identify/Select Expert Panels

PREP Process:

Selection of high-level panelists with broad experience in topic area after careful reviews for absence of conflict of interest. Panel members included:

• University Vice-Presidents• Deans and Associate Deans• Industry Experts (Company Vice Presidents,

etc.)• Evaluation Experts• Experts from other federal agencies

PREP Process:2. Develop Self-Study Reports

NPLs develop portfolio self-study reports. The reports include the following:• Section I: Agency and PREP Overview• Section II: Portfolio Description, including Logic Models and Graphics (e.g. Honey Combs)• Section III: Knowledge Area Descriptions• Section IV: Discussion of how portfolio meets R&D Criteria and their Dimensions

PREP Process:3. Compile Evidentiary Materials

• Track papers, citations, patents, products, educational efforts, adoption of products/ practices• Identify and present evaluation studies and special analyses conducted in programs covered in portfolios• Present budget tables to show portfolio priorities and emphases

PREP Process:4. Self Score

Portfolio NPLs score their self-study reports using an instrument developed in-house and based on OMB R&D criteria of relevance, quality, and performance. These self scores will eventually be compared to those assigned to the portfolios by the expert panels.

PREP Process:5. Convene Expert Panels

Panelists meet for 2 ½ days in Washington, DC.

Day 1: for orientation, short briefings by program managers and NPLs, along with Q&AsDay 2: for further review of documentation, discussion, deliberation, and recommendationsDay 3: to complete draft report containing score to be submitted for PART & BPI, and recommendations for portfolio improvement; debriefing by panel to CSREES

PREP Process:Panel Scoring Sheet

Expert Panel scores each dimension of each of three R & D criteria using customized anchors on a 3-point scale:

3= Exceeds expectations 2= Meets expectations

1= Needs improvement

Panel Scoring Sheet:OMB R&D Criteria & Dimensions

Relevance:1. Scope2. Focus on critical needs3. Identification of emerging issues4. Integration of CSREES programs5. Interdisciplinary integration

Panel Scoring Sheet:OMB R&D Criteria & Dimensions

Quality:

1. Significance of findings & outputs2. Stakeholder assessment3. Alignment of portfolio with current science4. Methodological rigor

Panel Scoring Sheet:OMB R&D Criteria & Dimensions

Performance:

1. Portfolio productivity 2. Portfolio completeness 3. Portfolio timeliness 4. Agency guidance relevant to portfolio 5. Portfolio accountability

Panel Scoring Sheet Example: RelevanceSection 1:

-Relevance-

Dimensions

Purpose of Dimension

Rating:

3

Rating:

2

Rating:

1

40%

of total

Total relevance score

1.1 Scope – coverage of the work of the full portfolio

Define & summarize needed & existing portfolio topics

Fully demonstrates exceptional depth

Portfolio coverage is static in depth

Portfolio is falling behind

40%

1.2 Portfolio’s ability to remain focused

Clarify & examine if portfolio focus on critical needs

Fully focused Adequately focused

Needs improvement

20%

1.3 Identification of emerging issues

Identify important new issues consistent with the portfolio mission

Contemporary & emerging issues identified

Missing some emerging issues

Needs coverage of important issues

20%

1.4 Integration of agency programs for portfolio

Demonstrate functional integration

REE fully integrated

Partially integrated

Insufficiently integrated

10%

1.5 Multi-disciplinary balance

Demonstrate disciplinary and scientific balance…

Extensive balance among relevant disciplines

Partial balance

Little balance 10%

Interim Annual Internal Review

• Update self-review document• Consider recommendations from Review Panels and describe Agency and Portfolio responses and, for some portfolios, develop strategic plans• Used as interim preparation for next external review at the fifth year

P&A Experience Working With Program Managers & NPLs

• Self-study reports development requires systematic collection and analysis of program data• Requires open communications between P&A and program units• Requires close collaboration between P&A staff and NPLs• All program units need to be encouraged to develop strategic plans based on panel recommendations• Given lack of readily available data, the process is extremely demanding

For more information, please callHenry M. Doan, Ph.D.

202-401-0791Or e-mail hdoan@csrees.usda.gov