Session 853Session 853 Ex Extending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programstending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programs

Board of Scientific Counselors Program Reviews Board of Scientific Counselors Program Reviews

of the U.S. Environmental Protection Agency’s of the U.S. Environmental Protection Agency’s Office of Research and DevelopmentOffice of Research and Development

Session 853Session 853 Ex Extending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programstending Organizational Capacity and Capability to Evaluate Federal Environmental Research Programs

Board of Scientific Counselors Program Reviews Board of Scientific Counselors Program Reviews

of the U.S. Environmental Protection Agency’s of the U.S. Environmental Protection Agency’s Office of Research and DevelopmentOffice of Research and Development

Lorelei KowalskiLorelei Kowalski

Designated Federal Official for the BOSCDesignated Federal Official for the BOSC

Office of Research and DevelopmentOffice of Research and Development

U.S. Environmental Protection AgencyU.S. Environmental Protection Agency

October 29, 2005October 29, 2005

2005 Joint Conference: Crossing Borders, Crossing Boundaries 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation AssociationCanada Evaluation Society & American Evaluation Association

Lorelei KowalskiLorelei Kowalski

Designated Federal Official for the BOSCDesignated Federal Official for the BOSC

Office of Research and DevelopmentOffice of Research and Development

U.S. Environmental Protection AgencyU.S. Environmental Protection Agency

October 29, 2005October 29, 2005

2005 Joint Conference: Crossing Borders, Crossing Boundaries 2005 Joint Conference: Crossing Borders, Crossing Boundaries Canada Evaluation Society & American Evaluation AssociationCanada Evaluation Society & American Evaluation Association

Federal Advisory Committee ActFederal Advisory Committee Act

• The 1972 Federal Advisory Committee Act (FACA), 5 U.S.C. The 1972 Federal Advisory Committee Act (FACA), 5 U.S.C. App. 2, created an orderly procedure by which Federal App. 2, created an orderly procedure by which Federal agencies may seek advice and assistance from citizens and agencies may seek advice and assistance from citizens and experts. experts.

• Enacted to promote meaningful public participation in Enacted to promote meaningful public participation in government decisions and to ensure that no particular government decisions and to ensure that no particular interest groups have unfair access to policy makers. interest groups have unfair access to policy makers.

• Maximizes public access to advisory committee deliberations Maximizes public access to advisory committee deliberations and requires balanced committee membership. and requires balanced committee membership.

• More than 950 advisory committees with more than 45,000 More than 950 advisory committees with more than 45,000 members advise the President and the Executive Branch on members advise the President and the Executive Branch on a variety of issues. a variety of issues.

Federal Advisory Committee Act Federal Advisory Committee Act RequirementsRequirements

• A Designated Federal Officer (DFO) (government employee) A Designated Federal Officer (DFO) (government employee) must attend each meeting. must attend each meeting.

• Meetings must be announced in advance and open to the Meetings must be announced in advance and open to the public, unless designated as closed for national security, or public, unless designated as closed for national security, or other appropriate reasons. The public is given an opportunity to other appropriate reasons. The public is given an opportunity to provide comments during each meeting.provide comments during each meeting.

• Any committee documents, including reports, minutes, agenda, Any committee documents, including reports, minutes, agenda, etc. must be made available to the public. etc. must be made available to the public.

• The committees must be “fairly balanced” in the points of view The committees must be “fairly balanced” in the points of view represented and bring a diversity of expertise and perspectives.represented and bring a diversity of expertise and perspectives.

Federal Advisory Committee ActFederal Advisory Committee Actat EPAat EPA

• EPA manages approximately 24 committees and 37 EPA manages approximately 24 committees and 37 subcommittees. subcommittees.

• Federal advisory committees are an important tool within Federal advisory committees are an important tool within EPA for building consensus and providing ideas and EPA for building consensus and providing ideas and recommendations from EPA’s diverse customers, partners, recommendations from EPA’s diverse customers, partners, and stakeholders. and stakeholders.

• Although not required under FACA, EPA requires Although not required under FACA, EPA requires subcommittees to adhere to the same public involvement subcommittees to adhere to the same public involvement requirements as Executive Committees.requirements as Executive Committees.

The Board of Scientific Counselors The Board of Scientific Counselors (BOSC)(BOSC)

The BOSC is a Federal Advisory Committee, established by EPA to provide advice, information, and recommendations about the ORD research program. •Includes up to 15 membersIncludes up to 15 members•Meets three to five times each year Meets three to five times each year •Has an approved charterHas an approved charter•Announces its meetings in the Announces its meetings in the Federal RegisterFederal Register•Opens its meetings to the publicOpens its meetings to the public•Has an internet websiteHas an internet website

The Board of Scientific Counselors The Board of Scientific Counselors MissionMission

Provide ORD advice and recommendations on:• Science and engineering research, programs and plans, Science and engineering research, programs and plans,

laboratories and centers, products, and management practiceslaboratories and centers, products, and management practices• Peer review policies, and utilization of peer review Peer review policies, and utilization of peer review • Program development and progress, research planning Program development and progress, research planning

process, and research program balanceprocess, and research program balance• Human resources planning, and the appropriate scope and Human resources planning, and the appropriate scope and

design of training programs for environmental research design of training programs for environmental research professionalsprofessionals

The Board of Scientific Counselors The Board of Scientific Counselors MembershipMembership

• Members constitute a distinguished body of scientists and engineers, drawn from academia, industry, and environmental communities, who are recognized experts in their respective fields.

• Members are Special Government Employees (SGEs), who are invited to serve on the Committee for their individual expertise.

• The BOSC can establish subcommittees, who must report their recommendations and advice to the BOSC Executive Committee for full deliberation and discussion.

• Subcommittees have no authority to make decisions on behalf of the BOSC, nor can they report directly to EPA.

• In FY05, the BOSC had eight active subcommittees .

Office of Research and Development Office of Research and Development Program ReviewsProgram Reviews

• The National Academy of Sciences has recommended The National Academy of Sciences has recommended independent expert review for evaluating federal research independent expert review for evaluating federal research programsprograms

• The Office of Management and Budget (OMB) highlights the The Office of Management and Budget (OMB) highlights the value of recommendations from independent expert review in value of recommendations from independent expert review in guidance to federal agencies.guidance to federal agencies.

• ORD chose to use the existing BOSC Committee to conduct ORD chose to use the existing BOSC Committee to conduct the expert review of ORD research programs.the expert review of ORD research programs.

Program Review PurposeProgram Review Purpose

• The program reviews provide prospective and retrospective evaluation of ORD research at the program level.

• Strengthen the research program and accountabilityStrengthen the research program and accountability• Communicate research progress and resultsCommunicate research progress and results• Verify clients application of research to environmental decisionsVerify clients application of research to environmental decisions• Help managers decide research investments/disinvestments Help managers decide research investments/disinvestments • Provide input to external evaluations and reports:Provide input to external evaluations and reports:

OMB’s Program Assessment Rating Tool Performance and accountability reports to Congress under

the Government Performance and Results Act (GPRA)

Evaluation CriteriaEvaluation Criteria

• OMB/OSTP* Investment Criteria for R&D: relevance, OMB/OSTP* Investment Criteria for R&D: relevance, quality, and performance—at the program levelquality, and performance—at the program level

• R&D investments must be planned to be R&D investments must be planned to be relevantrelevant to national to national priorities, agency missions, and customer needspriorities, agency missions, and customer needs

• Programs must maximize the Programs must maximize the qualityquality of the research they of the research they invest ininvest in

• R&D programs must demonstrateR&D programs must demonstrate performanceperformance by setting by setting annual and long-term goals and demonstrating progress annual and long-term goals and demonstrating progress toward outcomestoward outcomes

• OMB and OSTP encourage federal research managers to OMB and OSTP encourage federal research managers to characterize the program’s characterize the program’s scientific leadershipscientific leadership as well as well

*OSTP = Office of Science and Technology Policy*OSTP = Office of Science and Technology Policy

Program Reviews ConductedProgram Reviews Conducted

In FY05, the BOSC Executive Committee conducted program In FY05, the BOSC Executive Committee conducted program reviews of six ORD research programs by establishing the reviews of six ORD research programs by establishing the following subcommittees: following subcommittees: •Endocrine disrupting chemicalsEndocrine disrupting chemicals•Human healthHuman health•Ecological researchEcological research•Particulate matter/ozoneParticulate matter/ozone•Drinking waterDrinking water•Global changeGlobal change

Program Review ProcessProgram Review Process

• Each program review was coordinated by a DFO and Each program review was coordinated by a DFO and Subcommittee ChairSubcommittee Chair

• Process steps included formation of the subcommittee Process steps included formation of the subcommittee membership; coordination of a face-to-face meeting; and membership; coordination of a face-to-face meeting; and conference calls before and after that meetingconference calls before and after that meeting

• Review steps included development of charge questions; Review steps included development of charge questions; participation in conference calls and face-to-face meeting; participation in conference calls and face-to-face meeting; development of a draft report; vetting of the draft report through development of a draft report; vetting of the draft report through the BOSC Executive Committee; finalization of the report; and the BOSC Executive Committee; finalization of the report; and transmission of the report to ORD.transmission of the report to ORD.

• The BOSC completed 18 conference calls and 6 face-to-face The BOSC completed 18 conference calls and 6 face-to-face meetings in FY05 for the six subcommittees.meetings in FY05 for the six subcommittees.

Program Review ResultsProgram Review Results

• Final program review reports transmitted to ORD in FY05:Final program review reports transmitted to ORD in FY05: Endocrine disrupting chemicals: April 2005 Human health: August 2005 Particulate matter/ozone: August 2005 Ecological Research: August 2005

• A total of 72 recommendations were made in these reportsA total of 72 recommendations were made in these reports• ORD has already prepared responses to recommendations ORD has already prepared responses to recommendations

from two of these reports, and is working on responses for the from two of these reports, and is working on responses for the other two.other two.

Program Reviews Next StepsProgram Reviews Next Steps

• Lessons learned in implementing program reviews from the Lessons learned in implementing program reviews from the past year were discussed at a BOSC Executive Committee past year were discussed at a BOSC Executive Committee meeting in September 2005, including perspectives from both meeting in September 2005, including perspectives from both the Committee and ORD. the Committee and ORD.

• Measures to improve and streamline the process were Measures to improve and streamline the process were suggested and are being developed/implemented.suggested and are being developed/implemented.

• All agreed the effort was a worthwhile activity, and the BOSC All agreed the effort was a worthwhile activity, and the BOSC will continue to conduct program reviews for ORD.will continue to conduct program reviews for ORD.

• In FY06, the BOSC is planning to complete program reviews for In FY06, the BOSC is planning to complete program reviews for the Global Change and Drinking Water research programs, and the Global Change and Drinking Water research programs, and initiate program reviews for the Land (Hazardous Waste), Water initiate program reviews for the Land (Hazardous Waste), Water Quality, and Sustainability research programs.Quality, and Sustainability research programs.