Broader Impacts at NSF Proposal and Award Policies and Procedures Guide (PAPPG) (9 BI categories) 2017 American Innovation and Competitiveness Act (7 BI categories) Broader Impacts

Suzi Iacono, HeadOffice of Integrative Activities

U MichiganMarch 15, 2018

Broader Impacts at NSF

National Science Foundation’s mission

“To promote the progress of science; to advance the national health, prosperity, and welfare; to

secure the national defense...”

$7.5BFY 2017 Enacted

93%funds

research,education and

related activities

50,000proposalsevaluated

11,000awardsfunded

2,000NSF-fundedinstitutions

359,000people NSFsupported

$1.2BSTEM

education

231NSF-funded

NobelPrize

winners

$100Mto seed

public/privatepartnerships

NSF by the numbers

Examples of NSF’s significant investments

U

3-D PRINTINGBREAKTHROUGH

LIGOGRAVITATIONAL WAVES

SBIR

FOUNDATION FORTHE INTERNET

BARCODESPOPULARIZED

EARLY WEBSEARCH

HL TAUDISCOVERY

AUTONOMOUS CARSOFTWARE

NSF Merit Review Criteria

• Intellectual Merit: The Intellectual Merit criterion encompasses the potential to advance knowledge; and

• Broader Impacts: The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes.

NSF Strategic Plan 2018 - 2022

• STRATEGIC GOAL 2 Advance the capability of the Nation to meet current and future challenges.

• This goal flows from the latter part of the NSF mission statement—”to advance the national health, prosperity, and welfare; to secure the national defense; andfor other purposes.”

• This strategic goal echoes the “broader impacts” merit review criterion that was developed by the National Science Board. Within this strategic goal, NSF also seeks to advance the state of the practice of achieving broader impacts.

A Topic of Great Interest to NSF, NSB, and Congress: All Seeking to Clarify

1974 11 criteria for NSF review 1982 4 criteria (research competence, merit of the

research, utility, and effect on infrastructure)1986 NSF officially utilized “merit” review1996 Intellectual merit and broader impacts1998 Accountability for BI established 2000s “Archived” criteria (5 BI criteria)2010 America COMPETES Reauthorization (8 BI categories)2011 National Science Board report2016 NSF Proposal and Award Policies and Procedures

Guide (PAPPG) (9 BI categories)2017 American Innovation and Competitiveness Act (7 BI

categories)

Broader Impacts CategoriesAICA

The Foundation shall apply a broader impacts review criterion to identify and demonstrate project support of the following goals:

(1) Increasing the economic competitiveness of the United States.(2) Advancing of the health and welfare of the American public.(3) Supporting the national defense of the United States.(4) Enhancing partnerships between academia and industry in the United States.(5) Developing an American STEM workforce that is globally competitive through improved pre-kindergarten through grade 12 STEM education and teacher development, and improved undergraduate STEM education and instruction.(6) Improving public scientific literacy and engagement with science and technology in the United States.(7) Expanding participation of women and individuals from underrepresented groups in STEM.

Broader Impacts CategoriesArchived

• Advance discovery and understanding while promoting teaching, training, and learning, e.g., by training graduate students, mentoring postdoctoral researchers & junior faculty, involving undergraduates in research experiences, & participating in the recruitment, training, & professional development of K-12 mathematics & science teachers.

• Broaden participation of under-represented groups, e.g., by establishing collaborations with students & faculty from institutions & organizations serving women, minorities, & other groups under-represented in STEM.

• Enhance infrastructure for research and education, e.g., by establishing collaborations with researchers in industry & government labs, developing partnerships with international academic institutions & organizations, & building networks of U.S. colleges & universities.

• Broaden dissemination to enhance scientific and technological understanding, e.g., by presenting results of research & education projects in formats useful to students, scientists & engineers, members of Congress, teachers, & the general public.

• Benefits to society may occur, e.g., when results of research & education projects are applied to other fields of science & technology to create startup companies, to improve commercial technology, to inform public policy, & to enhance national security.

BI Original, AICA and PAPPGDifferences

• Overlaps and gaps• (5) BI Original = Advance Discovery category

+ while supporting college students/postdocs

+BUT no K – 12 mention• (7) AICA = STEM/Workforce development

+K through 12, but no 13 through post doc+no research infrastructure category

• (9) PAPPG = Separates students from STEM workforce development

+includes research infrastructure category

Why Broader Impacts?

• Since the beginning of NSF, more excellent projects than there are funds.

• Non-technical considerations come into play when there are many projects of quasi-equal scientific merit.

• Clear criteria are important–• Many changes over the years at NSF, in reducing criteria• NIH still uses a larger number of criteria.

How Does This Work?

• Since 1998, more strict accountability for BI at every stage of the merit review process.

• BI must be discussed and evaluated at each stage:• Proposal• Reviews• Panel Summary• Review analysis• Annual report & final report

• COVs are asked to assess BI treatment at each stage.

Merit Review Process

BI

BI BI BI

Merit Review Process

COV Reports on BI

• COVs, in general, have a very positive view of NSF’s merit review process and often offer suggestions for improvements.

• A majority of the COVs called for greater clarification about the meaning of the BI criterion.

• Since documentation was inconsistent, COVs are concerned about unnecessary randomness in the merit review process.

• A number of COVs suggest more extensive training for reviewers.

COV Comments

• Relative weighting of IM and BI is not consistent across the section and variations in expectations of what constitute quality BIs are seen in individual reviews, panel summaries and review analyses [SEP].

• There continues to be confusion (among PIs, reviewers, and COV members) about what sorts of activities constitute broader impacts, and whether they need to be specifically related to the research aspects of the proposal [CISE].

COV Comments

• There is evidence of a wide range of interpretation among both PIs and reviewers about what constitutes broader impact and how it should be weighted in the proposal ratings [CBET].

• The broader impact criterion appears to be used by reviewers more as a tie-breaker rather than a more substantial and equally weighted criterion [CBET].

• Most reviewers made an effort to report on both intellectual merit and broader impacts. However, what constitutes broader impacts and the relative weighting of these criteria in the review varies greatly [PHY-A].

Recent BI Analyses and Developments • Schienke, E.W., Tuana, N., Brown, D.A., Davis, K.J., Keller, K., &

Shortle, J.S. (2009). The role of the National Science Foundation Broader Impacts Criterion in enhancing research ethics pedagogy. Social Epistemology (317-336).

• Rothenberg, M. (2010). Making judgments about grant proposals: A brief history of the Merit Review Criteria at the National Science Foundation. Technology Innovations 12 (189 – 195).

• 2013: National Alliance for Broader Impacts (NABI) formed: a community of practice that fosters the development of sustainable and scalable institutional capacity and engagement in broader impacts activity.

• Watts, S.M., George, M., & Levey, D.J. (2015). Achieving broader impacts in the National Science Foundation Division of Environmental Biology. BioScience, 65(4), 397-407.

• Ongoing: Analyses of institutional data in NSF’s Office of Integrative Activities. 18

An Empirical Look at BI• A three-year process, so far

• Directorate-wide discussions• Internal staff• External ACs

• NSF Leadership Workshop in September 2015• Report to OMB in December 2015 + July 2016• Agency-wide strategic review in January – March 2016• Report to OMB in June 2017• Presentations at BI Summits, hosted by NABI

• Consensus hypotheses• Hypothesis 1: The application of BI has changed over time; we

should expect it to continue to evolve.• Hypothesis 2: The so-called COV “confusion” might be variation in

BI due to different award sizes, disciplines and solicitation language; inconsistent instructions from NSF; and/or some other reasons.

• Hypothesis 3: There are some common practices in panels, e.g., tie-breaking.

50.77% 54.30%

37.56%

11.34%8.84%

11.35%

13.41%17.18%

14.95%

9.90%6.69%

8.55%

14.59% 12.99%

27.57%

Advance Discovery Benefit To Society Broaden Participation Dissemination Enhance Infrastructure

Investment Type: Project Summary

RESEARCH RESEARCH INFRASTRUCTURE

EDUCATION

48.01% 49.27%

34.84%

8.42% 5.43%

13.73%

14.79%22.82%

17.22%

15.99%10.78%

13.34%

12.79% 11.69%20.87%

Advance Discovery Benefit To Society Broaden Participation Dissemination Enhance Infrastructure


EDUCATION

Investment Type: Panel Reviews

-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

AdvanceDiscovery

Benefit ToSociety

BroadenParticipation

Dissemination EnhanceInfrastructure

Discovery(Total:6124-Collected5448)

Learning(Total:845-Collected821)

Research Infrastructure(Total:607-Collected552)

Greater Emphasis in the Summary

Greater Emphasis in the Reviews

Investment Type: Differences Between Reviews and Project Summaries

Comments from Panel ReviewsSentences in panel summaries for awards with the phrase “Is A.” in proximity to a broadening participation term. • the team is world class and includes a (couple of?) young investigators (one is a

woman) working at an institution (plu) committed to diversity of color and s-e• the postdoctoral researcher to be supported by this grant, <PI_NAME>, is an

emerging woman scientist• the pi is also a woman in computational area of cognitive science, an under-

represented population, and has a track record of training students in underrepresented groups (minorities and students with financial need)

• the pi is a woman from an under-represented group, and so this project would help increase the visibility and retention of minority women in a stem field

• as a Hispanic woman, the pi is a member a woefully underrepresented group in the sciences, and she has a track record of mentoring younger emerging scientists

• however, as a woman, the pi is already making a significant impact as a role model for female students interested in science

• the graduate student involved with this work is a woman and, given her presentation at the recent sclerochronology conference -which earned her an award for the best student talk at the meeting - she is very well qualified and shows a great deal of potential for continuing a career as a female scientist

4.82% 3.56% 5.79%

38.50%

53.45%43.88%

25.20%

21.49%25.17%

6.64%

6.12%4.91%

15.15%7.19%

11.17%

2.55% 1.89% 2.42%7.16% 6.30% 6.66%

Advancing Health And Welfare Broaden Participation Developing Workforce

Economic Competitiveness Industry Participation National Security

Public Literacy


EDUCATION

Investment Type: AICA Project Summary

9.36% 5.80%10.94%

37.02%

61.01%45.54%

28.14%

21.10%

25.99%

5.20%

2.09%2.01%

13.11%4.01%

7.94%1.26% 0.91% 1.97%

5.91% 5.08% 5.61%



Public Literacy


EDUCATION

Investment Type: AICA Panel Reviews

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

Advancing HealthAnd Welfare


DevelopingWorkforce

EconomicCompetitiveness

IndustryParticipation

National Security Public Literacy

Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)

Greater Emphasis in Reviews

Greater Emphasis in Summaries

Investment Type: Differences Between Reviews and Project Summaries

Differences Across Investment Types by Student Group

0.000

0.500

1.000

1.500

2.000

2.500

3.000

3.500

4.000

4.500

Doctoral student Graduatestudent

Masters student Phd student Post doctoral Studentassessment

Undergrad Undergraduateresearch

Undergraduatestudent

Infrastructure Research Education

50.39% 48.41% 52.00% 54.82%47.54%

56.63%

33.25%

9.91% 10.89% 6.15%

10.47%

14.60%4.26%

27.01%

16.33%11.09%

20.97%10.93%

11.63% 16.27% 17.75%

12.43%

7.62%

6.67%7.76% 10.88%

9.66% 8.66%

10.93%

21.98%14.21% 16.02% 15.34% 13.18% 13.32%

Advance Discovery Benefit to Society Broaden Participation Dissemination Enhance Infrastructure

BIO CSE EHR ENG GEO MPS SBE

Directorates: Project Summary

49.12%42.86%

49.10% 51.02%46.57%

51.77%

34.38%

6.12%10.59%

6.91%8.09%

10.53% 3.74%

20.66%

19.64%

13.16%

19.56% 11.92% 13.40% 18.12% 18.00%

17.48%

12.78%

10.40%14.91% 15.45%

15.78%12.62%

7.64%

20.61%14.03% 14.07% 14.04% 10.59% 14.34%


Directorates: Panel Reviews


-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%




BIO CSE ENG GEO MPS SBEEHR

Directorates: Differences Between Reviews and Project Summaries

2.90% 4.36% 1.02% 4.34% 4.45% 2.55%

12.66%

44.52%35.50% 42.20% 32.83%

37.39%

51.53%

42.75%

32.76%

20.32%

28.24%

24.83%

33.44%

28.23%

6.39%

2.47%

6.03%

10.11%

9.18%

3.77%

3.81%

10.72%

4.97%

21.19%

9.46%22.50% 6.67%

7.84%

13.10%

0.36% 7.48%4.58% 2.28%

0.31%

0.79%

5.55%

12.01%5.13% 4.40% 4.04%

13.97%5.24% 8.83%



Public Literacy

Directorates: AICA Project Summary


6.58% 7.35%2.01%

6.35%15.77%

6.20%

24.83%

41.77% 43.03%44.79% 32.99%

35.32%50.46%

40.55%

37.59%

20.29% 27.13%

24.68%

32.04%29.59%

7.59%

1.14%

3.04%6.19%

8.15%

1.71%2.52%

5.30%

2.77%

18.88%11.47% 23.67%

4.49%

7.01%

7.99%

3.75%2.46%

1.83%

0.41%

0.54%

2.20%

9.95%3.66% 5.94% 2.34%

10.26%3.68%

11.54%

Advancing Health And Welfare Broaden Participation Developing WorkforceEconomic Competitiveness Industry Participation National SecurityPublic Literacy

Directorates: AICA Panel Reviews


-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

14.00%

Advancing Health And Welfare Broaden Participation Developing Workforce Economic Competitiveness

Industry Participation National Security Public Literacy




Directorates: Differences Between AICA Reviews and Project Summaries

50.39% 52%48.41% 49% 52.00%

56% 54.82% 54%47.54% 49%

56.63% 59%

33.25% 31%

9.91%11%

10.89%13% 6.15%

6% 10.47% 13%

14.60%17% 4.26%

5%

27.01% 34%

16.33%16%

11.09%11% 20.97%

16% 10.93% 9%11.63%

10%16.27% 14% 17.75%

15%

12.43%10%

7.62%7%

6.67% 7%7.76% 7% 10.88%

11% 9.66% 10% 8.66% 7%

10.93% 10%

21.98% 20%14.21% 15% 16.02% 16% 15.34% 13% 13.18% 13% 13.32% 13%


EHR ENG GEO MPS SBEBIO CSE

Project Summaries: Differences Between Awards – Declines by Directorate

Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline

-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%



Great Emphasis in Declines

Greater Emphasis In Awards

Project Summaries: Differences Between Awards – Declines by Directorate

49% 52%43%

50% 49%54% 51% 55%

47%52% 52%

58%

34% 38%

6%9%

11%

12%7%

6%8%

10%

11%

13%4%

4%

21%

26%

20%17%

13%

10% 20%16%

12%9%

13%

10%18%

13%18%

15%

17% 13%

13%10%

10% 10%15% 11% 15% 12% 16% 12% 13%

8%

8% 10%

21% 19%14% 13% 14% 16% 14% 14% 11% 14% 14% 13%

Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_InfrastructureEHR ENG GEO MPS SBEBIO CISE

Reviews: Differences Between Awards –Declines by Directorate

Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline Award Decline

-8%

-6%

-4%

-2%

0%

2%

4%

6%

BIO CISE EHR ENG GEO MPS SBE

Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_Infrastructure

Greater Emphasis In Awards

Greater Emphasis In Declines

Reviews: Differences Between Awards –Declines by Directorate

-8.0000%

-6.0000%

-4.0000%

-2.0000%

0.0000%

2.0000%

4.0000%

6.0000%

8.0000%


Greater Emphasis In Review

Greater Emphasis In Summary

Awards: Differences Between Reviews –Project Summaries


-10%

-8%

-6%

-4%

-2%

0%

2%

4%

6%

8%

Advance_Discovery Benefit_To_Society Broaden_Participation Dissemination Enhance_Infrastructure

Greater Emphasis In Review

Greater Emphasis in Project Summaries


Declines: Differences Between Reviews –Project Summaries

-8%

-6%

-4%

-2%

0%

2%

4%



Greater Discussion In Decline Reviews Compared to Proposals

Greater Discussion In Award Reviews Compared to Proposals

Differences Between Review Focus Difference & Proposal Focus Difference Awards – Declines

• Differences across investment categories, directorates, and BI evolution; some similarities, too

• Across the investment categories, people are important BI targets• For PIs, the focus is often Students and BP• For Reviewers, the focus is often BP and Workforce Development

• But Research Infrastructure and Dissemination are also important• PIs pay more attention to research infrastructure, while the Reviewers care

about Dissemination more than PIs

• PIs seem to use more of the available categories than the reviewers

Take Away Messages

• Across the Directorates, again people are the main targets• When looking at differences between PIs and Reviewers:

• PIs are focused on students, benefits to society and research infrastructure

• Reviewers are looking at dissemination more than the PIs• For the awards,

• Greater emphasis by reviewers on BP and Dissemination than warranted in the proposals

• For the declines,• Greater emphasis by reviewers on the students and research

infrastructure than in the proposals• Ration of ratios differences

• Award reviews compared to proposals• BP and dissemination are the emphasis areas

• Decline reviews compared to proposals• Students are the main emphasis area

Take Away Messages (2)

• Categories: Even though all categories are used by someone, some are used more than others

• Some categories may not be well understood or are rarely possible• Some categories may be seen as more influential than others e.g., in

getting an award, or in tie-breaking in a panel

• Primary BI focus: Young people• Least BI focus: Benefits to society

• Perhaps, these are too far-fetched for some or the opposite, taken for granted for others

• Except for SBE and GEO

• PIs: seem to use all BI areas, but not evenly• Most common use is what is close at hand, most tangible

• Reviewers: uneven use of the categories, too• Most common use is focused on what might be the most influential

• How important is BI in NSF merit review?• Not a 50-50 proposition, like two independent criteria• Instead they need to be aligned with the capacity of the research team

• Can they achieve what they say they will achieve?

What Does This Mean?

• How can we increase understanding of what BI is/could be?• Continue analysis and presentations on BI• Include all stakeholders, i.e., PI, reviewers, and PD• Decide what actions to take if we don’t like what we see in the analysis?• Include some BI survey questions in bi-annual merit review survey, including from the

different points of view of PI and Reviewer• How can we continue to increase clarity about merit review?

• Develop guidance for PIs, Reviewers, NSF staff, and COVs• Expect heterogeneity across research communities and types of programs• Our understanding of BI will evolve• Make it clear that PIs and Reviewers sometimes diverge on what is most

important• Increase the opportunities for evidence-based approaches as well as novelty• Assess impact of merit review orientation pilot

• What is the role of the institution and collective impact models?• Work with the NABI on institutional and collective BI • IM: Publications :: BI: ??

Open Questions for BI

Thank You!

[email protected]

+Awarded proposals, panel summaries, and post-award reports were assigned to 1 of 5 categories

Five groups of NSF documents related to NSF FY 2013 awards:

• Project summaries (N=8,042)• Project descriptions (N=8,823)• Panel summaries (N=1,264)• Final Project reports (N=1,586)• Public outcomes reports

(N=1,812)

Documents analyzed• Analyzed BI content of all

documents• Each document was

assigned to one of five categories

Methods

+Why different N’s? Encoding issues; BI section cannot be “found” in project summary; panels not required for all proposals; not all awards expired yet.

Data Analytics

1.39% 2.01% 0.64% 2.32% 1.82% 1.04% 7.37%

28.06%

17.97%

34.12%

20.23% 18.49%27.34%

27.76%

2.89%

2.28%

3.86%

2.63%2.59%

2.75%

1.12%0.67%

1.59%

1.22%

3.51%0.48%

0.73%

6.32%

28.60% 45.22%

24.05%

34.69%40.17%

30.90%

29.66%

3.07%

10.66%

7.45%13.60%

3.22% 4.12%

8.28%

0.22%

3.76%

3.58%

1.38%

0.15% 0.42%

3.51%

7.42%

2.58%

3.46%2.44%

6.73%2.76%

5.58%27.66%

13.94%

21.62% 19.20%26.35%

29.95%

10.39%

$404,439,723 $ 56 1,4 83,13 4 $534,735,189 $389,213,939 $550,681,144 $152,791,355

BIO(TOTAL:1070-COLLECTED 1062)

CSE(TOTAL:1247-COLLECTED 1231)

EHR(TOTAL 1208-COL L ECTED 931)

ENG(TOTAL:1455-COLLECTED 1435)

GEO(TOTAL:978-COLLECTED 958)

MP S(TOTAL:1972-COLLECTED 1966)

SBE(TOTAL:698-COLLECTED 693)

DIRECTORATES: PAPPG PROJECT SUMMARY Advance Well-Being Broaden Participation Develop Workforce Economic Competitiveness Enhance Infrastructure

Industry Participation National Security Public Literacy Stem Education

1.21%0.49% 2.33% 2.01% 0.90%

7.28%

29.26%23.24%

33.30%

22.00%19.60%

28.59%

24.42%

1.24%

0.88%

1.91%

1.14%1.68%

1.59%0.63%0.25%

0.76%

0.47%

4.47%

0.23%

0.41%2.78%

21.25%

42.88%27.20%

29.10%36.69%

25.30%

35.92%1.89%

10.04%

8.31% 15.43%

2.39%3.92%

4.66%

2.00%

1.77%1.19%

0.22%0.30%

1.28%

6.79%

1.95%

4.31% 1.53%

5.45%2.06%

6.73%38.22%

17.05%22.23% 22.83%

31.72%36.93%

16.30%

$404,439,723 $56 1 ,4 83,13 4 $534 ,735,189 $389 ,213,939 $550 ,681,144 $152 ,791,355

B IO(TOTAL:1070-COLLECTED 957)

CSE(TOTAL:1247-COLLECTED 1048)

EHR(TOTAL 1208-COL L ECTED 931)

ENG(TOTAL:1455-COLLECTED 1316)

GEO(TOTAL:978-COLLECTED 863)

MP S(TOTAL:1972-COLLECTED 1837)

SBE(TOTAL:698-COLLECTED 629)

DIRECTORATES: PAPPG PANEL REVIEWSAdvance Well-Being Broaden Participation Develop Workforce Economic Competitiveness Enhance Infrastructure

Industry Participation National Security Public Literacy Stem Education

NSF Broader Impacts: Thematic Analysis

April 2017

50Image courtesy MIT Museum

Scientific progress is one essential key to our security as a nation, to our better health, to more jobs, to a higher standard of living, and to our cultural progress..

Little Difference between Proposed and Reported Activities

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Description Final ReportAdvance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure

1,586 1,586

2.11% 1.17% 1.56%

22.59%

41.20%

22.28%

1.58%

1.11%

1.83%1.83%

0.35%29.93%

27.59%

46.89%

7.83%

2.60% 3.83%

0.76%

0.59%0.95%

3.53%

3.29%2.71%

29.85%22.11% 19.73%

$2,234,929,965 $161885448 $278,730,801

DISCOVERY(TOTAL:6124-COLLECTED6053)

LEARNING(TOTAL:845-COLLECTED844)

RESEARCH INFRASTRUCTURE(TOTAL:607-

COLLECTED602)

PANEL REVIEWS PAPPG FY 2014Advance Well-Being Broaden ParticipationDevelop Workforce Economic Competitiveness

Differences across Project Size

Differences between PI and Reviewer Perspectives

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Description PanelAdvance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure

8,823 1,264

Significant differences between what PIs propose & panelists perceive.

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Description Panel

Advance Discovery Benefit To Society Broadening Participation Dissemination Enhance Infrastructure

1,264 1,264

-0.4

%

-0.8

% -0.2

%

0.0% 0.2%

-0.1

%

-0.1

%

1.2%

5.3%

-0.8

%

1.8%

1.1% 1.3%

-3.3

%

-1.7

%

-1.4

%

-2.0

% -1.5

% -0.9

%

-1.2

% -0.5

%

-0.4

%

-0.8

%

-0.8

%

1.0%

-0.3

%

-0.3

%

-3.5

%

-7.4

%

-2.3

%

3.2%

-5.6

%

-3.5

%

-5.6

%

6.3%

-1.2

% -0.6

%

0.9%

1.8%

-0.8

% -0.2

%

-3.6

%

-0.1

%

-1.8

%

-1.8

%

-0.2

%

0.1%

-0.1

%

-2.2

%

-0.6

%

-0.6

%

0.9%

-0.9

%

-1.3

% -0.7

%

1.2%

10.6

%

3.1%

0.6%

3.6%

5.4%

7.0%

5.9%

-10.00%

-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

Difference Between Reviews and Project Summary FY 2014 PAPPGAdvance Well-Being Broaden Participation Develop Workforce

Economic Competitiveness Enhance Infrastructure Industry Participation

National Security Public Literacy Stem Education



BIO(Total: 1070-

Collected 957)

CSE (Total: 1247 –

Collected 1048)

EHRTotal 1208 –

Collected 931)

ENG(Total: 1455 –

Collected 1316)

GEOTotal: 978 –

Collected 863)

MPS(Total: 1972 –

Collected 1837)

SBE(Total: 698 –

Collected 629)

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%



DevelopingWorkforce




Difference Between Reviews and Project Summaries FY 2014Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)

AdvancingHealth And

Welfare

Broaden Participation

Developing Workforce

Economic Competitiveness

Industry Participation

National Security

Public Literacy

Alternate 2 of Previous: Added additional label text above bars

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%



DevelopingWorkforce




Difference Between Reviews and Project Summaries FY 2014Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)

AdvancingHealth And

Welfare

Broaden Participation

Developing Workforce

Economic Competitiveness

Industry Participation

National Security

Public Literacy

Alternate 3 of Previous: Added

label and emphasis text



-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

Advance Well-Being


DevelopWorkforce


EnhanceInfrastructure


NationalSecurity

Public Literacy Stem Education

Difference In Focus Between Reviews and Proposal Summaries FY 2014

Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)Research Infrastructure(Total:607-Collected552)

-0.4

%

-0.8

% -0.2

%

0.0% 0.2%

-0.1

%

-0.1

%

1.2%

5.3%

-0.8

%

1.8%

1.1% 1.3%

-3.3

%

-1.7

%

-1.4

%

-2.0

%

-1.5

% -0.9

%

-1.2

% -0.5

%

-0.4

%

-0.8

%

-0.8

%

1.0%

-0.3

%

-0.3

%

-3.5

%

-7.4

%

-2.3

%

3.2%

-5.6

%

-3.5

%

-5.6

%

6.3%

-1.2

% -0.6

%

0.9%

1.8%

-0.8

% -0.2

%

-3.6

%

-0.1

%

-1.8

%

-1.8

%

-0.2

%

0.1%

-0.1

%

-2.2

%

-0.6

%

-0.6

%

0.9%

-0.9

%

-1.3

% -0.7

%

1.2%

10.6

%

3.1%

0.6%

3.6%

5.4%

7.0%

5.9%

-10.00%

-8.00%

-6.00%

-4.00%

-2.00%

0.00%

2.00%

4.00%

6.00%

8.00%

10.00%

12.00%

Difference Between Reviews and Project Summary FY 2014 PAPPG

Advance Well-Being Broaden Participation Develop Workforce

Economic Competitiveness Enhance Infrastructure Industry Participation

National Security Public Literacy Stem Education

Greater Emphasisin Reviews


CSE EHR ENG GEO MPS SBEBIO

-8.0%

-6.0%

-4.0%

-2.0%

0.0%

2.0%

4.0%

6.0%

8.0%

Advance Well-Being


DevelopWorkforce


EnhanceInfrastructure


NationalSecurity

Public Literacy Stem Education

Difference In Focus Between Reviews and Proposal Summaries FY 2014

Discovery(Total:6124-Collected5448) Learning(Total:845-Collected821)

Research Infrastructure(Total:607-Collected552)

Greater Emphasis in Summary
