Science for the Masses

8/3/2019 Science for the Masses

1/4

Research-funding agencies are forevertrying to balance two opposing forces:

scientists desire to be left alone to dotheir research, and societys demand to

see a return on its investment.The European Commission, for example, has

tried to strike that balance over the past decadeby considering social effects when reviewingproposals under its various Framework pro-grammes for research. And the Higher Educa-tion Funding Council for England announcedlast year that, starting in 2013, research will beassessed partly on its demonstrable benefits tothe economy, society or culture.

But no agency has gone as far as the USNational Science Foundation (NSF), which will

not even consider a proposal unless it explicitlyincludes activities to demonstrate the projectsbroader impacts on science or society at large.The criterion was established to get scientistsout of their ivory towers and connect them tosociety, explains Arden Bement, director ofthe NSF in Arlington, Virginia.

Unfortunately, good intentions are notenough to guarantee success, says DiandraLeslie-Pelecky, a physicist at the University ofTexas in Dallas who is active in popular sciencewriting and other forms of outreach.

Leslie-Pelecky remembers a pilot projectshe carried out in 2001, when she was at the

University of Nebraska in Lincoln. In manyways, it was typical of the kinds of things that

NSF-funded researchers do to fulfil theirbroader-impacts requirement. She took three

female graduate students on weekly visits tolocal classrooms, where they spent 45-min-utes leading nine- and ten-year-old childrenin practical activities designed to teach themabout electricity and circuits. The visitorsalso talked about their lab work and careers.In addition, Leslie-Peleckydid something less typicalof broader-impacts efforts:she brought along educationresearchers to study the effectof this interaction on the chil-drens perception of scientists.

Those assessments were

startling, she says. After threemonths, most of the studentssaid that they still werent surewho these young teachers were except thatthey couldnt possibly be scientists. In theirminds, scientists were unfriendly, grey-hairedold men in white lab coats1.

And thats what I worry about with broaderimpacts, says Leslie-Pelecky. There are a lot ofpeople putting time and effort into [these sortsof activities] and they have no idea if theyremaking any difference or not.

Many NSF-funded researchers find thefoundations definition of broader impacts

to be, perhaps unsurprisingly, broad, andfrustratingly vague. Among the examples of

activities listed in the foundations proposalguide are: developing educational materi-

als for elementary, high-school and under-graduate students; involving these studentsin the research where appropriate; creatingmentoring programmes; maintaining andoperating shared research infrastructure;presenting research results to non-scientific

audiences such as policy-mak-ers; establishing international,industrial or governmentcollaborations; developingexhibits in partnership withmuseums; forming start-upcompanies; and giving pres-entations to the public.

Because it lacks conceptualclarity, the broader-impactsrequirement often leaves

researchers unsure about what to includein their proposals, and leads to inconsisten-cies in how reviewers evaluate applications.Broader impacts were designed to be open,but openness confuses a lot of people, saysLuis Echegoyen, the division director for NSFchemistry.

To make matters worse, the NSF has madelittle attempt to systematically track how itsbroader-impacts requirements are beingmet, or how much grant money is being

spent in the process. Nor does it have a sys-tem in place to evaluate the effectiveness of

Science for the massesThe US National Science Foundations insistence that every research project addresses broader

impacts leaves many researchers baffled. Corie Lok takes a looks at the system.

The criterion was

established to get

scientists out of

their ivory towers

and connect them

to society.

416

Vol 465|27 May 2010

416

NATURE|Vol 465|27 May 2010NEWS FEATURE

20 M ill P bli h Li it d All i ht d10


2/4

the various projects.These problems with the broader-impacts

requirement have been confirmed over the

past decade in studies from the NationalAcademy of Public Administration and else-where. In March, the NSFs oversight body,the National Science Board, launched a taskforce to examine how broader impacts canbe improved. Chaired by Alan Leshner, chiefexecutive of the American Association forthe Advancement of Science in WashingtonDC, the task force is not expected to make itsrecommendations until 2011. In the mean-time, a small number of academic institutionsare already exploring ways to make broader-impacts efforts work better.

After all, says Ralph Nuzzo, a chemist and

materials scientist at the University of Illinoisin UrbanaChampaign, most US scientistshave come to accept even if grudgingly that it is probably a good idea to demonstratethe wider implications of their work. Peoplewant to do the right thing, says Nuzzo. Its justhard to know what that is.

Scientists get creativeNonetheless, there have been some successes.At Rensselaer Polytechnic Institute in Troy,New York, for example, the NSFs NanoscaleScience and Engineering Center for DirectedAssembly of Nanostructures sponsors the

Molecularium project, which has producedteachers materials on nanoscience and ananimated three-dimensional IMAX filmcalledMolecules to the Max. At the Universityof WisconsinMadison, microbial biochemistDouglas Weibel and his group have prepared achild-friendly, interactive display about micro-scopy that they exhibit every yearat the universitys one-day publicscience exposition. At StanfordUniversity in California, chemi-cal engineer Andrew Spakowitzspends two to three hours aweek working with graduate and

undergraduate students to pro-vide workshops for patients atStanfords Lucile Packard Chil-drens Hospital, most of whomare unable to attend school.Spakowitz and his group cre-ate the workshops that covertopics such as pH and gravity,and lead the hands-on activi-ties at the hospital.

Some say that the broader-impacts criterion has helpedto catalyse a change in theresearch-focused culture of

academic science. It makesscientists think more explicitly

about how their work is connected to enhanc-ing benefits to society, says Robert Mathieu,chair of the astronomy department at the Uni-

versity of WisconsinMadison and directorof the universitys NSF-funded Center for theIntegration of Research, Teaching, and Learn-ing (CIRTL). As an example, Mathieu points tothe NSFs prestigious CAREER award for juniorfaculty members, which requires that applicantspropose educational activities, such as design-ing courses and carrying out public-outreachactivities, that are integrated with the proposedresearch. He has sat on severalreview panels, and says that theeducation section of proposalshas grown in length and sophis-tication over the years.

But despite the NSFs effortsto educate scientists aboutbroader impacts throughwebsites, workshops andconference sessions, most stillapproach the criterion with confusion anddread. Researchers often end up repackagingwhat theyre already doing. Overwhelmingly,says Echegoyen, the number one broaderimpact that most people in the chemistrydivision are using is training graduate studentsand postdocs.

One problem is that the kind of supportnetwork that researchers take for granted

working with collaborators, sharing ideas andadvice, learning from published results, attend-ing conferences is still rudimentary whenit comes to broader impacts. A useful modelcould be the network of technology-transferoffices that are found on many US campuses,which have been instrumental in helping

researchers to maximize the commercial effecof their research.

A preliminary network for broader impact

already exists. Stanford, for example, has anOffice of Science Outreach, which helpedSpakowitz to make the initial contactto get his project started at the hospitalAnd Mathieus centre at the University oWisconsinMadison is part of a networkof six CIRTLs located at research campusesuch as Vanderbilt University in NashvilleTennessee, and Texas A&M University in

College Station. The Wisconsin centre runs workshopand conducts individuaconsultations with facultymembers needing assistanc

with integrating broaderimpacts activities into theigrant proposals. The otheCIRTLs are moving towardsimilar sorts of programmes

Mathieu and his group are putting togetheplans to expand this network to 2025 universities. Their ultimate goal is for any USresearch university that wants its own CIRTLto have one, creating a community that sharebest practices among its researchers andother professionals, and develops the expertise to effectively broaden impacts. Mathieuestimates that establishing CIRTLs at th

nations top research universities would cosroughly US$100 million over five years.

Yet such ideas lead to a more fundamentaquestion. Is having every principal investigatoworking individually on broader impacts fowhich many are inexperienced and untrained the most efficient way of achieving the max

imum effect?Some scholars say no

In a paper published lasyear, Warren Burggren, biologist and dean of thCollege of Arts and Sciences at the University

of North Texas in Denton, writes that the job oimplementing broadeimpacts should fall to thresearchers institutionnot to the researcher himor herself2. The institution, be it college, department or centre, would pooa portion of the NSF grantobtained by its memberand hire the professionalneeded to broaden impacteffectively. Scientists should

still be involved, but thecoordination would happenStanford chemical engineer Andrew Spakowitz (right) teaches children at the Lucile

Packard Childrens Hospital as part of his NSF award.

It makes scientists

think more explicitly

about how their

work is connected toenhancing benefits

to society.

41

Vol 465|27 May 2010

41

NATURE|Vol 465|27 May 2010 NEWS FEATURE



3/4

at the institutional level. Ithink it will be more effi-cient, because youve got

people doing what theyretrained for, says Burggren.

Another idea, suggested byBarry Bozeman, a science-policy expert at the Univer-sity of Georgia in Athens, isfor the NSF to create specificresearch programmes withstrong broader-impact goalsaround areas in which theeffects are important and obvi-ous, such as climate change3.Bozeman says that the NSF isalready following this strategy

with awards that, for example,promote the recruitment andretention of women in academicscience.

The NSFs broader-impacts requirementtook its current form in 1997, when the founda-tion simplified the criteria used by reviewers toevaluate proposals. Two of the four criteria the intrinsic scientific merit of the project, andthe soundness of the teams approach weremerged into one, known as intellectual merit.And the other two the utility or relevanceof the project, and its effect on the infrastruc-ture of science and engineering

were collapsed into broaderimpacts.

For the first few years, manyproposers and reviewers ignoredthis second criterion, treating itwith the same disregard thatthey had previously expressedtowards utility or relevance. Itwas only in 2002 that the NSFcracked down, announcing that any proposalthat didnt separately address both the intel-lectual-merit and the broader-impacts criteriawould be returned without review.

The right trackOf the few small-scale efforts to track andassess the broader-impacts requirement, nonehas been conclusive. In 2008, for example,the NSF sent Congress a report on broaderimpacts, as mandated in the 2007 AmericaCOMPETES Act but included little morethan anecdotal descriptions of specificresearch projects. The chemistry divisionrecently contracted with an outside com-pany to assess the broader-impacts activitiesof a sample of its grantees, but the projectfell into limbo when the company dissolved.And the geosciences directorate carried out

informal surveys of broader-impacts activi-ties in the ocean and Earth sciences, whichThe film Molecules to the Maxwas created by an

NSF-funded nanoscience centre.

Outreach activities such as a board game (left) about the link between nanotechnology and society, and biochemistDouglas Weibel (right) presenting a hands-on microscope display to children, can help bring science to the public.

yielded some results, but also showed thatthe research and the broader-impacts workwere often so interwoven that it was difficultto tease them apart.

Mostly, evaluation happens as a by-productof other NSF activities routine reviews ofgrantees annual reports, for example, or theregular review of programmes at each divi-sion carried out by a committee of externalscientists.

By not tracking broader-

impacts activities, the NSFundervalues its true contributionto society, says Melanie Rob-erts, an assistant director at theColorado Initiative in MolecularBiotechnology at the Universityof Colorado in Boulder who hasanalysed the broader-impactsstatements from recent grant

abstracts. It is missing an opportunity to createa knowledge base of how to carry out broader-impacts activities effectively and reward thosewho do a good job.

The confusion that persists despite the NSFs

repeated attempts to clarify broader impacts

perhaps reflects more fundamental issuesabout the relationship between science andsociety, says Britt Holbrook, a philosopher ofscience at the University of North Texas. Is theNSF passing the buck by asking scientists tomeet what is essentially a political goal: dem-onstrating the benefits of science?

My hypothesis is that the NSF has passedsome of that burden to the people gettingfunded, says Holbrook, who has a g rantfrom the foundation to study how different

funding agencies incorporate societal impactsinto their review process. But when you dothat, you get push back from the scientificand engineering community because it goesagainst the traditional idea of peer review,which is designed to assess work at a techni-cal or scientific level.

And how does the NSF show impact, giventhat the agencys specialty, basic researchoften doesnt have an immediate pay-off, orelse has a pay-off that is difficult to quantify?Its a delicate balancing act, says Neal Lane, aphysicist at Rice University in Houston, Texas,who was the NSF director from 1993 to 1998

when the broader-impacts criterion was firstimplemented. Its important to get scientists tothink about how their work affects society, hesays. But one has to be careful not to push it toofar. If the NSF moves too far in the direction ofdoing things that have short-term benefits, thenI think that is not consistent with the NSFs mis-sion, and that would not be good for Americanscience, engineering and technology. Corie Lok is an editor for Nature in Boston,

Massachusetts.

1. Buk, g. a. et al.J. Elem. Sci. Educ.14, 110 (2002).2. Burren, W. W. Soc. Epistemol.23, 221 (2009).3. Bozemn, B. & Bordmn, c. Soc. Epistemol.23, 183

(2009).

See Editorial, page 398.

By not trackingbroader-impacts

activities, the NSF

undervalues its

true contribution

to society.

418

Vol 465|27 May 2010

418

NATURE|Vol 465|27 May 2010NEWS FEATURE



4/4

Copyright of Nature is the property of Nature Publishing Group and its content may not be copied or emailed to

multiple sites or posted to a listserv without the copyright holder's express written permission. However, users

may print, download, or email articles for individual use.

Documents

Science for the Masses