Achieving Broader Impacts in the National Science Foundation, Division of Environmental Biology

Since 1997, the National Science Foundation (NSF) has guided reviewers to use two criteria to evaluate proposals: intellectual merit (the potential to advance scientific knowledge) and broader impacts (the potential to benefit society). The introduction of the Broader Impacts Criterion (BIC) was met with mixed reception from scientists. Many felt that the BIC focused much needed attention and over- due recognition on scientists with a passion for engaging society; however, many judged the BIC a burden and a distraction from the science they were trained to conduct (Holbrook 2005, Chodos 2007, Alpert 2009, Mardis et al. 2012, Frodeman et al. 2013). Confusion and frustration over the implementation of the BIC has inspired many assessments of the BIC, which tend to fall into three categories: case studies, which demonstrate the benefits or shortcomings of the BIC; policy analyses of the debate surrounding the BIC or its impact; and quantitative assessments of pro- posed broader impact activities (BIA; see table 1).

This body of work has highlighted challenges in BIC implementation that include divergent political pressures on the NSF, a lack of institutional infrastructure, and variation among scientific cultures in the BIC’s perceived importance. Case studies and historical analyses provide excellent con- text and detail but may be limited in how their implications map onto policy and practices in individual divisions and directorates of the NSF. Quantitative assessments, on the
other hand, have generally been limited by NSF confidentiality rules to public abstracts, which do not always include BIAs, even if they are present in the corresponding proposal (Watts et al. 2013, but see NSB 2011). Confidentiality has also limited assessments of the reported outcomes of proposed BIAs.

Here, we use data sets of public abstracts, panel review summaries, declined proposals and project reports spanning 2000–2010 to evaluate the implementation of the BIC in the Division of Environmental Biology (DEB). By evaluating proposals from submission through the reporting stage, we are able to assess the BIC in the policy context that has influenced its implementation.