NSF’s Broader Impact Criterion

By Andrew Bernat, Kathleen Fisher, Susanne Hambrusch, and Jim Kurose

NSF proposals must address, and are evaluated according to, two fundamental criteria: Intellectual Merit and Broader Impact. Intellectual Merit is well understood (if frequently argued)—how well does the proposed research advance the field? Broader Impact, however, is not nearly as well understood and consequently often has played a more minor role in the review process. This might very well be changing. The purpose of this article is to provide context and information around recent discussions of Broader Impact, and to identify issues that the CISE academic research community may soon face.

NSF CISE’s 2009 divisional Committees of Visitors have noted the confusion and inconsistencies regarding Broader Impact, and have urged that the criterion be re-evaluated. Perhaps more importantly, Congress is also paying attention—the House-passed COMPETES Reauthorization Act of 2010 (http://democrats.science.house.gov/Media/file/Commdocs/HR5116.pdf) directs the NSF to apply the Broader Impact review criterion to achieve a number of societal goals, and to implement a plan for achieving this within six months of the Act becoming law. Congress has provided examples of what it considers Broader Impact:

1. Increased economic competitiveness of the United States.
2. Development of a globally competitive STEM workforce.
3. Increased participation of women and underrepresented minorities in STEM.
4. Increased partnerships between academia and industry.
5. Improved pre-K-12 STEM education and teacher development.
6. Improved undergraduate STEM education.
7. Increased public scientific literacy.
8. Increased national security.

Conspicuously missing from this list is a catchall bullet - “Whatever the PI wishes to consider as Broader Impact.” At the same time, the eight items cover a very broad range of possible activities. In particular, the list encompasses “broader impact” activities common in many NSF proposals - graduate student training, diversity programs, and undergraduate and graduate curriculum development, in addition to activities in which some researchers already participate.

CISE and NSF want to be out front on the issue of Broader Impact (BI). As part of the process of understanding the criterion and how BI should be evaluated, CISE funded a workshop to gather community input. The BIRDS (Broader Impacts for Research and Discovery) Summit was held in June 2010. CRA Board members Kathleen Fisher, Susanne Hambrusch and Jim Kurose and CRA Executive Director Andy Bernat were invited to attend. The report of the workshop organizers is/will be posted at .

At the workshop, it was noted that, beyond the prodding of Congress, there are many reasons to care about BI and to connect science to society in order to:

• Ensure better public understanding of science and engineering.
• Ensure better public appreciation of research, its purpose and impact.
• Inspire the young to enter science and engineering.

It was also noted that researchers supported by NSF have a long-standing, implicit compact with taxpayers to use public funds to pursue avenues that will, in the aggregate, ultimately benefit society. Broader Impact seeks to ensure that this commitment is met.

At the BIRDS workshop, participants discussed examples of broader impact activities lying within the context of the COMPETES language. These included:

• Develop educational materials for elementary, high-school and undergraduate students.
• Involve high-school and undergraduate students in research where appropriate.
• Create or participate in existing effective mentoring programs.
• Develop, maintain and operate a shared research infrastructure.
• Establish international, industrial or government collaborations.
• Form start-up companies.
• Present research results to non-scientific audiences from policy-makers to average citizens.
• Give presentations about the field to the public to foster life-long learning.
• Develop exhibits in partnership with museums.

These examples include not only the “traditional” activities, but also those focused on improving the infrastructure for research with multiplicative affects. As is clear from these examples, broader impact activities are typically more focused on direct impact than the “innovation” that is a hallmark of intellectual merit. Indeed, it is often preferable to leverage current validated efforts than to attempt new ones for the sake of BI innovation.

As CISE develops its understanding of broader impacts, based on community advice, there are many issues to raise and to address. For example:

• Does the measure of broader impact depend on the level of effort involved, on the impact, or on some other measure?
• How should reviewers evaluate broader impacts statements?
• How much should broader impacts “count” compared to intellectual merit?
• How should NSF monitor progress of broader impacts statements?
• How should PIs be held accountable for their current and past statements?
• How does one measure the success of a broader impacts activity?
• What is the cost of broader impacts and where should funding come from? Line item in budget?  Separate program?

The House version of America Competes has potential implications for CS departments and their institutions as well, as it “requires principal investigators applying for Foundation research grants to provide evidence of institutional support for the portion of the investigator’s proposal designed to satisfy the Broader Impacts Review Criterion, including evidence of relevant training, programs, and other institutional resources available to the investigator from either their home institution or organization or another institution or organization with relevant expertise.”  This suggests that departments, universities and professional organizations could look more closely at existing or future activities that can be strengthened by faculty involvement and which translate to broader impact activities. NSF CISE is interested in beginning a dialogue with departments planning to provide such opportunities for their faculty, particularly in the areas of education and broadening participation in computing. For more information, contact Jan Cuny (jcuny@nsf.gov) for more information.

Andrew Bernat is CRA’s Executive Director.
Kathleen Fisher (AT&T Labs Research), Susanne Hambrusch (Purdue University), and Jim Kurose (UMass) are members of CRA’s Board of Directors.