Toward the upcoming dates for research proposals:
Strongly substantive recommendations on the Broader Implications Section (and relationship to Intellectual Merits).
With good wishes ! -Mary
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>From the Grant Writers Application Workbook
Writing the Broader Impacts Section of Your Research Proposal
The stand-alone Broader Impacts (BI) section, which is a major cause of angst for many applicants, isn't really that hard to write - provided that you understand what the National Science Foundation is trying to accomplish through this part of the application. Put simply, NSF wants every dollar it invests in basic research and education to have social, as well as scientific, consequences. That kind of social awareness distinguishes NSF and has always impressed us.
It is also necessary that you understand that BI activities must evolve from what is proposed for research. In other words, they cannot be completely independent of the research. Rather, they have to grow naturally out of the proposed investigations, either directly or indirectly (see Merit Review Principle #2, below). That is why we recommend that you write the BI section after you have completed all of the other, research parts of the Project Description.
You should plan for this section to occupy one-to-two of the fifteen pages that are allowed for the Project Description of a standard grant application. While it is essential that meaningful Broader Impacts activities be proposed, they cannot be of such a scale that they would interfere with your ability to conduct the research that is proposed elsewhere in the Project Description.
NSF is serious about what is proposed here - to the extent that, if the BI section isn't comparable in strength and quality to the application's Intellectual Merit side, it is highly unlikely that your proposal will be funded, even if it were to receive a perfect score for Intellectual Merit.
MERIT REVIEW PRINCIPLES, REVIEW CONSIDERATONS AND DESIRED SOCIETAL OUTCOMES IN THE CONTEXT OF BROADER IMPACTS
Merit review principles, merit review considerations used in evaluation, and desired societal outcomes have been detailed in Chapter 6. However, they are sufficiently informative of what needs to be written here that they bear repeating - but now in the context of Broader Impacts. [In what follows, we have inserted illustrative modifying words in brackets.]
Merit Review Principles:
1. "All NSF projects [including Broader Impacts] should be of the highest quality and have the potential to advance, if not transform, the frontiers of knowledge."
This Principle tells you that your Broader Impacts activities must be distinguished and they have to have the potential to produce positive societal impact of a kind that is relevant to NSF. Offering activities that are either predictable, "canned", or that appear to be tacked on to what is obviously a pure research proposal could, and probably would, put your entire application at risk.
2. "NSF projects, in the aggregate, should contribute more broadly to achieving societal goals. These broader impacts may be accomplished through the research itself, through activities that are directly related to specific research projects, or through activities that are supported by, but are complementary to, the project. The [Broader Impacts] project activities may be based on previously established and/or innovative methods and approaches, but in either case must be well justified."
The second Principle mandates that, overall, the results obtained from your project must include more than the knowledge that will be derived from the research. There have to be societally important outcomes, as well. The current NSF mandate that your Broader Impacts activities must grow directly or indirectly from the proposed research comes from this Principle. In addition, Principle #2 decrees that what is proposed must be justified, i.e., the BI activities have to meet a need that is relevant to NSF.
3. "Meaningful assessment and evaluation of NSF-funded [Broader Impacts] projects should be based on appropriate metrics, keeping in mind the likely correlation between the effect of broader impacts and the resources provided to implement projects. If the size of the activity is limited, evaluation of that activity, in isolation, is not likely to be meaningful. Thus, assessing the effectiveness of these activities may best be done at a higher, more aggregated, level than the individual project."
The third and final Principle suggests that, even though NSF does not require evaluation of all BI activities (http://www.nsf.gov/bfa/dias/policy/merit_review/mrfaqs.jsp#8) you will likely be well served by offering an evaluation plan for your BI activities (see discussion of distinguishing approaches later in this Chapter). Do they work and, if not, how can they be improved? The third Principle also implies that you have to invest sufficient resources in this part of the application if you want it to succeed. With respect to success, when the National Science Board assessed that aspect of past BI activities, they were surprised to find that applicants often didn't follow through with what they had proposed. That finding is one of the things that led NSF to change, effective January 14, 2013, how Broader Impacts activities should be proposed and evaluated. It is also what has led to the following sentence in the Grant Proposal Guide: "With respect to the third principle, even if assessment of Broader Impacts outcomes for particular projects is done at an aggregated level, PIs are expected to be accountable for carrying out the activities described in the funded project." (emphasis ours). Even though applicants have always been accountable for what they would propose, the last phrase in that sentence puts everyone on notice that NSF will now be enforcing what they haven't enforced very well in the past. So, don't propose anything under Broader Impacts that you can't/won't be able to deliver.
Merit Review Considerations for Broader Impacts:
1. "What is the potential for the proposed [Broader Impacts] activity to benefit society or advance desired societal outcomes?"
This consideration doesn't need any additional elaboration, in our opinion.
2. "To what extent do the proposed [Broader Impacts] activities suggest and explore creative, original, or potentially transformative concepts?"
As stated earlier, under Principle #1, the BI activities that you propose can't be predictable, "canned", or appear to be tacked on. They need to be genuinely creative and original. That translates into your need to do the same kind of literature review for your Broader Impacts section that you would instinctively do for your research-related sections. Most applicants don't do such a review. If you do, it will prevent you from proposing things that have already been tried/done by others. In addition, it will almost surely help to spark in you new Broader Impacts ideas/approaches. If you are going to claim transformative potential for what you propose, that's terrific, as long as you can credibly defend that assertion, i.e., that implementation of your Broader Impacts idea(s) will literally revolutionize the area of your focus. To that point, NSF defines transformative research as that which "involves ideas, discoveries, or tools that radically change our understanding of an important existing scientific or engineering concept or educational practice or leads to the creation of a new paradigm or field of science, engineering, or education. Such research challenges current understanding or provides pathways to new frontiers." (http://www.nsf. gov/about/transformative_research/definition.jsp) Most applicants who ascribe transformative potential to their BI activities are caught out by reviewers, who consider their claim to be overreaching. Be careful not to make that mistake.
3. "Is the plan for carrying out the proposed [Broader Impacts] activities well-reasoned, well-organized, and based on a sound rationale? Does the plan incorporate a mechanism to assess success?"
Most of this consideration doesn't require elaboration. The second sentence should be regarded as another, unequivocal indicator that an evaluation plan in your BI section would likely be well received.
4. "How well qualified is the individual/team/organization to conduct the proposed [Broader Impacts] activities?"
Everything that you propose under Broader Impacts must be within your/your team's capabilities. If unpaid outside services are required to make the project feasible, be sure to accompany your application with a letter or letters of collaboration that commit expertise to the project that, otherwise, would be missing. To be credible, paid-for services need to be included in your Budget. For example, a fee-for-service consultant may be needed to ensure that evaluation of your Broader Impacts (and Intellectual Merit) activities is fully objective.
5. "Are there adequate resources available to the PI (either at the home organization or through collaborations) to carry out the proposed [Broader Impacts] activities?"
This consideration should be seen as a follow on from Merit Review Principle #3. Resources often are not thought about in the context of BI activities. They are just as important to this part of the application as they are to the parts that pertain to research. For example, many Broader Impacts projects require access to resources/subjects that are not under the control of the applicant. In such a circumstance, reviewers will want to see a letter or letters of collaboration that confirm your access.
Societal Outcomes Desired by NSF (from most to least specific):
1. Full STEM participation of women, persons with disabilities and underrepresented minorities;
2. Improved STEM education and educator development at any educational level;
3. Development of a diverse, globally competitive STEM workforce;
4. Enhanced [STEM] infrastructure for research and education;
5. Increased public scientific literacy and public engagement with [STEM] science/technology;
6. Increased partnerships between academia, industry and others;
7. Improved national security;
8. Increased economic competitiveness; and
9. Improved well being of individuals in society.
It isn't necessary, in our view, to go over each of the desired societal outcomes that is listed, above; the meaning of most of them is readily apparent. For your project to be credible, you should choose a specific subdivision of one of these desired-societal outcomes, especially if it is one of those that is general in nature. The examples that we will provide at the end of this Chapter will clarify what we mean by, "choose a specific subdivision".
APPROACHES THAT WILL HELP TO DISTINGUISH YOUR BROADER IMPACTS SECTION
All applicants include a description of their proposed Broader Impacts activities - and so should you. But if that's all you provide, not only will your BI section be deficient, it won't stand out from others. Here are things that you can include that will set your application's BI section apart.
Focus on a Problem that is Important to NSF
Ideally, you want what you propose for BI activities to help solve a problem that is relevant to NSF - i.e., one that the agency cares about. Relevant problem areas are reflected by the list of desired-societal outcomes listed, above. Note that the first three, and probably the fourth and fifth, of the categories listed above are STEM related. They are the easiest to target, in our opinion. Numbers six through nine are increasingly general in nature. As noted earlier, if you were to choose one of the latter as the basis for your BI activities, you would need to clarify what specific subdivision under the general area would be targeted, e.g., "sensors" under "Improve national security."
Include an Evaluation Plan
As noted earlier, in our opinion, an evaluation plan will help your project to stand out in a positive way. For that reason, we recommend that you include such a plan. Both formative (process) and summative (outcomes) evaluation should be proposed. If you have never created such a plan before, this is not the time to propose what you "think" is adequate. If you, like most applicants, have no formal training in how to write and implement an evaluation plan, read The 2010 User-Friendly Handbook for Project Evaluation (http://informalscience.org/images/research/ TheUserFriendlyGuide.pdf), which is published by NSF. You can easily read it in an hour or two. Doing so will ensure that you write a highly convincing evaluation plan. For example, one of the things that you will learn is that you need to engage an outside evaluator. Having an objective, disinterested person perform the evaluation is essential to the plan's credibility. Proposing you or someone else from your project as the evaluator is akin to having a fox guarding the chicken coop.
If you know during development of your proposal who the evaluator will be, we strongly recommend that you involve him/her in the creation of your evaluation plan.
Many institutions offer evaluation resources - e.g., a fee-for-service group of experienced evaluators who are available to help plan and implement evaluation strategies. If that is the case at your institution, include such an individual in your Budget as a consultant, but only if s/he will be paid from the grant. If the proposed evaluator is a member of your larger campus community and, therefore, would be unable (or unwilling) to accept payment, include him/her as a "human resource" in the Facilities, Equipment and Other Resources section (see Chapter 18 of this Workbook). Including such a person in the Budget or Budget Justification would constitute voluntary committed cost sharing, which is prohibited by NSF.
If the evaluator will be paid from the grant, a letter of collaboration from him/her should not be included with the proposal; the fact that paid effort appears in the Budget is sufficient. If the evaluator will not be paid from the grant, s/he should provide a letter of collaboration. It should be nothing more than that: agreement to serve as the evaluator (see Chapter 19 in this Workbook). A mistake that is commonly made is to include the evaluation plan in the letter of collaboration, which would likely be interpreted as an attempt to circumvent the page limitation on the Project Description.
Finally, because credible evaluation is a requirement for both the research and Broader Impacts parts of your application, we strongly recommend that you ask someone who is a skilled evaluator to serve as a member of your Pre-Submission Review Committee (see Chapter 20).
Include a Dissemination Plan
As noted above, BI activities cannot be of a scale that they interfere with completion of the proposed research. So, how do you propose a small BI project and still have a viable claim for positive societal impact? The answer to that question is a dissemination plan that amplifies the positive impact of your small-scale project. We strongly recommend that such a dissemination plan be presented as part of your Broader Impacts section.
For example, if your BI project requires implementation of activities at local high schools, you shouldn't propose to do so at all eight such institutions that are within a 30-mile driving radius of your university. Doing so would represent nearly a full-time commitment, which is counter to the concept of Broader Impacts. Rather, the appropriate approach, in our opinion, would be to propose your BI activities at two or three of the schools, in conjunction with a strong evaluation plan. You would then disseminate what works on that small scale. For example, you might recruit colleagues at other institutions to help you test your findings elsewhere. They would provide letters of collaboration that would accompany your proposal. Such an approach offers the potential for amplification of the results of your relatively small-scale project and increases, therefore, the likelihood that they will have positive impact.
