Research, Technology & Development Evaluation
Josh Schnell, PhD
Principal, Academic & Government Consulting
Clarivate, United States
Madeleine Wallace, PhD
Chief Executive Officer
Windrose Vision
Fairfax, Virginia, United States
Lena Leonchuk, PhD
Senior Innovation Policy Analyst
RTI International, United States
Jeffrey Alexander, PhD
Director, Innovation Policy
RTI International
Washington, District of Columbia, United States
Location: Grand Ballroom 10
Abstract Information: In the United States, major federal research funders, notably the National Institutes of Health (NIH) and the National Science Foundation (NSF), are launching and expanding programs that aim to increase the diversity of the scientific workforce. Recent studies suggest that investigators who identify as people at color face barriers in applying for funding and in the selection process for awards, creating in particular disadvantages for faculty members in the tenure and promotion process. This contributes to the lack of representation of minority populations in the academic workforce, where surveys indicate that over 70% of doctorate-holding faculty identify as White (NSF, 2019 Survey of Doctoral Recipients). Agencies must also consider the scope of the term “diversity” in assessing whether their programs are inclusive. Diversity can encompass individuals from under-represented groups in terms of geography (e.g., states that consistently receive lower research funding), institutional characteristics (e.g., Historically Black Colleges and Universities), and family background (e.g., first-generation college graduates). To evaluate these programs, funders need appropriate data on the demographics of applicants and investigators to establish baseline levels of participation and to track progress in raising the representation of minority researchers. Access to more accurate and comprehensive data on representation among awardees can also enable agencies to be more effective at outreach efforts that encourage submissions from under-represented groups. The panelists in this session are engaged in projects analyzing data on research grant applicants and awardees. They have encountered a number of challenges in collecting relevant data, as many funders collect demographic data on a voluntary basis only. Moreover, standardized demographic categories may distort the realities of the barriers that under-represented minorities face in R&D funding and the narratives that programs can use to explain their methods and objectives. For example, Asian-Americans are excluded in many cases from the category of “under-represented minorities” used by the NSF, but a recent study suggests that even this demographic group may face systematic discrimination in research funding (Chen et al., 2022). The panelists will give examples of the challenges in understanding the diversity of the scientific research workforce in demographic and other terms, specify gaps in the available data that prevent accurate analysis of disparities in participation and advancement, and potential solutions under development to fill those gaps. The panel will offer perspectives from evaluators, researchers, and research funders.
Relevance Statement: Research funders are prioritizing efforts to enhance diversity, equity and inclusion (DEI) in their research funding programs. The National Science Board of the NSF states that by increasing the representation of people of color in the research workforce, the United States will gain from the talents of the “missing millions” who could be scientists and engineers but are excluded from these professions due to structural and systematic discrimination. The learning agendas of federal research agencies include issues related to diversity and representation as key topics for evaluation plans. The panelists’ work on evaluating such programs have uncovered data gaps and other operational limitations that show how perceptions about the barriers to participation faced by people of color can be shaped by myths and misunderstandings rather than empirical evidence and analysis. This session is intended to advance the field beyond diagnosing and framing the discussions around DEI and research funding, moving the community to address how to evaluate DEI initiatives in operational settings. This panel will discuss the need for data-driven story-telling to enable more informed decision-making in program design, operation, and evaluation in this area. Several recent studies suggest a level of discrimination present in funding processes, including proposal development, peer review, and post-award evaluation, but the quality of data is difficult to determine. Filling missing demographic data retroactively is fraught with the potential for inferential errors, but advances in data linkage provide some new approaches with potential to resolve these gaps. At the same time, federal regulations regarding privacy and confidentiality may complicate or prevent efforts to link disparate datasets, creating new challenges. This panel will use this particular class of federal programs to explore fundamental issues in evaluation, such as the use of advanced analytics, the difficulties in validating evidence to understand systematic patterns in discrimination, and how data-driven storytelling can be a more effective tool in developing diagnostics and interventions.
Presenter: Josh Schnell, PhD – Clarivate
Presenter: Jeffrey Alexander, PhD – RTI International
Presenter: Lena Leonchuk, PhD – RTI International
Presenter: Madeleine F. Wallace, PhD – Windrose Vision