Delayed Ignition: Investing more evaluation time and resources to better convey the story for undeserved participants.

Abstract Information: More inclusive programs and projects require a more inclusive evaluation. Establishing the processes and strategies to make an evaluation more inclusive for diverse parties is not always straightforward for those historically undervalued and underserved. In this Ignite talk, evaluators will describe a project’s approach to devote more time and resources to develop, conduct, and report on a more inclusive evaluation. The strategies include multiple approaches and methods to check on instrument design, data collection, analysis, and reporting to interested parties. The examples will come from the evaluation of the NASA Neurodiversity Network (N3) program where developers and evaluators have made a concerted effort to acknowledge and address asymmetric power relations that arise from various intersecting identities that stakeholders bring to the project, and ways in which the project acknowledged the common experiences stakeholders bring while also honoring the individual experiences and stories. The overarching goal of the N3 program is to make science learning opportunities more engaging and accessible to neurodiverse students. With this in mind, researchers have engaged neurodiverse youth in testing and reviewing curriculum content such as building and launching rockets, developed distanced internship experiences that pair autistic youth with professional scientists, and engaged with intern alums to provide feedback on the program as well as the data collection instruments themselves. A significant thread throughout these multiple touchpoints has been the importance of building community, positive representation, and creating a safe space for youth to learn, grow, and take ownership of their stories. This session is intended for evaluators interested in seeing practical examples of evaluators’ journey towards more equitable evaluation practices when working with historically undervalued groups.

Relevance Statement: This proposal contributes to the knowledge base on promising inclusive program evaluation design and utilization practices. This Ignite talk aims to engage evaluators to consider the role of language and communication with intended audiences and program developers through the different evaluation phases. In addition, we reflect on existing practices of ableism in research and evaluation that harm intended learners with disabilities (Bottema-Beutel et. al., 2021). This work necessitates for evaluators to self-reflect on their practices and use of language and subsequently encourage program stakeholders to take steps to use language that is non-albeist to minimize harm and maximize the benefits of programs working on broadening STEM participation. The Ignite talk also tackles the challenges that growing developmental programs face regarding small sample sizes and the tension this puts on evaluators to accurately and transparently represent evaluation findings while safeguarding participants' identities in reporting to support program development effectively. Reference: Bottema-Beutel, K., Kapp, S. K., Lester, J. N., Sasson, N. J., & Hand, B. N. (2021). Avoiding ableist language: Suggestions for autism researchers. Autism in Adulthood.