STEM Education and Training
Alicia Kiremire, PMP (she/her/hers)
Owner
FlowStream Management LLC, United States
.jpg "Benjamin Williamson, EdM (he/him/his) photo")
Benjamin Williamson, EdM (he/him/his)
Research and Evaluation Scientist
CYBER.ORG
Bossier City, Louisiana, United States
Location: Room 208
Abstract Information: In this session, participants will learn how purposively-sampled focus groups were used by CYBER.ORG’s evaluators to identify impacts of its work over three years. Funded by the US Cybersecurity & Infrastructure Security Agency (CISA), CYBER.ORG provides K-12 cybersecurity education programming for teachers and students across the nation.
Long-term impacts of a STEM education initiative are an important story to understand and to tell. Compared to numerical data alone, impact stories enriched by participants’ lived experiences can build greater empathy and human connection in the reader. In turn, these stories can nurture relationships among program participants, implementers, funding agencies, and other stakeholders. For instance, as participants share their stories and program implementers can respond in future work, participants feel more understood, their needs met, and the relationship strengthened. Similarly, relationships between program implementers and funding agencies can continue to deepen as funders encounter personal stories demonstrating the impacts of the meaningful work they are funding.
To uncover the story of program impacts, several evaluation methods can be considered. Longitudinal surveys are the most traditional method used to invite perspectives from program participants over a number of years. However, in STEM K-12 projects, especially those funded by time-bound grants, extended longitudinal studies may not be a feasible option due to time, cost, sampling, and contact restraints. Retrospective pre-post surveys are also an option; but if they are quantitative in nature, participant stories cannot fully be expressed and instead are limited to the indicators determined by the researcher.
In this interactive demonstration, a joint external and internal evaluation team will share why we chose purposively-sampled focus groups to explore impacts of CYBER.ORG’s teacher professional development and classroom curriculum over three years. We will share our processes of sampling and data collection, which centered on retrospective questioning around the experiences of consistent participants. We will discuss key themes from focus group results, including some unforeseen outcomes identified by participants themselves. Finally, we will discuss benefits, limitations, and potential biases of this method compared to others and will invite session participants to consider which evaluation methods could best tell the story of their own STEM education program’s long-term impacts.
Relevance Statement: This session is relevant to the conference theme, “The Power of Story,” because the presenters sought to uncover the story of a program’s long-term impacts on participants and their communities. Participant stories are expected to build empathy and connection in stakeholders and funders, an application of the research of Zak (1) and Manney (2). A qualitative data collection approach gave space and voice to the stories of participants’ lived experiences. The presenters did not claim generalizability of results. Instead, they worked to understand and tell the story of a specific sub-group of program participants.
The session is relevant to AEA and the evaluation field by aligning with AEA’s Guiding Principles of Systematic Inquiry and Respect for People as well as the AEA competencies of Methodology and Context. Our methodological approach was chosen collaboratively, with forethought around specific data needs, the context of CYBER.ORG programs and its participants’ data, and available resources for the evaluation. The retrospective nature of focus groups was similar to the retrospective surveying method that has been discussed by AEA 365 contributors (3). Qualified focus group participants were K-12 teachers who had attended at least one CYBER.ORG professional development event and had taught CYBER.ORG’s cybersecurity content consistently for the last two to three years. The evaluators welcomed the perspectives of focus group participants, two thirds of whom teach at Title I schools. And through the retrospective focus groups, participants played a large role in identifying the outcomes of interest (whereas in a longitudinal study or even retrospective pre-post survey, outcomes not evident to researchers may not be included in the study design).
Finally, the session is relevant to the STEM TIG because it describes how the presenters measured impacts of a STEM education program. Impact measurement is critical, especially for grant-funded projects in STEM education. When many grant periods (and therefore evaluation contracts) are only a year long and projects serve rapidly developing STEM fields, there may not be time for a multi-year longitudinal study. This requires additional options for measuring long-term impacts. Furthermore, the impacts identified by our focus groups are likely of interest to other STEM education projects: bolstering equity in Title I schools, empowering female teachers and students in STEM classrooms, developing teachers with education and/or industry backgrounds, and influencing student career paths. The tangible equity outcomes were especially encouraging to find in focus group participant stories and quotes.
References:
1. Zak, P. (2013). How stories change the brain. Greater Good Magazine (University of California Berkeley). Available: HERE
2. Manney, P. (2008). Empathy in the time of technology: How storytelling is the key to empathy. Journal of Evolution & Technology. Vol. 19 (1) pp. 51-61. Available: HERE
3. Engelman, S., & Campbell, B. (2013). The advantages of using retrospective surveys. AEA 365 Blog. Available: HERE