Development of an Evaluation Model for the Great Lakes Bay Regional STEM Initiative

Abstract Information: We never know what sparks someone’s STEM story: A community STEM program, an interactive chemistry activity, or a lesson writing code for a video game. This paper discusses how one evaluation plan is a first step in documenting students’ stories and improving STEM programming. The Great Lakes Bay Regional STEM Initiative (GLBR STEM Initiative) is currently seeking quality data to provide evidence of program success, specifically for the “Access and Equity Mini-Grants Program” (AEMGP), which provides funding for out-of-school time (OST) STEM programs in underserved populations. The AEMGP has been successfully delivered to hundreds of participants involving dozens of teachers. However, measurement tools have not been established, thus critical program outcomes are generally unknown creating stress with funders who are now requesting evidence of positive program impacts. Therefore, this project resulted in an evaluation plan the AEMGP can implement in the future to obtain desired data. This task seemed straightforward; however, three unique circumstances complicated evaluation design efforts: 1) the program is two-tiered with the first phase focusing on training the teachers and the second phase focusing on implementation of the STEM program, 2) the AEMGPs occur within a variety of student-serving organizations, such as libraries and community programs, resulting in a wide range of teacher experience, program formats, and classroom settings, and 3) a disconnect exists between the information funders desire and information that can be reasonably gathered during the programs’ relatively short duration (6-10 hours). The final evaluation plan also needed flexibility for desired generalizability among multiple settings of the AEMGPs as well as adaptability to other programs sponsored by the GLBR STEM Initiative. Ultimately, evaluators developed a unique, innovative evaluation plan that integrates the needs of the local STEM community with the macro-level information sought by funders and the parent organization (STEM Learning Ecosystem Community of Practice). The designed evaluation plan focuses on an AEMGP that takes place within a local summer program that enrolls high-risk, low-achieving students in an effort to improve academic achievement and student behavior. This AEMGP runs eight hours and is for students in third through fifth grades. Because this evaluation’s focus is quite specific, two factors have been incorporated to mitigate challenges: 1) all program participants (teachers and students) need to be included in the sample, with an anticipated sample size of 20 teachers and 150 students and 2) the surveys need to be easily administered without taking up too much program time yet must meet the needs of the broader evaluation requirements. To fulfill these criteria, a one-group pretest-posttest design was selected for the administration of a brief PnP survey to collect response outcomes. This survey includes six questions for teachers and four questions for students targeting respondents’ feelings toward their STEM experience and addressing the needs of all stakeholders. The presentation will detail how challenges were overcome to design an evaluation plan that successfully collects information desired by funders without burdening directors and how it can be adapted to multiple settings and programs, thus continuing to spark future STEM stories.

Relevance Statement: The Great Lakes Bay Regional STEM Initiative (GLBR STEM) has many stories to tell: Stories of teachers gaining confidence in their STEM-teaching skills, of young students getting excited about science and math, and of former program participants voluntarily seeking opportunities to learn more about STEM. However, this organization struggles to provide depth to these stories due to lack of supporting data. This struggle stems from several factors, such as budgetary and time constraints, but even more so because of the disconnect between the information that funders require and the information that can be reasonably collected within the confines of program delivery. The aim of this project was to develop an evaluation plan that is simple in design but rich in data collection that could be implemented on a micro level while still satisfying the needs of the larger organization. The contribution of this evaluation plan is that it addresses a common issue of non-profit organizations: The need to conduct multiple evaluations that are responsive to the variety of programs offered and that can be integrated into an overall analysis of the evaluands. The evaluation plan is responsive to specific STEM programs as it is easily adaptable to multiple formats and lengths. For example, when a specific evaluation plan serves as a foundational schema or structure for future evaluations, the necessity of the GLBR STEM Initiative to use valuable resources to create a new and unique evaluation plan for each program is minimized, if not eliminated. The analysis of the data collected will guide shareholders to make informed decisions regarding funding and provide immediate feedback for program improvement. The results of these evaluations will culminate in richer, more in-depth stories of STEM engagement by both teachers and students.