111 - A Mixed Methods Approach to Educational Evaluation: Lessons Learned From a STEM Program

Abstract Information: This paper presents a cumulative summary of an evaluation project of the INcubating Engineers for Food Innovation (iNEST) Program which is a collaborative project among several institutions of higher education including the University of Tennessee at Chattanooga, University of Tennessee at Knoxville, and Tennessee Technological University, between 2021 and 2023. Based on some unforeseen programmatic challenges, including those presented by the COVID-19 pandemic, the evaluation approaches had to be modified to enable the evaluators to tell a holistic story regarding different aspects of the program. In this presentation, lessons learned from the evaluation will be discussed, including how we used triangulation, to achieve the most from the process. In addition, information from this paper will inform evaluators about possible ways to alter evaluation designs through formative assessment.

Relevance Statement: The goal of the USDA funded INcubating Engineers for Food Innovation (iNEST) Program project is to create a reputable and sustainable food science educational training program to offer unique opportunities for undergraduate engineering students to participate in interdisciplinary instructional and research activities in food science and technology. The project kicked off during the COVID era, and the first year was challenging in terms of both implementation of instruction, research, and evaluation activities, especially given that all activities had been planned to be held in person. During the second year of the program (2022-2023), lessons learned during the first year were implemented, and the program ran more smoothly. This presentation will highlight different assessment approaches that were utilized in order to provide a holistic picture of the whole project, enabling the “story of the iNEST program to be told,” and for the program to be an example for future researchers and implementers of similar programs. Evaluation of the project utilized a mixed methods approach which included snapshot assessments (surveys) following Professional Development and Effective Learning workshops, interviews with course instructors and the PI team, focus groups, and document analysis. All assessments were conducted virtually. Key issues to be addressed during this presentation include: 1. One of the key reasons for the success of the evaluation and of the project overall emanated from the transparent communication between the evaluators and the PI team, with buy-in from all stakeholders to implement formative assessment feedback. During planning and reflection meetings with the faculty and staff, the evaluators provided actionable items for each relevant stakeholder category. 2. Secondly, the use of retrospective pretest surveys to assess (perceptions of) knowledge and skills gained is one of the key pieces of the evaluation approach, which has been very informative in terms of shedding light on the teaching and learning of food science and technology. 3. Methods and data triangulation have been very useful to enable the whole story of the project to be told from different angles, and by different stakeholders. 4. Follow-up with program participants 6 months after completing the program regarding how they are implementing the skills learned during the program. We believe sharing our experience evaluating this (still ongoing) program will help other evaluators reflect on their approaches to evaluation in order to be able to tell their stories more effectively.