152 - Paradigm shifts created by a pandemic: Changes in focus and voice in evaluation

Abstract Information: The purpose of this poster is to discuss the power of adaptation in the face of adversity with an emphasis on using storytelling to evaluate projects. Due to closures of schools and community programs, the question became: how do we evaluate without robust quantitative data? Story telling through qualitative data gathering became something that we looked upon as doable. Two projects that used qualitative methodologies to produce evaluation reports will be shared: The first discussed project highlights reviewing retroactive data from Ohio’s 21st Century Community Learning Centers to inform the statewide evaluation of this federally funded program. The second discussed project examines qualitative observational and focus group data presented in story format to evaluate health and research literacy in a community for an NIH funded project. The questions to prompt the discussion will seek stories from the participants that highlight how methodologies have changed at their institutions. Outcomes of the discussion are focused on how evaluation processes have changed with particular emphasis on what changes are likely permanent. We also bring an emphasis to how stories have strengthened and improved the evaluation process. While stories are the emphasis of this poster, other ways of evaluating and reporting will be shared as well.

Relevance Statement: The COVID 19 pandemic with its sudden onset, life and death urgency, and lack of a foreseeable resolution ushered in an imminent threat to human well-being forcing changes to everyday life as we knew it, and to the usual project evaluation procedures. Even though the pandemic is now controlled with vaccines and better treatment measures, many ideas and processes of evaluation have changed. Some of these changes were temporary, but the literature documents that there is a difference in employer expectations of the workplace and how one chooses to do that work [Carroll & Conboy, 2020; Kodama, 2020; Sein, 2020]. This poster is relevant in that likely all AEA registrants were impacted by the pandemic in some aspect. Having the ability to share experiences and learn how practices may have adapted over the past couple of years will benefit attendees. Ultimately, this poster discussion will aim towards improving the professional practice of all attendees. This poster aligns with each of the AEA evaluator competencies as one of the outcomes of the session will be to help evaluators reflect on their own competencies within their individual works. For instance, we will address how throughout these evaluations we wrestled with maintaining solid qualitative methodology while prioritizing the unique context that programs were facing during this time. We had to develop innovative planning and reporting techniques to meet the needs of the projects as well as facilitating constructive interactions with clients. This poster also aligns with each of the AEA guiding principles. Specifically, as our systematic methodologies have adapted towards storytelling in the face of the pandemic, we’ve had to reconsider and reflect upon the limitations and strengths of these more qualitative approaches while using the data available to the evaluations. Both projects strived to balance the needs of the clients while ultimately providing an evaluation that served the common good for all stakeholders. Especially with limited data availability and quick shifts in methodologies, this commitment to equity was (and continues to be) of utmost importance. By discussing two very different topics (an educational evaluation and a community health evaluation), participants will be able to see how methodological adaptations may be applied to a variety of projects.