20 - Evaluators as crafters of stories: examining an interactive, multi-method, and self-reflective process of telling stories of after school programs in NYC

Abstract Information: This proposal aims to describe an interactive, multi-method, self-reflective process involving the stories evaluators share when presenting data on multi-year, after-school programs targeting underserved communities in New York. This in-depth analysis will be based on the experiences and interpretive practices gathered from 1:1 guided conversations and a focus group with a team of nine evaluators of diverse cultural backgrounds and areas of expertise. The team works under L&G Research and Evaluation Consulting, a women- and minority-owned (MWBE) evaluation consulting company based in NYC, partnering with more than fifty community-based organizations (CBO) and public agencies. The reflections will focus on the evaluators’ approach to writing a mid-year implementation progress report for 21st CCLC after-school programs, addressing the triangulation of sources, creativeness, and positionality in crafting a program narrative. We will reflect on the tensions between developing narratives that accurately reflect program fidelity to implementation while integrating the different perspectives of stakeholders regarding their own lived experiences with implementing programs and recognizing the power evaluators hold in crafting the final narrative. The narratives evaluators incorporate into reports attempt to account for compliance requirements and the different perspectives of key stakeholders and bring attention to the local context in which programs are situated. Our reports are also informed by participatory evaluation practices. They include multiple sources of information like operational program data, stakeholders’ accounts, advisory meeting notes, program quality self-assessment meetings, and observations of program activities. Integrating practitioners’ knowledge of the program with operational data and stakeholders' voices is critical when telling the story behind the data and increasing data utilization. Thus, this self-reflection process will shed light on the mechanics behind putting together a narrative encompassing evaluation protocols, compliance analysis, and the local context. While also acknowledging the lenses the evaluator carries based on their background and identities - race/ethnicity, gender, age- and how those factors shape how a narrative is formed.

Relevance Statement: This proposal aims to share and discuss the process used when crafting the story behind data used in evaluating 21st CCLC after-school programs in New York. At L&G Research and Evaluation Consulting, we developed tools and procedures driven by the utilization-focused evaluation (Patton, 2008) and the AEA Guiding Principles frameworks. This participation in the AEA 2023 conference will be an opportunity to share insights and engage in a field-level discussion about the participatory evaluation of programs (Guijt, 2014), triangulation of sources, and crafting accounts to share data in ways that maximize utilization of data. It is an opportunity to examine the quality of our knowledge-building process, communication strategies, and interactions with stakeholders. This roundtable will also contribute to discussing the significance of the four evaluator competencies: pragmatism, intentional, inclusive, and dynamic (Chicago 2018) in our work. When discussing aspects of the evaluator’s positionality and own voice in the stories we tell, we will harness how the AEA guiding principles for evaluators -systematic inquiry, competence, integrity, respect for people, and common good and equity- are at play in the complex process of crafting a story about the data used in evaluating education programs. References AEA Guiding Principles for Evaluators - https://www.eval.org/Portals/0/Docs/AEA_289398-18_GuidingPrinciples_Brochure_2.pdf Guijt, I. (2014). Participatory Approaches. Methodological Briefs: Impact Evaluation 5. UNICEF Office of Research, Florence. Retrieved from: https://www.betterevaluation.org/sites/default/files/Participatory_Approaches_ENG.pdf AEA The 2018 AEA Evaluator Competencies - https://www.eval.org/Portals/0/Docs/AEA%20Evaluator%20Competencies.pdf Patton, M. Q. (2008). Utilization-focused evaluation (4th ed.). Thousand Oaks, CA: Sage Publications.