131 - Zoom Out Before Zooming in: Building Evaluation Capacity in a School District by Telling the Story of Implementation

Abstract Information: In the summer of 2022, our evaluation firm was contracted to assist a large school district which had received emergency relief funding to abate the severe learning losses, disengagement and absenteeism in their schools following the Covid-19 pandemic. Our research team was tasked with evaluating seven different initiatives across 38 schools, all of which were entering year two of a three year grant cycle. District administrators were understandably eager for us to tell them the story of impact by answering the ultimate evaluation question: What measurable effect are these initiatives having on students? We began by learning about each program, and soon found that all were experiencing growing pains commonly found in the early stages of implementation. As evaluators we knew that it was too early to look at outcomes for these interventions, and we saw an opportunity to facilitate a conversation in the district about the importance of implementation fidelity. This demonstration session will share our journey to shift the way that this school district approached their assessment of new interventions, encouraging them to “zoom out” and first tell the story of program implementation, before “zooming in” to examine the effect on students. We will share the four pillars we identified, inspired by the core capacities of implementation science, which we used as a framework for our evaluation reports. We will also discuss the immense importance of building trust and establishing collaborative relationships with district and program leaders, to allow for open and honest conversations. We believe this tale of building evaluation capacity has implications beyond education, for any field where new initiatives could easily be labeled unsuccessful and therefore dismissed, when in fact, they were never given the opportunity to fully operate as intended.

Relevance Statement: We believe that presenting and sharing the lessons learned from our evaluation capacity building work would be beneficial to a wide audience of evaluators. Our approach strongly aligns with the following AEA Evaluator Competencies: Interpersonal ‒ we carefully and intentionally cultivated an authentic partnership with district leaders based on open and honest communication, which established a solid base from which to encourage a shift in thinking and an openness to new concepts; and Context ‒ we sought input from a variety of stakeholders and continually shaped the evaluation in collaboration with the district, creating a simple yet thorough framework* for our reports, to ensure information was clear and actionable. We believe that our discussion of implementation science ‒ including information about drivers, accelerants, practice profiles, and fidelity assessments ‒ would be of great value to attendees who are unfamiliar with these tools and how they can be used to frame a formative evaluation. We also hope to ignite discussion about the intersection of evaluation and organizational capacity building, including where our responsibilities lie as evaluators. In our story, we felt compelled to think beyond the programs we are evaluating today, and to consider all the initiatives the district will launch, evaluate and make decisions about in the future. When we combine our work as evaluators with organizational capacity building, we can have a much wider and longer-lasting effect on high-quality program design and development within that organization, which will ultimately lead to increased impact for program participants.

*Our model adapts concepts emanating from the work of NIRN, the National Implementation Research Network; SISEP, the State Implementation and Scaling-up of Evidence-based Practices team; and ECTA, the Early Childhood Technical Assistance Center.