144 - Evaluating State Education Policies: Current Challenges and Lessons Learned Using Examples from Early Literacy Evaluations and Education and Career Planning Research

Abstract Information: State education agencies (SEAs) are tasked with ensuring that every child in the state has access to a high quality education. They have talent and resources at their disposal, and often have legislative mandates providing direction to their efforts. But SEAs face unique and challenging obstacles to enacting effective education policies and programs. Many states constitutionally provide a high degree of local control to districts, meaning that SEAs are limited in their ability to implement policy statewide. Evaluations of SEA programs, initiatives, and even enactments of state legislation must navigate related challenges, for instance using heterogeneous outcome data, accounting for staggered implementation timelines and approaches, and assessing different interpretations of requirements. Given the nuance of local contexts and the sweeping nature of state education policy, the evaluation goal is to tell two stories: 1) what is the experience of real individuals under this policy, with all of their unique experience and complexity, and 2) how can the overall impact of the policy be described? This poster will present some common challenges faced by evaluators undertaking state-level evaluations. Using examples from current SEA partnerships in Colorado, Alaska, Massachusetts, and Arizona, our team will share important lessons learned, work-arounds, and approaches that can support effective and inclusive state-wide evaluation. We provide examples from both the K-5 and 9-12 grade spans, discussing evaluations of early literacy policy and college and career readiness programs.

Relevance Statement: Evaluations of state-wide education policies and legislation face unique challenges, given the local control so often protected for districts, the heterogeneity of outcome data, and the vast differences in implementation and policy coherence. This poster shares examples from ongoing partnerships with three SEAs to illuminate the context behind these challenges, and to share important lessons learned, work-arounds, and evaluation approaches that can support strong state-level evaluations. Viewers may consider the importance of centering equity in evaluation, given the need to summarize findings across diverse stakeholder groups and contexts. Our team will be present to discuss the balancing act of being inclusive in telling the story of how a policy or program is enacted, while also providing actionable summaries and findings for SEAs. Viewers can expect to walk away with both usable approaches for their own policy evaluations, and important questions to pose to their evaluation team as they begin planning such evaluations.