135 - Building Accessible, Equitable, and Actionable Evaluation Systems

Abstract Information: At South Carolina First Steps, the state’s only dedicated early childhood agency, we are disrupting the status quo of program evaluation in state government. We are actively working to make evaluation accessible, equitable, and actionable and to shrink the gap between those who evaluate programs, those who design programs, and those who participate in programs. As a state agency, our organization collects data on the reach and impact of our programs on children and families across all 46 counties in South Carolina.

Our organization is legislatively mandated to contract with external evaluators to evaluate our agency's impact on a five-year schedule through analysis of retrospective data. These evaluations typically culminate with a dense report and a presentation of findings to staff and the Board of Trustees. Program participant and service deliverer involvement is minimal in these evaluations, as they are typically focused on administrative data. In the past, these time and money-intensive evaluations were the only tools we had to tell our story from an evaluation perspective. As our organizational capacity has grown, our approach to evaluation has expanded to include the use of rapid-cycle feedback loops. Feedback loops are a new evaluation tool that is used to collect, interpret, and respond to feedback in real-time. Feedback loops are led internally and are used as a complementary strategy to externally led, agency-wide evaluations.

The speaker will introduce the concept and five phases (i.e., design, collect, interpret, respond, and close the loop) of a feedback loop. Next, the speaker will highlight two short vignettes on how these loops have been implemented to gather feedback from two perspectives in our First Steps 4K program – the parents/caregivers of a First Steps 4K student and the teachers/directors of a First Steps 4K provider. South Carolina First Steps provides state-funded, free, full-day four-year-old kindergarten in non-public school settings to Medicaid-eligible children statewide in a program known as First Steps 4K.
To make evaluation accessible, the speaker will share examples of how we strengthened capacity for collecting, analyzing, reporting, and disseminating data. During the 2020-2021 school year, we launched our First Steps 4K family engagement survey – a biannual survey to gather data on parent-reported outcomes – including the ability to enter the workforce and pursue postsecondary training – and parent-reported child outcomes related to kindergarten readiness. Over the past three years, we have heard from over 1,000 First Steps 4K families. In spring 2022, we expanded our evaluation of First Steps 4K to include feedback loops with First Steps 4K providers.
To make evaluation equitable, the speakers will illustrate how we uplift diverse perspectives and how we provide opportunities for families and providers to participate throughout the five phases of a feedback loop.
To make evaluation actionable, the speakers will illustrate how evaluators, program staff, service deliverers, and program participants worked together to co-design and implement changes in response to feedback.

Relevance Statement: The speakers will use short stories to illustrate how an early childhood state agency blended principles of community-based participatory research and storytelling to rapidly collect, interpret, and respond to feedback. External, retrospective program evaluations provide a comprehensive report on our agency’s reach and impact. However, the ability to perform intermittent “pulse checks” on programs provides an informal, yet complementary strategy, to monitor our agency’s performance and impact in real-time. By using rapid-cycle feedback loops, our agency can gather information on what is working well, what needs to be improved, and implement data-informed changes in response to the feedback. Feedback loops can be a low-cost, low-burden evaluation tool that can build a foundation for creating accessible, equitable, and actionable evaluation systems.  Although a feedback loop can illuminate areas of success and opportunities for growth within a program, a feedback loop also involves self-reflection on the process itself. After each loop, evaluators are encouraged to reflect and respond to ways that different steps in the loop can be refined before the next loop begins. Thus, this new evaluation tool promotes continuous quality improvement not only of our programs, but also stimulates growth of the people and the process of evaluation itself.