80 - Evaluating federally funded efforts to improve permanency in the child welfare system: how stories fueled evaluation and evaluation fueled the story of success.

Abstract Information: In 2018, five organizations were awarded federal grants to Strengthen Child Welfare Systems (SCWS) through collaborative efforts to improve outcomes. SCWS evaluators how they evaluated the impact of direct practice and systemic reform efforts in Texas and Washington. This session will begin with a description of the federal expectation for impacting safety, permanency, and well-being, as measured by the federally mandated Child and Family Services Review (CFSR) process. Presenters will discuss some of the successes and challenges in assessing grant impact on federal CFSR outcome measures at a statewide level. In addition to measuring impact based on CFSR outcomes, SCWS evaluators successfully gathered evidence of grant impact on child welfare outcomes in other ways. First, SCWS evaluators will share how they captured the stories of people with lived experience to fuel insights throughout the evaluation process. Presenters will talk about the successes and challenges in utilizing interviews, focus groups, and surveys to gather the experiences and perspectives of those who have been touched by SCWS interventions to better understand each project's implementation. Presenters will then describe how the evaluation fueled the story about the impact of the grant. Presenters will emphasize how utilizing mixed-methods approaches enabled them to weave together different types of data to tell the story of shifting outcomes beyond those related to the CFSR. Presenters will describe how they have developed processes for incorporating people with lived experience throughout the grant process, from intervention design to data interpretation and reporting.

Relevance Statement: The SCWS funding announcement was developed to provide resources to focus on positively impacting CFSR outcomes. As the grant cycle comes to a close, this session offers an opportunity to learn about how the evaluators were able to assess the intended impact of the grant, but also gather information that provides a richer context for understanding how the grants were successful. The AEA theme of storytelling is particularly relevant to this project. In the last five years, child welfare systems and researchers have begun to understand the importance of incorporating the voices of people with lived experience into both program development and improvement, as well as evaluation practice. The SCWS evaluators evolved over the last five years to embrace efforts to gather the stories of people who experienced the child welfare system as parents or children as a source of evaluation data, but also as contributors in the development of interventions, as a source of qualitative data for the evaluation, and as key partners in understanding research findings. This session allows participants to hear about each state’s efforts to evaluate the impact of both direct practice and systemic reform efforts in their states. The evaluators will share how they were able to assess the impact on proximal and other expected long-term outcomes. This discussion will be framed by the concept of storytelling. Within this session, we will share the SCWS evaluator’s perspectives on the following questions, allowing time for conversation with audience members about their experiences. Story fuels evaluation: • How have you gathered the experience of people with lived experience? • What can be learned from the stories and experiences of people with lived experience that is not captured by more traditional evaluation approaches? • How can child welfare evaluators authentically capture the experience of the clients who interact with a bureaucratic system to inform system improvements? Evaluation fuels story: • How did this perspective of people who experience the intervention contribute to your understanding of the evaluation findings? • How have you incorporated people with lived experience into strategy and evaluation development and into the analysis process? Session participants will leave with not only an understanding of this federal grant initiative and its impact in five states, but also a new approach to authentically incorporating lived experience and stories into intervention design, evaluation design, data sources, and partners in making meaning of evaluation findings and designing recommendations.