19 - Sharing Challenges and Lessons Learned from Evaluating Family Poverty Reduction Programs: How Can We Effectively Tell the Program's Story?

Abstract Information: This birds of a feather session aims to bring together evaluators of family poverty reduction programs to share their insights and best practices. We want to create a space for attendees to explore how complex program data can be weaved together to tell meaningful and easy-to-consume stories. The path out of poverty is filled with ups and downs, which makes tracking and communicating families' progress out of poverty challenging. Relatedly, tracking whether individuals who live in poverty as children are progressing towards exiting poverty as adults is also challenging. This session will include a discussion, a case study, and an opportunity to network with evaluators doing similar work. To kick off the discussion, the facilitators will share reflections from a rich case study about a program that works alongside families for 20 years to break the generational cycle of poverty. We will share our lessons learned in measuring family well-being, education, and employment. We will also share strategies for writing meaningful evaluation conclusions. Reflection questions to spark conversation include: How can we boil down complex evaluation findings into meaningful evaluation conclusions? What are promising strategies to tell complex stories in ways that are relatable and interpretable to the clients? What does it mean to switch from a focus on measuring self-sufficiency to well-being? Participants will leave the session with a better understanding of the challenges of evaluating poverty reduction programs and strategies for addressing them in their own work.

Relevance Statement: Family poverty reduction programs aim to address the intergenerational cycle of poverty by simultaneously supporting children's and parents' well-being and development. These programs are often complex and multifaceted, with a range of components and strategies to improve families' outcomes. Evaluating the effectiveness of such programs requires a nuanced understanding of the social and economic factors that contribute to poverty and an ability to measure the impact of interventions across multiple domains, including family well-being, education, and employment.
Importance to the Field of Evaluation: This session will explore several challenging themes common to many evaluations. First, how do you use evaluative conclusions to effectively tell a meaningful story about a complex program? Second, how do you use qualitative and quantitative data to tell meaningful stories about complex programs that don't overwhelm readers? These two issues represent the competing tension that comes with trying to uphold multiple quality evaluation standards simultaneously (i.e., How do you conduct an evaluation that is useful, and meaningful, while also rigorous and evidence-based?). Our discussion on how to move from measuring more dated concepts, such as self-sufficiency, to more wholistic concepts such as family well-being will be relevant to members of the human service evaluation TIG.

Implications for Evaluation Theory or Practice: One of the main topics we hope to address is how to apply Scriven's logic of evaluation (i.e., evaluation theory) in the context of complex human service programs (i.e., in practice). In other words, we will be exploring whether it is possible to have concise evaluative conclusions that are meaningful to complex programs. When programs are complex, do evaluative conclusions become overly complex and an ineffective communication tool? (e.g., Is it meaningful to tell clients their programs are partially effective?)

Benefit to Audience: Evaluating these programs can be challenging since the interventions tend to be complex (i.e., targeting multiple ambitious outcomes) and the road out of poverty can be nonlinear. We are interested in creating a space for evaluators doing similar work to network and learn from each other and share their insights, best practices, tools, and innovative approaches.

Relevance to Conference Theme: This session is related to the conference theme in two ways. First, telling stories is not only an important part of evaluation practice, but it is also an important way we, as evaluators, learn from each other. Birds of a feather sessions are an opportunity for evaluators who work on similar types of projects to share their stories of successes and challenges. In the complex environment of human services, there is rarely a single right answer. Hearing stories from others about what has worked for them may spark ideas for attendees. The second way this session is relevant to the theme is a discussion of how to both use story to dive into the nuances of program impact, as well as how to contract on details and use evaluation conclusions to summarize complex data to tell an effective and easy-to-consume program-level story.