Mixed Methods Evaluation
Shannon Stagman, M.A.
Director, Advisory Services
Arabella Advisors
BROOKLYN, New York, United States
Julie Slay, Ph.D.
President and Founder
Prism Partners Group, Washington, United States
Location: Room 309/310
Abstract Information: Incorporating storytelling in evaluation efforts allows us to create more space for qualitative data, ground findings in a broader context that helps deepen our understanding, and support ongoing learning and sense-making efforts in a way that’s more accessible for mixed audiences. In a perfect world, each evaluation effort would have the perfect data set to address its research question and tell the story of a given program, project, or place. In reality, however, that data set often does not exist and cannot be created with the resources at hand, and some evaluation methodologies are limited in ways we cannot control. Evaluators, therefore, must find ways to address these limitations without introducing their own biases in the process.
In this interactive session, we will explore the decisions evaluators must make when tasked with distilling an incomplete or imperfect data set into a coherent narrative. We will ground the session in the case study of a partner collaborative among three nonprofit organizations that wanted to assess state-level progress on climate justice, democracy, and journalism work over time. We will provide participants with the aims of the evaluation, the research questions we sought to answer, and the data sets we identified to support this learning effort, before dividing participants into small groups to develop a story about a state based on the available data. We will then share back our state summaries, discuss the differences between them, and dig into the choices made that led to each story. Through the discussion, we will explore the limitations of data to tell stories, the biases that informed our storytelling decisions, and how to identify and overcome the features of our own training as evaluators that can make storytelling challenging.
Relevance Statement: As the philanthropy sector increasingly invests in movement-building as a force for change, stories of movement momentum are an important way to understand emerging impacts such as the benefits of a strengthened advocacy ecosystems, narrative shifts, and champion-building. When evaluating advocacy and systems-change work, it is critical for evaluators to look beyond traditional programmatic impact metrics and share bigger-picture signals of progress that demonstrate the importance of long-term investment in this work.
Many fields use stories to convince and educate stakeholders for various purposes—fundraising, education, and lobbying, for instance. Storytelling does not necessarily require a specific set of methodologies, and emerging data sets and approaches appear to be useful in understanding ecosystem impacts and narrative change (e.g., social media monitoring, social listening). However, many evaluators are either unfamiliar with these approaches or don’t know how to access or use them. Though the field has largely finished debating whether qualitative data is credible, there is a gap in understanding how evaluators can rely on new methodologies like media monitoring to tell meaningful stories, as well as where we should exercise caution around their constraints. Many texts outline ways to ensure reliability and validity in traditional qualitative methodologies (Lin, 1994; Patton, 2002; Bickman & Rog, 1998), but fewer have explored the use of social media analysis in evaluative contexts (Young et al., 2020).
In this session, we will use our own experience employing a mix of traditional and newer qualitative methodologies related to evaluating advocacy and movement-building. Our example details an effort to understand progress toward improved climate and energy policies and practices, increased representation of pro-democracy legislative leaders, and amplification of voices of communities of color in media coverage in five states in the US. Data sources include legislative and court-case tracking, election outcomes data, surveys and qualitative interviews, and media monitoring databases. We will share some of the factors we considered when selecting and using these data sources to analyze and derive meaningful findings, as well as how we crafted stories of three different social movements working in informal alignment.