True stories: Centering authenticity in bundled evaluations of programs tackling society’s biggest challenges

Abstract Information: Lincoln and Guba (1985) developed the concept of authenticity in qualitative research to mean that findings are credible not only from the perspective of participants but also in relation to the broader implications of the research. This means that participants play a role in how their experiences and perspectives are represented in the research and that they learn something from having participated in the research. This session will describe how evaluators centered authenticity in a multifaceted AmeriCorps evaluation project devoted to evidence-building in national service programs.
Since 2021, AmeriCorps, the federal agency that funds national service programs to make positive impact in communities, has funded three “bundle” evaluations, which are being carried out by an independent research firm. The evaluations are descriptive or outcome-based and it is the first time that the agency is looking at how organizations are implementing Climate Change, Substance Use Recovery Coaching and State-Level Volunteer Management Capacity Building programs. In complement to the evaluations, we facilitated a yearlong, topic-tailored evaluation capacity building space for each of the bundle groups.
The pairing of evaluation capacity building with the bundled evaluations is a two-pronged approach to building evidence for each intervention model. First, in the short term the capacity building helped participants contextualize evaluation activities, such as data collection requests, and offered opportunities to contribute to data analysis and reporting as well as serving as a peer-learning space. Second, the capacity building empowered participating organizations to build on the bundle evaluation in the long-term by planning and implementing evaluations in their own specific contexts going forward and also providing live feedback to the evaluators and agency. Building the evaluation capacity of evaluation participants also supports educative authenticity, one of the five dimensions of authenticity identified by Lincoln and Guba.
In this panel, presenters will share how they adapted the notion of authenticity into the three studies and the evaluation capacity building initiative in order to include participants’ feedback, pivot content delivery and evaluation strategies, and overall, make the evidence journey more accessible to each of the groups. We will tell the story on the differences and similarities within the groups and the lessons learned along the way that inform both the specific focus area of the programs but also broader evaluation and capacity building issues.

Relevance Statement: American society has entered an era of “pernicious polarization” (Mounk, 2022), fueling distrust experienced across sectors. In evaluation, trust between evaluators and participants is crucial for carrying out a study successfully. To build trust and engagement among research and evaluation participants, authenticity in all aspects of the relationship is essential. As Lincoln and Guba (1985) and James (2008) suggest, authenticity not only enhances internal and external validity and evaluators’ ability to tell true stories, but it has the potential to change participants and society through a worthy topic of study.
This session is relevant for individuals who wish to learn more about how to center authenticity in evaluation from multiple perspectives. In addition, this session is also pertinent to those who are interested in understanding more about connecting evaluation and evaluation capacity building as a strategy for building evidence. Finally, this session is relevant for organizations and individuals who are interested in bundled evaluation models. Additional details about the relevance of this session are as follows:
• Evaluators will find this session relevant because it describes how this approach can cultivate deeper, more productive relationships between evaluator and evaluation participant. Specifically, we will describe evaluation strategies that center authenticity—empowering evaluation participants as co-creators at all stages of the evaluation process. For example, we will describe how we engaged grantees to provide feedback on the evaluation bundle research questions; offer input on ICF’s data collection approach, including feedback on surveys and protocols and our plans for administering them; and contribute to final evaluation products. Taken together, participants’ engagement fostered more buy-in for the bundle evaluation, greater validity in findings, and, going forward, we anticipate organizations will better use evaluation findings to inform practice.
• Funders will find this session relevant because it describes an approach they can use to grow the evidence base around a general topic of interest in the near and longer-terms. In the near term, authentic engagement can support for valid research by supporting participants’ feedback on research questions, methods, and interpretation. In the longer-term, this approach can help organizations design and implement their own evaluations in ways that extend the impact of the initial bundle evaluation.
• Organizations implementing programs will find this session relevant because it describes ways they can grow evidence around topics that are relevant to them and tell their stories through evaluation. In particular, authentic dialogue can help organizations make better use of the bundle evaluation findings by improving their ability to interpret the findings and apply it to their situation.
References
James, N. (2008). Contributor for qualitative research terms. In L. M. Given (Ed.), The SAGE handbook of qualitative research methods (p. 118). Thousand Oaks, CA: Sage Publications.
Lincoln, Y. S., & Guba, E. G. (1985) Naturalistic inquiry. Thousand Oaks, CA: Sage.
Mounk, Y. (2022). The doom spiral of pernicious polarization. The Atlantic. https://www.theatlantic.com/ideas/archive/2022/05/us-democrat-republican-partisan-polarization/629925/