32 - From Planning to Action: Lessons Learned from Engaging Stakeholders in Evaluating the Impacts of a Research Coordination Center Supporting Projects Promoting COVID-19 Testing Equity.

Abstract Information: Engaging stakeholders throughout the evaluation process is crucial to ensuring that evaluation findings and recommendations are used to improve program implementation, especially in a complex intervention landscape. We assessed the coordination, leadership, and community engagement activities provided by the RADx-UP (Rapid Acceleration of COVID-19 Diagnostics for Underserved Population) CDCC to 137 projects across the United States. The Rapid Acceleration of Diagnostics for Underserved Populations (RADx-UP) promotes testing equity and addresses COVID-19-related health disparities among underserved populations across the United States. The Coordination and Data Collection Center (CDCC) supports the 137 projects within RADx-UP. Our evaluation team assessed CDCC’s coordination, leadership, COVID-19 testing support, and community engagement activities. We present 3 case study examples describing the benefits and lessons learned from engaging and collaborating with diverse CDCC stakeholders in a rapidly evolving intervention landscape. We used an ethnographic approach to examine three case studies that best illustrate the challenges faced and corrective actions taken while engaging stakeholders during critical evaluation milestones (planning and design, data collection, dissemination and advocating for changes). To organize the lessons learned throughout the evaluation process (from planning to action), we built consensus as a team, identifying these three concrete scenarios (case studies). Our data sources included internal meeting memos, after-action review notes, and other documents to contextualize these scenarios. Within each case study, we described how we engaged stakeholders in planning evaluation activities, developing survey tools, and disseminating findings. Then, we summarized themes across the case studies, describing the needs and recommendations per case study. Some of the challenges we faced include sustaining stakeholders’ motivation to use evaluation findings as the program nears completion and difficulties in promptly closing the feedback loop from evaluation data collection to dissemination. It is useful for evaluators to think through how they can proactively maximize opportunities for meaningful stakeholder engagement. Benefits of engaging stakeholders include securing buy-in; strengthening the perceived value of evaluation and survey relevance; and increasing the level of influence of evaluation findings across diverse stakeholder groups. Evaluators engaged in similar initiatives can utilize our summarized lessons and key tips to strengthen stakeholder engagement throughout evaluation processes.

Relevance Statement: Stakeholder engagement strengthens evaluation design, the relevance of evaluation data collection instruments, and the capacity to use evaluation results for decision-making. Unfortunately, complex intervention programs comprise diverse stakeholders with competing implementation and evaluation needs. Therefore, program evaluators should be innovative in addressing the rapidly involving needs of stakeholders, share information timely, and use targeted dissemination strategies to motivate the use of evaluation findings to improve program quality. Published evaluation approaches (such as developmental evaluation, utilization-focused evaluation, and participatory evaluation) have outlined guidance to identifying stakeholders and engaging them in shaping evaluation design, interpretation, dissemination, and utilization of recommendations to strengthen program implementation and impacts. We build upon these guiding principles by presenting a detailed, contextual summary of the lessons we gained from engaging with multiple, diverse stakeholders in a complex, multi-center consortia program. These lessons will be relevant to other evaluators assessing the impacts of similar programs. Our team engages with CDCC program stakeholders to continuously monitor the quality of program delivery and increase CDCC's capacity to rapidly incorporate and use evaluation recommendations. In this proposal, we present the benefits and lessons learned from engaging stakeholders in evaluating CDCC's outcomes and impacts. We described how we ensured that evaluation recommendations were disseminated to the appropriate audiences and consequently used to improve the program leadership, organization structure, and coordination of implementation activities. Although there are multiple benefits for stakeholder engagement and empowerment in program evaluation, numerous factors can impede success if evaluators are not intentional about how, why, and what they want to achieve throughout the engagement process. Our team rediscovered the importance of starting any evaluation endeavor with the audience in mind. We took a pragmatic approach to engaging stakeholders, leveraging our subject matter expertise to complete deliverables while pulling stakeholders in for high-level yet targeted discussions to maximize limited time and personnel resources. The key tips and thematic lessons summarized by our team will be useful for evaluators engaging with diverse stakeholders in a complex research consortium or any public health initiative.