78 - Multi-country evaluation of interventions to strengthen data quality and use for COVID-19 vaccination: a mixed-methods evaluation from DRC, Kenya, Niger, Senegal, Tanzania, and Vietnam

Abstract Information: This poster documents the results of a mixed methods evaluation on the implementation and outcomes of digital health interventions to support COVID-19 vaccination in DRC, Niger, and Vietnam. Substantial investments were made in digital health policies, programs, and interventions to support COVID-19 vaccination, such as national systems to track vaccines and cold chains, information systems to record and manage patient vaccination data, and digital tools to plan vaccination service delivery. This mixed-methods, real-world evaluation employed a range of approaches such as root cause analysis, causal link monitoring, and outcome harvesting to assess whether and how countries’ digital health interventions implemented for COVID-19 vaccination improved vaccination planning, uptake, or contributed negatively or positively to the broader digital health ecosystem. Implemented during an emergency context on complex interventions and programs, the evaluation was designed to immediately inform programmatic adaptation and improvement. The poster describes the methodology used across the countries and findings from this evaluation, including whether and how various digital health interventions affected the improved uptake of safe and effective COVID-19 vaccines. 

Relevance Statement: This poster is relevant to evaluators who are faced with evaluating complex interventions in complex settings and for those who are interested in reflecting on what we learned as evaluators of policies, programs, and interventions during the COVID-19 pandemic. Substantial investments were made in digital health policies, programs, and interventions to support COVID-19 vaccination, such as national systems to track and monitor vaccines and cold chains, information systems to record and manage patient vaccination data, and digital tools to plan vaccination service delivery. In an emergency context, these investments were often made without clear agreement on program logic or theory of change, or how these investments would be monitored or evaluated. The challenges in conducting such an evaluation are many, including a lack of clearly defined theories of change and existing indicators, changing interventions and contexts, cross-country synthesis of disparate contexts, and an absence of a counterfactual for determining magnitude of impact. This evaluation was designed to explore positive and negative intended and unintended consequences on related programs and systems, such as national childhood vaccination programs and national digital health ecosystems. Furthermore, evaluations with a similar approach across multiple countries can provide a richer narrative, by highlighting contextual differences that contributed to different methodologies and evaluation outcomes. This poster will be highly relevant to anyone who is interested in learning more about using qualitative approaches and mixed methods to evaluate the effects and contributions of specific interventions on broader environments, as well as those asking the same evaluation questions across different interventions and settings.