Creating a New Approach in Ethiopia: When Evaluation is the Story

Abstract Information: Last Mile Health’s (LMH) mission is to save lives in the world’s most remote communities - what we call the Last Mile - by promoting national Community Health Worker (CHW) programs. In 2021-2022, LMH partnered with Ethiopia’s Ministry of Health to design and pilot an innovative integrated refresher training for CHWs—a module blending digital self-learning and in-person elements—with the goal of decreasing costs while improving learning gains. A rigorous evaluation was designed to test the learning material, measure knowledge and skills, while continually listening to and improving from the user experience. Participants made strong gains in both knowledge and practical skills, while the costs of training were reduced by nearly 40% compared to conventional in-person training. The program team, evaluators, and communications team all worked hand-in-hand to tell this remarkable story through reports, briefings, conferences, blog posts, and more. Based on the success of the pilot module, the Ministry of Health asked LMH to expand this approach to additional training modules to be used with its cadre of more than 40,000 CHWs. This represents an enormous opportunity to build and scale the success of effective, cost-effective training—ultimately improving accessibility, knowledge, and skills for health workers who provide life-saving primary care in rural and remote communities. This Ignite presentation will tell the story of this pilot program, its evaluation, and the stories that they generated. While individual stories and narratives played a role throughout the evaluation, this example shows how evaluations tell stories themselves—and the impact those stories can have.

Relevance Statement: As AEA has noted with the theme of this conference, storytelling is essential to evaluation and serves many purposes. It is critical for evaluators and data analysts to be well-versed in the importance of the narrative, how stories themselves create impact, how data becomes a story that explains a phenomenon, and how telling a story gives individuals a voice. In this, evaluators have many responsibilities - to their subjects, to their clients, to the organizations or programs they examine - to the wider public who needs to hear their stories in ways that are compelling, data-driven, and most of all - authentic. This presentation provides an example of how stories from feedback during user testing and throughout the evaluation helped guide the work, as well as how evaluation results were used to tell a compelling and credible story of success, one that inspired new donors and convinced a national government to act. It resulted in the Ministry of Health committing to scaling the program across its entire community health worker training content, and at a national scale, enabling it to reach 40,000 community health workers and millions of people in the most remote and underserved corners of Ethiopia. It reminds us that none of this work happens in isolation - it requires partnership among program staff who design and implement with excellence, and evaluators who study with rigor and openness. And it sometimes requires a communications team that can translate results to a compelling and convincing story, package them in ways that are approachable, understandable to wide audiences, visually appealing, and easy to grasp, and that drive the reader to action. Stories can make us feel something new, think differently, or care more deeply. At their best, they drive action, and this presentation shows one way that this happens. Reference: RMNCH IRT Blended Learning Pilot Abridged Report: https://lastmilehealth.org/wp-content/uploads/2022/07/Blended-IRT_RMNCH_Pilot_Abridged_Report_v08.pdf