149 - Lessons learned from the realistic evaluation of a project introducing a technological innovation in West African primary health care centres

Abstract Information: The AIRE project is funded by Unitaid and is implemented by a consortium of NGOs. It aims to introduce a medical innovation, pulse oximeters (PO), in 202 primary health centers in four West African countries (Burkina Faso, Guinea, Mali and Niger). The objective is to improve the diagnosis and treatment of children under five years of age. Indeed, these tools make it possible to detect severe cases leading to respiratory distress, and thus identify the need to refer children to hospitals to receive adequate care, particularly oxygen. This is the first time this tool has been introduced for routine use in these settings.

The idea behind our research was to reconstruct the story and the journey of this innovation in health centres. What has happened since its introduction? In what context(s) is the PO used/adopted by health staff? What explains their decision to use/adopt (or not) the PO? What are the implications of adopting (or not adopting) the PO? We believe that realistic evaluation is particularly relevant for "telling the story" of a project in the context of an evaluation. First, it allows the use of history in evaluation, by paying particular attention to the contexts of intervention. Secondly, it allows to listen to and take into account the different voices of the project (of the implementation teams, of the "beneficiaries", etc.) and comparing experiences (everyone's story), expectations and achievements. Finally, it allows to reconstruct the global story of the introduction of the innovation. It is the links between the context, the intervention, the mechanisms and the outcomes, and the sequence between the different configurations that will enlighten the data and give it the consistency of a story. We therefore considered that a realistic evaluation was particularly suitable to answer our research questions.

To carry out this evaluation, we assembled an international research team and conducted several research workshops to prepare and conduct the analysis. The first workshop aimed to train the teams in the use of the approach; the second helped to organize the data collection according to the realistic approach. The last workshop allowed us to launch the analysis, reformulating together the list of hypotheses and immersing ourselves together in the empirical data. We conducted over 200 interviews in the four countries at different times throughout the project, carried out observations of under-five consultations and held group discussions with the implementation teams. All analyses, including empirical testing of realistic hypotheses, were conducted using the qualitative analysis software Nvivo.

We propose to present both the empirical results of this realistic evaluation and a methodological reflection on the challenges encountered. Based on our experience, we will discuss the challenges related to the complexity of operationalising this evaluation approach, to the integration of this type of evaluation into a more global research project, as well as its contributions and limits for the evaluation of health programmes.

Relevance Statement: Firstly, we believe that our proposal may be important in the field of evaluation as it is one of the largest realistic evaluations carried out, in terms of resources and coverage, in West Africa. It seems important to us to present our empirical results, but also and above all the intellectual and operational paths we took, and the challenges we encountered. We have collected data in several countries, worked jointly with various NGOs and researchers from very different backgrounds. We have learned a lot and reflected on our practices, both in terms of the operationalisation of the approach and the method, but also in terms of the dissemination and popularisation of these scientific evaluation methods.

Secondly, it seems to us that the realist evaluation approach corresponds perfectly to the theme of the conference, since its objective is to reconstruct and tell the story of a programme, from its conception to its experimentation and appropriation by the beneficiaries, up to its outcomes. To do this, it aims to take into account the stories of each of the people involved in the project, in order to understand how they reacted to it.

Finally, realist evaluation is a growing branch of evaluation that is expanding beyond the health field. This approach allows us to reflect on evaluative practices and methods (notably through its anchorage in very developed ontological reflections) and to participate in consolidating and developing the theory-based branch of evaluation. Theoretical writings on this approach are beginning to be numerous, but there is still a need for guidance on the concrete operationalisation of the approach at its various stages: how to reconstruct the intervention theory with the field teams? How to formulate and build the list of hypotheses? How to concretely "test" the hypotheses in the field and through the empirical material? How to move from a realistic case study analysis to a cross-sectional analysis? Our presentation will aim to show how, as a research team (more or less initially experienced in the realist approach), we answered these questions in the light of the various constraints encountered.