102 - Lending credibility to your evaluation story with data quality and assurance approaches – strategies from an education intervention longitudinal evaluation in Somalia

Abstract Information: Stories that emerge from an evaluation are sometimes unexpected or at odds with commonly held beliefs, causing stakeholders and funders to question the validity of evaluation findings. Since any story is only as strong as the data supporting it, evaluators must employ rigorous methods to ensure data quality and be able to effectively communicate procedures and practices in a way that makes resulting stories believable. This panel will share the data quality and assurance methods from a large longitudinal study based in Somalia. We will share lessons learned over four data collection cycles and the resulting tools and strategies we designed to increase data collection rigor and fidelity, while streamlining data management and analysis. The panelists will describe learning and recommendations for best practice related to data collection tools, enumerator training and oversight, and data management and analysis. Panelists include both external evaluation experts from the United States and in-country colleagues based in Somalia. Their expertise includes methods in both qualitative and quantitative data studies. This talk will be of interest to evaluators and researchers working with large or dispersed data collection teams and those wishing to increase efficiency and effectiveness of data collection and analysis, while enhancing quality assurance.

Relevance Statement: Assuring and communicating data quality are critical elements of effective evaluation stories. Evaluators must use care in selecting data collection tools and methods that are appropriate for the project, rigorous, culturally sound, meet stakeholders' requirements and expectations, and take into account the needs and characteristics of stakeholders and subjects. This panel will share an example of a complex international project that required the concerted efforts of a multinational, multi-disciplinary team to devise and implement strategies for ensuring data quality in a challenging data collection environment. We will start the panel off with a presentation on data collection tools and techniques, describing structural and procedural approaches for enhancing fidelity of data collection processes for a large and dispersed team in a challenging environment. This presentation will provide concrete examples of tools and approaches this evaluation adopted to enhance data quality and data management efficiency, without compromising data richness. The second presentation will focus on training and refresher methodologies for enumerators, interviewers, and researchers. This presentation will describe train-the-trainer strategies and methodologies, mentoring, and oversight processes for maximizing fidelity of data collection in the field. The last presentation will emphasize data documentation, cleaning, and visualization techniques to ensure data quality. This panel will be of interest to evaluators and partners in large, international, and cross-cultural projects who may engage large teams or third-party enumerators to collect evaluation data. It will provide examples of tools and methods that make it easier to collect and organize data. Together, the presentations will invite us to reconsider longitudinal techniques to examine program or project impacts over time and to embrace the challenges that come with large or dispersed data collection teams.