Meta-Evaluation: Exploring Quality Assurance Processes in Evaluation Practice

Abstract Information: Meta-evaluation is the evaluation of an evaluation. It entails analyzing the quality of the evaluation process and findings rather than evaluating the program or intervention under consideration. Meta-evaluation is a powerful instrument for improving the reliability and effectiveness of evaluation activities, though little empirical research has focused on describing it in practice. This paper describes which standards a sample of AEA evaluators use and where they draw their quality assurance criteria from. The paper also explores how deeply various stakeholders are engaged in quality assurance and what mitigating circumstances might influence quality assurance processes. It reports on a study of practitioners from the American Evaluation Association membership who responded to an online survey on their quality assurance processes. Respondents stated that they use several instruments to assure quality in their work, including published and unpublished standards, principles, and guidelines, as well as processes that involve stakeholder input at various levels of evaluation. Using a fishbone, or Ishikawa diagram, we systematically identify and classify the factors assuring evaluation quality into: (I) standard use; (II) ethical guidelines; (III) stakeholder engagement; (IV) data quality; (V) training; and (VI) organizational learning. We then report how these factors are linked to quality assurance in the evaluation process.