183 - Epistemic (in)justice in evaluation: theoretical progress and empirical evidence on the connection between authentic storytelling and internal validity in impact evaluations

Abstract Information: There is an increasing desire for evaluators to credibly show the impact of a program while simultaneously taking care that their efforts serve the social good and are justice-informed. This process includes building awareness of injustices that evaluation may affect, both positively and negatively. Epistemic injustice is a relatively new concept within feminist philosophy, only recently applied to evaluation, that focuses on the knowledge component of power relations, and the wrongs that can occur to someone based on their capacity as a knower. How a participant tells a story, where, why, and within which power structure all influences the determination of impact. Applied to evaluation, especially impact and outcome evaluation, epistemic injustice provides an especially timely and salient theoretical lens through which evaluators can enhance the validity of their findings from both a technocratic and participatory evaluation perspective. This poster presents the findings from empirical research on the conditions for epistemic injustice in international development impact evaluations, based on data from evaluation reports and narrative interviews with working evaluators. The paper will be especially relevant for outcome and impact evaluators seeking practical guidance to understand how to use “justice as their signposts” (House, 1980) in the context of modern evaluation challenges.

Relevance Statement: The presentation will present one aspect of a dissertation study on how causal validity, as a measure of truth, and epistemic injustice, as a measure of justice, manifest and are negotiated in international development impact evaluation. Inspired by seminal evaluation theorist Ernest House’s call for evaluators use justice as their signposts to enhance the validity of evaluation findings (House, 1980), together with Donald Campbell’s work on experimental design (Campbell, 1970), this research explores the application of the epistemic injustice to impact evaluation practice/praxis. Apparent tension between participatory and technocratic evaluation approaches of impact evaluation have left evaluators unfairly caricaturized and dichotomized between those that seek truth, and those that seek justice. One connection between epistemic injustice and values in evaluation is that when an evaluator does not understand or account for the values of their evaluand, especially when there are other injustices present (Byskov, 2020), they risk committing an epistemic injustice (Schwandt & Gates, 2021). There are many other examples that are both theoretically grounded and empirically validated. In applying epistemic injustice to international development impact evaluation, I consider the extent to which a polysemy of impact contributes to epistemic injustice and the relative necessity of methodological pluralism in preventing injustice. The political economy of evaluation and its use in public decision-making is complex, and the existing structure may limit evaluators' choices in the service of truth and justice. This research and subsequent presentation has implications for metaevaluation, the integration of theory into evaluation practice, and is timely in that it centers a very specific aspect of storytelling: the connection of power and knowledge. It may be relevant for audience members interested in incorporating justice-aware methods into their quantitative evaluation designs, and also for evaluators seeking an additional theory to enhance their justice-aware evaluation praxis. Campbell, D. T. (1970). Considering the case against experimental evaluations of social innovations. Administrative Science Quarterly, 15(1), 110–113. http://www.jstor.org/stable/2391194 Byskov, M. F. (2021). What makes epistemic injustice an “injustice”? Journal of Social Philosophy, 52(1), 114–131. https://doi.org/10.1111/josp.12348 Schwandt, T. A., & Gates, E. F. (2021). Evaluating and valuing in social research. The Guilford Press.