How Systems Change and Transformation Change the Plot of Our Evaluation Story: Findings from an Inter-disciplinary Literature Review, 2011-2021

Abstract Information: Amidst the covid-19 pandemic and the urgency to address wicked problems, a plot change unfolds across evaluation and related fields. Since the field’s origins, evaluators story their craft as a practice that contributes to social betterment by assessing the value of policies and programs using systematic, social science methods. In the last decade, a new story is rising up about how evaluators can and should co-facilitate systems change and transformation. Drawing on an interdisciplinary review of over one hundred references, this paper argues that an emerging plot calls on evaluators to step into new roles and new scenes and that the script has not yet been written. Scholars and practitioners across the seven disciplines reviewed (i.e., evaluation, health, sustainability, organizational management, and philanthropy) all call for a big plot change but are murky on the details. We share our analysis of this literature highlighting where stories align across disciplines around what it means to work in complexity and leverage change to systems and where they diverge and remain open about the roles monitoring, learning, and evaluation play. We reflect on why this is an exciting opportunity for the field.

Relevance Statement: In a recent issue of the American Journal of Evaluation, scholars Zenda Ofir and Deborah Rugg (2021) describe evaluation at a critical juncture: “the aftermath [of the COVID-19 pandemic] brings a unique opportunity - a chance for all humanity to reimagine and reinvent our societies and their relationship with the natural ecosystems on which everything depends, redefine our personal and professional commitments and priorities, and revisit our worldviews and how we conduct ourselves. This is also the chance to change how the field of evaluation is viewed and developed” (p. 47). In our view, the most pressing opportunity for the evaluation field is to reimagine our role in social change and transformation. Historically, the field has narrowly focused on assessing discrete projects, policies, and change initiatives using linear and mechanistic logic models and experimental and quasi-experimental designs premised on counterfactual and attribution-based logic of causation (Schwandt & Gates, 2022). The trend of systems thinking and complexity science challenges these assumptions and argues for non-linear, interconnected models of change and contribution analysis and adaptive learning and management, amidst other shifts (Gates, Walton, & Vidueira, 2021). The trend of equity, social justice, and decolonization further challenges the field to ensure that evaluation processes and uses counter systemic oppression - questioning the root causes of problems, facilitating equitable participation and deliberation, and ensuring interventions and evaluations are not complicit in unjust ways of thinking and working (EEI & GEO, 2021). These two trends provide a backdrop for calls to rethink and reshape the way we practice evaluation to more purposefully, directly examine and even forward changes to large-scale systems. While this has meant considerable buzz around the terms “systems change” and “transformation,” there has been little empirical research on the topic within the evaluation field and a wide range of work, often using distinct terminology, in related fields. Moreover, given how well established a “project mentality” (Patton, 2021) and linearity are to our ways of thinking about change and evaluation, it can be easy to interpret these trends within prior assumptions and, in doing so, constrain evaluation role and influence. Our paper draws together an extensive, interdisciplinary body of academic and grey literature to identify key cross-cutting findings. These will be of interest to those commissioning, leading, and/or evaluating systems change and transformation initiatives.