27 - Introduction to Equitable Systems Change Evaluation Framework Theory and Practice

Abstract Information: More organizations are shifting away from isolated and fragmented projects to larger, more equity-centered and culturally responsive systems change initiatives. However, relatively few social change leaders have developed the organizational capacity, robust strategies, collaborative partnerships, and learning and evaluation methods that are needed to support successful systems change. Many evaluation guides and frameworks focus on improving a single practice, such as assessing equity-centered strategies, using a systems lens, or working more effectively with diverse communities. Culturally responsive and equitable evaluation (CREE) and systems- and complexity-informed evaluation (SCIE) are relatively new examples of such single-focus evaluation approaches. Explicitly linking these approaches into a coherent whole has the potential to increase the rigor, validity, reliability, and value of evaluation practice. To advance the practice of evaluating equity-centered systemic and structural change initiatives, CREE and SCIE approaches have been combined to create the Equitable Systems Change Evaluation Framework (ESCEF). This workshop will provide an overview of the ECSEF and show how CREE and SCIE principles are woven into the framework’s seven-step process. assessing the situation; clarifying the initiative, co-designing the evaluation, adapting data collection methods and tools, and analyzing, interpreting, and reporting findings. This workshop will provide mini-lectures, case studies, tools, and interactive exercises to help participants apply the ESCEF to their evaluation practice in a wide range of contexts.

Relevance Statement: Over the past decade, government, non-profit, and philanthropic entities, have been working to address increasingly complex community, regional, and global problems, such as accelerating climate change, food insecurity, homelessness, dislocation, structural racism, and other inequities in access to resources, services, and opportunities. More organizations have responded to these challenges by shifting away from isolated and fragmented programs and projects to larger, more equity-centered and culturally responsive systems change initiatives. However, relatively few social change leaders have developed the organizational capacity, robust strategies, learning and evaluation methods or collaborative partnerships to design and implement successful systems change initiatives (Co-Impact, 2022: “Philanthropy for Systems Change Handbook”). Organizations need new approaches to update their learning and evaluation systems to keep pace with these shifts in practice. Evaluation guides and frameworks typically focus on improving specific practices, such as assessing equity-centered strategies, incorporating complex systems concepts, or working effectively with diverse communities. For example, culturally responsive and equitable evaluation (CREE) and systems- and complexity-informed evaluation (SCIE) are relatively new evaluation approaches that have received increased attention within the evaluation field. Each approach provides a unique but complementary way of understanding how people, programs, and initiatives are influenced by cultural factors and power relations that are situated within complex institutional, community, and political systems. Historically, these approaches have represented distinct schools of thought within evaluation practice and have been explored by program evaluators independent of each other, with few exceptions. Explicitly linking these approaches into a coherent whole has the potential to increase the rigor, validity, reliability, and value of evaluation practice. Doing so would provide a platform to more accurately capture the “story” of complex initiatives and opportunities to increase their effectiveness and impact. Understanding the potential value of integrating CREE and SCIE approaches, a team of NORC researchers (the presenters of this workshop) developed of the Equitable Systems Change Evaluation Framework (ESCEF) to advance the practice of evaluating equity-centered systemic and structural change initiatives. This framework works well for explicitly integrating equity into the evaluation design and approach, and engaging people with lived experience, organizations providing services and supports to the community, and other experts and interested parties. Such evaluations may also call for systems mapping and network analysis; identification of systems gaps; assessment of system assets, resources, and decision-making processes; clarification of theories of transformational change, storytelling and other innovating methods for capturing emergent ‘ripple effects’ of systemic interventions, and the use of iterative learning cycles and developmental evaluation approaches to facilitate the ongoing evolution and adaptation of initiatives in dynamic environments. This workshop is designed to improve participants’ understanding of the ECSEF, showing how CREE and SCIE principles are woven together through the framework’s seven-step process. assessing the situation; clarifying the initiative, co-designing the evaluation, as well as creating or adapting data collection methods and tools, analyzing and interpreting data, and sharing evaluation findings. The workshop will present mini-lectures, provide illustrative case studies, and use tools and interactive exercises to help participants learn to apply the framework to their evaluation practice.