Systems in Evaluation
Audrey Moore, PHD (she/her/hers)
Principal Researcher
Mathematica
Fairfax, Virginia, United States
Audrey Moore, PHD (she/her/hers)
Principal Researcher
Mathematica
Fairfax, Virginia, United States
Ivonne Padilla, MPA/ID (she/her/hers)
Researcher
Mathematica, United States
Laura Meyer, MPP
Researcher
Mathematica, United States
Rachel Blum (she/her/hers)
Senior Technical Advisor, Youth & Workforce Development
Education Development Center, Inc. (EDC), United States
Erin McCarthy
Officer, Learning and Evaluation
Hilton Foundation, United States
Location: Grand Ballroom 3
Abstract Information: The use of data on education and youth systems through storytelling can facilitate change at many levels. It can make data more accessible to communities, help policymakers understand pain points, and facilitate an understanding of where change is taking place. This panel brings together a series of evaluations that use storytelling in different ways to drive systems change. The first presentation entitled Measuring Systems Change: Improving access and equity to middle school education presents the findings from a longitudinal political economy analysis. The presentation focuses on how the evaluation team used longitudinal data to understand institutional changes related to new systems that were developed to improve teacher recruitment and improve decision-making around budgeting middle school delivery. The presentation uses graphics and a delphi process to illustrate how complex systems and a complicated network of actors reduced the sustainability of systems change interventions. The second presentation, entitled Improving the quality and relevance of professional education and training focuses on an intervention aimed at strengthening the TVET governance system and improving the alignment of training with the needs of the private sector. The presentation shows how interventions aiming to improve the TVET system in El Salvador benefited from incorporating the narratives of stakeholders involved to make the intervention politically feasible to implement and sustainable. It also presents lessons learned related to considering past experiences to increase the capacity of the private sector to identify technical skills needed and building stronger relationships among the private sector and education actors. The third presentation, entitled Developing an early childhood development (ECD) systems measurement framework, highlights how local researchers engaged with key stakeholders to modify a systems diagnostic tool to measure change in Early childhood education in each country, monitor and assess systems change and use the data to help drive the integration of ECD in each country. The presentation will highlight the different stories and paths of each country. The final presentation, entitled Using analytic frameworks to capture unanticipated systems-level changes, shows how a story-telling process can help public- and privately funded youth development projects articulate their contributions to systems change, especially when such contributions are not originally embedded within the traditional program monitoring processes. Using a youth workforce development case study from Rwanda, this presentation offers a living Youth Systems Framework for benchmarking the state of a youth system and capturing changes to that system over time.
Relevance Statement: Systems have many definitions. Meadows (2008) defines systems as “an interconnected set of elements that is coherently organized in a way that achieves something (function or purpose).” Holland (1998) defines it as “a configuration of interaction, interdependent parts that are connected through a web of relationships, forming a whole that is greater than the sum of its parts.” Others such as Barbasi (2002) and Gates, Walton & Vidueira (2021) focus on systems as “a set of interrelated elements that interact to achieve a purpose or systems that are nested and networked within each other.” Regardless of the definition, systems are complex and interconnected, and any changes to one component of the system lead to a series of additional intended or unintended changes that ripple through the system. Systems change is an intentional process designed to alter a system by shifting its function or structure through purposeful interventions across the system. Systems change aims to bring about lasting change by altering underlying conditions which make the system operate in a particular way – and change in these systems can be difficult to create and even harder to measure. Systems change is difficult to measure for a number of reasons. Systems change occurs when many changes come together over time, when different stakeholders interact with one another and respond to different incentives, interests, mental models, relationships, and power dynamics. These changes can be difficult to measure both within and across different systems. First, where do you draw boundaries to measure change? People and organizations have different perspectives on where boundaries of the system are, so while boundaries are needed, they are often arbitrary making it difficult to fully account for what needs to be measured. Second, any attempt to measure systems change requires people to fully understand the cause-and-effect relationships. These relationships may be multi-dimensional and hard to measure. Positive youth development programs may face difficulties in capturing and articulating this complex change process to public or private donors who often require “bite-sized” results reporting. Finally, measuring systems change take time – often years, something donor programs may or may not be willing to do. The use of storytelling in evaluation brings data to life. It simplifies findings, may graphically depict the narrative story the data tells us, and can be more powerful in helping the communities that we serve understand whether and how changes affect education and youth systems. It makes seemingly boring and uninteresting—or unwieldy— data tangible and accessible to everyone and is important to the work many evaluators do to support communities. The use of storytelling in each of the proposed presentation has helped policymakers and communities better understand the barriers and facilitators of change, and consider targeted interventions that catalyze long term changes and make complex problems easier to understand and resolve over time. These presentations contribute to the field of evaluation by providing examples of how to translate data into easily digestible information that is accessible to all audiences and helps increase uptake.
Presenter: Erin McCarthy – Hilton Foundation
Presenter: Audrey S. Moore, PHD (she/her/hers) – Mathematica
Presenter: Ivonne Padilla, MPA/ID (she/her/hers) – Mathematica
Presenter: Laura Meyer, MPP – Mathematica
Presenter: Rachel Blum (she/her/hers) – Education Development Center, Inc. (EDC)