Systems in Evaluation
E. Jane Davidson, Ph.D.
Founder and Principal Consultant
Real Evaluation, United States
E. Jane Davidson, Ph.D.
Founder and Principal Consultant
Real Evaluation, United States
Thomaz Chianca, PhD (he/him/his)
Managing Partner
COMEA Relevant Evaluations
Rio de Janeiro, Rio de Janeiro, Brazil
Mariana Beirao Xavier, MA (she/her/hers)
Senior Evaluation Manager
Laudes Foundation
Amsterdam, Netherlands
Merav Bat-Gil, MA
Development Manager, Europe
Business & Human Rights Resource Centre
Location: White River Ballroom A
Abstract Information: Systems transformation is a nuanced and multifaceted undertaking that is best assessed using stories of influence and change from those who are closely watching it unfold. Laudes Foundation has made a huge transition from a logframe- and KPI-based approach to evaluation to a more flexible, complexity-ready approach that uses rubrics to describe the essence of what a transformed system needs to look like, as well as what good progress should translate to along the way. Key to the system is a learning agenda and a group of learning champions, who are supported to bring stories and other evidence to life using real-time, rubric-guided evaluative sensemaking processes. Partner organizations (grantees) select the rubrics that best capture the aspects of the system they are seeking to change, then gather stories of systemic change and other mixed-method evidence to assess baseline and progress on their focus outcomes. Insights from partner learning reports are then synthesized within each programming area using a sensemaking process led by Laudes’ learning champions. This yields a rich understanding of how well the needed change is emerging in the industries and regions where Laudes supports change. Cross-organization learning conversations help extract nuanced intelligence about systemic change, which then feeds into Laudes’ strategic adaptation process. In this lively, interactive demonstration session, participants will hear from the architects of the rubric-enhanced system, the internal evaluation leaders overseeing its application, and one of Laudes’ partner organizations, who will share their experience of using rubrics to tell their story of systems change contribution.
Relevance Statement: Rubrics, storytelling, and sensemaking are powerful tools that can help foundations like Laudes drive systemic change. However, successfully implementing these tools requires careful planning, strategic buy-in, and capacity building at all levels. In this demonstration session, we will explore Laudes' journey towards harnessing the power of rubrics, storytelling, and sensemaking, and provide practical insights on how to get started, create the infrastructure, build capacity, pace the process, take context into account, avoid fear of evaluation, and shift mindsets towards evaluative thinking. This session is designed for professional evaluators and decision-makers who are interested in learning how to use rubrics to assess meaningful, complex, systemic change and to guide collective learning and strategic adaptation. Laudes Foundation has radically shifted their measurement and learning system from a quantitative KPI- and logframe-based focused approach to a more nuanced, systems change-savvy, rubric-enhanced approach to measuring systemic change, one that can flex as change emerges. This process gained force in 2020-2021 when a systemwide theory of change (https://www.laudesfoundation.org/how-we-work) and a set of evaluative rubrics that clarify what success needed to look like (https://www.laudesfoundation.org/grants/rubrics) were developed. Our presenters from Laudes will share the strategies they used to persuade the foundation to make such a bold change to their measurement and learning system. We will share tips on how to pace the process to ensure that it creates appetite rather than groans, and how to build capacity among foundation staff and partners to use rubrics validly. Context is critical when evaluating change, particularly in difficult contexts where you would expect different results than in easier contexts. In our session, we will discuss how Laudes Foundation has taken context into account when evaluating its systemic change initiatives. We will share strategies for avoiding the fear of evaluation by emphasizing collective learning, support, and adaptation, rather than predetermined targets. This shift in mindset requires a move away from relying on colorful and meaningless KPI dashboards and towards using evaluative thinking, mixed evidence, and reasoning to assess important, meaningful, complex changes. One common misconception about rubrics is that all rubric ratings end up being quantitative. In our presentation, we will explain how Laudes Foundation has successfully used qualitative rubrics to support its systemic change initiatives. We will also share insights on how to shift mindsets towards evaluative thinking, which involves a focus on continuous learning, critical reflection, and the use of evidence to guide decision-making. Finally, we will discuss how Laudes Foundation has built capacity for rubric-based evaluation among its staff and partners. We will share practical insights on how to develop and deliver effective and engaging training materials on how to use rubrics to guide collective learning and adaptation.