Youth Focused Evaluation
E. Jane Davidson, Ph.D.
Founder and Principal Consultant
Real Evaluation, United States
Thomaz Chianca, PhD (he/him/his)
Managing Partner
COMEA Relevant Evaluations
Rio de Janeiro, Rio de Janeiro, Brazil
Thomaz Chianca, PhD (he/him/his)
Managing Partner
COMEA Relevant Evaluations
Rio de Janeiro, Rio de Janeiro, Brazil
Nicole Dulieu, M.A.
Head of Evidence & Learning
Save the Children, United States
Margarita Franco, MA (she/her/hers)
Global Sponsorship Program Quality Advisor
Save the Children, United States
Erika Barrera, Juridical Science (she/her/hers)
Program Coordinator
Save the Children
San Salvador, San Salvador, El Salvador
Location: Room 102
Abstract Information: Save the Children’s Sponsorship program supports children, their families, and communities in 20 countries to create change together so that all children survive, learn, are protected, and live free from discrimination. A priority and challenge is centering the voices of children as co-designers of the change and capturing their own stories about the outcomes affecting them. Sponsorship’s new rubric-enhanced MEAL system, currently being piloted in El Salvador, Zambia, and the Philippines, provides a nuanced way of capturing important, intangible outcomes such as children’s experiences of inequality and discrimination, equitable family environments, and empowered communities creating change for and in collaboration with children. Rubrics, applied correctly, can help shift power to communities for design and evaluation because they describe lived reality (as a progression from the current to the aspirational state) rather than representing it with abstract indicators. The El Salvador team will share how they contextualized the rubrics for their local culture and context and then used them with children’s, families’, and community stories to assess the baseline situation. This helped them rethink their initial co-designed program and create plans to capture evidence of transformational change as it emerges. Rubrics-enhanced evaluation lends itself particularly well to telling a compelling story for visual and verbal readers, weaving together a diverse mix of evidence (stories, photos, and numbers) in a way that engages every kind of audience. Finally, we will discuss how to get buy-in for a switch to this approach within an organization that is heavily invested in logframes and indicators.
Relevance Statement: Evaluators seeking to unleash the power of story are faced with a difficult conundrum. Stories are rich and powerful sources of evidence, but it can be hard to clearly show how well change is progressing over time. Rubrics methodology provides a breakthrough here. A rubric describes what the situation looks like at different levels of performance on an outcome. When the main evidence is stories, rubrics describe what kinds of stories we will see if the situation is at different performance levels. Sponsorship’s community development efforts focus on the children most impacted by inequality and discrimination. The rubrics used to assess change describe the kinds of stories we would typically expect to hear at five levels, for example: • Children are fully included, valued, and supported to thrive. • Children are substantially included and free from discrimination. • Children are nominally included but many still experience barriers. • Children experience little overt discrimination, but social exclusion is a significant issue for some. • Some children are ostracized, abandoned, or experiencing abuse or serious discrimination. Presenters from the El Salvador country office will share their experiences of applying the rubrics to understand the local issues and to establish a baseline for the forthcoming 10 years of work supporting those communities. Key to this was the process of contextualizing the rubrics for the local culture and context and gathering story-based evidence to assess the prevalence and severity of children’s experiences. Contextualization involved working with children to identify which groups were most impacted by inequality and discrimination in the impact area and then listening to their (and their caregivers’ and others’) stories of social exclusion, discrimination, abuse, and abandonment – as well as their stories of inclusion and support. Country office teams were then coached in ‘sensemaking’ processes where diverse sources of evidence were synthesized alongside the rubrics to provide a clear assessment of how harmful or conducive the situation was for children. Participants will learn how rubrics can be designed to provide a clear, shared language about outcomes while still allowing room for nuanced contextualization to the local culture and context. We will share the main challenges we have faced to develop and pilot the rubrics in real-life complex contexts. Rubric-enhanced evaluation lends itself particularly well to storytelling; they help convert rich, story-based evidence into easily visualizable ways of showing trackable progress. By weaving together a diverse mix of evidence, including stories, photos, and numbers, rubric-enhanced evaluation provides a compelling and comprehensive – yet concise – story of progress towards transformational change in the areas an initiative seeks to influence. This is particularly important for engaging stakeholders and funders, who are often more responsive to stories than to dry data but who love to see trackable progress. Save the Children’s Head of Evidence and Learning will share how to obtain support for adopting a rubrics-based approach in an organization that is heavily invested in logframes and indicators. Plenty of time will be available for questions in this lively, interactive demonstration session!