Use and Influence of Evaluation
Veridiana Mansour Mendes, n/a
Evaluation Officer
WIPO, Brazil
Adan Ruiz Villalba, n/a
Head of Evaluation
WIPO, Spain
Julia Engelhardt, n/a
Senior Evaluation Officer
WIPO, Ecuador
Location: Room 309/310
Abstract Information: If we could draw an analogy between evaluation and storytelling, an ideal evaluation process would be a classic quest story told in a liner narrative with a happy ending. In this story, evaluators would be protagonists, as they take the lead and confront multiple challenges along the way to deliver a credible and useful evaluation. Evaluands would be sidekicks (aka “allies to the protagonists”) because they are the evaluation’s main counterparts and beneficiaries, but cannot affect its independence. The happy ending would be the use of the evaluation by evaluands and other relevant stakeholders (aka “meaningful extras”).
In practice, evaluators do not always manage to tell this story. We, evaluators, face many difficulties to confront challenges and achieve the planned happy ending; what is more, we often place responsibility on evaluands for our frustrations at challenging, unproductive and unused evaluations. “They lack commitment” or “they resist changes” are evaluators’ common complaints against evaluands, and many of us have already developed negative biases towards them. If truth be told, sometimes we treat them as the antagonists of an evaluation – or maybe even frequently.
The issue is that despite doing our best to promote evaluands’ interest and participation, we may have never fully treated them as our sidekicks. Research shows that people tend to reject what they do not understand; this naturally applies to evaluators an evaluands. Therefore, to be and act as allies, sidekicks must know and comprehend what protagonists do….but do we really promote evaluands’ knowledge and understanding about evaluation? Do evaluands truly know what is and what is not an evaluation? Do evaluands fully understand the process and the jargon we use? According to experience, the answer to these questions is probably “no”.
In view of that, we may have our share of responsibility for turning evaluands into antagonists; this means that we also have some power to revert it to start telling more stories with happy ending. In this context, the WIPO Evaluation Section has been using storytelling techniques in different stages of the evaluation process to develop evaluands’ knowledge and understanding about evaluation and its benefits, and hence build their commitment to become real allies and satisfied users. Pilot initiatives showed that simple actions such as eliminating jargon, explaining relevant concepts with analogies, and simplifying evaluation frameworks and tools with storytelling techniques and behavior insights could lead to positive results in terms of participation, engagement and use.
During this skill-building workshop, participants will: i) be introduced to the WIPO Evaluation Section’s approach to enhance evaluands’ commitment to evaluations through storytelling and behavior insights; ii) learn storytelling techniques that can help transform evaluands into sidekicks when used in the beginning of the evaluation process (e.g., theory of change); and iii) apply storytelling techniques in a hypothetical evaluation’s theory of change to engage evaluands in the evaluation process.
Relevance Statement: How to improve evaluation use and influence? Evaluators from all over the world have been trying to answer this question. From participatory approaches to mechanisms to follow-up the implementation of recommendations, significant progress has been made in this regard. Yet, the challenge remains.
This is not surprising. To some extent, we all fail to act on evidence. How many of us have postponed that promise of exercising more or smoking less, even after reading many research papers on the benefits of healthy habits? Deep inside, we all know that having evidence is not always enough to promote changes, but it does not mean that we should stop seeking solutions to improve evaluation use and influence.
We have done a lot already. We employ a variety of approaches, methods and tools available in the field of social sciences. We use several strategies to engage stakeholders throughout an entire evaluation process. We develop impressive infographics. We produce neutral and unbiased technical reports to provide stakeholders with quality evidence. We have multiple mechanisms to monitor the implementation of recommendations. So, what else can we do?
What if we rethink the role of the evaluand in the evaluation process? What if we step out of our comfort zone and start building their commitment with better stories? The power of storytelling to influence actions has already been proved by neuroscience. Paul J. Zak (2015), for example, found that “compelling narratives cause oxytocin release and have the power to affect our attitudes, beliefs and behaviors”. Jaaskelainen et al (2020), for instance, found that “high inter-subject correlation of brain activity during narrative processing seems to predict the efficacy of a narrative” in influencing decision-making.
The WIPO Evaluation Section is trying to transform evaluands into evaluators’ sidekicks without jeopardizing the independence of the evaluation. We piloted storytelling techniques in different stages of recent evaluation processes to enhance evaluands’ knowledge and understanding about evaluations, and we noticed that they became more enganged and committed. We have indications that this commitment have contributed to increase evaluation use and influence, but we are not able to prove the causal relationship between the pilot approach and evaluation use because we only have a few examples. Since we do not conduct many evaluations per year, we would like to share our experience with other evaluators and develop their skills on this approach to have more examples that in the future could either confirm or reject our hypothesis. If confirmed, we strongly believe that this could bring value to the evaluation community.