Director of Evaluation & Learning The Audacious Project, TED, Portugal
Abstract Information: Evaluation and films have a long yet unsatisfying history together. We have seen it mostly either on the inscrutable low ratings for great movies, or on the unfathomable criteria used in awards ceremonies. But as audio-visual content gains ground and becomes the undisputable leader in how ideas are spread, it becomes urgent to go beyond “is it any good?”. It is time to ask: “how good is it?”, “for whom?”, and “why?”. The idea of communication as a tool for social change has been around for several decades. It became particularly popular on social marketing campaigns for public health. In recent years, it spread to other formats, including films and documentaries on topics as diverse as climate change, human rights, or animal welfare. The perceived effect some documentaries might have had on awareness and policy change seemed to have sparked an emerging field within the film industry: impact production. That said, and while impact producers are increasingly more common in film production teams and guides on how to produce impact have been written and shared widely, a question seems unanswered: did the “impact produced” had any impact? We argue that without evaluation we are left at best with a list of good intentions. In this session we will present the case of the film “Where the Waves Break” and how evaluative thinking and evaluation were used throughout the different phases of its production. “Where the Waves Break” is a documentary feature film about migration, belonging, and race that premiered in 2023. In its first phase (development), the team developed the film’s preliminary Theory of Change (ToC). The ToC was instrumental in both development and pre-production, as it helped refine the target audiences and the list of potential partners, as well as to develop the script and define where the film would be shot. During post-production, the team purposefully sampled and surveyed a group of representatives of target audiences to test which messages they would identify and how they would interpret them. That informed the finalization of the film and the distribution strategy. Alongside film festival exhibitions, the film was shown to target audiences and followed by roundtables with specialists on the topics. Audience was surveyed on their knowledge, attitudes, and intended behavior. In this session we will present the emerging findings, discuss the state of the art in film evaluation, and we will bring up both the challenges faced by the team and how they have tried to overcome them.
Relevance Statement: We believe this topic is relevant for the evaluation field, and particularly to the Arts, Culture, and Museums TIG, because while the idea of communication as a tool for social change has been around for several decades, we believe not enough attention has been paid to evaluating it – particularly films or documentaries. Exemplary of that: a preliminary search for the keywords “film” and “documentary” on both the American Journal of Evaluation and New Directions of Evaluation retrieved no articles on evaluating films. On the other hand, the concept of documentaries as a tool for social change has gained quite a lot of traction in recent years outside of the traditional evaluation circles. One good example of that is DocSociety’s Impact Field Guide & Toolkit, originally published in English and since then translated into seven other languages. And while the guide does talk about the importance of evaluation, there is no evidence that evaluations have been done systematically. We believe evaluators should be part of this discussion and we think this presentation may contribute to it.
