Cluster, Multi-site and Multi-level Evaluation
Kathleen O'Connor, PhD (she/her/hers)
Consultant, Evaluation and Learning
TCC Group
Philadelphia, Pennsylvania, United States
Lisa Frantzen, MBA
Associate Director, Evaluation & Learning
TCC Group, United States
Location: Room 206
Abstract Information: As evaluators, we are storytellers. We tell stories that are rooted in data provided by others, stories that are not our own. It is critical, then, that we put careful thought into making sure we tell these stories in effective ways and to the right audiences. Effective storytelling methods and relevant audiences differ based on contextual factors. We must ask: Whose story are we telling? Who should hear the story? What are the most effective ways to share a particular story with different audiences? There is no one-size-fits-all approach to reporting on evaluation findings. In this roundtable session, we will share our experience evaluating a mentor-led, Community of Practice-based capacity-building program over the past three years and the distinct methods we employed to understand and tell the stories of (1) the individual experiences of program participants and (2) the collective experiences of each Community of Practice. We used various methods to center the stories of individual participants and of the Communities of Practice, including journey mapping and a case study approach. Participants in this session will walk away with concrete examples of storytelling decisions and practices. We will briefly present our approaches and lessons learned and will then facilitate a conversation with session participants. We will invite participants to share the ways they have approached reporting evaluation findings that tell the stories of participants at individual and collective levels. We will ask: What methods have you found to be effective? In what ways have you adapted your reporting for different audiences? How have you involved participants in the meaning making and storytelling process?
Relevance Statement: As evaluators, we are storytellers. We tell stories rooted in data provided by others, stories that are not our own. It is critical, then, that we put careful thought into making sure we tell these stories in effective ways and to the right audiences. Effective storytelling methods and relevant audiences differ based on contextual factors. We must ask: Whose story are we telling? Is it an individual’s, a cohort’s, an organization’s, an initiative’s? Who should hear the story? What are the most effective ways to share a particular story with different audiences? There is no one-size-fits-all approach to reporting on evaluation findings. Evaluating capacity-building programs that are implemented within a cohort or Community of Practice potentially requires measurement of outcomes at multiple levels: the individual participants, the cohorts or Communities of Practice, the organizations for whom the participants work, and the communities within which the participants belong. This calls for careful study design and implementation; it also calls for unique and creative reporting methods to ensure that the stories of individual and collective experiences and outcomes are told in ways that remain true to both types of experiences. In this roundtable session, we will share our experience evaluating a mentor-led, Community of Practice-based capacity-building program over the past three years and the distinct methods we employed to understand and tell the stories of (1) the individual experiences of program participants and (2) the collective experiences of each Community of Practice. We used various methods to center the stories of individual participants and of the Communities of Practice, including journey mapping and a case study approach. As a field, we ought to be moving toward more equitable evaluation practices. How and with whom we share evaluation findings is an important piece of this. As the tellers of others’ stories, we should continually seek out ways to effectively lift the voices of those in the communities in which and with which we conduct evaluations. Considering the ways we share evaluation findings so that they accurately and adequately tell the stories of particular stakeholders – stories of individual or collective experiences – is critical to this. Also critical is ensuring that the reporting methods we use to tell these stories are relevant and appropriate to the particular audiences with whom we are sharing. This requires, first, identifying the appropriate audiences and then, tailoring reports to those audiences. Participants in this session will walk away with concrete examples – provided by us and by other session participants – of these storytelling decisions and practices. We will briefly present our approaches and lessons learned and will then facilitate a conversation with session participants. We will invite participants to share the ways they have approached reporting evaluation findings that tell the stories of participants at individual and collective levels. We will ask: What methods have you found to be effective? In what ways have you adapted your reporting for different audiences? How have you involved participants in the meaning making and storytelling process?