Skip to main content
Evaluation 2023 Main banner
Icon Legend
Presentation Icons
PD Workshop
PD Workshop
Virtual Offering
Virtual Offering
Presidential Strand
Presidential Strand
Plenary Session
Plenary Session
BOF
BOF
Ignite
Ignite
Back

Qualitative Methods

Creating Impact Stories for Federally Funded Capacity Building Centers

Friday, October 13, 2023
3:45 PM – 4:45 PM ET
Location: Grand Ballroom 8

Location: Grand Ballroom 8

Abstract Information: The US Department of Education (ED) funds a set of Comprehensive Centers to provide evidence-based, capacity-building support to State Educational Agencies. ED encouraged Center evaluators to develop Impact Stories to describe how Center support projects are affecting change in educational policy and practices. As the evaluators for the Region 5 and Region 14 Comprehensive Centers, the presenters on this panel will discuss how they employed various approaches to collecting compelling evidence to demonstrate how Center efforts support educators, administrators, and students. Two papers employed a story circle methodology to solicit participant perspectives of impact, while a third leveraged existing quantitative data, new interview data, and strategic communication support to tell a story of impact. The Impact Stories we created are short, engaging, and compelling documents that convey these stories through the voice, perceptions, and experiences of participants. We will discuss how the different methodologies (i.e., story circles and mixed method data analysis) guided our creation of meaningful and exciting evaluation finding briefs that are responsive to funder requirements and highly accessible to non-technical audiences. Session attendees will walk away with a general understanding of and practical advice around employing our approaches to contextualize, enhance, and convey evaluation stories.

Relevance Statement: Evaluations are meant to be understood and consumed to aid in program improvement and evidence-based decision-making (CDC, 2023). While numbers and statistical significance are important, humans are motivated by stories. However, moving beyond more traditional ways of reporting evaluation results to tell those stories to a broader audience is often challenging. This proposed panel presents and discusses two approaches to overcoming this challenge to evaluation storytelling: a story circle methodology approach and a mixed methods integrated partnership with communications professionals. Created by John O’Neal, a Story Circle gathers people together, through a facilitated discussion, to tell personal stories and share personal experiences, in a safe and respectful setting (National Public Housing Museum, n.d.a). It was created as an alternative to theater evaluation but has implications and use across program evaluation efforts (National Public Housing Museum, n.d.b). “Story Circles differ from focus groups, another group process for evaluation that can be generative, in at least two key ways: the democratic structure of Story Circles gives everyone equal time, and Story Circles center the story as a narrative form, rather than inviting participants to answer a series of questions (Imagining America, 2018).” Story circles leverage stories as a powerful human expression, accommodate diverse voices and perspectives, and document findings that emerge organically from participants rather than from preconceived protocols or instruments (Sukop, Torbin, and Fischman, 2007). Two presenters used this methodology to document the perceived impact of technical assistance on education practice challenges. We developed impact stories that rely heavily on participant voice, providing less analysis and synthesis than traditional qualitative efforts. The first two presentations will discuss how we executed story circles, one in an in-person, and one in a hybrid setting. We will provide practical advice about how to execute a story circle, including participant guidelines and troubleshooting during data collection. We will also discuss lessons learned in employing an innovative storytelling technique and recommendations for making storytelling a part of evaluators’ toolkits. The third presentation will also discuss a more traditional mixed methods approach. This approach included evaluation and communications experts forming a collaborative partnership which informed both the data collection approach and development of the final Impact Stories. First, the evaluation team conducted quantitative survey data as part of the larger Comprehensive Center evaluation. Next, both evaluation and communication team members used the quantitative results to inform the development of specialized qualitative protocols used by both evaluation and communication team members. Once complete drafts were available, discussion and revision occurred in collaboration with the project respondents to ensure accuracy, representation, and voice within the stories. Centers for Disease Control and Prevention (CDC). (2023). https://www.cdc.gov/evaluation/index.htm Imagining America. (2018). Story Circles as Ongoing and Collaborative Evaluation—Roadside Theater’s “Story to Performance” – Assessing the Practices of Public Scholarship. https://epistemicjusticeiarslce2018.files.wordpress.com/2020/04/story-circles-as-ongoing-and-collaborative-evaluatione28094roadside-theatere28099s-_story-to-performance_.pdf National Public Housing Museum. (n.d.a). https://www.nphm.org/story-circle-toolkit National Public Housing Museum. (n.d.b). Story Circle Toolkit. https://static1.squarespace.com/static/5356a3bfe4b04b5c0a1dca51/t/5fd7981121747156cf02f035/1607964690317/StoryCircleToolkit112420.pdf Sukop, S., Tobin, J., and Fischman, G.E. Storytelling Approaches to Program Evaluation: An Introduction. http://korwinconsulting.com/wp/wp-content/uploads/2015/10/StorytellingApproachestoProgramEvaluation_Final1.pdf