Internal Evaluation
Lexi Jones, MS
Evaluation Coordinator
South Carolina First Steps, United States
Location: White River Ballroom E
Abstract Information: In recent years, South Carolina First Steps has reframed our approach to evaluation to encompass both formal and informal evaluation and to marry quantitative and qualitative data. For many years, evaluation has relied heavily on the value of quantitative data and the “power of numbers.” Evaluation focused on population-level data and statistical significance. An evaluator collected, analyzed, and reported results based on data gathered from participants. Manuscripts and reports presented numbers that were often detached from the people represented by these numbers. In this chapter of evaluation, participants had a passive role in the evaluation process. In recent years, a new narrative in the book of evaluation is emerging – one that uplifts the value of anecdata and the “power of story.” The roles and responsibilities of an evaluator are evolving to include data visualization and storytelling. Principles of storytelling respect the dignity and self-identity of the human experience behind the numbers. Rather than simply reporting on data, our organization is relating to data – bringing to life the stories behind the statistics. South Carolina First Steps is an early childhood state agency and 501(c)(3) nonprofit that focuses on preparing children for school and lifelong success. At South Carolina First Steps, we are rewriting the role of a “program participant” in the evaluation process. Rather than passively providing information, we are inviting participants to be actively engaged in the five phases of a feedback loop (i.e., design, collect, interpret, respond, close the loop). In 2020, we launched “Your Voice Matters”, an ongoing series of rapid-cycle, high-quality feedback loops that uses community voice to inform changes that promote continuous quality improvement within our organization. Based in principles of community-based participatory research and human-centered design, our organization is building equitable evaluation ecosystems that uplift the value, voice, and expertise of the individuals who participate in our programs. To date, South Carolina First Steps has completed 17 feedback loops with over 4,000 respondents from across South Carolina. We have gathered feedback from families participating in a school transition program, parents participating in our First Steps 4K program, childcare providers receiving technical assistance, and childcare providers implementing our school transition and First Steps 4K programs. At the core of our feedback work, we ask individuals three key questions: What are we good at? What can we do better? What ideas do you have to improve this program? Through “Your Voice Matters,” we are empowering individuals to shape evaluation processes and program changes that are designed by communities, for communities. This session will be an opportunity for evaluators to discuss how their organization is gathering, incorporating, and responding to client voice in their evaluations. The speaker will moderate a discussion around the role of “anecdata” in internal evaluation, co-designing evaluation processes with clients, and using principles of storytelling and data visualization to bring data to life.
Relevance Statement: The field of evaluation has evolved and expanded to capture the dynamic nature of the human experience. New tools, theoretical frameworks, analyses, and processes are emerging that blend the narrative and numeric impact of a program. Numbers are necessary, but not sufficient to drive change within an organization. Without an image or a story, numbers lack an emotional and compelling call to action. The proposed session will highlight an emerging process that uses client voice to make data relatable and actionable. Principles of participatory research and human-centered design will provide a foundation for the discussion. The concept of a rapid-cycle, feedback loop is an exemplar of the evaluator competencies of context and interpersonal interactions. A feedback loop is context-dependent, with results providing real-time data on participant experience and outcomes. The speaker will highlight how one organization was intentional in soliciting feedback from individuals who are often underrepresented in evaluation; communicating transparently the process, results, and action steps from the evaluation to multiple stakeholders; and strengthening relationships with community members by respecting the dignity and diversity of experiences in their community. The session will serve as a peer-to-peer learning opportunity for attendees to discuss innovative methods of engaging clients in evaluation. The speaker will share lessons learned from engaging participants in feedback loops within state government and non-profit settings. When program participants are invited to participate throughout the evaluation process, the evaluator must focus on two guiding principles – respect for people and common good and equity. The facilitator will share a short story that highlights how one organization “set the table” for conducting a rapid-cycle, feedback loop. As evaluators, we can think of ourselves as hosts to a dinner party – our guests may include program staff, grant funders, communications members, program participants, and other key stakeholders. As a “host”, we have a responsibility to create an atmosphere that welcomes diverse perspectives and experiences, while also ensuring the integrity and rigor of the evaluation. Throughout the process, evaluators facilitate conversations between “guests” that focus on transparency and clear communication, removing technical jargon that may disrupt conversations. As an evaluator, we may need to nurture our emotional intelligence so that we can balance and manage the multiple relationships and social interactions that can impact and contribute to the evaluation. The evaluator acknowledges the historical, environmental, cultural, and social contexts that may impact the current evaluation. The evaluator then synthesizes the multiple experiences and data and invites the “guests” to help shape the narrative of the story that is trying to emerge from the data.