19 - Once upon a program: Using the the SCM to tell a program's story

Abstract Information: Every evaluation tells a story. We (evaluators) find elements of characters (e.g., sponsors, implementers, and participants); settings (i.e., social and organizational contexts); plot (i.e., the historical, current, and anticipated program circumstances); conflict (usually a decision to be made about a program); and a theme (the reason for a program to exist). However, often evaluations are told in a way that does not make use of the full power of storytelling, although good stories help us relate to and make sense of the world around us (Anderson, 2010; Bietti et al., 2019). The Success Case Method (SCM) is a rapid impact evaluation approach that is widely used to tell the story of training and development success. Participants will be introduced to a case through which facilitators weave in-depth lessons and group activities. After providing an overview of the SCM, its main elements, and innovations, we will focus on how individual stories identified via questionnaires and interviewing contribute to telling the broader program story. Stories featured in this workshop exemplify women issues and racial issues in organizational learning contexts; we will illustrate how SCM is able to give voice to marginalized groups within an organization by using the independent evaluator narrator. The facilitators bring more than 20 years of experience in training and development. Together, they recently developed a SCM certification for practitioners with no evaluation experience. This workshop is designed for evaluators who want to extend their toolkit and include the SCM as one approach in their repertoire.

Relevance Statement: Relevance Brinkerhoff’s (2003) Success Case Method (SCM) is not a new method but has evolved over more than four decades. Widely practiced in training and development contexts, the approach has also been used for program evaluation in international development, social service, and education contexts (e.g., Coryn et al., 2009; Piggot-Irvine et al., 2009; Olson et al., 2011; Medina et al., 2015). As such, it remains an important approach, especially in situations when randomized controlled trials or quasi-experiments are not a feasible option for impact evaluation. Implications for theory and practice The workshop will integrate new theoretical insights based on recent interviews with Robert Brinkerhoff to situate the SCM approach in the broader “Garden of Evaluation” (Montrosse-Moorhead, Schroeter, & Becho, 2023). It also draws on the facilitators’ experiences in implementing success case studies for corporate, multilateral, and nonprofit organizations around the globe highlighting shifts in the way the SCM is practically realized in response to technological changes and shifts in learning design during the pandemic. Insights into new developments and adaptations of the method in relation to leadership development interventions will be shared with particular attention to increased efficiencies and changes in human interaction. Value Target groups for this session include graduate students, researchers, and evaluators with little or no prior knowledge of the method. Practitioners, organizational development professionals, trainers, and research and evaluation administrators and sponsors who wish to expand their knowledge and use of SCM may also benefit. By the end of the session, participants will understand SCM theory and methodology, and practice key elements of designing, implementing, and weaving SCM into rapid impact evaluations. As such, the workshop is closely aligned with the Guiding Principles for Evaluators and builds Evaluation Competencies across categories. SCM is a nimble evaluation approach that allows for fast identification of impact barriers and opportunities for program improvement during the development and implementation stages. This workshop features new insights into theoretical aspects of the SCM. At the same time, we share practical knowledge and build capabilities to implement SCM in practice. The field of evaluation benefits in that the SCM has been updated to account for theoretical and methodological innovation that benefits practitioners. References Anderson, K. E. (2010). Storytelling. In H. J. Birx (Ed.), 21st Century Anthropology: A Reference Handbook (pp. 277-286). London, UK: SAGE Publications. Bietti, L. M., Tilston, O., & Bangerter, A. (2019). Storytelling as adaptive collective sensemaking. Topics in cognitive science, 11(4), 710-732. Brinkerhoff, R. O. (2006). Telling Training's Story: Evaluation Made Simple, Credible, and Effective. San Francisco, CA: Berret-Koehler Publishers, Inc. Brinkerhoff, R. O. (2003). The success case method: find out quickly what’s working and what’s not. San Francisco, CA: Berret-Koehler Publishers, Inc. Coryn, C. L., Schröter, D. C., & Hanssen, C. E. (2009). Adding a time-series design element to the success case method to improve methodological rigor: An application for nonprofit program evaluation. American Journal of Evaluation, 30(1), 80–92. Montrosse-Moorhead, B., Schroeter, D., & Becho, L. (2023, March 16). The Garden of Evaluation Approaches. https://doi.org/10.17605/OSF.IO/MKXG