Systems in Evaluation
Kirsten Kainz, PHD (she/her/hers)
Systems Transformation Consultant
self-employed
Wilmington, North Carolina, United States
Kirsten Kainz, PHD (she/her/hers)
Systems Transformation Consultant
self-employed
Wilmington, North Carolina, United States
Kirsten Kainz, PHD (she/her/hers)
Systems Transformation Consultant
self-employed
Wilmington, North Carolina, United States
Kathleen (Kate) Gallagher, PHD (she/her/hers)
Director of Research and Evaluation
Buffett Early Childhood Institute, University of Nebraska, United States
Kathleen (Kate) Gallagher, PHD (she/her/hers)
Director of Research and Evaluation
Buffett Early Childhood Institute, University of Nebraska, United States
Kathleen (Kate) Gallagher, PHD (she/her/hers)
Director of Research and Evaluation
Buffett Early Childhood Institute, University of Nebraska, United States
Location: Room 103
Abstract Information: Agent-Based Modeling is a simulation tool for visualizing and parameterizing the dynamic unfolding of complex phenomena that results from the interaction of agents in environments. In this skill building workshop participants will learn about a tool for facilitating interested parties through a systemic investigation of their change initiative and develop stories of what is happening based on a set of questions relevant to the design of an agent-based model. By the end of the workshop participants will have learned about key ideas in agent-based modeling, learned about a conceptual tool for facilitators based on agent-based modeling principles, and discussed using the tool related to their own or imagined equity-focused systems change initiatives.
Relevance Statement: The rationale for this skill-building workshop integrates principles and methods from complexity and systems science with evaluation principles and collective sense-making through storytelling. Specifically, we borrow our initial foundation from a convening of the Institute of Medicine of the National Academies of Sciences, Engineering, and Medicine in the United States (Institute of Medicine, 2014) that generated a set of recommendations for evaluating complex interventions. Those recommendations highlighted evaluative thinking (Buckley et al., 2015) and sense-making practices (Weick, 1995): introduce evaluative thinking early in project development; create and evolve theories of change as lessons are learned; replace Did it work? questions with more nuanced questions about what worked, what worked less well, what could be scaled, and what could be sustained; create an ecology of evidence use that includes strategies for learning from evidence to support systems change; and invest sufficient resources, time, commitment, trust, and relationships to ensure improvement. We build onto that foundation the application of agent-based modeling principles for understanding dynamic unfolding of complex phenomena such as social systems change initiatives (Willensky & Rand, 2015). And we complete the integration by rendering the key questions underlying agent-based modeling into a conceptual tool for facilitating sense-making and storytelling (McCracken et al., 2022) about systems change initiatives using soft-systems methods and critical systems methods (Checkland & Scholes, 1999; Ulrich & Reynolds, 2020). Participants in the workshop will learn about the foundations and integrations underlying the conceptual tool, and discuss applying the tool or other related tools for making sense of complex equity-focused change initiatives. Our experience suggests that when used with groups of people with interests in a system change initiative the conceptual tool moves storytelling beyond simple naming of initiative components to clearer statements about how agents and environments interact in ways that yield intended and unintended consequences. References Buckley, J., Archibald, T., Hargraves, M., & Trochim, W. M. (2015). Defining and teaching evaluative thinking: Insights from research on critical thinking. American Journal of Evaluation, 36(3), 375-388. Institute of Medicine. (2014). Evaluation design for complex global initiatives: Workshop summary. Washington, DC: The National Academies Press. https://doi.org/10.17226/18739 Weick, K.E. (1995), Sensemaking in Organizations, London, Sage. Wilensky, U., & Rand, W. (2015). An introduction to agent-based modeling: modeling natural, social, and engineered complex systems with NetLogo. Mit Press. Checkland, P., & Scholes, J. (1999). Soft systems methodology in action. John Wiley & Sons. Ulrich, W., & Reynolds, M. (2020). Critical systems heuristics: The idea and practice of boundary critique. In Systems approaches to making change: A practical guide (pp. 255-306). Springer, London. McCracken, J., Wolff, L., Murikumthara, D., Fyfe, R., Snow, T., & Dusseldorp, T. (2022). Places, systems and stories: The role of storytelling in place-based initiatives. AQ-Australian Quarterly, 93(2), 8-40.