Theories of Evaluation
Eric Einspruch, Ph.D.
Principal
ELE Consulting, LLC, United States
Jennifer Billman, Ph.D.
Assessment Coordinator & Associate Professor of Biology
HACC, Central Pennsylvania's Community College, United States
Location: Room 202
Abstract Information: AEA 2023 invites us to explore how our individual and collective stories contribute to and shape the narrative of evaluative thinking and practice. The stories we live, engage, and retell reflect our understandings of reality (ontologies) and our ways of knowing about that reality (epistemologies). Grounded in varying ontological origin stories, these ways of knowing include empirical, traditional, and revealed knowledge, and each contributes to the narrative of our evaluations. Though often viewed in conflict with one another, these knowledges exist as a Knowledge Trinity (Billman, 2019, 2023), equal yet differentiated (Goodchild, 2021). Viewed this way, these knowledges provide us opportunities to engage across ontologies and epistemologies in search of spaces that foster connection rather than separation (Goodchild, 2021). In search of these connection spaces, this Think Tank invites participants into the ethical space (Ermine, 2007) between ontologies and epistemologies to explore the hidden, deeper level thoughts, interests, and assumptions (p. 195) underpinning evaluative thinking. During this Think Tank, we will 1) review the ontological origin stories of evaluative thinking, 2) explore the three types of knowledge and their intersections, and 3) review examples pulled from the published literature that demonstrates how each knowledge story influences evaluative thinking. Participants will leave with increased understanding of the importance of ontological considerations in their work and practical ways to engage multiple ways of knowing in their evaluations.
Relevance Statement: On February 27, 2023, the System in Evaluation Working Group of the European Evaluation Society sponsored a debate, In Search of the Golden Mean, between Michael (Mike) C. Jackson and Michael Quinn Patton. During this debate Jackson articulated Critical Systems Thinking’s use of multiple truths reflecting multiple ontologies and epistemologies to solve complex problems. Patton emphasized the need for evaluators to shift their unit of analysis from projects and programs to systems to advance transformational changes. To make this shift, evaluators must understand the theoretical, epistemological, and ontological origins of the different approaches. We must ask the following types of questions: How does practicing without this understanding impact the stories our evaluations’ tell? How do we understand our own ontological and epistemological stories, both individually and collectively? How do we know if our methodologies are privileging one story over another? To answer these questions, we argue that we must understand our ontological origin stories and the epistemologies emerging from them in order to have any hope that our evaluative work can address the complexities of the polycrisis we now face.
How one approaches the systems complexities of this polycrisis reflects one’s held ontology and epistemology. Too often, transformational systems change gets delayed because individuals holding different ontologies and epistemologies struggle to interact in ways that lead to productive outcomes. In this Think Tank we apply Legal Cree scholar Willie Ermine’s “ethical space” framework to the intersection between ontologies and their related epistemologies within our evaluative thinking. Ermine argued that despite surface conversations, “what remains hidden and enfolded are the deeper level thoughts, interests and assumptions that will inevitably influence and animate the kind of relationship the two can have” (p. 195). This ethical space brings differing world views (i.e., ontologies) into productive dialogue with each other.
Here, we create an ethical space for exploration of the ontological origin stories of the major evaluation paradigms and their varying epistemologies, traditional knowledge, revealed knowledge, and empirical knowledge. Seeking a dialogue in the ethical space between these epistemologies, we explore what we already know about their use in evaluation through the published literature to identify practical steps evaluators can take to conduct evaluations in this ethical space.
Ermine, W. (2007). The ethical space of engagement. Indigenous Law Journal, 6(1), 194–203. https://jps.library.utoronto.ca/index.php/ilj/article/view/27669
Goodchild, M. (2021). Relational systems thinking: That’s how change is going to come, from our earth mother. Journal of Awareness-Based Systems Change, 1(1), 75–103. https://doi.org/10.47061/jabsc.v1i1.577
European Evaluation Society, System in Evaluation Working Group. (2023, February 27). In Search of the Golden Mean [Video]. YouTube. https://youtu.be/lnyfpC8E4O8