President and Senior Evaluator Pyxis Management, Inc., California, United States
Abstract Information: Throughout its history, the evaluation field has developed numerous theories about how evaluation can and should be practiced. Many evaluation theories are informed by stories written by experienced and knowledgeable evaluation practitioners. These stories are integral to the knowledge of the discipline and are often viewed as the field’s collective understanding of how evaluation can and should be practiced. Yet, research suggests that the influence of evaluation theories on evaluation practice is minimal. So, what knowledge does inform the day-to-day practice of evaluators?
Many theorists propose that evaluation practice is influenced by practical knowledge inherent to the doing of evaluation. Some have even argued that practical knowledge has a greater influence on evaluation practice than evaluation theories in some instances. However, little is known about practical knowledge evaluation suggesting much of the knowledge about our field remains untapped.
A multiple-methods research design aimed to address this gap by exploring practical knowledge in evaluation and how it’s integrated into evaluator education. Qualitative and quantitative data collected through document review and the repertory grid technique suggested a holistic epistemology of evaluation practice grounded in four interrelated ways of knowing. Additional insights into the underlying dimensions of these ways of knowing, including knowing in action and expressions of knowing, were also interpreted. Qualitative data collected through semi-structured interviews suggested that these ways of knowing are reflected in the sample of evaluation educational programs, albeit to varying degrees.
Collectively, these findings provide new insights into an underlying epistemology of evaluation practice that recognizes practitioners as knowers and knowledge creators and reframes the connection between theory and practice. The findings also raise questions about the preparation of future evaluators, including what aspects of practice are more or less valued by educational programs.
Relevance Statement: In his seminal book, Evaluating with Validity, House (1980) claims that “the social import of evaluation is enormous; its self-understanding relatively minute” (p.11). Since then, the field has grown in many ways, including the emergence of new evaluation theories and advancements in professionalization. Yet, the extent to which self-understanding has improved is unclear.
Some theorists propose that practical knowledge is essential in evaluation practice (e.g., Christie, 2003; Christie & Masyn, 2010; Shadish, W. R., & Epstein, 1987). Schwandt (2008) describes practical knowledge as “the tact, dispositions, and considered character of decision making called for in various situations faced in ‘doing’ the practice” (p. 33). It is informed by stories practitioners create as they interact with and theorize about problems and learn from experience. Despite its perceived importance to evaluation practice, limited research has been conducted on practical knowledge in evaluation, which may contribute to a lack of self-understanding within the field.
Recently, dissertation research conducted by the author helped address this gap by investigating practical knowledge in evaluation, specifically focusing on the ways of knowing that underlie and guide practitioners, and the presence of these ways of knowing in evaluator educational programs. Quantitative and qualitative data were collected from various sources using a multi-methods design (i.e., document review, repertory grid method, and semi-structured interviews).
During the presentation, attendees will be introduced to a holistic epistemology of evaluation practice based on four interrelated ways of knowing: knowing self, knowing others, knowing the discipline, and knowing the common good and equity. Important insights into the underlying dimensions of the ways of knowing, including creative processes that actuate knowing and methods evaluators use to express what has become known through their practice will also be presented. Last, the presenter will discuss how this epistemology is integrated into evaluation educational programs, albeit to varying degrees.
This presentation will tell a new tale about the epistemology of evaluation practice that challenges existing assumptions about how the field conceptualizes theory, practice, and knowledge construction. Attendees will gain new insights into the importance of practical knowledge in evaluation practice, the role of practitioners as theorists and knowers, and the relationship between theory and practice. Collectively, this information represents another step in the field's journey towards self-understanding and will position attendees to appreciate the unique nature of evaluation practice, stimulating new thinking about the knowledge of evaluation, professionalization, and evaluator education.
Christie, C. A. (2003). Editor’s notes. In C. Christie (Ed.), The Practice‐theory relationship in evaluation. New Directions for Evaluation, 97, 1–6
Christie, C. A., & Masyn, K. (2010). Latent Profiles of evaluators’ self-reported practices. Canadian Journal of Program Evaluation, 23(2), 225–254
House, E. R. (1980). Evaluating with validity. Sage Publications
Schwandt, T. A. (2008). The relevance of practical knowledge traditions to evaluation practice. In N.L. Smith & P.R Brandon (Eds.), Fundamental issues in evaluation (pp. 30–41). Guilford Press
Shadish, W. R., & Epstein, R. (1987). Patterns of program evaluation practice among members of the evaluation research society and evaluation network. Evaluation Review, 11(5), 555–590.
