Research on Evaluation
Melvin Mark, PhD
Professor Emeritus of Psychology
Penn State University, United States
Melvin Mark, PhD
Professor Emeritus of Psychology
Penn State University, United States
Melissa Goodnight, PhD
Assistant Professor of Educational Psychology
University of Illinois, United States
James Herbert, PhD
Associate Professor
University of South Australia, Australia
Karen Kirkhart, PhD
Professor Emerita
Syracuse University, School of Social Work
Syracuse, New York, United States
Abstract Information: Evaluation’s potential for making a positive difference is an important part of the rationale for the field. Classic concepts of evaluation use, and more recent notions about evaluation influence – as well as the research on evaluation (RoE) associated with these topics --are visible signs of evaluators’ collective concern for what (if anything) evaluation brings about. Kirkhart’s (2000) seminal paper on evaluation influence provided a model with three dimensions: source (i.e., whether evaluation results or evaluation processes stimulated an evaluation consequence); intention (whether the consequence was intended or unintended); and time (whether the consequences emanate from the onset of the evaluation or from its end or even later). Henry and Mark (2003) and Mark and Henry (2004), with an expansion by Mark (2006, 2011), describe and organize into categories a set of change processes that can be used to describe the pathway from an evaluation to its consequences. One reason for this influence process model – and a clear connection to the conference theme – is that it enables the telling of a more complete story of how an evaluation led to its downstream consequences. (In contrast, familiar types of evaluation use focus on specific parts of the story; e.g., process use focuses on the beginning of the story while instrumental use focuses on the end). Recently Goodnight (in press) integrated the Kirkhart and the Mark/Henry frameworks and included two other dimensions: (1) motivation or reasons for engaging in an evaluation, such as social betterment (see Henry & Mark, 2003) or social justice for culturally marginalized groups and (2) context, including language, history, culture and politics. This panel describes and illustrates how concepts from the evaluation influence literature can be employed to tell a more complete story of a program, its evaluation, and the consequences of the evaluation. A subheading in Melissa Goodnight’s (in press) recent paper captures the panel’s focus: “Finding the language to tell an evaluation story.” In this panel, Goodnight will present her integrated influence model. She will draw on her research on a large-scale monitoring and evaluation system for primary schools in rural districts across India, highlighting the importance of context in telling the story of an evaluation through RoE. In the second presentation, James Herbert will draw on (1) the Mark-Henry evaluation influence model, (2) his own research on the influence of the evaluations of two child protection programs, and (3) other research on evaluation influence. Herbert will address methodological challenges and options for studying evaluation influence, as well as highlight lessons to date from such research. In the third presentation, Mel Mark will offer suggestions for methods for studying evaluation influence, both for efforts to tell the (relatively) full story and for “short stories” that focus on select segments of a lengthier influence pathway. He will also highlight the importance of considering evaluation influence both to enrich and to tell the broader story of evaluation and its consequences. Karen Kirkhart, a key contributor to the literature on evaluation influence, will serve as discussant.
Relevance Statement: “Evaluation influence” includes consequences of evaluation that “evaluation use” may not. This is one reason that, in addition to research on evaluation use – historically one of the largest and most notable topics examined in research on evaluation – it is important to add further research on evaluation influence. Research on evaluation influence is illustrated and discussed in this session, as are different research questions one might examine about evaluation influence – in part in the hope of helping stimulate additional research on evaluation aimed at this topic. As the presentations note, the evaluation influence literature includes models that characterize what about an evaluation contributes to the evaluation consequence, what the processes are that might be involved in bringing about that evaluation consequence, and what the context is in which this evaluation and its consequence come about. Incorporating these ideas in studies of evaluation influence should help us tell a more complete story of a given evaluation and for sketching the story of evaluation in general and its benefits. Evaluation influence will typically be more difficult to study than evaluation use. In the case of more direct evaluation use, people near the program – especially the funders, staff, or clients – often will be able to tell you how the evaluation was used (if it was). Evaluation’s contribution might be also apparent in the archival record (e.g., if the evaluation is cited during a legislative deliberation). For more remote cases of evaluation influence, however, different methods are likely to be needed. One benefit of this session, then, and an important one for those interested in research on evaluation, is to highlight methods and research questions appropriate for the study of evaluation influence. Generating more stories of broader evaluation influence is also vital for communicating the value of evaluation to potential evaluation sponsors. Otherwise, we risk underselling the benefits of evaluation. In addition, research on evaluation influence has promise for helping to guide evaluation practice. Certain practices may contribute to evaluation influence more than others, and growing the evidence base about influence will help identify such practices, ideally with evidence about the circumstances in which they are more or less beneficial.
Presenter: Karen E. Kirkhart, PhD – Syracuse University, School of Social Work
Presenter: Melvin M. Mark, PhD – Penn State University
Presenter: Melissa R. Goodnight, PhD – University of Illinois
Presenter: James L. Herbert, PhD – University of South Australia