A different kind of story: Answering the question “What design should I use?”

Abstract Information: After teaching evaluation (and research methods) for over twenty years, I thought there must be a better way to address the age old, always asked question, “What evaluation design should I use?” There are many factors to consider in choosing a design and the newness of learning about various research designs can be overwhelming in and of itself to a student new to evaluation. To further overwhelm my students, I teach them about the many approaches to evaluation and the methods that can be used to evaluate programs. Several years ago, I began to develop a framework to integrate this information as a way to help students make sense of evaluation designs, approaches, and methods. Further, I wanted this framework to enable students to think deeply about their own ideology with regard to evaluation, as opposed to adopting my preferences or those of the authors they may have read about in class. In short, I wanted them to synthesize the information they had learned and use the model I had developed as a way to guide their design choices. I wanted them to begin to think about their own professional identity as an evaluator. This session will present the framework developed and engage in a discussion with participants as a way to continue to improve the framework so that it can best serve students as they learn about and synthesize the fundamental yet immense information about evaluation design, methods, and approaches.

Relevance Statement: It is critically important that new (and old) evaluators understand the many factors that influence the choice of evaluation design. I also believe it is important, just as we purposefully choose an evaluation design, that as evaluators, we are intentional about the evaluation approaches we use. Further, in choosing both evaluation designs and approaches, evaluators should be conscious of the implications of these choices. The evaluation methods we choose also influence an evaluation, including the most basic issue that the evaluation team needs to have access to evaluators who are skilled in collecting and analyzing data associated with the evaluation methods used. In teaching evaluation design, methods, and approaches to my graduate students as separate units in the course, yet asking them to integrate them in projects, I could see how difficult it was to understand how it all fit together. I began working on a framework to help students think about various factors in order to make choices about design and methods, as well as to explore their own ideology with regard to evaluation. The model includes both influencers and calibrators. Influencers either expand or restrict the evaluation design, methods, and approaches an evaluator can choose. Influencers include the resources we have available for evaluation, the control we have over the evaluation environment, the skills we have on our evaluation team, and the access we have to stakeholders. Calibrators impact our own ideology as an evaluator. These are factors I ask my students to think deeply about, rather than adopting my or another instructor’s evaluation preferences. Calibrators include preferences with regard to research methods, evaluation design, and the role of an evaluator. Calibrators shape our evaluation ideology, which in turn influences our choice of evaluation design, methods, and approaches. At the same time, the choice of evaluation design, methods, and approaches is moderated by the influences of resources, environment, skills, and access. I have been using this model for a few years now in the classroom and have received positive feedback from students. However, I believe it can be improved and that there are undoubtedly considerations that may be missing. I also recognize that some topics may be simplified as a way to assist students new to evaluation to synthesize this large body of information, yet it does introduce them to ongoing conversations in evaluation regarding the role of the evaluator, degree of stakeholder involvement, manipulation of evaluation environment, resource allocation, and importance of skilled evaluators. I hope this session can be an opportunity to share the framework and gain input on ways it might be improved as a teaching tool to evaluators in training.