Debating rigor through a framework of shared objectives and tradeoffs

Note: This presentation is co-presented with Giovanni Dazzo and Bret Barrowman. This presentation will present contrasting views about rigor, but will ultimately argue that rigor is not the purview of one method or paradigm, dispelling unproductive dichotomies that may frustrate conversations about evaluation quality. Presenters, themselves from sometimes opposing paradigms, will share characteristics of rigor that cut across many paradigms. While acknowledging that there are many decisions about rigor that evaluators will not agree on, such as various forms of validity, presenters argue there are many still where they will find common ground. Methodological pluralism, when built on a foundation of shared values, can improve the evidence landscape that evaluators seek to build (rigor) and why evaluation matters for program participants (relevance). Building from a situational typology, each presenter within this panel will offer distinct – and differing - insights and recommendations of tradeoffs and modifications that evaluators can make to increase rigor and relevance in a given situation. These will be applicable to a wide variety of applications: exploratory or subject matter research, measurement and performance evaluation, and qualitative or quantitative evaluation. These perspectives will represent a spectrum of critical theory, critical realist, and positivist-informed paradigms with a focus on what these paradigms mean to the practical implementation of program design and evaluation.