165 - The Story of Research on Evaluation (RoE): Continuing A Thorough Investigation on Recent Trends

Abstract Information: This study extends a previous systematic review of research on evaluation (RoE; Coryn et al., 2017) by examining the following five years, resulting in an overall review of RoE published between 2005 and 2019. We pulled articles published in 14 evaluation journals between 2015 and 2019 and screened them for whether or not they were RoE. Articles considered RoE were then coded based on Mark’s (2008) taxonomy of research on evaluation. We will share the trends in the additional five years of data and discuss implications for the field and study of evaluation.

Relevance Statement: As the field of evaluation continues to grow, we must bring awareness to the relevance of research on evaluation (RoE) in our careers as evaluators. Based off the work we do, we get to decide where this field continues to go. RoE gives the ability to share stories on what we have done as evaluators for our field and where we continue to go. RoE allows us to take a more in-depth look at the purpose, plans, and ideas shared that have led to successful evaluations while being able to share obstacles that we have met along the way. When last surveyed, 96% of a sample of AEA members (n = 196) indicated that RoE was of value, and that RoE affects their thinking about evaluation either some (51%) or a lot (46%; Coryn et al., 2016). AEA members even agreed RoE improves, informs, and guides the practice of evaluation (41%), provides empirical evidence (21%), builds on a body of knowledge (18%), and finds the most effective methods of evaluation (18%).

Coryn et al. (2017) further examined a decade of RoE (2005 to 2014), finding that roughly 8% of articles in evaluation journals were RoE. There also seemed to be an increase in RoE articles from 13 in 2005 to 41 in 2014. Therefore, there seems to be a gap between evaluators' values towards RoE and the prevalence of RoE being published in evaluation journals. We wanted to examine whether this trend of increased RoE over the years was continuing.

We followed up on Coryn et al. (2017) by looking at the next five years of data, from 2015 to 2019. We used the definition of RoE provided by Coryn et al. (2017) and then coded studies screened as RoE against Mark’s (2008) framework: subjects of inquiry (i.e., practice, consequences, contexts, professional issues, domain-specific) and modes of inquiry (i.e., description, tool development/validation, causal analysis, classification, values inquiry). We will present the findings for the next five years of data and compare to the previous ten years of data from Coryn et al. (2017). We will share the trends in the additional five years of data and discuss implications for the field and study of evaluation.

References:
Coryn et al. (2016). Does research on evaluation matter? Findings from a survey of American Evaluation Association members and prominent evaluation theorists and scholars. American Journal of Evaluation, 37(2), 159–173. https://doi.org/10/f8qqvk

Coryn et al. (2017). A decade of research on evaluation: A systematic review of research on evaluation published between 2005 and 2014. American Journal of Evaluation, 38(3), 329–347. https://doi.org/10/ggf5td

Mark, M. M. (2008). Building a better evidence base for evaluation theory: Beyond general calls to a framework of types of research on evaluation. In N. L. Smith & P. R. Brandon (Eds.), Fundamental issues in evaluation (pp. 111–134).