Learning From Practice: Scoping Review Methods and Early Findings on Youth Participatory Evaluation

Abstract Information: Evaluation reports, or gray literature, are a treasure trove of information on the story of evaluation practice. What gets done in evaluation and what can be learned from practitioners is there to be collected, yet systematic reviews often focus on peer reviewed publications and miss the wealth of information in the gray literature. This research on evaluation study sought to change that through a scoping review of evaluation reports, with a focus on whether and to what extent youth were included. With the rising popularity of youth participatory evaluation practices, it seems important to confirm the extent to which youth are being included in evaluations of programs that serve them and what participation looks like. This presentation will share the systematic review protocol and tools used to conduct the review, and lessons learned about conducting systematic reviews along the way. In particular, researchers will discuss how and why they used the Arksey and O’Malley’s scoping review methods framework and integrated extant checklists (PRISMA-ScR, ENTREQ, COREQ, & CHESS). Early findings on youth participatory evaluation practices will be highlighted. This study demonstrates the potential of using evaluation reports to learn more from practitioners and inform future research. The session will close with time for reflection on the implications of the study’s methods and findings for future research on evaluation.

Relevance Statement: Participatory evaluation practices have grown over time, at least in rhetoric (Bitar et al., in press; Montrosse-Moorhead et al., 2019). There have been increasing calls within both academic and practice spaces (e.g., publications, blogs, webinars, etc) about how to be more collaborative within evaluation, both for the purposes of utilization and empowerment (c.f., Chouinard & Cram, 2019; Cousins, 2019; ). Yet, it remains difficult to do in practice, particularly to include those who are traditionally withheld from roles with decision-making power, such young people, the primary beneficiaries of youth programs. Little is known on the extent to which participatory practices are actually happening, despite existing frameworks (Checkoway, 2003; Richards-Schuster & Platcha Elliott, 2019; Sabo, 2003) and resources available on how to facilitate youth participation (c.f., UNICEF, 2018) . Systematic reviews offer one way to dig deeper into existing literature to uncover themes through thorough, transparent, and replicable processes. Unfortunately, systematic reviews historically have tended to focus on peer-reviewed articles, missing the treasure trove of gray literature (e.g., evaluation reports). This research on evaluation study sought to change that through a scoping review. The database used for this study was the publicly available repository through the Center for Advancement of Informal Science Education. The study design was grounded in Arksey and O’Malley’s (2005) scoping review framework. Given that evaluation reports included quantitative and qualitative data, a new protocol for conducting mixed-method scoping reviews was created. This new protocol drew from several extant checklists specific to quantitative or qualitative methods (PRISMA-ScR, ENTREQ, COREQ), and from an evaluation-specific checklist (CHESS).This session will provide an overview of our methodology, the steps of our protocol, the tools that we used (e.g., Zotero, Covidence), and lessons learned from the process. Early findings on youth participatory evaluation practices will be highlighted. This session will be of interest to researchers and practitioners, including youth participatory practitioners. This study demonstrates the potential of using evaluation reports to learn more from practitioners and inform future research. The session will close with time for reflection on the implications of the study’s methods and findings for future research on evaluation.