Organizational Learning & Evaluation Capacity Building
Theresa Melton, PhD
Assistant Professor, Youth Development Leadership
Clemson University
Clemson, South Carolina, United States
Theresa Melton, PhD
Assistant Professor, Youth Development Leadership
Clemson University
Clemson, South Carolina, United States
Edmond Bowers, Ph.D.
Associate Professor
Clemson University
Clemson, South Carolina, United States
Edmond Bowers, Ph.D.
Associate Professor
Clemson University
Clemson, South Carolina, United States
Amanda Teye, PhD
Professor, Department of Political Science
James Madison University, United States
Amanda Teye, PhD
Professor, Department of Political Science
James Madison University, United States
Location: Room 208
Abstract Information: Process and outcome evaluations are conducted to assess program implementation and/or impact, with the intent to inform subsequent actions to strengthen programming. Yet, an organization’s capacity for evaluation can impact the evaluator’s ability to conduct a meaningful, systematic evaluation as well as the organization’s ability to utilize information made available from that evaluation. Fortunately, evaluators can work with practitioners to help strengthen their organization’s processes and structures, as well as employee knowledge and skills. Strengthening this capacity for evaluation not only can strengthen the relationship between researchers and practitioners (Melton et al., 2022), it can also strengthen the organizational culture of evaluation and the ability to support meaningful evaluation (Goh et al., 2004). In this roundtable discussion, researchers share findings from a workshop with over 70 youth-serving organizations, in which they jointly identified barriers to evaluation capacity and resources that can help promote an organization’s capacity for evaluation. Following that brief presentation, the group will work together to discuss the role evaluators can take in addressing barriers and facilitating procedures to build an organization’s capacity for evaluation.
Relevance Statement: Process and outcome evaluations are conducted with the goal of assessing program implementation and/or impact, with the intent to inform action. Yet, an organization’s capacity for evaluation can impact an evaluator’s ability to conduct a meaningful, systematic evaluation. Fortunately, evaluators can work with practitioners to help strengthen their organization’s processes and structures, as well as employee knowledge and skills. Strengthening this capacity for evaluation not only can strengthen the relationship between researchers and practitioners, it can also strengthen the organizational culture of evaluation and ability to support meaningful evaluation. In an effort to learn more about how to improve organizational capacity for evaluation, the research team conducted a workshop with over 70 organizations, focusing specifically on helping practitioners identify current capacities for evaluation, challenges that prevented evaluations from being conducted and/or from informing practice, and next steps to strengthen their capacities for evaluation. As part of this process, workshop participants responded to a survey which included an evaluation capacity checklist with the following domains: evaluation planning, data collection (outcomes and processes, analysis/communication, culture of evaluation within organization, and stakeholder engagement. Participants were asked to indicate whether items within each domain 1) needed redone, 2) hadn’t been completed, 3) was in progress, or 4) was established. Of the 70+ workshop participants, over 50 mentoring organizations participated in the survey, representing over 18 states. The majority of participating organizations employed over 20 staff members (44.4%), served over 200 youth annually (68.7%), and served youth of all ages (47.3%). Overall, participants were either in progress or had developed most of the items needed to plan the evaluation. However, additional needs were identified for data collection strategies, especially developing a system to help capture data once collected, and using the data/communicating the data also emerged as an area to focus more efforts on. Most organizations identified limited stakeholder engagement. Although larger organizations (e.g., staff greater than 20, budget over $1 million), reported having similar or less capacity for evaluation than midsized (13-20 staff, 500k to 1 mill budget) organizations. Across all domains, scores were higher among those reporting use of an information management system (IMS), though only half of the organizations present reported using an IMS. In an open-ended prompt, organizations commonly noted “time,” “resources,” and “training” as obstacles to evaluation. In this roundtable discussion, we will present findings from that workshop, highlighting which areas of evaluation (e.g., evaluation planning, data collection, data use and communication, and/or stakeholder engagement) were identified by practitioners as challenging. With this information, we hope to lead a meaningful conversation among evaluators around concepts such as 1) what is the role of evaluators in helping to build the evaluation capacity of programs, 2) evaluation as an iterative process that can be used to strengthen the culture of evaluation within an organization and support buy-in, and 3) specific ways that evaluators can partner with organizations to help build their capacity for evaluation. Additionally, we will discuss the value of identifying an organization’s capacity prior to beginning an evaluation.