Democracy and Governance
Salome Tsereteli-Stephens (she/her/hers)
Director - Learning, Evaluation, and Impact
American Bar Association Center for Global Programs, United States
Jeanette Tocol, Evaluation and Learning Advisor (she/her/hers)
Evaluation and Learning Advisor
American Bar Association Rule of Law Initiative, United States
Paola Suarez, MEL Expert (she/her/hers)
MEL Regional Manager
American Bar Association Rule of Law Initiative
Bogotá D.C., Distrito Capital de Bogota, Colombia
Victoria Alegre, MEL Expert (she/her/hers)
Regional MEL Manager
American Bar Association Rule of Law Initiative, Peru
Location: Room 103
Abstract Information: Many development programs across the world provide trainings on various topics with an assumption that when individuals gain knowledge, it will result in long term changes for both the individual and the organizations they represent. But practitioners and academics alike struggle with establishing clear linkages between individual changes in knowledge and skills with behavioral and organizational changes. How can evaluators use available tools to tell the story of change among the individuals and organizations that development programs support? This panel discussion will present how ABA ROLI programs have used the Kirkpatrick Model of training evaluation to measure reaction, learning, behavior and results all throughout the training, administering measurement tools in parallel to implementation of the training event. Panelists will reflect on their experiences of how the Model has been used to collect training and change stories in multi-year training programs for justice actors in Peru, Colombia and in a global justice collaborative fellowship program, including what challenges and lesson were learned in the process.
Relevance Statement: Development programming that targets government institutions often focuses on training officials with the aim to change the quality of public or sector specific services and overall performance. In describing the training story under this theory, evaluators are most commonly faced with various measurement challenges. First, evaluators have to situate the individual changes that are anticipated within the broader frame of the organizational change context. Second, any changes that may be anticipated in individuals will occur within the context of the individuals' conditions – their expectations, needs and reactions to specific training material as well as their ability to create change within the organization. Third, any measurable effect (impact) to the organization of a specific training may have to be propelled by a significant number of individuals and other necessary resources. And lastly, any contribution from individual changes resulting in organizational changes may need to be determined in parallel. The Kirkpatrick model for training evaluation presents potential to address some of these challenges. The model is an approach to assess across four levels – experience, learning, behavior and result. ABA ROLI’s experience in programs may provide useful insight on the best ways to use the model, examples of how tools were developed, and what lessons have been learned especially in reflecting on when it has been useful and not useful in providing a full story picture of trainings in DRG programming. The first presentation will focus on ABA ROLI’s experience in Colombia Programs where the Model is applied to help improve the design and impact of courses offered to judges and judicial staff. In this example ABA ROLI will present level 1 to 3 data collection tools, and the various ways the training story is relayed and used for both ABA ROLI programming and sustained improvement of training programs by counterparts. The second presentation will focus on ABA ROLI’s experience in Peru where the Model is used in key trainings for judges and prosecutors to demonstrate individual results and observe contributions of which to organizational performance. The presentation will feature the various ways the tools have been developed, the challenges experienced and the ways the evaluator has worked around those challenges to arrive at a training story that may be shared with counterparts. The third presentation showcases how ABA ROLI applied an adjusted approach for using the Model for a global fellowship program aimed at improving the collaboration of various justice actors in specific countries though mentorship support and study tours in the U.S. The presentation will focus on showing the types of analysis that can be drawn to provide a training story for fellowship and the change plans that were continuously monitored up to 24 months post assistance.
Presenter: Salome Tsereteli-Stephens (she/her/hers) – American Bar Association Center for Global Programs
Presenter: Paola Andrea Suarez, MEL Expert (she/her/hers) – American Bar Association Rule of Law Initiative
Presenter: Victoria Alegre, MEL Expert (she/her/hers) – American Bar Association Rule of Law Initiative