Assessment Designer, Educational Testing Service American University, Measurement and Evaluation Alum, 2023, United States
Abstract Information: Evaluation orthodoxy holds that an external evaluator is needed to tell a truthful, objective story. This presentation describes an instance where an internal team evaluated its processes using traditional tools and the participatory writers’ room approach. Eight test developers were tasked with using Agile methods to create practice tests. The team answered the questions:
Main Question: How effective were Agile methods for test development?
To what extent did Agile team members use methods that were different from standard test development methods?
To what extent were Agile methods more effective than standard test development methods?
What were the benefits and challenges of using Agile methods?
The evaluation followed a utilization-focused approach with a mixed-methods design that included Likert-scale questions, open-ended questions, and a focus group discussion. Survey responses indicated significant differences between Agile and non-Agile practices in five areas. While participants generally rated Agile methods as more effective, only one area’s survey result indicated significant effectiveness. Overall, Agile practices were rated as more effective for creating test forms, but this result was not significant. Wilcoxon signed ranks values and visuals will be presented. The Agile team used the writers’ room approach to create the final report. This participatory approach captured additional ideas, enhanced engagement, and promoted ownership. The final report offered recommendations for using technology, using Agile on operational teams, and providing Agile training. This case illustrates that using a writers’ room approach to develop an evaluation report promotes team collaboration and discussion of important themes that can guide effective decision-making.
