Revolutionizing Program Evaluation: Harnessing the Power of Artificial Intelligence tools like ChatGPT

Abstract Information: Program evaluation has traditionally been a time-consuming and resource-intensive process due to the nature of the work involved and the skills required. However, with the advent of artificial intelligence (AI) tools such as ChatGPT, the evaluation process has the potential to be revolutionized. Through this Ignite presentation, I will highlight 20 ways evaluators can utilize ChatGPT and similar free or inexpensive AI tools to conduct more efficient, accurate, and effective evaluations. AI tools can help speed up the process of program evaluation by providing fast and efficient ways to analyze data. In addition to existing AI tools that can analyze massive samples of quantitative data, natural language processing tools such as ChatGPT can understand and analyze text data from surveys, interviews, and other sources. This can save time compared to manually coding and analyzing text data. AI tools like OpenAxis can help find higher-quality datasets and generate visualizations such as graphs and charts in a fraction of the time. Tools like ChatGPT can also help evaluators in reviewing their evaluation questions and ensuring the evaluation questions align with the purpose of evaluation and the indicators used. It can also help review data collection tools such as surveys, interview protocols, and focus group protocols. Additionally, AI tools like Microsoft Designer, Adobe Firefly, and Photosonic can help design quick presentations and art for evaluators. Finally, I will also discuss the challenges, limitations, and ethical considerations around the use of AI in program evaluation.

Relevance Statement: This presentation on harnessing the power of Artificial Intelligence (AI) tools like ChatGPT in Program Evaluation is highly relevant and important to the field of evaluation, as it addresses one of the most pressing challenges facing evaluators today: how to conduct program evaluation more efficiently, effectively, and ethically. This presentation is also very timely, as just this week, leading academic publishers like Elsevier and Cambridge University Press have declared that AI tools like ChatGPT can be used in research production and manuscript preparation even though they may not be listed as co-author. This presentation's implications for evaluation theory and practice are significant, as it suggests that AI has the potential to transform the way in which evaluation is conducted. Specifically, it highlights how AI tools can help to address some of the key limitations of traditional evaluation methods, such as the time and resource-intensive nature of manual data collection and analysis, the potential for human error and bias, and the limited scope of statistical and qualitative analyses. By harnessing the power of AI, evaluators can conduct program evaluations more efficiently, allowing them to make data-driven decisions in a more timely manner. For evaluators, the value of this presentation lies in its ability to provide practical guidance on how to integrate AI into their evaluation practice. The presentation will highlight a number of specific AI tools and techniques that can be used to enhance various stages of the evaluation process, from developing evaluation plans to data collection, and analysis to reporting. By providing concrete examples of how AI can be applied to evaluation practice, the presentation helps evaluators to understand how AI can be leveraged to improve their work. This presentation is highly relevant to this year’s AEA theme as it can also help evaluators generate better narratives and stories that highlight the impacts and outcomes of programs. For example, evaluators could input a large set of interview transcripts into ChatGPT, which could analyze the text data and identify key themes and topics to create narratives that could be used to communicate the impact of the program to a wider audience, such as funders, policymakers, or the general public, in a way that is both compelling and data-driven. Furthermore, the presentation emphasizes the importance of ethical considerations in the use of AI in evaluation practice, which is critical for evaluators to keep in mind as they adopt new technologies. As AI continues to evolve and become more ubiquitous in society, the need for evaluators to engage in ongoing reflection and dialogue about the ethical implications of their work will only become more pressing. Overall, this presentation represents an important contribution to the field of evaluation by highlighting the potential of AI to revolutionize evaluation practice and providing practical guidance on how to integrate AI into evaluation workflows. By adopting AI tools and techniques, evaluators can improve the efficiency, accuracy, and effectiveness of program evaluation while also being mindful of the ethical implications of their work.