Integrating Technology into Evaluation
Crystal Luce (she/her/hers)
student
University of Colorado - Denver, Colorado, United States
Antonio Olmos, Ph.D.
Executive director
Aurora Research Institute, United States
Rachel Carlson, n/a
Research Assistant
Aurora Research Institute, United States
Location: Grand Ballroom 9
Abstract Information: OpenAI such as ChatGPT is starting to make its mark on the world around us. Stories of OpenAI have been in the news, and shows such as “Last Week Tonight with John Oliver” and “South Park.” Questions remain for its use in fields such as program evaluation and its ability to tell an accurate story. The process of program evaluation involves collecting, analyzing, and summarizing the data to tell an effective story and make decisions based on the insights gained. ChatGPT, a language model developed by OpenAI, offers a range of features that can assist program evaluators in collecting, analyzing, and reporting data. The possibilities of OpenAI seem limitless. The potential areas of use include: help with statistical analysis (what test may be appropriate when, reminders of assumptions, and help with completing the analysis), help with writing code for programs (such as Python, SQL, and R), help with literature reviews (finding articles and summarizing them), help with finding and creating questions (tests, surveys, interviews, and focus groups), help with qualitative analysis (sentiment analysis and themes), and help with data summarizing (qualitative and quantitative). In this demonstration, we wish to highlight some of the potential uses of ChatGPT for evaluators and discuss some of the potential consequences. Tools such as ChatGPT can be useful for program evaluators and our storytelling abilities; the applications are infinite. However, left unchecked, this technology can have lasting negative ramifications, and we must be aware of the potential harm that could be created. In this demonstration, we hope to show both the benefits and account for the costs of such technology.
Relevance Statement: Program evaluation involves data collection, analysis, and synthesis to be able to tell an effective story. However, the manual tasks of these elements can be draining on valuable resources such as time and money. However, AIlearning, such as ChatGPT, have the potential to enhance the work and thus storytelling ability of evaluators. ChatGPT, developed by OpenAI, is a state-of-the-art language model that can perform a variety of natural language processing tasks. Program evaluators can utilize ChatGPT in multiple ways to enhance their evaluation processes, including data analysis, report generation, and survey design. One benefit of ChatGPT is its ability to help conduct statistical analyses. It can help perform a range of analyses, such as regression, ANOVA, and factor analysis, to identify the relationship between program inputs, outputs, and outcomes. This can help evaluators determine which program activities are most effective in achieving desired outcomes, and which ones may need to be adjusted or eliminated. Another key benefit of ChatGPT is its natural language processing capabilities. It can process unstructured data, such as open-ended survey responses, social media posts, and interview transcripts, and identify patterns, themes, and sentiments. In addition to natural language processing and statistical analysis, ChatGPT can also assist with survey design. It can suggest appropriate question formats, such as Likert scales or multiple-choice questions, and recommend response options based on previous responses. This can improve the quality and accuracy of survey data, making it easier for evaluators to draw meaningful conclusions about the program's impact. ChatGPT can also assist with data visualization, generating graphs, charts, and tables to help evaluators communicate their findings to stakeholders. ARI has been using ChatGPT as an assistant to our work. We may have forgotten how to write a specific code, assumptions of statistical tests, or need help summarizing a large number of articles, and the program is an easily accessible tool for our work. Though we love using ChatGPT and its multiple applications, we also understand the potential risks of using such technology. The data that ChatGPT works from can be flawed, and as the old adage goes, “garbage in, garbage out.” We wish to demonstrate not only its many uses but also how we try to take into account its potential drawbacks and trust the final results.