Natural Language Processing (NLP) is a branch of Artificial Intelligence (AI) where the aim is for computers to understand and draw insights from human language, text, and speech data. NLPs are revolutionizing the way people analyze and generate text data. This offers exciting new ways for processing qualitative data for evaluation purposes. Tools like ChatGPT, Bard and LLaMA captured headlines in 2023 and brought increased awareness within the evaluation community of both the potential and perils of NLP. MERL Tech formed a Community of Practice (COP) to bring together evaluators who are curious about using emerging NLPs in their work. At this session, we will share some of the ways that evaluators in the cop are experimenting with emerging NLPs, the kinds of bias that they are encountering, the consequences of these biases for evaluation, and ideas on how to address them.
