Integrating Technology into Evaluation
Cynthia Phillips, PhD
Principal
Decision Catalyst, LLC
Glenshaw, Pennsylvania, United States
Lisa Gajary, PhD
Principal
Caspian Strategy & Analytics LLC
Dublin, Ohio, United States
Anand Desai, PhD (he/him/his)
Senior Fellow
Clarivate
Washington, District of Columbia, United States
Location: Room 103
This roundtable session focuses on how evaluators might couple two technologies—large language models (LLM) and computational simulation modeling—to re-imagine the development of Theories of Change (ToC) for evaluating investments in research. We start by elaborating a five-step model that serves as the foundation for developing a ToC representing the path from research to societal benefit in research, development, and innovation (RDI) systems. This model was developed using LLMs focused on case studies describing the translation of foundational research to societal benefits including commercialization (a subset of the 6,975 cases submitted to the 2014 United Kingdom’s Research Excellence Framework). Given that in complex RDI systems, there is the potential for multiple pathways and feedbacks involved in reaching a specific objective or bringing about change, we show how computational modeling can aid in specifying feedback loops within an RDI-system ToC. We show this by focusing on the knowledge transfer step in the ToC, and then by using Jordan’s (2013) team science logic model, we formalize aspects of this transfer step as a computational simulation model. The simulation model prototype that is produced provides an entry point for the in silico examination of the feedback loops associated with accelerating the translation of research to societal and comparing funding outcomes for curiosity-driven and use-inspired research. Both the conceptual framework and simulation model will serve as points of departure for a discussion on how to incorporate feedback in ToCs throughm computational simulation modeling.