Program Theory and Theory-Driven Evaluation
Charles Gasper, M.A.
Evaluation Director
TCC Group, New York, United States
Stephanie Coker, MPA; BA in Political Science and Government
Senior Manager, Strategic Learning and Evaluation
CVS Health
New York, New York, United States
Rose Konecky, n/a
Evaluation and Learning Consultant
TCC Group
South Windsor, Connecticut, United States
Location: Grand Ballroom 10
Abstract Information: Who do you typically engage when developing a theory of change for a program or organization? Is it the usual group of program designers, implementers, and maybe a representative or two from the pool of individuals the program or organization is supposed to serve? Do you tie social science research to the various connections within that theory of change? You might if you have a subject matter expert on your team, but most likely, that individual isn’t present. So, we develop logic models based on the viewpoints of a select few individuals and argue that the model represents the programmatic and/or organizational intents to address the social science challenge.
We know it is important to hear from the communities these organizations and programs serve. We know that it is better to also be able to speak to the social science that could underly these. Yet we don’t because there is a cost in time and funding that seems unreasonable to purchasers of the work. Yet, what if we told you that you could engage more of the community? What if we told you could hear the voice of those the programming and organizational efforts are supposed to effect, along with the possibility of engaging all of the program implementers, other key relevant voices in the field, practically anyone you think might have something valuable to contribute?
In this expert lecture, we will discuss how we have used concept mapping to hear the voice of the community, voices we never hear in the development of logic models. We’ll also share some emerging work we are doing with Ais such as ChatGPT to provide some insight as to whether and how strongly the connections amongst the activities and interconnected outcomes are found in social science.
Relevance Statement: In consulting practice, we often develop logic models representing programmatic theories of change to support our evaluation work. The practice of developing these models has evolved over years from reflecting predictions of effects based on social science research to engaging those who designed, implemented, and manage those programs. We often talk about engaging those affected, but let’s be honest, we tend to only select a couple of individuals to participate in the logic model development sessions. We justify this by arguing the costs of bringing so many people together to discuss the activities and outcomes of programming is very high. As a result, we don’t hear from the affected, we only hear really from those who are responsible for the programs’ design, implementation, and management. We speak about democratizing evaluation, yet the core underpinning of the description of a program or organization only reflects the views of a select few. This description often drives what we evaluate and while we might involve more in our data collection and use methods that are intended to hear more voices, our underlying structure can be flawed having not heard those voices in the first place.
Techniques and technology have evolved over the years and the pandemic forced members of the evaluation community to get a bit more creative in the areas of data collection. But there is opportunity to be more creative. Concept mapping has been a viable technique for over three decades and while individuals such as William Trochim have worked to make it more accessible, it is underutilized as an evaluative tool. We have also seen the rise of AI to serve in various functions in data analysis and aggregation. We believe that the marriage of these techniques and technologies can help open doors and get us closer to truly democratized evaluation.
Through our presentation, participants will learn how concept mapping can be adapted as a tool for supporting the development of more representative theories of change, better understand how AI can return social science research to the design and evaluation of connections among activities and outcomes, and how to harness these to better tell the story of a program or organization.