150 - Linking Logic Models, Evaluation Questions, and Survey Design to Elicit Participants’ Perspectives in Theory-Driven Evaluation

Abstract Information: This session shares our story of applying theory-driven evaluation projects with diversity, equity and inclusion (DEI) outcomes. We will present two cases from higher education where a logic modeling process elicited practitioner perspectives and ultimately informed the collection of survey data to understand participants’ perspectives. Stories about program goals and implementation were captured in logic models, which were necessarily drawn with very different structures to reflect each program’s nature. In one case, the logic model was designed early in the process while the program was still being developed and played an important role in internal communications. In the second case, the logic model was designed after years of implementation, and played an important role in both formative and summative evaluation efforts. We will map the applied work we did to move from the theoretical logic models through the evaluation questions to survey design. The discussion will focus on two particular outcomes, one about instructors’ awareness about social identities and another about students’ sense of inclusivity. We will share resources for survey design in educational contexts and offer attendees the opportunity to reflect on how these examples relate to their own work.

Relevance Statement: This presentation focuses on the applied work that is essential for implementing theory-driven evaluation in educational contexts. We present examples from recent internal evaluations in higher education that had diversity, equity, and inclusion (DEI) outcomes. In both evaluation projects, practitioner’s perspectives were privileged in the design of logic models and participants’ perspectives informed the primary outcomes. We will briefly note resources for fundamental topics, such as the design of logic models (McLaughlin & Jordan, 2015; Meyer et al., 2022) and psychological measures (AERA, 2018). We will map the connections between logic model, evaluation questions, and survey items. In one project, an online course for instructors, the logic model served to clarify goals and design formative and summative evaluation plan components. We will share the logic model and show how we mapped a DEI outcome to a data collection tool. We will specifically consider an outcome about participants’ awareness of social identities and follow that outcome from the logic model through the evaluation plan to the survey instrument. With both a Likert scale and open-ended items, this survey will provide an example where reliability needed to be addressed. The second example is from a campus-wide effort to increase the use of interactive teaching methods in undergraduate classrooms. In this case, the logic model served as a springboard for generating detailed evaluation questions. It helped program staff reflect on the project’s broad goals and prioritize short-term evaluation tasks. We will share the simplicity of the final logic model, the complexity of related evaluation questions, and focus on one evaluation question about whether active learning strategies foster a sense of inclusivity. We will describe how we adapted items from the research literature and our university’s large-scale student survey to design a related set of both open- and closed-ended items for program evaluation purposes. We will use this example to highlight the importance of validity. We will invite participants to reflect on their own work and this session will promote multiple Evaluator Competencies: practitioners’ use of and reflection about theory-driven approaches (see 1.5 and 2.9); supporting the collection of valuable data by engaging stakeholders (see 2.10 and 3.2); and the value of documentation (see 4.6). This session will be particularly valuable to attendees who see an increasing need for understanding people’s perspectives by using psychological measures but do not have experience with selecting or designing item sets. It adheres to the Guiding Principles of methodological competence (B1) and showing respect for people by eliciting the perspectives of participants who might not typically be prioritized (D1), thus making efforts to address any evaluation’s potential for amplifying marginalized voices (E2). References American Educational Research Association. (2018). Standards for educational and psychological testing. American Educational Research Association. McLaughlin, J. A., & Jordan, G. B. (2015). Using logic models. Handbook of practical program evaluation, 62-87. Meyer, M. L., Louder, C. N., & Nicolas, G. (2022). Creating with, not for people: theory of change and logic models for culturally responsive community-based intervention. American Journal of Evaluation, 43(3), 378-393.