4 - Evaluating institutional change efforts: evaluators as story collectors and story analysts

Abstract Information: Telling the story of institutional transformation can be complex, given that institutional transformation is often a long-term process. Shifting institutional priorities, resources, and personnel shape the stories of transformation initiatives. Moreover, effective institutional transformation initiatives must also facilitate change on multiple levels with stakeholders and participants at each level seeing the story through a different lens. As evaluators, we often operate just outside of the institutional actors engaged in the initiative. We provide the implementation team with structures or frameworks to help them tell both the overall program story and the stories of individual stakeholders of all types. We observe how each part of the institutional system tells the story of the transformation, how these stories change over time, where these stories converge and diverge, and how these stories weave together to create a complete and complex tapestry. In this session we will describe the different ways in which we have worked to document or make the stories of institutional change explicit and use them to understand institutional transformation.

Relevance Statement: As evaluators, we often operate just outside of the institutional actors engaged in the initiative. From this vantage point, we can observe how each part of the institutional system tells the story of the transformation. We weave the disparate story threads together to create a cohesive tapestry that tells the complete story across time and actor. In this session we will describe our evaluation of Purdue’s ten-year institutional transformation program, IMPACT, initiated with the goal of radically changing instruction at the university. We will discuss the different ways in which we have worked to document or make these stories explicit. We will also share how our long-term story collection role has allowed us to use changes in stories as an indicator of culture change at the institution. IMPACT, like many transformation projects, began as a grand idea with only a broad stroke outline. The details of the project emerged, clarified, and changed over time, driven by changing leadership, priorities, and indicators of success. As the project evaluators, we work alongside the program implementors as facilitators to help define, document, and continually refine the program model. In this way we are effectively collecting the story of the program developers—aiding them in explicitly describing and chronically their motivations, intentions, and expectations for the program over time. For example, over its ten-year history, IMPACT shifted markedly from a program focused primarily on institutional transformation through course redesign and course restructuring to a focus on institutional transformation through empowerment and development of reflective, engaged faculty. In part, this shift in focus was driven by stories we collected from stakeholder groups across the university in the initial phases of the program. Comparing the stories each group told about the purpose, motivation and early impacts of the program was useful for understanding the interests and expectations of key administrative program stakeholders. As semi-external to the project, we were able to collect these stories confidentially such that all stakeholders could candidly share their perspectives and understandings of the program and its impacts. We could then communicate these stories back to the program implementation team along with key areas of convergence and divergence among stakeholders’ stories. We’ve also been able to collect the stories of the direct participants in this institutional change effort (faculty participating in professional development activities). Changes in these accounts over time provide compelling indicators of a culture shift at the institution. For example, early participants expressed strong beliefs that time spent on developing teaching practice was a potential threat to academic career advancement; on-the-other-hand, current participants perceive faculty professional development of teaching practice as an asset for advancement. The stories have been indicative of a profound change in the value placed on the practice of teaching as part of a tenure-track career, the importance of student learning as a key objective of faculty, and the role of effective instruction by faculty in achieving student learning. This session will be of interest to evaluators and partners involved institutional transformation for enhanced teaching and learning.