Organizational Learning & Evaluation Capacity Building
Margaret Hargreaves, M.P.P.
Senior Research Scientist
NORC at the University of Chicago
Central Lake, Michigan, United States
Freya Bradford, MSW
Director of Systems Change & Learning
Rotary Charities of Traverse City, United States
Brandon Coffee-Borden, MPP (he/him/his)
Senior Research Scientist
NORC at the University of Chicago, United States
Location: Room 104
Abstract Information: In complex social systems, building organization-level evaluation capacity benefits from a systemic approach that takes internal and external contextual factors into account. Internal factors include the organization’s supply and demand for evaluation and evaluation capacity building (ECB). External factors include the political, economic, social, and cultural contexts in which the organization is operating (Scott Chaplowe, 1-22-22: OL-ECB TIG AEA365 Blog). Over the past decade, more organizations have responded to complex environmental changes by shifting from programmatic initiatives to larger-scale, systems change initiatives (Co-Impact 2022, Philanthropy for Systems Change Handbook). Their evaluation and ECB practices need to be updated to keep pace with these organizational changes. This session describes how Rotary Charities of Traverse City (RC), a regional foundation in Northwest Lower Michigan, successfully updated its evaluation system and ECB practices in 2022 to better assess and tell the story of its systems change initiatives. This panel session provides two perspectives of the revision of RC’s evaluation and ECB practices – the evaluation consultant and the foundation. The evaluation consultant will discuss the process and methods featured in the evaluation and ECB revision project. The Foundation’s Director of Systems Change and Evaluation will discuss RC’s use of storytelling and other methods to assess and report on the outcomes and impacts of its systems change initiatives.
Relevance Statement: In complex social systems, building organizational evaluation capacity benefits from a systemic approach that takes internal and external contextual factors into account. Internal factors include the organization’s supply (existing evaluation capacity) and demand (incentives and motivations) for evaluation and evaluation capacity building (ECB). External factors include the political, economic, social, and cultural contexts in which the organization is operating (Scott Chaplowe, 1-22-22: OL-ECB TIG AEA365 Blog). Other considerations for effective evaluation capacity building identified by Chaplowe include meaningful engagement with stakeholders in the process, attention paid to the delivery and use of evaluation practices and findings, the ability to be flexible and adapt to changing evaluation needs, and the ability to align and integrate evaluation functions, including ECB, into other organizational goals and activities. Over the past decade, government, non-profit, and philanthropic entities, have been responding to increasingly complex community, regional, and global problems, such as accelerating climate change, food insecurity, homelessness, dislocation, and inequitable access to resources, services, and opportunities. More organizations have responded to these environmental changes by shifting from programmatic initiatives targeting the knowledge, skills, and behaviors of individuals and organizations to larger-scale, systems change initiatives targeting the conditions that cause or hold these problems in place (Co-Impact, 2022: “Philanthropy for Systems Change Handbook”). This session will share the story of how Rotary Charities of Traverse City (RC), a regional foundation in Northwest Lower Michigan. Updating an enterprise-wide evaluation and ECB system to address any one of these elements is a challenging task. Integrating all four elements (principles-based, systems- and complexity-informed, (SCIE), and culturally responsive and equitable (CREE) evaluation and storytelling approaches) into an organization-wide evaluation system is arguably more difficult. From January 2022 through March 2023, RC worked with an evaluation consultant to conduct a comprehensive review and update RC’s evaluation goals, theory of change, learning and evaluation questions, data collection methods, tools and metrics, analytic approaches, reporting practices, and evaluation capacity building activities. This panel sessions describes RC’s evaluation project from two perspectives – the evaluation consultant and the Foundation. The evaluation consultant will discuss the process and methods featured in the project, including the use of AEA guiding principles and the Equitable Systems Change Evaluation Framework (ESCEF). The Foundation will discuss RC’s use of storytelling and other methods to assess and report on the outcomes and impacts of its systems change initiatives. These activities include RC’s project, “Stories of Change: How a Systems Change Approach is Transforming a Region” and RC’s adoption and training of RC staff, board, grantees, and stakeholders on new methods, including Collective Community Capacity assessments and Ripple Effect Mapping – a storytelling-approach to assess the outcomes and impacts of RC systems change initiatives.
Presenter: Brandon Coffee-Borden, MPP (he/him/his) – NORC at the University of Chicago
Presenter: Margaret B. Hargreaves, M.P.P. – NORC at the University of Chicago
Presenter: Freya Bradford, MSW – Rotary Charities of Traverse City