Cluster, Multi-site and Multi-level Evaluation
Emily Leung, MPH
Lead, Research and Data Analytics
CAI (Cicatelli Associates Inc.), United States
Ruchi Mehta, MPH
Project Director & Evaluator
CAI (Cicatelli Associates, Inc.), United States
Alice Douglas, MPH
Director, Learning Collaboratives
CAI (Cicatelli Associates Inc.), United States
Amelia Gonzalez, MA
Technical Assistance Regional Lead (TAP-in)
CAI (Cicatelli Associates Inc.), United States
Rama Murali, MPH, MA
Deputy Director (TAP-in)
CAI (Cicatelli Associates Inc.), United States
Rebecca Cohen, MD, MPH
HIV Clinical Specialist and Associate Medical Director
Division of HIV and STD Programs at the Los Angeles County Department of Public Health, United States
Lindsay Senter, MPH
Vice President, Research and Evaluation
CAI (Cicatelli Associates Inc.), United States
Location: Room 301
Abstract Information: Learning Collaboratives (LC) consist of teams of peers from different organizations/agencies convening regularly for shared learning opportunities to implement an intervention, systems-level changes, and/or organizational changes using data-driven and quality improvement methods. This presentation will describe CAI’s (Cicatelli Associates Inc.) approach to developing and implementing a mixed-methods, adaptable, Multi-level LC Evaluation Framework to evaluate CAI’s portfolio of LCs. The Framework describes rigorously evaluating LCs at four levels – facilitator, participating agencies’ systems, participating agencies’ staff, and client/priority population. To illustrate the Framework, we will present two case studies: 1) a HRSA HIV/AIDS Bureau-funded project, Technical Assistance Provider-innovation network, that is facilitating an LC on implementing rapid antiretroviral therapy services at seven HIV service providers in California 2) the Pre-Release Substance Use Disorder (SUD) project, where CAI is building the capacity of seven community-based organizations throughout the U.S. to deliver a curriculum on SUD as a chronic condition for justice-involved individuals. Data collected from this multi-level evaluation framework enables us to tell a rich story for different audiences for program improvement, advancement, and dissemination.
Relevance Statement: Learning Collaboratives (LC), first conceived by the Institute for Healthcare Improvement (IHI) in 1994, is a tool used extensively in healthcare settings where teams convene to make “breakthrough” improvements on a specific area over 6-18 months. LCs consist of learning sessions led by content and improvement experts, action periods between learning sessions where teams implement systems- and organizational-level changes or interventions, regular reporting and review of performance data [1]. CAI, a public health non-profit organization, has a long history of successfully designing and implementing LCs to build the capacity of community-based (CBO) and healthcare organizations to strengthen systems, thereby improving organizational and client outcomes. CAI has developed a Multi-level LC Evaluation Framework, detailing a standardized method of evaluating LCs to ensure we have a clear roadmap of intended outcomes. This framework has enabled us to tell a comprehensive story of multi-level impacts systematically and effectively to improve, advance, and disseminate our work. This session will be presented in three distinct sections. Firstly, we will describe how CAI has adapted and implemented the IHI LC model in our capacity-building work with agencies, aiming to improve organizational and systems changes and health outcomes. Then, we will introduce our Muti-level LC Evaluation Framework. The Framework ensures that evaluation questions are established and performance measures are developed and tracked at four different levels– 1. facilitator-level (effective LC implementation), 2. participating agencies’ systems (process, strategy or intervention implementation, level of engagement, and overall systems-level outcomes); 3. participating agencies’ staff (knowledge, attitudes, self-efficacy, engagement, and/or skills to implement), 4. client/priority population (resulting client outcomes (e.g., improved HIV outcomes)). Standardized data collection tools are used but new tools tailored for each LC are routinely developed. We will then describe two case studies that implemented the Framework. Firstly, we will describe the Technical Assistance Provider-innovation network (TAP-in) project, where we worked with the Los Angeles County Division of HIV and STD Programs (LA DHSP) to conduct an LC with seven contracted HIV clinic sites to assist with protocol and systems development to provide rapid antiretroviral therapy (ART) services. The data collected has been critical in telling a multi-level story to help clinics understand program improvement, implementation progress, and community impact, and to inform TAP-in’s and LA DASP’s expansion of the LC to additional cohorts. The second case study, Project SUCCEED (Substance Use as a Chronic Condition Engagement and Education) LC, brings together CBOs that serve justice-involved clients dealing with substance use disorder (SUD). The LC goals are to train CBO staff to deliver a curriculum on SUD as a chronic condition to the justice-involved individuals they serve while concurrently building staff capacity, establishing relationships, and strengthening agency systems and partnerships through peer learning. We evaluated the LC’s impacts on CBOs’ organizational practices, their individual staff, and the justice-involved individuals they serve. This case study will discuss key findings and implications of our findings at each level of the Framework. [1] Baker GR. Collaborating for Improvement: The Institute for Healthcare Improvement’s Breakthrough Series. New Med. 1997;1:5-8.