43 - Designing Adequately Powered Cluster and Multisite Randomized Trials to Detect Main Effects, Moderation, and Mediation

Abstract Information: The purpose of this workshop is to train researchers and evaluators how to plan efficient and effective cluster and multisite randomized studies that probe hypotheses concerning main effects, mediation, and moderation. We focus on the conceptual logic and mechanics of multilevel studies and train participants in how to plan cluster and multisite randomized studies with adequate power to detect multilevel mediation, moderation, and main effects. We introduce participants to the free PowerUp! software programs designed to estimate the statistical power to detect mediation, moderation, and main effects across a wide range of designs. The workshop will combine lecture with hands-on practice with the free software programs. The target audience includes researchers and evaluators interested in planning and conducting multilevel studies that
investigate mediation, moderation, or main effects. Participants should bring a laptop to the session.

Relevance Statement: Several recent studies of evaluations spanning e.g., education, psychology, social work have examined the design purpose and capacity of several recent waves of multilevel studies (e.g., Spybrook and Raudenbush, 2009; Spybrook, Shi, and Kelcey, 2016; and Spybrook, Zhang, Kelcey, & Dong 2020). They found that many studies drew on weak statistical designs that were not adequately powered for main, mediation and moderator effects. However, they also found that more recent waves increasingly used appropriate designs that yielded adequate power. These papers suggest both the state of evaluation design across multiple fields is less than sufficient but also that initial professional development efforts in this area have yielded some steady improvement of study designs. In this way, sustained efforts to build the capacity of researchers in the field in terms of study design capture a core facet to advancing the quality of evaluation research.

More recently, the field has also emphasized the need to complement ‘what works’ questions with explanatory investigations. For instance, researchers have grown increasingly interested in moderation question focused not just on what works, but also for whom an intervention works for and under what circumstances an intervention works. Likewise, an increasingly common line of inquiry focuses on mediational questions—that is, designs that provide a more comprehensive assessment of an intervention by specifically mapping out the causal connections implied by the theory underlying an intervention. As the number of CRTs and multisite randomized trials (MRTs) funded grows and the number of researchers involved in planning CRTs and MRTs to detect mediation, moderation and main effects increases, the need for further capacity building is critical. Understanding the context and pathways in which an intervention is likely to be effective are fundamental to making results scalable and applicable to a wide range of schools, an integral part of the mission of AEA. For these reasons, it is critical to continue to disseminate this work and continue to build capacity among researchers planning multilevel study designs. The tools disseminated through this workshop will increase
researchers’ capacity to plan effective and efficient studies of multilevel moderation and mediation.