Abstract Information: Stochastic, random-effects extensions to familiar tree-based machine learning (ML) models were applied to data contained in Oregon’s state longitudinal data system. We used the generalized semi-parametric stochastic mixed-effects model to predict the on-track to graduation status of 9th grade students. ML methods were used to identify complex relationships between the large number of student and school variables contained in the data and the implementation status of a statewide intervention designed to support students at risk of not graduating. In the presentation, we outline how ML methods present an alternative to traditional parametric modeling approaches, and demonstrate how ML methods can more readily facilitate the identification of disproportionate or inequitable outcomes for underrepresented subgroups. We also demonstrate the use of modern data visualization techniques in order to enhance communication with key stakeholders. Implications for evaluation practice are discussed.
Relevance Statement: With increasing interest in the identification of disproportionate or inequitable outcomes for underrepresented population groups, evaluators now regularly implement designs and specify models that allow for the estimation of heterogenous treatment effects. In typical practice, a set of theory-derived covariates and a limited number of interaction terms are tested. Evaluators judiciously limit the number of terms included in the specification, so as to not overparameterize the model relative to sample size, and reduce multicollinearity and statistical power concerns. As a result however, the evaluator’s ability to identify complex higher order relationships or unexpected moderation between a treatment condition and group membership is diminished. In contrast, algorithm-driven models which leverage machine learning (ML) methods can provide the evaluator with an alternative means to find patterns among large numbers of variables which were either not specified or were more complex than were specified a priori. In this space, a relatively well-known algorithm-driven or ML model is the classification and regression tree or its variants (e.g., random forest, bagged or boosted trees). ML relies on recursive partitioning, whereby increasingly homogenous subgroups are formed, with respect to the outcome, by separating groups along several covariates (Capitaine, Genuer, & Thiébaut, 2021). The result is algorithmic identification of subgroups for which the specified parametric model fits differently.
In this presentation, we demonstrate the use of tree-based models to predict 9th grade on track to graduation with data from Oregon’s state longitudinal data system. Students are classified as on track to graduate if they have completed 25% of the coursework needed to graduate by the end of their freshmen year in high school. The key intervention variable is the implementation of 9th grade student success teams. Success teams review student course taking patterns, grades, and earned credits toward graduation, and offer tutoring support, provide mechanisms for credit recovery, and direct students to school sanctioned academic and health resources. We use tree-based mixed effects random forest models to identify patterns among the large number of student and school variables contained in the dataset, including piecewise relationships, multi-term interaction effects, and non-linear associations between predictor and outcome. For evaluators, the identification of complex subgroup relationships enables a more robust reporting to stakeholders and potentially more efficient use of targeted supports for participants at heightened risk of negative outcomes. We also use these methods to identify patterns within a subset of incorrectly classified cases (i.e., students who are labeled as being on track to graduation and do not graduate within four years, false negatives, and those who do graduate in four years while being labeled as not on track, false positives) to gain greater understanding of the contextual factors that underlie the misclassification. We believe that the novel findings that can be obtained from application tree-based models align with the conference theme, The Power of Story, and expect the presentation will attract a range of AEA and Cluster, Multi-site & Multi-level Evaluation TIG members with interests in the application of modern data science methods in program evaluation contexts.
