37 - Reaching the Finish Line: Impacts of Coaching on College Completion

Abstract Information: A college degree typically helps graduates earn more when they enter the labor force and is a key driver of economic mobility. However, many students who enroll in college do not finish, reflecting the numerous academic financial, and social emotional barriers students may face to succeeding in college and completing a degree, from being underprepared for college coursework to struggling to afford tuition or other expenses like books and housing. The COVID-19 pandemic brought additional challenges and amplified existing ones. Funded by The Boston Foundation, Success Boston brings together Boston Public Schools (BPS), local colleges and universities, the City of Boston, and various nonprofit organizations. In 2022, the initiative recommitted to improving BPS graduates’ postsecondary completion rates, with a target rate of 70 percent across all groups of students. A core strategy of Success Boston, transition coaching offers support to students in addressing challenges they face in college. The Success Boston Coaching (SBC) model matches recent graduates of BPS with a coach, who meets with them regularly during their first two years of college and supports them as they navigate academic, financial, and social barriers. Our quasi-experimental study looks at SBC’s effects on students’ four-, five-, and six-year postsecondary completion rates, across more than 9,000 students across 50 colleges who initially enrolled in college in 2015, 2016, or 2017. To do this, we conducted propensity score matching to construct a comparison group with similar demographic and baseline academic achievement to SBC students, and who enrolled in the same college in the same year. To measure SBC’s effects, for each outcome, we estimate a linear regression model that includes indicators for matching blocks and matching characteristics, among the matched comparison and SBC students. We find that students who received SBC graduated in four years at rates that are 21 percent higher than those of noncoached students and in five years at rates that are 15 percent higher. These effects are meaningful because completing college in four years or five years, rather than in six or more years, can help students avoid the costs of additional years of college and allow them to enter the labor market sooner. At six years after entering college, differences in graduation rates between coached and noncoached students were no longer statistically significant, suggesting that some noncoached students had caught up to the coached students at this point. Findings suggest that the coaching program could be an important component in helping move citywide postsecondary completion toward its of 70 percent target. That said, only about 49 percent of coached students, and a similar proportion of all BPS students, complete college in six years, well short of the 70 percent target. BPS students will continue to need support, perhaps through extending coaching beyond students’ first two years in college, for the city to achieve the initiative’s goal.

Relevance Statement: The 2023 conference’s theme is storytelling: the ability to use evaluations to present narratives, and describe how programs or initiatives evolve over time, and what improvements have been made and what still needs to be done. Our study tells the story of a transition coaching program, borne out of a commitment to doubling what, at the program’s onset, was a 35% postsecondary completion rate for Boston Public Schools (BPS) graduates, to reach a 70% target across all students. Although coaches from nonprofit organizations directly deliver the coaching, the coaching is part of a broader Success Boston initiative that brings together nonprofits, the City of Boston, the Boston Foundation, BPS, and dozens of local colleges to help BPS graduates overcome some of the challenges they may face in college. The study examines the extent to which this coaching succeeds in helping students complete college. Examining more than 9,000 students enrolling in more than 50 colleges across three cohorts, we look at the effects of the Success Boston Coaching (SBC) program on graduation rates four, five, and six years after entering college. As part of the storytelling, we situate our findings—of significant positive impacts after four and five years, but not six—in the context of the initiative’s history, as well as its current objectives and framework. We also consider how the existing cross-sector collaboration may shed light on helpful practices and strengths, and also be used to improve coaching efforts and other college completion initiatives going forward, particularly in line with the Success Boston equity framework focus on addressing structural and systematic barriers in higher education. Our study also represents an example of a rigorous quasi-experimental design that can generate useful and policy-relevant evidence. Given that it was not feasible to conduct an experimental design in this context, we used a quasi-experimental design that compares SBC students—regardless of their coaching participation level—with a comparison group of similar students, and that can account for as many of the observable confounders as possible. Guided by the current methodological research on best quasi-experimental design practices, we constructed such a comparison group using a matching process that had two features: matches were local (with matching blocks based on cohort, high schools, and college) and also focal (matching was done using baseline characteristics believed to predict both selection into treatment and the outcome, including gender, race/ethnicity, high school academic achievement, socioeconomic status). Within each block, we estimated a separate propensity score based on student-level factors related to outcomes, such as high school GPA, income level, and socio-economic status (SES), and used those scores to match treatment and comparison students in the same block. Moreover, we use a doubly robust approach in which we include matching blocks and matching characteristics in our impact model, for each sample of matched treatment and comparison students. Our estimates can help generate important insights about SBC’s effects and suggest an evaluative approach that may be useful in other contexts as well.