Abstract Information: The presentation describes the impact assessment of a STEM education promotion program in Brazilian public schools. Sponsored by the World Education & Development Fund, the program reaches 1,600 high schools by offering teacher training in STEM subjects and providing science education infrastructure for schools. The evaluation method adopted is Staggered Difference in Difference, a cutting-edge program evaluation technique applied in cases where treatment units enter the program in a phased manner. Using a public microdata panel from Brazil, it was possible to conclude that the program has a positive impact of up to 2.5% on student grades in Natural Sciences, but the impact is not sustained over time and is only observed in the first two years of implementation.
Relevance Statement: The main contribution of this presentation is methodological. The schools in the evaluated program are not randomly selected, which required the adoption of a non-experimental method. In addition, another challenge is that new schools are admitted to the program in a staggered manner, which makes it difficult to establish a clear control group. These are common problems in impact evaluation scenarios and even make it difficult to adopt one of the most traditional non-experimental evaluation methods, Difference-in-Differences. Recent econometrics literature has discussed ways to evaluate program impact in this scenario. One of the most recent articles - Callaway and Sant'Anna (2021), "Difference-in-Differences with Multiple Time Periods", Journal of Econometrics (https://doi.org/10.1016/j.jeconom.2020.12.001) - proposes an efficient and feasible way to circumvent the problem by estimating "Group-Time Average Treatment Effects," parameters that can be grouped to enable both impact analyses with a temporal cut and a cut by program entry cohort. Additionally, this method already has implementation in major statistical software. Therefore, our presentation seeks, through our case study, to discuss the problem, present the methodological solution, and expose our impressions on the advantages and disadvantages of such methodology.
