55 - Evaluating military maternity leave policy: How our analytic decisions impact the stories we tell

Abstract Information: In recent years, the U.S. Department of Defense has enhanced supportive policies for female servicemembers; for example by expanding paid parental leave. Prior to 2015, all services offered new mothers 6 weeks of leave. In 2015, the Navy and Marine Corps expanded to offer 18 weeks, while the Army and Air Force maintained their 6-week leave policy. Another policy change occurred in 2016 when all military service branches shifted to offer 12 weeks. While this immediately doubled the leave available in the Army and Air Force, it dramatically reduced the leave available for Navy and Marine Corps mothers. Evaluating the impact of these policy changes sheds light on the benefits of maternity leave – and the impacts of health policies in general.
Our evaluation team sought to examine how changes in maternity leave policy were related to health outcomes in the first year postpartum. What we learned through our analyses, however, was the critical importance of each design and analytic decision, as it can greatly impact the evidence we provide and the stories we tell.
Our analyses used existing personnel and medical data, drawn from a secure data repository that houses extensive military records. The evaluation sample included servicemembers who gave birth between 2014 and 2019 (N = 48,730). Participants were identified using live birth diagnostic codes from healthcare administrative data. Diagnostic codes were also used to identify postpartum health outcomes of interest. We used three of the chronic conditions categorized by the Functional Comorbidity Index (FCI; Sears & Rundell, 2020) to indicate common postpartum health concerns (depression, anxiety, and back conditions). Additionally, the MSI matrix (Hauret et al., 2010) served as a guide to identify musculoskeletal conditions, both inflammation/pain and injury (i.e., dislocation, derangement, sprain, or stress fracture).
Our first approach was to evaluate the associations between maternity leave policies and mothers’ health outcomes with a Cox regression analysis wherein we used maternity leave policy (i.e., 6, 12, or 18 weeks) as the primary predictor of health outcomes, while including as covariates known correlates of postpartum health (e.g., age, race) and potentially confounding factors (e.g., service branch, year). These analyses identified many strong benefits of longer leave policy: mothers with longer leaves had significantly lower rates of depression, back conditions, and injuries. Although we incorporated control variables to account for issues such as service branch and year, our Cox regression analysis approach did not fully capture the nuance within our data, and a different approach yielded different insights.
In our second analytic approach, we used a time series and difference-within-difference strategy. This strategy emphasizes the differences between branches at certain times, rather than the leave policy itself. By taking both a branch and time focus, we saw different patterns of results. These analyses demonstrated that when maternity leave decreased (i.e., 18 to 12 weeks), health problems increased among postpartum mothers, rather than demonstrating a decrease of health concerns with increased maternity leave access. Both analytic approaches provide compelling, but different pictures, of the impact of changing maternity leave policies.

Relevance Statement: This policy evaluation focused on an important and highly publicized military issue: the health of female service members. We aimed to understand how different military maternity leave policies impacted female servicemembers. Our results provide valuable information by evaluating the implications of these wide-sweeping policy changes. With the recent expansion of parental leave across all service branches, it is critical to identify evidence related to this key policy.
While we found important benefits of offering longer leave policies, what we learned along the way was even more critical for approaches to evaluation design and analysis. We were reminded that data can tell very different stories depending on our conceptual, theoretical, design, and analytic choices. As evaluators, our approaches to project design, data analysis, and data visualization have rippling effects into what we learn, what we recommend to program and policy decision-makers, and the stories we tell about the people who use the programs we evaluate. It is crucial for our teams to consider not only the conceptual or theoretical foundation of our selected approaches, but also the practical realities of the data before us. Our decisions must be rooted in an understanding of theory and statistics so that we are conducting appropriate and rigorous evaluations. However, we also must ensure that we fully understand and explore the data so that we can accurately and ethically reflect the truth of the people our evaluations represent.
Examining data from multiple angles can help us tell a more coherent, more accurate, and often more complex story. Although stakeholders and funders may prefer a simple and direct explanation, we must maintain our dedication to understanding the complexities and seek to communicate nuances in ways they can be easily understood and adapted by those in charge of the program.