Once Upon a Time, There Was a P-Value… Our Evolving Story of Statistical Significance

Abstract Information: For years, our evaluation team analyzed and reported quantitative findings using the conventional alpha value, 0.05, to indicate statistical significance. As our data analysis plans grew more robust, we began to incorporate effect sizes and found ourselves facing a dilemma: Our calculated p-values often hovered above 0.05, even when we detected large effects. What did this mean for our interpretation of the data? For users’ interpretations?

Realizing that our stories of significance influence how users interpret and react to findings, we began to evolve our thinking. Why DID we predetermine alpha at 0.05? It was not mandated by the American Statistical Association. Indeed, there exists a Line of Enlightened Judgement for choosing an optimal alpha based on sample size, prior beliefs, and minimizing losses to Type I and II errors.

This presentation covers our experience dethroning alpha=0.05 and re-writing our stories of statistical significance using a more nuanced, evaluation-centric process.

Relevance Statement: Our team develops and implements multi-year, multi-level evaluations of individual behaviors and the policy, systems, and environments that can profoundly influence behaviors. We have quantitatively assessed individual behaviors for thousands of students, the environments of hundreds of early care and education providers, the quality of tens of physical activity resources, success factor strength for a small number of food systems coalitions… Each evaluation comes with its own, unique statistical analysis plan appropriate to its purpose.

But until recently, there was one thing common to all of these plans: our significance level. For years, we analyzed and reported quantitative findings using the conventional alpha, 0.05, to indicate statistical significance. We also recognized 0.01 (highly significant) and 0.001 (very highly significant). When it came to interpreting the results, we knew p-values were just one of many considerations, and we took care to acknowledge sample size and the magnitude of change in our interpretations. Even so, it seemed the glorious asterisk—that tiny p< 0.05 indicator—was getting much more attention from our users than it warranted. We wondered: What role are we playing in focusing attention on that asterisk?

In response, we sought to offer users another metric by which to gauge results: the effect size. As we incorporated effect sizes into our work, we faced a new dilemma. For evaluations with relatively small sample sizes, our calculated p-values often hovered above 0.05—even when we detected large effects. For samples in the thousands, we saw that many “significant” changes had low-to-no measurable effect. How, then, should we interpret the findings? And how were we to help those unfamiliar with concepts like effect sizes and p-values use the data we shared to make judgements of their own?

Realizing that our stories—told through numbers, data visualizations, and text—influence how users interpret and react to findings, we began to evolve our thinking. Why DID we set alpha at 0.05? Intuitively, it made general probabilistic sense, and most of our sample sizes fell within a range that worked well for this threshold. But as we dove headlong into the (somewhat sordid) history of p-values¹, the American Statistical Association Statement on P-Values², and other literature surrounding p-values, we altered our approach. We considered the Line of Enlightened Judgement for choosing an optimal alpha based on sample size, prior beliefs, and minimizing losses to Type I and II errors. This presentation offers an overview of our experience re-writing our stories of statistical significance. As we dethrone alpha=0.05, we advance a new “advisory council” of sample size, losses from error, prior beliefs, and our overarching evaluation purpose. We have begun to engage our priority users in these conversations and acknowledge the vital role that lived user experience plays in how we interpret findings.

1. Kennedy-Shaffer L. (2019) Before p< 0.05 to Beyond p< 0.05: Using History to Contextualize p-Values and Significance Testing. The American Statistician, 73(S1): 82-90, DOI:10.1080/00031305.2018.1537891.

2. Wasserstein RL, Lazar NA. (2016) The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2):129-133, DOI:10.1080/00031305.2016.1154108.