Learning to Love Uncertainty (Why We Need to Stop Using “Not Statistically Significant)

I want to talk to you about why you should stop saying “not statistically significant” based on sample size alone.

The term “not statistically significant” should only be applied to a hypothesis, not a sample size, and even then it’s an arbitrary line we’ve drawn to lump results into a false dichotomy of “certain” or “uncertain”, instead of talking about what level of uncertainty we are actually dealing with.

“Statistical significance” relates to how strongly the data you have can disprove your hypothesis.

Let’s say we have a survey about how much people like the finale episode of the Sopranos on a scale from 1–10. One of the hypotheses we’re trying to disprove is that men liked the episode more than women. We sample some people out of the millions who watched the episode. Depending on the amount of answers we have, the kind of answers we have, and the variance between answers, the difference between between men and women may or may not be “statistically significant”; a cutoff drawn somewhere along the line from very certain to totally uncertain. In this case, the term is being applied to the significance of the difference.

But there’s another way that people use “not statistically significant” all the time that is absolutely unhelpful, hurtful and down right incorrect.

In these cases it’s used as a communication convention, a shortcut norm to save people from having to engage with higher amounts of uncertainty. It’s when people have a small sample size and they decide that they are so uncertain about how well…