It’s mentioned on the podcast pretty much every week. But what does “statistical significance” actually mean?
In this episode of The Studies Show, Tom and Stuart start 2024 off with the most exciting subject possible: p-values. THRILL as they discuss statistical misconceptions! MARVEL as they talk about how “effect size” differs from “statistical significance”! CHUCKLE as they resort to endless coin-flipping analogies! And GASP as they discuss ways to stop scientists from “hacking” their p-values and ending up with misleading research!
The Studies Show is brought to you by Works in Progress magazine - an online magazine full of essays about science, technology, and human progress. Works in Progress is the kind of magazine that makes its readers massively better-informed about every subject it covers, with deeply-researched articles by experts in the relevant fields - and it’s all free. Check it out at their site right here.
Show notes
89% of psychology textbooks get p-values wrong
Letter on how the research on power-posing went wrong
The classic “false-positive psychology” paper on how p-hacking can get you any result you want
How p-hacking is just “overfitting” by another name
List of weasel terms like “approaching significance”
Reading on a screen before bed “might be killing you”!
The (much less scary) relevant study
Tom’s BuzzFeed News article on the idea to lower the p-value threshold to 0.005
The original paper, plus the response arguing that scientists should “justify their alpha”
Registered Reports, and how they can deter p-hacking
Credits
The Studies Show is produced by Julian Mayers at Yada Yada Productions.
Episode 23: Statistical significance