That ≈0.01 SD effect is actually 0.0087 +/- 0.0043 (standard error), giving a 95% confidence interval of roughly 0.0003 to 0.0171 standard deviations.
So the effect could be tiny.
Worse, the report says it excludes observations outside the 0.5th and 99.5th percentiles of the 2021–22 fall-to-spring score-change distribution. I'm not sure how this is defensible. Either you trust your measurement instrument or you don't.
If you're using a difference-in-differences design, excluding observations can change what you're measuring, particularly if the exclusions differ between treatment and control groups or over time. Without trimming those observations, the estimated effect drops from 0.0087 SD to 0.0058 SD (and is no longer statistically significant).
The authors preregistered their study, but there's nothing in the report indicating that the trimming of 'extreme' values was part of that preregistration...
Sounds like they noticed a few extreme outliers were pulling the difference down and hacked it out. Methodologically that’s really sus to do this after the study is done. But generally I think it’s okay to exclude extreme outliers - we generally assume Gaussian distributions, but the real world happens and messes things up. Having a few kids that are 5 or 6 sigma out pulling your averages - I can believe the rules are different for them. But this should have been decided ahead of time.
I emailed the authors and received a reply. That part of the study was not preregistered.
My message to them:
Their reply:
Aside:
Wow! I love this.
I love that you bothered to email them with a thought out question.
I love that they bothered to respond with a thought out answer and they took you quite seriously.
I love that you bothered to tell us about the exchange.
Thank you so much for all of this. This is what we should all strive to do with our time and as a community.
Alright mate
Yeah! Fuck follow-through!
Interesting that their take is that the outliers represent measurement errors. Plausible and reasonable. But the fact that it changes the conclusion of the study deserves skepticism.
The outliers would have made the claimed effect even stronger. They were trying not to overhype their findings.
Phenomenal. This is why I come to Hacker News. Thanks so much.
Respect! Good science