Blogs
What “significant” means
The word was captured by a calculation. You are allowed to ask for it back.
In ordinary speech, significant means large enough to matter. In a results table, it usually means that a particular calculation came out below a threshold, often 0.05. Those are not the same sentence. Confusing them is how a tiny difference becomes a press release, and how a real difference in a small experiment gets binned as “no result.”
What the number is
A p-value answers a narrow question: if the world were really the null case — no difference, no association — how often would a study like this one produce a result at least as extreme as the one in front of us? A small p-value means “surprising, if nothing is going on.” It does not mean the effect is large. It does not mean the effect is caused by the thing you hope. It does not mean the result will appear again. Sample size can manufacture surprise out of a difference too small to act on. A huge, real effect can fail the threshold if you only measured four plants.
A fictional trial, labeled as fictional
Suppose two fertilizer regimes and a hundred students each growing one pot of the same bean. Regime B grows plants 2 millimetres taller on average, the scatter is tiny because the room is controlled, and the p-value is 0.01. The result is statistically surprising and agriculturally dull. Suppose instead a field trial with eight farms, a 20 percent yield difference, and a p-value of 0.08 because farms are messy. The second result missed a conventional threshold and is the one a farmer might still want repeated. Nothing in this paragraph is data from a real study. It is a picture of why the threshold and the importance come apart.
What to ask instead
How big was the difference, in units a person can picture? How many independent units were there — students, plots, patients, not repeated measurements dressed up as samples? What was the control? Was the comparison the one the study claimed to care about before the data arrived? The quiz has a question on this for a reason. If you remember one line: significance is about surprise under a model. Importance is about the world. Report both, and do not let either word do the other’s job.
Leave a Reply