The P Value and Significance

The P Value and Significance

The p value is the number reported by nearly every statistical test, and it is also one of the most misunderstood.

The correct definition

A p value is the probability of observing data at least as extreme as what you saw, assuming the null hypothesis is true. A small p value means your data would be surprising if there were truly no effect, which counts as evidence against the null.

What it is not

• It is not the probability that the null is true.
• It is not the probability your result happened by chance in plain terms.
• A large p value does not prove the null; it only shows a lack of strong evidence.

Significance thresholds

If the p value falls below a chosen alpha, often 0.05, the result is called statistically significant and we reject the null. This cutoff is a convention, not a law of nature.

Practical warnings

• Statistical significance is not the same as practical importance. A tiny effect can be significant with a huge sample.
• Testing many hypotheses inflates false positives, so corrections are needed for multiple comparisons.

Key idea

A p value is the chance of data at least as extreme under the null, and small values give evidence against the null but never prove it true or false.