The Hypothesis Testing Framework
Hypothesis testing is a formal procedure for deciding whether data provides enough evidence to support a claim. It frames the question as a contest between two hypotheses.
The two hypotheses
- The null hypothesis states that there is no effect or no difference. It is the default we assume true.
- The alternative hypothesis states that there is an effect.
We never prove the null. Instead we ask whether the data is too surprising under the null to keep believing it.
The procedure
- Choose a test statistic that summarizes the data.
- Assume the null is true and find how extreme the statistic is.
- If the result is sufficiently unlikely under the null, we reject the null in favor of the alternative.
Two kinds of error
- A Type one error rejects a true null, a false alarm.
- A Type two error fails to reject a false null, a missed effect.
The threshold called alpha, often 0.05, controls the Type one error rate we are willing to tolerate.
Key idea
Hypothesis testing pits a null of no effect against an alternative, rejecting the null only when data is too unlikely under it, balancing Type one and Type two errors.