Hypothesis testing and p-values
Null and alternative hypotheses, p-values, Type I and II errors, power, and statistical vs practical significance.
A hypothesis test asks whether the evidence in a sample is strong enough to conclude something about the population, such as "the new supplier's parts are heavier."
The steps
- State the null hypothesis (no difference, no effect) and the alternative .
- Choose the significance level α (usually 0.05) before looking at the data.
- Calculate the test statistic and its p-value.
- If , reject : the difference is statistically significant. Otherwise, you do not have enough evidence (which is not the same as proving there is no difference).
What a p-value is
The probability of getting a result at least as extreme as the one observed, if were true. A small p-value means the data would be surprising if nothing had changed.
Errors and power
- Type I error (α): concluding there is a difference when there is none (a false alarm).
- Type II error (β): missing a real difference.
- Power (1 − β): the chance of detecting a difference of a given size. Plan for 80% or more.
Power grows with sample size, with the size of the difference, and with lower process variation.
Practical significance
With enough data, a difference of 0.0001 mm becomes "significant". Always ask whether the difference is big enough to matter for the product, and look at the confidence interval of the difference.
One-sided and two-sided tests
Use a one-sided test only if you decided beforehand that only one direction matters (for example, "is the new coating's life longer?").
Questions and answers
My p-value is 0.06. Is there an effect or not?
The evidence is not strong enough at α = 0.05, but it is not proof of no effect either. Look at the size of the difference and its confidence interval, and consider collecting more data.
Can I run many tests and report the significant ones?
Running many tests inflates false positives. With 20 tests at α = 0.05 you expect one "significant" result by chance. Use corrections (Bonferroni, Tukey for ANOVA) or plan fewer, focused tests.
