t-tests and ANOVA
Comparing one, two or many means: 1-sample, 2-sample and paired t-tests, one-way and two-way ANOVA, and multiple comparisons.
These tests compare averages. They assume roughly normal data (or reasonable sample sizes) and independent observations.
t-tests
- 1-sample t: is the mean different from a target? Example: is the average fill 500 g?
- 2-sample t: are two independent group means different? Example: supplier A vs supplier B. Use Welch's version, which does not assume equal variances.
- Paired t: the same units measured twice (before and after, gauge 1 vs gauge 2). It tests the mean of the differences and is much more sensitive than a 2-sample test for paired data.
ANOVA
Analysis of variance compares three or more means at once by splitting total variation into variation between groups and within groups. The F statistic is the ratio of the two.
- One-way ANOVA: one factor (four machines).
- Two-way ANOVA: two factors and their interaction (machine and shift).
- Eta squared (SS between / SS total) says what share of the variation the factor explains.
A significant ANOVA says at least one mean differs. To find which, use Tukey's comparisons (all pairs), Dunnett's (each against a control) or Hsu's MCB (compare with the best).
Assumptions to check
- Residuals roughly normal (probability plot of residuals).
- Similar variances across groups (Levene's test); if not, use Welch's ANOVA.
- Independent observations; randomize the run order.
Questions and answers
Can I just run several 2-sample t-tests instead of ANOVA?
That inflates the false alarm rate. Use ANOVA followed by a multiple comparison method.
My groups have very different sample sizes. Is that a problem?
ANOVA handles unequal sizes, but it becomes sensitive to unequal variances. Use Welch's ANOVA when sizes and variances both differ.
