Resources What's New Support Download Contact Sales: Arthur-Industrial@outlook.com
Home › Resources › Hypothesis testing and p-values

Hypothesis testing and p-values

Null and alternative hypotheses, p-values, Type I and II errors, power, and statistical vs practical significance.

A hypothesis test asks whether the evidence in a sample is strong enough to conclude something about the population, such as "the new supplier's parts are heavier."

The steps

  • State the null hypothesis (no difference, no effect) and the alternative .
  • Choose the significance level α (usually 0.05) before looking at the data.
  • Calculate the test statistic and its p-value.
  • If , reject : the difference is statistically significant. Otherwise, you do not have enough evidence (which is not the same as proving there is no difference).

What a p-value is

The probability of getting a result at least as extreme as the one observed, if were true. A small p-value means the data would be surprising if nothing had changed.

Errors and power

  • Type I error (α): concluding there is a difference when there is none (a false alarm).
  • Type II error (β): missing a real difference.
  • Power (1 − β): the chance of detecting a difference of a given size. Plan for 80% or more.

Power grows with sample size, with the size of the difference, and with lower process variation.

Practical significance

With enough data, a difference of 0.0001 mm becomes "significant". Always ask whether the difference is big enough to matter for the product, and look at the confidence interval of the difference.

One-sided and two-sided tests

Use a one-sided test only if you decided beforehand that only one direction matters (for example, "is the new coating's life longer?").

Questions and answers

My p-value is 0.06. Is there an effect or not?

The evidence is not strong enough at α = 0.05, but it is not proof of no effect either. Look at the size of the difference and its confidence interval, and consider collecting more data.

Can I run many tests and report the significant ones?

Running many tests inflates false positives. With 20 tests at α = 0.05 you expect one "significant" result by chance. Use corrections (Bonferroni, Tukey for ANOVA) or plan fewer, focused tests.

In RS-1: The Compare Groups tab has tests for means, variances, proportions and counts, nonparametric and equivalence tests.

← All articles