Learn · updated 2026-10-28
Hypothesis Tests: T Test, ANOVA and Chi-Square, and Which One to Use
A hypothesis test checks whether a difference in your data is bigger than random variation alone would produce; the p-value is the chance of a difference at least that big if there were really none.
Every test here answers one question: is the difference you see bigger than the noise would make on its own? The test you pick depends on what you measured and how many groups you compare. Pick the wrong one and the p-value can mislead.
| Your question | Data | Test |
|---|---|---|
| Do two groups have different means? | Measurements, two independent groups | 2-sample t test (Welch) |
| Did the same parts change, before vs after? | Measurements, paired (same parts twice) | Paired t test |
| Do three or more groups differ? | Measurements, 3+ groups | One-way ANOVA |
| Does the defect rate depend on the shift (or line, or supplier)? | Counts in categories | Chi-square test |
Reading the p-Value
The p-value is the chance of seeing a difference at least this big if there were really no difference. Below 0.05 (the usual α) you call the difference statistically significant; above it you haven't shown a difference, which is not the same as showing there is none. Always look at the size of the difference and its confidence interval too: with enough data, a difference too small to matter becomes "significant".
Worked Example: One Data Set, Two Tests
Cushny and Peebles's data, made famous by Student in 1908: extra hours of sleep for 10 patients on each of two drugs. The same 10 patients took both, so the data are paired.
| Test | t | df | p | Verdict at α = 0.05 |
|---|---|---|---|---|
| 2-sample t (Welch), treating them as independent | -1.861 | 17.78 | 0.0794 | Not shown |
| Paired t, using each patient's difference | -4.062 | 9 | 0.0028 | Different |
The same numbers give opposite verdicts. The paired test removes the patient-to-patient variation and looks only at each patient's change: drug 2 added 1.58 hours on average, and the change is consistent from patient to patient. Use the paired test whenever the same parts, people or machines are measured twice; ignoring the pairing throws that information away. These t values are exactly the ones R prints for this data set.
Three or More Groups: One-Way ANOVA
Plant weights under a control and two treatments, 10 plants each (the PlantGrowth data). ANOVA compares the variation between the group means with the variation within groups: F = 4.846, p = 0.0159, so at least one group differs. ANOVA doesn't say which; compare the groups' confidence intervals or run a follow-up test. Running three separate t tests instead inflates the chance of a false alarm.
Counts: The Chi-Square Test
Day shift: 20 defective, 30 good; night shift: 35 defective, 15 good. If the shift didn't matter, each cell would hold 27.5 or 22.5. χ² = Σ(observed − expected)² ÷ expected = 9.09 with 1 degree of freedom, p = 0.0026: the defect rate depends on the shift (an illustrative table; no continuity correction). The test needs expected counts of about 5 or more in every cell.
Sources
- R Core Team, R datasets documentation: sleep (Student, 1908) and PlantGrowth (Dobson, 1983); the t and F values above are those printed by R's own examples
- Student (1908), The Probable Error of a Mean, Biometrika 6(1), 1–25
- NIST/SEMATECH e-Handbook, §7.3 Comparisons based on data from two processes
The LimitLine Newsletter
About one email a month: new guides, tool updates and one practical SPC tip. Unsubscribe in one click.