LimitLine

Learn · updated 2026-10-28

Hypothesis Tests: T Test, ANOVA and Chi-Square, and Which One to Use

By Nicholas Williford, Lean Six Sigma Black Belt · About LimitLine

A hypothesis test checks whether a difference in your data is bigger than random variation alone would produce; the p-value is the chance of a difference at least that big if there were really none.

Every test here answers one question: is the difference you see bigger than the noise would make on its own? The test you pick depends on what you measured and how many groups you compare. Pick the wrong one and the p-value can mislead.

Your questionDataTest
Do two groups have different means?Measurements, two independent groups2-sample t test (Welch)
Did the same parts change, before vs after?Measurements, paired (same parts twice)Paired t test
Do three or more groups differ?Measurements, 3+ groupsOne-way ANOVA
Does the defect rate depend on the shift (or line, or supplier)?Counts in categoriesChi-square test

Reading the p-Value

The p-value is the chance of seeing a difference at least this big if there were really no difference. Below 0.05 (the usual α) you call the difference statistically significant; above it you haven't shown a difference, which is not the same as showing there is none. Always look at the size of the difference and its confidence interval too: with enough data, a difference too small to matter becomes "significant".

Worked Example: One Data Set, Two Tests

Cushny and Peebles's data, made famous by Student in 1908: extra hours of sleep for 10 patients on each of two drugs. The same 10 patients took both, so the data are paired.

-202460.7 H-1.6 H-0.2 H-1.2 H-0.1 H3.4 H3.7 H0.8 H0 H2 H0.75Drug 11.9 H0.8 H1.1 H0.1 H-0.1 H4.4 H5.5 H1.6 H4.6 H3.4 H2.33Drug 2
Each dot is one patient; the rust bar is the mean. The groups overlap a lot, but nearly every patient slept longer on drug 2.
TesttdfpVerdict at α = 0.05
2-sample t (Welch), treating them as independent-1.86117.780.0794Not shown
Paired t, using each patient's difference-4.06290.0028Different

The same numbers give opposite verdicts. The paired test removes the patient-to-patient variation and looks only at each patient's change: drug 2 added 1.58 hours on average, and the change is consistent from patient to patient. Use the paired test whenever the same parts, people or machines are measured twice; ignoring the pairing throws that information away. These t values are exactly the ones R prints for this data set.

Three or More Groups: One-Way ANOVA

Plant weights under a control and two treatments, 10 plants each (the PlantGrowth data). ANOVA compares the variation between the group means with the variation within groups: F = 4.846, p = 0.0159, so at least one group differs. ANOVA doesn't say which; compare the groups' confidence intervals or run a follow-up test. Running three separate t tests instead inflates the chance of a false alarm.

Counts: The Chi-Square Test

Day shift: 20 defective, 30 good; night shift: 35 defective, 15 good. If the shift didn't matter, each cell would hold 27.5 or 22.5. χ² = Σ(observed − expected)² ÷ expected = 9.09 with 1 degree of freedom, p = 0.0026: the defect rate depends on the shift (an illustrative table; no continuity correction). The test needs expected counts of about 5 or more in every cell.

Run these tests in LimitLineHypothesis tests: t tests, ANOVA, proportions and chi-square, with a plain-English verdict (a Pro tool; Free users can try it on the example).

Sources

  1. R Core Team, R datasets documentation: sleep (Student, 1908) and PlantGrowth (Dobson, 1983); the t and F values above are those printed by R's own examples
  2. Student (1908), The Probable Error of a Mean, Biometrika 6(1), 1–25
  3. NIST/SEMATECH e-Handbook, §7.3 Comparisons based on data from two processes

The LimitLine Newsletter

About one email a month: new guides, tool updates and one practical SPC tip. Unsubscribe in one click.

We'll email you once to confirm before anything else is sent. Privacy