LimitLine

Validation: checking LimitLine’s math

Generated 2026-10-21 · code version aa089a2 (2026-10-21) · Node.js 22.16.0

341 of 341checks pass, across 29 reference cases

What was tested

A script loads LimitLine’s calculation modules, spc-core.js, stats-core.js and lean-core.js, runs each case below, and writes this page. These are the same source files that are built into the app at limitline.org/app. The script runs them in Node.js, not in a browser; both use the same IEEE double-precision arithmetic.

Each row compares one number from LimitLine with the expected value. Under each expected value is where it comes from:

Published values are rounded, so the tolerance is set by how many digits the source prints. These are the same tolerances LimitLine’s automated tests use.

What this does not cover

Variables control charts 47/47

Individuals (I-MR), X̄-R and X̄-S limits, and the control-chart constants they use.

NIST individuals chart: flow rate, 10 batches

9/9 pass

Data: nist-6322-flow-rate.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Moving ranges (rounded to 0.1)2, 2.3, 1.4, 3.5, 3.4, 1.4, 0.2, 1.2, 1.5printed in the source2, 2.3, 1.4, 3.5, 3.4, 1.4, 0.2, 1.2, 1.5noneexact matchPASS
Mean (centre line)50.81printed in the source50.810± 1e−4PASS
Average moving range MR̄1.8778printed in the source1.877777777778−2.2e−5± 1e−4PASS
Individuals UCL55.8041printed in the source55.80408983452−1.0e−5± 0.001PASS
Individuals LCL45.8159printed in the source45.815910165481.0e−5± 0.001PASS
MR chart UCL = 3.267 × MR̄6.1347726computed here from the source’s data and formula6.1347−7.3e−5± 0.001PASS
Points beyond the 3-sigma limits0printed in the source0noneexact matchPASS
Phase I: limits from the first 10 of 13 points, UCLThree extra points (60, 50, 51) added after the baseline must not move the limits.55.8041printed in the source55.80408983452−1.0e−5± 0.001PASS
Phase I: limits from the first 10 of 13 points, LCL45.8159printed in the source45.815910165481.0e−5± 0.001PASS

NIST table of control-chart constants A2, D3, D4 (n = 2 to 10)

27/27 pass

LimitLine derives d2 and d3 by numerical integration and c4 from the gamma function; this compares the resulting factors with NIST’s printed table.

Show all 27 checks
StatisticExpectedLimitLineDifferenceToleranceResult
A2, n = 21.88printed in the source1.879969818288−3.0e−5± 0.001PASS
D3, n = 20printed in the source00± 0.001PASS
D4, n = 23.267printed in the source3.266523688828−4.8e−4± 0.001PASS
A2, n = 31.023printed in the source1.0233259525863.3e−4± 0.001PASS
D3, n = 30printed in the source00± 0.001PASS
D4, n = 32.575printed in the source2.574593665255−4.1e−4± 0.001PASS
A2, n = 40.729printed in the source0.7285974499089−4.0e−4± 0.001PASS
D3, n = 40printed in the source00± 0.001PASS
D4, n = 42.282printed in the source2.2820546448095.5e−5± 0.001PASS
A2, n = 50.577printed in the source0.5768190730159−1.8e−4± 0.001PASS
D3, n = 50printed in the source00± 0.001PASS
D4, n = 52.115printed in the source2.114496137029−5.0e−4± 0.001PASS
A2, n = 60.483printed in the source0.48324654313692.5e−4± 0.001PASS
D3, n = 60printed in the source00± 0.001PASS
D4, n = 62.004printed in the source2.003831266449−1.7e−4± 0.001PASS
A2, n = 70.419printed in the source0.41928346042232.8e−4± 0.001PASS
D3, n = 70.076printed in the source0.07570367850434−3.0e−4± 0.001PASS
D4, n = 71.924printed in the source1.9242963214963.0e−4± 0.001PASS
A2, n = 80.373printed in the source0.3725274556687−4.7e−4± 0.001PASS
D3, n = 80.136printed in the source0.1361723798821.7e−4± 0.001PASS
D4, n = 81.864printed in the source1.863827620118−1.7e−4± 0.001PASS
A2, n = 90.337printed in the source0.3366969357212−3.0e−4± 0.001PASS
D3, n = 90.184printed in the source0.18401834324911.8e−5± 0.001PASS
D4, n = 91.816printed in the source1.815981656751−1.8e−5± 0.001PASS
A2, n = 100.308printed in the source0.30826327064752.6e−4± 0.001PASS
D3, n = 100.223printed in the source0.22302445808462.4e−5± 0.001PASS
D4, n = 101.777printed in the source1.776975541915−2.4e−5± 0.001PASS

Published c4 and d2 values

4/4 pass
StatisticExpectedLimitLineDifferenceToleranceResult
c4, n = 20.7979printed in the source0.79788−2.0e−5± 1e−4PASS
c4, n = 50.94printed in the source0.93999−1.0e−5± 1e−4PASS
c4, n = 250.9896printed in the source0.989644.0e−5± 1e−4PASS
d2, n = 253.931printed in the source3.93063−3.7e−4± 0.001PASS

X̄-R and X̄-S limits recomputed by hand with NIST’s printed constants

7/7 pass

The data are a small LimitLine test set (5 subgroups of 5), not a published example. Expected values are computed in this script from the raw data with NIST’s printed A2 = 0.577, D4 = 2.115 and c4 = 0.9400, without using LimitLine’s code.

Data: xbar-r-test-5x5.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Grand mean X̿10.036computed here from the source’s data and formula10.0360± 1e−12PASS
X̄ chart UCL (X̿ + 0.577 R̄)10.33604computed here from the source’s data and formula10.33594591797−9.4e−5± 5e−4PASS
X̄ chart LCL (X̿ − 0.577 R̄)9.73596computed here from the source’s data and formula9.7360540820329.4e−5± 5e−4PASS
R chart UCL (2.115 R̄)1.0998computed here from the source’s data and formula1.099537991255−2.6e−4± 5e−4PASS
R chart LCL (D3 = 0 for n = 5)0computed here from the source’s data and formula0noneexact matchPASS
X̄-S chart UCL (X̿ + 3 s̄ / (c4 √5))10.32015959673computed here from the source’s data and formula10.320162619733.0e−6± 1e−4PASS
S chart UCL (B4 from c4 = 0.9400)0.4158743549349computed here from the source’s data and formula0.41589416701482.0e−5± 0.001PASS

Attribute control charts 27/27

c, p, np, u and Laney p′ charts.

NIST c chart: defects on 25 wafers

4/4 pass

Data: nist-6331-wafer-defects.csv

StatisticExpectedLimitLineDifferenceToleranceResult
c̄ (centre line)16printed in the source160± 1e−4PASS
UCL28printed in the source280± 1e−4PASS
LCL4printed in the source40± 1e−4PASS
Wafers beyond the limits (1-based)24printed in the source24noneexact matchPASS

NIST p chart: 30 wafers of 50 chips

8/8 pass

NIST prints the fraction defective per wafer. Counts are fraction × 50. Limits are computed here from those data with the standard formula p̄ ± 3√(p̄(1 − p̄)/n), independently of LimitLine’s code.

Data: nist-6332-wafer-proportions.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Total defectives347computed here from the source’s data and formula347noneexact matchPASS
p̄ = 347 / 15000.2313333333333computed here from the source’s data and formula0.23133333333330± 1e−12PASS
UCL0.4102391185947computed here from the source’s data and formula0.41023911859470± 1e−12PASS
LCL0.05242754807193computed here from the source’s data and formula0.052427548071930± 1e−12PASS
UCL to 4 decimals0.4102computed here from the source’s data and formula0.41023911859473.9e−5± 0.001PASS
Wafers beyond the limits (1-based)15, 23computed here from the source’s data and formula15, 23noneexact matchPASS
np chart centre = n·p̄11.56666666667computed here from the source’s data and formula11.566666666670± 1e−12PASS
np chart UCL20.51195592974computed here from the source’s data and formula20.511955929740± 1e−12PASS

Montgomery u chart: defects in computers, 20 samples of 5

3/3 pass

The centre line is printed. The limits are computed from the printed centre line as 1.93 ± 3√(1.93/5) and compared at 2 decimals.

Data: montgomery-u-computers.csv

StatisticExpectedLimitLineDifferenceToleranceResult
ū (centre line)1.93printed in the source1.930± 1e−12PASS
LCL0.07computed here from the source’s data and formula0.06613305195891→ 0.07−3.9e−3same at 2 decimalsPASS
UCL3.79computed here from the source’s data and formula3.793866948041→ 3.793.9e−3same at 2 decimalsPASS

Montgomery u chart: dyeing rolls of cloth, unequal inspection units

5/5 pass

The centre line is printed. Per-roll limits are computed from ū = 153/107.5 with ū ± 3√(ū/n) and compared at 2 decimals.

Data: montgomery-u-cloth-rolls.csv

StatisticExpectedLimitLineDifferenceToleranceResult
ū (centre line)1.423printed in the source1.4232558139532.6e−4± 5e−4PASS
UCL, roll 2 (8 units, widest)2.69computed here from the source’s data and formula2.688626427923→ 2.69−1.4e−3same at 2 decimalsPASS
UCL, roll 3 (13 units, narrowest)2.42computed here from the source’s data and formula2.41589419127→ 2.42−4.1e−3same at 2 decimalsPASS
LCL, roll 20.16computed here from the source’s data and formula0.1578851999839→ 0.16−2.1e−3same at 2 decimalsPASS
LCL, roll 30.43computed here from the source’s data and formula0.4306174366369→ 0.436.2e−4same at 2 decimalsPASS

Laney p′ chart: 16 months, sample sizes 8,755 to 22,300

7/7 pass

Data: spcforexcel-laney-p.csv

StatisticExpectedLimitLineDifferenceToleranceResult
p̄0.4774printed in the source0.4773725037135−2.7e−5± 1e−4PASS
σz = MR̄ / 1.128 (article’s MR̄ 10.4108)9.229432624113computed here from the source’s data and formula9.229389043457−4.4e−5relative ± 2e−3PASS
Jan-07 LCL (%)32.96printed in the source32.95668793931→ 32.96−3.3e−3same at 2 decimalsPASS
Jan-07 UCL (%)62.52printed in the source62.51781280338→ 62.52−2.2e−3same at 2 decimalsPASS
Jul-07 LCL (%)38.24printed in the source38.23883615181→ 38.24−1.2e−3same at 2 decimalsPASS
Jul-07 UCL (%)57.24printed in the source57.23566459089→ 57.24−4.3e−3same at 2 decimalsPASS
Months outside the limitsJul-07printed in the sourceJul-07noneexact matchPASS

Process capability 11/11

Cp, Cpk, Pp, Ppk and the Cpk confidence interval.

NIST capability example: USL 20, LSL 8, mean 16, s 2

9/9 pass

NIST gives summary statistics only. To drive LimitLine’s code, a 6-point data set with mean exactly 16 and sample s exactly 2 is built, and σ within is set to 2 to mirror the handbook. In the app, Cp/Cpk use σ within = MR̄/d2, so loading this file there reproduces Pp = 1.000 and Ppk = 0.667, not Cp/Cpk.

Data: nist-616-capability-mean16-s2.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Data set mean16computed here from the source’s data and formula160± 1e−4PASS
Data set s2computed here from the source’s data and formula20± 1e−4PASS
Cp1printed in the source10± 1e−4PASS
Cpu0.6667printed in the source0.6666666666667−3.3e−5± 1e−4PASS
Cpl1.3333printed in the source1.3333333333333.3e−5± 1e−4PASS
Cpk0.6667printed in the source0.6666666666667−3.3e−5± 1e−4PASS
Pp (overall s = 2)1printed in the source10± 1e−4PASS
Ppk (overall s = 2)0.6667printed in the source0.6666666666667−3.3e−5± 1e−4PASS
One-sided (USL only): Cpk = Cpu0.6667printed in the source0.6666666666667−3.3e−5± 1e−4PASS

Cpk 95% confidence interval: Cpk 1.17, 50 points

2/2 pass

LimitLine uses Bissell’s approximation Cpk ± 1.96·√(1/(9n) + Cpk²/(2(n − 1))).

StatisticExpectedLimitLineDifferenceToleranceResult
Lower 95% limit0.92printed in the source0.9206097774316→ 0.926.1e−4same at 2 decimalsPASS
Upper 95% limit1.42printed in the source1.419390222568→ 1.42−6.1e−4same at 2 decimalsPASS

Gauge R&R (measurement systems analysis) 13/13

Crossed gauge R&R by the ANOVA method. Like Minitab’s default, LimitLine removes the part × operator interaction when its p-value is above 0.25.

AIAG gauge study: 10 parts × 3 operators × 3 trials, ANOVA mean squares and variance components

9/9 pass

Data: aiag-msa4-gage-rr.csv

StatisticExpectedLimitLineDifferenceToleranceResult
MS part9.81799printed in the source9.817992716049→ 9.817992.7e−6same at 5 decimalsPASS
MS operator1.58363printed in the source1.583631111111→ 1.583631.1e−6same at 5 decimalsPASS
MS repeatability (pooled)0.03997printed in the source0.03997327635328→ 0.039973.3e−6same at 5 decimalsPASS
Part × operator interaction p-value0.974printed in the source0.974106404329→ 0.9741.1e−4same at 3 decimalsPASS
Interaction removed from the model (p > 0.25)removedprinted in the sourceremovednoneexact matchPASS
Variance component, operator0.0514553printed in the source0.05145526115859→ 0.0514553−3.9e−8same at 7 decimalsPASS
Variance component, part-to-part1.08645printed in the source1.086446604411→ 1.08645−3.4e−6same at 5 decimalsPASS
Variance component, total gauge R&RThe printed 0.0914253 is the sum of the rounded 0.03997 + 0.0514553; the unrounded sum is 0.0914285, so the check allows 5 parts in 100,000.0.0914253printed in the source0.091428537511873.2e−6relative ± 5e−5PASS
Variance component, total variation1.17788printed in the source1.177875141923→ 1.17788−4.9e−6same at 5 decimalsPASS

AIAG gauge study: % contribution (ANOVA method)

4/4 pass

Data: aiag-msa4-gage-rr.csv

StatisticExpectedLimitLineDifferenceToleranceResult
% contribution, Total gauge R&R7.76printed in the source7.762158675209→ 7.762.2e−3same at 2 decimalsPASS
% contribution, Repeatability3.39printed in the source3.393676878861→ 3.393.7e−3same at 2 decimalsPASS
% contribution, Reproducibility4.37printed in the source4.368481796348→ 4.37−1.5e−3same at 2 decimalsPASS
% contribution, Part-to-part92.24printed in the source92.23784132479→ 92.24−2.2e−3same at 2 decimalsPASS

Regression and numerical accuracy 22/22

NIST Statistical Reference Datasets with certified values. Longley is a notoriously ill-conditioned regression; NumAcc2 and NumAcc4 test whether means and standard deviations keep their digits when values are large and close together. “Correct digits” is the log relative error, LRE = −log10(|value − certified| / |certified|).

NIST StRD Longley: 6 predictors, 16 observations

17/17 pass

Data: nist-strd-longley.csv

Show all 17 checks
StatisticExpectedLimitLineDifferenceToleranceResult
Coefficient, Intercept−3482258.634596NIST certified value−3482258.634596−6.5e−9relative ± 1e−8PASS
Coefficient, GNPDEFL15.06187227137NIST certified value15.061872271382.5e−12relative ± 1e−8PASS
Coefficient, GNP−0.03581917929259NIST certified value−0.03581917929259−6.5e−16relative ± 1e−8PASS
Coefficient, UNEMP−2.020229803817NIST certified value−2.020229803817−3.1e−15relative ± 1e−8PASS
Coefficient, ARMED−1.033226867174NIST certified value−1.033226867174−3.1e−15relative ± 1e−8PASS
Coefficient, POP−0.05110410565358NIST certified value−0.051104105653583.2e−15relative ± 1e−8PASS
Coefficient, YEAR1829.151464614NIST certified value1829.1514646146.1e−12relative ± 1e−8PASS
Standard error, Intercept890420.3836074NIST certified value890420.38360759.1e−8relative ± 1e−8PASS
Standard error, GNPDEFL84.91492577477NIST certified value84.914925774788.9e−12relative ± 1e−8PASS
Standard error, GNP0.03349100777224NIST certified value0.033491007772253.4e−15relative ± 1e−8PASS
Standard error, UNEMP0.4883996816517NIST certified value0.48839968165175.0e−14relative ± 1e−8PASS
Standard error, ARMED0.2142741631617NIST certified value0.21427416316172.2e−14relative ± 1e−8PASS
Standard error, POP0.2260732000694NIST certified value0.22607320006942.3e−14relative ± 1e−8PASS
Standard error, YEAR455.4784991422NIST certified value455.47849914234.7e−11relative ± 1e−8PASS
Residual standard deviation304.854073562NIST certified value304.8540735623.1e−11relative ± 1e−9PASS
R²0.9954790045773NIST certified value0.9954790045773−1.3e−15relative ± 1e−10PASS
F statistic330.2853392346NIST certified value330.2853392345−6.8e−11relative ± 1e−8PASS

NIST StRD NumAcc4: 1001 values near 10,000,000

3/3 pass

8.3 correct digits is the best possible for s here: 10000000.1 cannot be stored exactly in binary floating point. Stata 17’s published scores are shown for comparison.

Data: nist-strd-numacc4.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Mean, correct digits (Stata 17: 15.7)10000000.2NIST certified value10000000.215.7 correct digits1.9e−9≥ 15 correct digitsPASS
Standard deviation, correct digits (Stata 17: 8.3)0.1NIST certified value0.10000000055888.3 correct digits5.6e−10≥ 8 correct digitsPASS
Lag-1 autocorrelation, correct digits (Stata 17: 10.7)−0.999NIST certified value−0.998999999981410.7 correct digits1.9e−11≥ 10.5 correct digitsPASS

NIST StRD NumAcc2: 1001 values near 1.2

2/2 pass

The lag-1 autocorrelation is computed by the stats module but not shown in the app today.

Data: nist-strd-numacc2.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Standard deviation, correct digits (Stata 17: 15.6)0.1NIST certified value0.115.6 correct digits−2.8e−17≥ 15.5 correct digitsPASS
Lag-1 autocorrelation, correct digits (Stata 17: 16.0)−0.999NIST certified value−0.999all digits1.1e−16≥ 15.5 correct digitsPASS

Run charts 27/27

Anhøj run-chart rules: a signal when the longest run on one side of the median is too long or the line crosses the median too few times.

Anhøj limits for 10 to 30 useful points

21/21 pass

Each row feeds LimitLine a series with exactly that many points off the median and reads back its two limits.

Show all 21 checks
StatisticExpectedLimitLineDifferenceToleranceResult
n = 10: longest-run limit / fewest crossings6 / 2printed in the source6 / 2noneexact matchPASS
n = 11: longest-run limit / fewest crossings6 / 2printed in the source6 / 2noneexact matchPASS
n = 12: longest-run limit / fewest crossings7 / 3printed in the source7 / 3noneexact matchPASS
n = 13: longest-run limit / fewest crossings7 / 3printed in the source7 / 3noneexact matchPASS
n = 14: longest-run limit / fewest crossings7 / 4printed in the source7 / 4noneexact matchPASS
n = 15: longest-run limit / fewest crossings7 / 4printed in the source7 / 4noneexact matchPASS
n = 16: longest-run limit / fewest crossings7 / 4printed in the source7 / 4noneexact matchPASS
n = 17: longest-run limit / fewest crossings7 / 5printed in the source7 / 5noneexact matchPASS
n = 18: longest-run limit / fewest crossings7 / 5printed in the source7 / 5noneexact matchPASS
n = 19: longest-run limit / fewest crossings7 / 6printed in the source7 / 6noneexact matchPASS
n = 20: longest-run limit / fewest crossings7 / 6printed in the source7 / 6noneexact matchPASS
n = 21: longest-run limit / fewest crossings7 / 6printed in the source7 / 6noneexact matchPASS
n = 22: longest-run limit / fewest crossings7 / 7printed in the source7 / 7noneexact matchPASS
n = 23: longest-run limit / fewest crossings8 / 7printed in the source8 / 7noneexact matchPASS
n = 24: longest-run limit / fewest crossings8 / 8printed in the source8 / 8noneexact matchPASS
n = 25: longest-run limit / fewest crossings8 / 8printed in the source8 / 8noneexact matchPASS
n = 26: longest-run limit / fewest crossings8 / 8printed in the source8 / 8noneexact matchPASS
n = 27: longest-run limit / fewest crossings8 / 9printed in the source8 / 9noneexact matchPASS
n = 28: longest-run limit / fewest crossings8 / 9printed in the source8 / 9noneexact matchPASS
n = 29: longest-run limit / fewest crossings8 / 10printed in the source8 / 10noneexact matchPASS
n = 30: longest-run limit / fewest crossings8 / 10printed in the source8 / 10noneexact matchPASS

Anhøj Figure 2 case: 24 points, 2 on the median

6/6 pass

The series reproduces the figure’s pattern (1 = above the median, −1 = below, 0 = on it), not its original values.

Data: anhoj-figure2-pattern.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Useful points (not on the median)22printed in the source22noneexact matchPASS
Longest run10printed in the source10noneexact matchPASS
Crossings3printed in the source3noneexact matchPASS
Longest-run limit7printed in the source7noneexact matchPASS
Fewest-crossings limit7printed in the source7noneexact matchPASS
Signalyesprinted in the sourceyesnoneexact matchPASS

Lean tools 7/7

Published worked examples for OEE, takt time and heijunka.

OEE worked example

4/4 pass
StatisticExpectedLimitLineDifferenceToleranceResult
Availability0.8881printed in the source0.8880952380952−4.8e−6± 5e−5PASS
Performance0.8611printed in the source0.8610813226095−1.9e−5± 5e−5PASS
Quality0.978printed in the source0.97804991956835.0e−5± 5e−5PASS
OEE0.7479printed in the source0.74793650793653.7e−5± 5e−5PASS

Takt time

1/1 pass
StatisticExpectedLimitLineDifferenceToleranceResult
Takt time, seconds (480 min ÷ 240 units)120printed in the source1200± 1e−12PASS

Heijunka: leveling by volume

2/2 pass
StatisticExpectedLimitLineDifferenceToleranceResult
Level rate per day100printed in the source1000± 1e−12PASS
Buffer needed to cover Monday100computed here from the source’s data and formula1000± 1e−12PASS

Cross-checks against independent statistical software 187/187

These are not published examples. ref/ref_gen.py ran SciPy and statsmodels on seeded random data and stored their answers in ref/refs.json; LimitLine must reproduce them. They test the same calculations by a second, independent implementation, across many more inputs than textbook examples cover.

Probability distributions

92/92 pass

t, F and chi-square tail areas, the normal CDF, t quantiles, and log-gamma (Python math.lgamma). The p-values for every test in LimitLine come from these functions.

Show all 92 checks
StatisticExpectedLimitLineDifferenceToleranceResult
t CDF (t = -3.2, df = 1)0.09641124797923independent software output0.09641124797923−5.6e−17relative ± 1e−10PASS
t CDF (t = -3.2, df = 3)0.02466592148135independent software output0.024665921481353.8e−17relative ± 1e−10PASS
t CDF (t = -3.2, df = 9)0.005415651294951independent software output0.005415651294951−8.7e−19relative ± 1e−10PASS
t CDF (t = -3.2, df = 30)0.001619300855977independent software output0.0016193008559779.3e−18relative ± 1e−10PASS
t CDF (t = -3.2, df = 200)0.0007991768707151independent software output0.000799176870715−4.1e−17relative ± 1e−10PASS
t CDF (t = -1.1, df = 1)0.2348538278116independent software output0.2348538278116−2.8e−17relative ± 1e−10PASS
t CDF (t = -1.1, df = 3)0.1758415974673independent software output0.1758415974673−2.2e−16relative ± 1e−10PASS
t CDF (t = -1.1, df = 9)0.1499413809284independent software output0.14994138092843.3e−16relative ± 1e−10PASS
t CDF (t = -1.1, df = 30)0.1400404590438independent software output0.1400404590438−2.1e−15relative ± 1e−10PASS
t CDF (t = -1.1, df = 200)0.1363271860987independent software output0.13632718609871.6e−14relative ± 1e−10PASS
t CDF (t = 0.4, df = 1)0.6211189415908independent software output0.62111894159082.2e−16relative ± 1e−10PASS
t CDF (t = 0.4, df = 3)0.6420324230128independent software output0.64203242301281.1e−16relative ± 1e−10PASS
t CDF (t = 0.4, df = 9)0.6507579541643independent software output0.6507579541643−1.1e−16relative ± 1e−10PASS
t CDF (t = 0.4, df = 30)0.6540047417429independent software output0.65400474174296.7e−16relative ± 1e−10PASS
t CDF (t = 0.4, df = 200)0.655208297837independent software output0.6552082978369−1.2e−14relative ± 1e−10PASS
t CDF (t = 2.5, df = 1)0.8788810584092independent software output0.87888105840920relative ± 1e−10PASS
t CDF (t = 2.5, df = 3)0.956146676496independent software output0.9561466764960relative ± 1e−10PASS
t CDF (t = 2.5, df = 9)0.9830690861585independent software output0.98306908615850relative ± 1e−10PASS
t CDF (t = 2.5, df = 30)0.990942175466independent software output0.9909421754660relative ± 1e−10PASS
t CDF (t = 2.5, df = 200)0.9933884136791independent software output0.99338841367913.3e−16relative ± 1e−10PASS
t CDF (t = 6, df = 1)0.9474315432887independent software output0.94743154328870relative ± 1e−10PASS
t CDF (t = 6, df = 3)0.9953636425539independent software output0.99536364255390relative ± 1e−10PASS
t CDF (t = 6, df = 9)0.999898750339independent software output0.9998987503390relative ± 1e−10PASS
t CDF (t = 6, df = 30)0.9999993028616independent software output0.99999930286160relative ± 1e−10PASS
t CDF (t = 6, df = 200)0.9999999954424independent software output0.99999999544240relative ± 1e−10PASS
F upper tail (F = 0.3, df 1, 5)0.6074354940759independent software output0.60743549407591.1e−16relative ± 1e−10PASS
F upper tail (F = 0.3, df 3, 12)0.8248035702258independent software output0.82480357022581.1e−16relative ± 1e−10PASS
F upper tail (F = 0.3, df 6, 9)0.9217416608607independent software output0.9217416608607−1.1e−16relative ± 1e−10PASS
F upper tail (F = 0.3, df 20, 40)0.9973296186574independent software output0.99732961865740relative ± 1e−10PASS
F upper tail (F = 1.2, df 1, 5)0.3232601902798independent software output0.32326019027985.0e−16relative ± 1e−10PASS
F upper tail (F = 1.2, df 3, 12)0.351512640724independent software output0.3515126407241.6e−15relative ± 1e−10PASS
F upper tail (F = 1.2, df 6, 9)0.3865692436842independent software output0.3865692436842−5.6e−17relative ± 1e−10PASS
F upper tail (F = 1.2, df 20, 40)0.3035030346267independent software output0.3035030346267−4.4e−15relative ± 1e−10PASS
F upper tail (F = 4.5, df 1, 5)0.08735930812736independent software output0.08735930812736−5.6e−17relative ± 1e−10PASS
F upper tail (F = 4.5, df 3, 12)0.02457347884095independent software output0.02457347884095−7.3e−17relative ± 1e−10PASS
F upper tail (F = 4.5, df 6, 9)0.02214050292969independent software output0.02214050292969−2.1e−17relative ± 1e−10PASS
F upper tail (F = 4.5, df 20, 40)2.5950300463e−5independent software output2.5950300463e−5−2.3e−19relative ± 1e−10PASS
F upper tail (F = 30, df 1, 5)0.002764960301305independent software output0.002764960301305−5.2e−18relative ± 1e−10PASS
F upper tail (F = 30, df 3, 12)7.3730324758e−6independent software output7.3730324758e−6−1.2e−20relative ± 1e−10PASS
F upper tail (F = 30, df 6, 9)1.8525315594e−5independent software output1.8525315594e−5−5.8e−20relative ± 1e−10PASS
F upper tail (F = 30, df 20, 40)4.7700820249e−18independent software output4.7700820249e−18−4.3e−32relative ± 1e−10PASS
Chi-square upper tail (x = 0.5, df = 1)0.479500122187independent software output0.479500122187−2.2e−16relative ± 1e−10PASS
Chi-square upper tail (x = 0.5, df = 2)0.7788007830714independent software output0.7788007830714−2.2e−16relative ± 1e−10PASS
Chi-square upper tail (x = 0.5, df = 5)0.9921232932326independent software output0.99212329323260relative ± 1e−10PASS
Chi-square upper tail (x = 0.5, df = 17)0.9999999999488independent software output0.99999999994880relative ± 1e−10PASS
Chi-square upper tail (x = 3, df = 1)0.08326451666355independent software output0.083264516663559.7e−17relative ± 1e−10PASS
Chi-square upper tail (x = 3, df = 2)0.2231301601484independent software output0.2231301601484−5.8e−16relative ± 1e−10PASS
Chi-square upper tail (x = 3, df = 5)0.6999858358786independent software output0.69998583587863.3e−16relative ± 1e−10PASS
Chi-square upper tail (x = 3, df = 17)0.9999304982629independent software output0.99993049826290relative ± 1e−10PASS
Chi-square upper tail (x = 11, df = 1)0.0009111188771537independent software output0.0009111188771537−8.7e−19relative ± 1e−10PASS
Chi-square upper tail (x = 11, df = 2)0.004086771438464independent software output0.0040867714384643.5e−18relative ± 1e−10PASS
Chi-square upper tail (x = 11, df = 5)0.05137998348307independent software output0.05137998348307−9.0e−17relative ± 1e−10PASS
Chi-square upper tail (x = 11, df = 17)0.8565639878403independent software output0.85656398784033.3e−16relative ± 1e−10PASS
Chi-square upper tail (x = 40, df = 1)2.5396285895e−10independent software output2.5396285895e−10−5.2e−26relative ± 1e−10PASS
Chi-square upper tail (x = 40, df = 2)2.0611536224e−9independent software output2.0611536224e−90relative ± 1e−10PASS
Chi-square upper tail (x = 40, df = 5)1.4933679001e−7independent software output1.4933679001e−7−2.1e−22relative ± 1e−10PASS
Chi-square upper tail (x = 40, df = 17)0.001294198533743independent software output0.001294198533743−4.3e−18relative ± 1e−10PASS
Normal CDF (z = -8)6.2209605743e−16independent software output6.2209605743e−169.4e−30relative ± 1e−12PASS
Normal CDF (z = -5)2.8665157188e−7independent software output2.8665157188e−71.2e−21relative ± 1e−12PASS
Normal CDF (z = -3)0.00134989803163independent software output0.001349898031633.3e−18relative ± 1e−12PASS
Normal CDF (z = -1.5)0.06680720126886independent software output0.06680720126886−2.6e−16relative ± 1e−12PASS
Normal CDF (z = 0)0.5independent software output0.50relative ± 1e−12PASS
Normal CDF (z = 0.7)0.7580363477769independent software output0.75803634777691.1e−16relative ± 1e−12PASS
Normal CDF (z = 2.2)0.9860965524865independent software output0.98609655248650relative ± 1e−12PASS
Normal CDF (z = 4)0.9999683287582independent software output0.99996832875820relative ± 1e−12PASS
Normal CDF (z = 6)0.9999999990134independent software output0.99999999901340relative ± 1e−12PASS
t quantile (p = 0.025, df = 1)−12.70620473617independent software output−12.706204736179.2e−14relative ± 1e−9PASS
t quantile (p = 0.025, df = 4)−2.776445105198independent software output−2.7764451051980relative ± 1e−9PASS
t quantile (p = 0.025, df = 15)−2.13144954556independent software output−2.131449545564.4e−14relative ± 1e−9PASS
t quantile (p = 0.025, df = 100)−1.983971518524independent software output−1.9839715185237.7e−14relative ± 1e−9PASS
t quantile (p = 0.1, df = 1)−3.077683537175independent software output−3.0776835371751.1e−13relative ± 1e−9PASS
t quantile (p = 0.1, df = 4)−1.533206274059independent software output−1.533206274059−2.6e−14relative ± 1e−9PASS
t quantile (p = 0.1, df = 15)−1.34060560785independent software output−1.34060560785−7.8e−15relative ± 1e−9PASS
t quantile (p = 0.1, df = 100)−1.290074761347independent software output−1.2900747613464.2e−14relative ± 1e−9PASS
t quantile (p = 0.9, df = 1)3.077683537175independent software output3.077683537175−1.1e−13relative ± 1e−9PASS
t quantile (p = 0.9, df = 4)1.533206274059independent software output1.5332062740592.5e−14relative ± 1e−9PASS
t quantile (p = 0.9, df = 15)1.34060560785independent software output1.340605607857.5e−15relative ± 1e−9PASS
t quantile (p = 0.9, df = 100)1.290074761347independent software output1.290074761346−4.2e−14relative ± 1e−9PASS
t quantile (p = 0.975, df = 1)12.70620473617independent software output12.70620473617−8.2e−14relative ± 1e−9PASS
t quantile (p = 0.975, df = 4)2.776445105198independent software output2.7764451051988.9e−16relative ± 1e−9PASS
t quantile (p = 0.975, df = 15)2.13144954556independent software output2.13144954556−4.4e−14relative ± 1e−9PASS
t quantile (p = 0.975, df = 100)1.983971518524independent software output1.983971518523−7.7e−14relative ± 1e−9PASS
t quantile (p = 0.995, df = 1)63.65674116287independent software output63.65674116287−2.0e−13relative ± 1e−9PASS
t quantile (p = 0.995, df = 4)4.60409487135independent software output4.60409487135−1.1e−13relative ± 1e−9PASS
t quantile (p = 0.995, df = 15)2.946712883475independent software output2.946712883475−1.4e−13relative ± 1e−9PASS
t quantile (p = 0.995, df = 100)2.625890521438independent software output2.625890521438−5.1e−14relative ± 1e−9PASS
log-gamma (0.1)2.252712651734independent software output2.2527126517344.4e−16± 1e−12PASS
log-gamma (0.5)0.5723649429247independent software output0.5723649429247−8.9e−16± 1e−12PASS
log-gamma (1)0independent software output−8.881784197e−16−8.9e−16± 1e−12PASS
log-gamma (2.5)0.2846828704729independent software output0.28468287047291.3e−15± 1e−12PASS
log-gamma (10)12.80182748008independent software output12.801827480087.1e−15± 1e−12PASS
log-gamma (171.3)708.114947039independent software output708.114947039−2.3e−13± 1e−12PASS

Descriptive statistics (37 values)

7/7 pass

Quartiles use the (n + 1)p method that Minitab uses (numpy percentile method “weibull”). The mean and standard deviation feed every tool; skewness, kurtosis and quartiles come from the same module but the app doesn’t show them today.

Data: refs-describe.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Mean5.681897368676independent software output5.6818973686760relative ± 1e−13PASS
Standard deviation4.471880036468independent software output4.4718800364680relative ± 1e−13PASS
Skewness (adjusted)1.207487634087independent software output1.207487634087−4.4e−16relative ± 1e−12PASS
Kurtosis (adjusted)1.292806393565independent software output1.292806393565−1.8e−15relative ± 1e−12PASS
Q12.197822189793independent software output2.1978221897930relative ± 1e−13PASS
Median5.170722741533independent software output5.1707227415330relative ± 1e−13PASS
Q37.449728222047independent software output7.4497282220470relative ± 1e−13PASS

Anderson-Darling normality test (4 samples)

8/8 pass

Data: refs-anderson-darling.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Sample 1 (n = 25): AD statistic0.2022517255367independent software output0.20225172553670relative ± 1e−10PASS
Sample 1: p-value0.8634339963817independent software output0.86343399638170± 1e−9PASS
Sample 2 (n = 40): AD statistic0.7966172718201independent software output0.7966172718201−7.1e−15relative ± 1e−10PASS
Sample 2: p-value0.03566547290245independent software output0.035665472902461.5e−15± 1e−9PASS
Sample 3 (n = 200): AD statistic0.4711891733073independent software output0.47118917330732.8e−14relative ± 1e−10PASS
Sample 3: p-value0.2429460060544independent software output0.2429460060544−3.9e−14± 1e−9PASS
Sample 4 (n = 15): AD statistic0.5809110602341independent software output0.5809110602341−3.6e−15relative ± 1e−10PASS
Sample 4: p-value0.1091912875371independent software output0.10919128753712.3e−15± 1e−9PASS

Hypothesis tests: t tests, ANOVA, Levene, chi-square, two proportions

25/25 pass

Data: refs-t-tests.csv · refs-anova-groups.csv

Show all 25 checks
StatisticExpectedLimitLineDifferenceToleranceResult
1-sample t (μ0 = 9.5): t0.01935998176627independent software output0.019359981766270relative ± 1e−12PASS
1-sample t: p0.984779347686independent software output0.984779347686−2.6e−14relative ± 1e−10PASS
1-sample t: 95% CI low8.601612726835independent software output8.6016127268351.1e−14relative ± 1e−11PASS
1-sample t: 95% CI high10.41502743645independent software output10.41502743645−1.1e−14relative ± 1e−11PASS
1-sample t: p (one-sided, greater)0.492389673843independent software output0.492389673843−1.3e−14relative ± 1e−10PASS
Welch 2-sample t: t−0.8824736934754independent software output−0.8824736934754−1.1e−16relative ± 1e−12PASS
Welch: p0.3838189220058independent software output0.3838189220058−5.0e−16relative ± 1e−10PASS
Welch: df33.44907190079independent software output33.449071900797.1e−15relative ± 1e−12PASS
Welch: 95% CI low−2.591163598292independent software output−2.5911635982922.6e−14relative ± 1e−10PASS
Welch: 95% CI high1.02280181955independent software output1.02280181955−2.6e−14relative ± 1e−10PASS
Pooled 2-sample t: t−0.8172053632849independent software output−0.8172053632849−1.1e−16relative ± 1e−12PASS
Pooled: p0.4187755041075independent software output0.4187755041075−1.5e−15relative ± 1e−10PASS
Pooled: 95% CI low−2.72513188634independent software output−2.72513188634−4.4e−15relative ± 1e−10PASS
Paired t: t−2.976068678971independent software output−2.9760686789714.4e−16relative ± 1e−12PASS
Paired t: p0.008476036680099independent software output0.0084760366800990relative ± 1e−10PASS
Paired t: 95% CI high−0.2006319767431independent software output−0.2006319767431−6.3e−15relative ± 1e−10PASS
One-way ANOVA: F4.764705737524independent software output4.764705737524−1.8e−15relative ± 1e−12PASS
One-way ANOVA: p0.006885553282113independent software output0.006885553282113−4.7e−17relative ± 1e−10PASS
Levene (median): W0.6922417635841independent software output0.6922417635841−1.1e−16relative ± 1e−12PASS
Levene: p0.5629570705917independent software output0.56295707059174.4e−15relative ± 1e−10PASS
Chi-square, 3×3 table [[12,30,18],[20,25,35],[8,14,20]]: χ²6.468289325657independent software output6.468289325657−8.9e−16relative ± 1e−12PASS
Chi-square: p0.1667994104129independent software output0.16679941041290relative ± 1e−10PASS
Chi-square: df4independent software output4noneexact matchPASS
2 proportions, 45/300 vs 30/280 (pooled): z1.537094376756independent software output1.5370943767560relative ± 1e−12PASS
2 proportions: p0.1242701998111independent software output0.1242701998111−2.5e−16relative ± 1e−10PASS

Multiple regression: 3 predictors, 30 observations

30/30 pass

Data: refs-regression.csv

Show all 30 checks
StatisticExpectedLimitLineDifferenceToleranceResult
Coefficient, intercept1.874689665505independent software output1.874689665505−1.8e−15relative ± 1e−11PASS
Coefficient, x11.361563143009independent software output1.361563143009−2.2e−16relative ± 1e−11PASS
Coefficient, x2−0.8739299498415independent software output−0.8739299498415−4.4e−16relative ± 1e−11PASS
Coefficient, x30.1496412458402independent software output0.14964124584022.5e−16relative ± 1e−11PASS
Standard error, intercept0.1689927843108independent software output0.1689927843108−5.6e−17relative ± 1e−10PASS
Standard error, x10.1769743853737independent software output0.1769743853737−2.8e−17relative ± 1e−10PASS
Standard error, x20.2010811903905independent software output0.20108119039052.8e−17relative ± 1e−10PASS
Standard error, x30.2230514708441independent software output0.2230514708441−5.6e−17relative ± 1e−10PASS
p-value, intercept2.3419510988e−11independent software output2.3419510988e−112.1e−25± 1e−8PASS
p-value, x13.6501351758e−8independent software output3.6501351758e−86.6e−24± 1e−8PASS
p-value, x20.0001886853370603independent software output0.0001886853370603−1.2e−18± 1e−8PASS
p-value, x30.5082121352493independent software output0.50821213524930± 1e−8PASS
Adjusted R²0.8107432932159independent software output0.81074329321590relative ± 1e−11PASS
Model p (F test)3.7114932556e−10independent software output3.7114932556e−104.7e−24± 1e−8PASS
Leverage, row 10.1062954486162independent software output0.1062954486162−6.9e−17relative ± 1e−10PASS
Leverage, row 20.1553432805676independent software output0.15534328056762.8e−17relative ± 1e−10PASS
Leverage, row 30.06425586121756independent software output0.06425586121756−2.8e−17relative ± 1e−10PASS
Leverage, row 40.1440919363799independent software output0.1440919363799−5.6e−17relative ± 1e−10PASS
Leverage, row 50.2929633545587independent software output0.29296335455871.1e−16relative ± 1e−10PASS
Standardized residual, row 1−0.1175203952685independent software output−0.11752039526852.1e−15relative ± 1e−9PASS
Standardized residual, row 2−0.2228715870875independent software output−0.22287158708753.1e−15relative ± 1e−9PASS
Standardized residual, row 3−0.6462074454988independent software output−0.64620744549882.0e−15relative ± 1e−9PASS
Standardized residual, row 4−0.9651196491279independent software output−0.96511964912792.6e−15relative ± 1e−9PASS
Standardized residual, row 50.297067676452independent software output0.2970676764522.3e−15relative ± 1e−9PASS
VIF, x11.572894585264independent software output1.572894585264−2.2e−16relative ± 1e−9PASS
VIF, x21.026032907857independent software output1.0260329078574.4e−16relative ± 1e−9PASS
VIF, x31.604908810459independent software output1.6049088104590relative ± 1e−9PASS
Prediction at (0.5, −0.2, 0.1)2.745221351562independent software output2.745221351562−1.8e−15relative ± 1e−11PASS
95% confidence interval, low2.323258410863independent software output2.323258410863−1.1e−14relative ± 1e−10PASS
95% prediction interval, high4.651613048762independent software output4.6516130487624.3e−14relative ± 1e−10PASS

Non-normal data: Box-Cox, lognormal and Weibull fits

7/7 pass

Maximum-likelihood fits with the location fixed at 0, as scipy’s floc = 0. These drive LimitLine’s non-normal capability.

Data: refs-lognormal-60.csv · refs-weibull-80.csv

StatisticExpectedLimitLineDifferenceToleranceResult
Box-Cox λ (maximum likelihood)−0.07761589539388independent software output−0.077615866431762.9e−8± 1e−6PASS
Lognormal σ0.4395476313864independent software output0.43954763138645.6e−17relative ± 1e−7PASS
Lognormal μ0.972522082197independent software output0.972522082197−1.1e−16relative ± 1e−7PASS
Lognormal 99.865th percentile9.88634847448independent software output9.886348474487.1e−15relative ± 1e−6PASS
Weibull shape2.000148110365independent software output2.0001505280742.4e−6relative ± 1e−5PASS
Weibull scale11.79568421492independent software output11.795715014063.1e−5relative ± 1e−5PASS
Weibull 0.135th percentile0.433653560681independent software output0.43365642448972.9e−6relative ± 1e−4PASS

Gauge R&R two-way ANOVA (10 parts × 3 operators × 2 trials)

4/4 pass

Data: refs-gage-rr-synthetic.csv

StatisticExpectedLimitLineDifferenceToleranceResult
SS part63.83469679353independent software output63.834696793537.1e−15relative ± 1e−10PASS
SS operator0.3980597608446independent software output0.39805976084479.5e−15relative ± 1e−10PASS
Interaction p-value (vs error)0.8499763059015independent software output0.8499763059015−2.7e−15± 1e−9PASS
Pooled repeatability SS = interaction SS + error SS0.8816023027272independent software output0.88160230272721.8e−15relative ± 1e−10PASS

Designed experiment: 2³ full factorial, 2 replicates, full model

14/14 pass

Data: refs-doe-2x3-replicated.csv

Show all 14 checks
StatisticExpectedLimitLineDifferenceToleranceResult
Coefficient, A4.139202170081independent software output4.1392021700811.8e−15relative ± 1e−11PASS
p-value, A7.2262702212e−7independent software output7.2262702212e−7−3.0e−21± 1e−8PASS
Coefficient, B−2.607738056833independent software output−2.6077380568331.2e−14relative ± 1e−11PASS
p-value, B2.3519608957e−5independent software output2.3519608957e−55.4e−19± 1e−8PASS
Coefficient, C0.4531543108019independent software output0.45315431080196.1e−15relative ± 1e−11PASS
p-value, C0.1685042157022independent software output0.1685042157022−5.1e−15± 1e−8PASS
Coefficient, A×B1.597856365582independent software output1.597856365582−4.9e−15relative ± 1e−11PASS
p-value, A×B0.0006956174417653independent software output0.00069561744176531.4e−17± 1e−8PASS
Coefficient, A×C−0.04963003276434independent software output−0.049630032764342.0e−15relative ± 1e−11PASS
p-value, A×C0.8724222505913independent software output0.87242225059139.9e−15± 1e−8PASS
Coefficient, B×C0.1982409079729independent software output0.1982409079729−1.5e−15relative ± 1e−11PASS
p-value, B×C0.5263965727899independent software output0.5263965727899−1.9e−15± 1e−8PASS
Coefficient, A×B×C0.1543435210803independent software output0.15434352108033.7e−15relative ± 1e−11PASS
p-value, A×B×C0.6200488147326independent software output0.6200488147326−7.1e−15± 1e−8PASS

How to rerun these yourself

  1. Download a dataset below. Each is a plain CSV with a header row, in the order the source prints it.
  2. The Individuals (I-MR) tab with capability, the Pareto and the Run chart are in the free plan. Subgroup and attribute charts and the other analysis tools need Pro.
  3. Open LimitLine and paste the file’s contents into the tool listed. LimitLine lists the header word under “Ignored non-numeric entries”, which is expected.
  4. Compare the result with the case’s expected values above. Or open the same file in Minitab, JMP, R or Excel and compare all three.

Settings to match when you compare with another package: I-MR sigma is MR̄/1.128 (average moving range). Gauge R&R uses the ANOVA method and removes the interaction when p > 0.25. Quartiles use the (n + 1)p method. The 2-sample t test is Welch’s (no equal-variance assumption) unless “Assume equal variances” is ticked. All results are also in machine-readable form in results.json.

FileWhat it isIn LimitLineIn Minitab or elsewhere
nist-6322-flow-rate.csv10 rowsNIST §6.3.2.2 flow rate, 10 batchesIndividuals (I-MR) tab: paste the column (rows are batches 1 to 10).Minitab: Stat › Control Charts › Variables Charts for Individuals › I-MR, column flow_rate.
xbar-r-test-5x5.csv5 rows5 subgroups of 5 (LimitLine test set)Subgroups (X̄-R / X̄-S) tab: paste the five data lines (one subgroup per line).Minitab: Stat › Control Charts › Variables Charts for Subgroups › Xbar-R, “Observations for a subgroup are in one row of columns”.
nist-6331-wafer-defects.csv25 rowsNIST §6.3.3.1 defects per wafer, 25 wafersAttributes tab, c chart: paste the 25 counts (one per line).Minitab: Stat › Control Charts › Attributes Charts › C.
nist-6332-wafer-proportions.csv30 rowsNIST §6.3.3.2 defective chips, 30 wafers of 50Attributes tab, p chart: paste the lines “defectives, sample size” (for the np chart, also enter sample size 50).Minitab: Stat › Control Charts › Attributes Charts › P, subgroup sizes column sample_size.
montgomery-u-computers.csv20 rowsMontgomery p. 318 (via IMSL): defects in 20 samples of 5 computersAttributes tab, u chart: paste the lines “defects, units”.Minitab: Stat › Control Charts › Attributes Charts › U, subgroup sizes column units.
montgomery-u-cloth-rolls.csv10 rowsMontgomery p. 321 (via IMSL): defects in 10 rolls of cloth, units of 50 m²Attributes tab, u chart: paste the lines “defects, units”.Minitab: Stat › Control Charts › Attributes Charts › U, subgroup sizes column units.
spcforexcel-laney-p.csv16 rowsSPC for Excel Laney p′ example, 16 monthsAnalysis tools › EWMA · CUSUM · more › Laney P′: paste the lines (the first two numbers on each line are used: defectives, sample size).Minitab: Stat › Control Charts › Attributes Charts › Laney P′.
nist-616-capability-mean16-s2.csv6 rows6 values with mean 16 and s 2 (built to match NIST §6.1.6)Individuals (I-MR) tab with LSL 8 and USL 20: paste the values and compare Pp and Ppk.Minitab: Stat › Quality Tools › Capability Analysis › Normal (subgroup size 1); compare overall Pp/Ppk.
aiag-msa4-gage-rr.csv90 rowsAIAG MSA (4th ed.) gauge study as printed by SPC for Excel: 10 parts, operators A/B/C, 3 trialsAnalysis tools › Gauge R&R: paste the lines “part, operator, measurement” (the trial column at the end is ignored).Minitab: Stat › Quality Tools › Gage Study › Gage R&R Study (Crossed), ANOVA method.
nist-strd-longley.csv16 rowsNIST StRD Longley (y = TOTEMP first, then 6 predictors)Analysis tools › Regression: paste the file with its header row (first column = Y).Minitab: Stat › Regression › Regression › Fit Regression Model, response TOTEMP, the other six as continuous predictors.
nist-strd-numacc4.csv1001 rowsNIST StRD NumAcc4, 1001 valuesIndividuals (I-MR) tab: the centre line is the mean. The app rounds what it shows, so the digit-level result is in results.json.Minitab: Stat › Basic Statistics › Display Descriptive Statistics (increase displayed digits).
nist-strd-numacc2.csv1001 rowsNIST StRD NumAcc2, 1001 valuesIndividuals (I-MR) tab: the centre line is the mean. The app rounds what it shows, so the digit-level result is in results.json.Minitab: Stat › Basic Statistics › Display Descriptive Statistics.
anhoj-figure2-pattern.csv24 rowsPattern of the qicharts “Run charts with R” Figure 2 case (above/below/on the median)Analysis tools › Run chart: paste the column. It shows the longest run about the median (10) and flags the shift.Any run-chart tool that applies the Anhøj rules (for example qicharts in R).
refs-describe.csv37 rows37 gamma-distributed values used for the descriptive-statistics cross-checkIndividuals (I-MR) tab: the centre line is the mean. The app rounds what it shows, so the digit-level result is in results.json.Minitab: Stat › Basic Statistics › Display Descriptive Statistics.
refs-anderson-darling.csv280 rows4 samples (n = 25, 40, 200, 15) used for the Anderson-Darling cross-checkAnalysis tools › Capability+: paste one sample’s values at a time and add any spec limit; the Anderson-Darling result is shown.Minitab: Stat › Basic Statistics › Normality Test, Anderson-Darling.
refs-t-tests.csv59 rowsSamples a (n 18), b (n 23), c (n 18, paired with a) for the t-test cross-checksAnalysis tools › Hypothesis tests: paste sample a (and b, or c for the paired test). 1-sample target 9.5.Minitab: Stat › Basic Statistics › 1-Sample t / 2-Sample t / Paired t.
refs-anova-groups.csv39 rows4 groups (n 8, 12, 10, 9) for the one-way ANOVA and Levene cross-checksAnalysis tools › Hypothesis tests › one-way ANOVA or equal variances: paste each group.Minitab: Stat › ANOVA › One-Way; Stat › ANOVA › Test for Equal Variances (Levene).
refs-regression.csv30 rows30 observations, 3 correlated predictors, for the OLS cross-checkAnalysis tools › Regression: paste with the header row; predict at 0.5, -0.2, 0.1.Minitab: Stat › Regression › Regression › Fit Regression Model.
refs-lognormal-60.csv60 rows60 lognormal values for the Box-Cox and lognormal-fit cross-checksAnalysis tools › Capability+ (Box-Cox or Lognormal): paste the column and add a spec limit.Minitab: Stat › Quality Tools › Individual Distribution Identification.
refs-weibull-80.csv80 rows80 Weibull values for the Weibull-fit cross-checkAnalysis tools › Capability+ (Weibull): paste the column and add a spec limit.Minitab: Stat › Quality Tools › Individual Distribution Identification (2-parameter Weibull).
refs-gage-rr-synthetic.csv60 rowsSimulated gauge study, 10 parts × 3 operators × 2 trialsAnalysis tools › Gauge R&R: paste the lines “part, operator, measurement”; LSL 3, USL 7.Minitab: Stat › Quality Tools › Gage Study › Gage R&R Study (Crossed), ANOVA method.
refs-doe-2x3-replicated.csv16 rows2³ factorial, 2 replicates (16 runs), coded −1/+1Analysis tools › DOE, step 2: paste the runs with the header row (factors A, B, C, last column = response).Minitab: Stat › DOE › Factorial › Define Custom Factorial Design, then Analyze Factorial Design (full model).

Sources cited

Code fingerprint

SHA-256 of the calculation modules tested for this page (commit aa089a2):

FileSHA-256
spc-core.js44183f9bd23f9ba791f7a14a882d479b113fc5a9d3c421befc2c5ae88bb0fbe4
stats-core.js6a42c8190f66b3a51947a99fadb392258ca6ee035ed971a2246e3e1176a84b66
lean-core.jsd142a61e3e3cddcbcfd72a6261d5582f3943bea4a3ab4ea3bb813e51d793550b

Found a number that disagrees with your software? Email [email protected] with the dataset and both results.

← Back to LimitLine · Privacy policy · Terms of service