Validation: Checking LimitLine’s Math
- Published references: 291 of 291 checks pass in 25 cases (NIST handbook and StRD, AIAG gauge study, Montgomery, SPC for Excel, qicharts and qicharts2, OEE.com, Lean Enterprise Institute).
- Independent software: 484 of 484 checks pass in 9 cases (SciPy and statsmodels on the same data).
- Every dataset is downloadable as CSV, so you can rerun any case in LimitLine, Minitab or another package.
- What a pass means: LimitLine’s number equals the reference within the tolerance shown on its row, for that dataset. It shows the formulas are implemented correctly; it doesn’t prove your data meets the method’s assumptions.
- The code tested is the code you run: the math modules are published at limitline.org/source. Their combined fingerprint is
64e4b90d; the app shows the same one under its version, so you can confirm they match.
- Variables control charts 47/47
- Attribute control charts 27/27
- Process capability 11/11
- Gauge R&R (measurement systems analysis) 13/13
- Regression and numerical accuracy 22/22
- Run charts 97/97
- Time-weighted charts (EWMA, CUSUM) 65/65
- Lean tools 7/7
- Power and sample size 2/2
- Cross-checks against independent statistical software 484/484
What Was Tested
A script loads LimitLine’s calculation modules (spc-core.js, stats-core.js and lean-core.js), runs each case below, and writes this page. These are the same source files that are built into the app at limitline.org/app. The script runs them in Node.js, not in a browser; both use the same IEEE double-precision arithmetic.
Each row compares one number from LimitLine with the expected value. Under each expected value is where it comes from:
- printed in the source: the value as published.
- NIST certified value: from the NIST Statistical Reference Datasets.
- computed here from the source’s data and formula: the source gives the data and the method but not that number, so the script computes it separately, without LimitLine’s code.
- independent software output: SciPy or statsmodels on the same data.
Published values are rounded, so the tolerance is set by how many digits the source prints. These are the same tolerances LimitLine’s automated tests use.
What This Does Not Cover
- The screens. Typing or pasting data, file import (CSV, Excel, Word, PDF, scanned images), on-screen rounding and chart drawing are not tested here. The app shows rounded values; this page shows full precision.
- Features without a published reference on this page: G and Laney U′ charts, Nelson rules 2 to 8 (tested only on made-up patterns), Pareto, the “Which chart?” guide, non-normal capability indices, unreplicated designs (Lenth’s method), and the offline help assistant. LimitLine’s own unit tests cover these with hand calculations, but they are not shown here because there is no published worked example to compare with.
- X̄-R and X̄-S charts are checked against NIST’s printed constants on a LimitLine test set, not against a published worked example.
- Not a regulatory validation. This page is not an IQ/OQ/PQ package, computer-system validation or a 21 CFR Part 11 assessment, and no outside body has reviewed it. If your quality system requires software validation, you can use these datasets in your own protocol.
- Some citations are partial. The Montgomery u-chart data are cited by page number as reproduced in the IMSL documentation, which doesn’t state the edition. The Minitab Gauge R&R figures were transcribed from Minitab output, which has no public link.
Variables Control Charts 47/47
Individuals (I-MR), X̄-R and X̄-S limits, and the control-chart constants they use.
NIST Individuals Chart: Flow Rate, 10 Batches
9/9 passData: nist-6322-flow-rate.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Moving ranges (rounded to 0.1) | 2, 2.3, 1.4, 3.5, 3.4, 1.4, 0.2, 1.2, 1.5printed in the source | 2, 2.3, 1.4, 3.5, 3.4, 1.4, 0.2, 1.2, 1.5 | none | exact match | PASS |
| Mean (center line) | 50.81printed in the source | 50.81 | 0 | ± 1e−4 | PASS |
| Average moving range MR̄ | 1.8778printed in the source | 1.877777777778 | −2.2e−5 | ± 1e−4 | PASS |
| Individuals UCL | 55.8041printed in the source | 55.80408983452 | −1.0e−5 | ± 0.001 | PASS |
| Individuals LCL | 45.8159printed in the source | 45.81591016548 | 1.0e−5 | ± 0.001 | PASS |
| MR chart UCL = 3.267 × MR̄ | 6.1347726computed here from the source’s data and formula | 6.1347 | −7.3e−5 | ± 0.001 | PASS |
| Points beyond the 3-sigma limits | 0printed in the source | 0 | none | exact match | PASS |
| Phase I: limits from the first 10 of 13 points, UCLThree extra points (60, 50, 51) added after the baseline must not move the limits. | 55.8041printed in the source | 55.80408983452 | −1.0e−5 | ± 0.001 | PASS |
| Phase I: limits from the first 10 of 13 points, LCL | 45.8159printed in the source | 45.81591016548 | 1.0e−5 | ± 0.001 | PASS |
NIST Table of Control-Chart Constants A2, D3, D4 (n = 2 to 10)
27/27 passLimitLine derives d2 and d3 by numerical integration and c4 from the gamma function; this compares the resulting factors with NIST’s printed table.
Show All 27 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| A2, n = 2 | 1.88printed in the source | 1.879969818288 | −3.0e−5 | ± 0.001 | PASS |
| D3, n = 2 | 0printed in the source | 0 | 0 | ± 0.001 | PASS |
| D4, n = 2 | 3.267printed in the source | 3.266523688828 | −4.8e−4 | ± 0.001 | PASS |
| A2, n = 3 | 1.023printed in the source | 1.023325952586 | 3.3e−4 | ± 0.001 | PASS |
| D3, n = 3 | 0printed in the source | 0 | 0 | ± 0.001 | PASS |
| D4, n = 3 | 2.575printed in the source | 2.574593665255 | −4.1e−4 | ± 0.001 | PASS |
| A2, n = 4 | 0.729printed in the source | 0.7285974499089 | −4.0e−4 | ± 0.001 | PASS |
| D3, n = 4 | 0printed in the source | 0 | 0 | ± 0.001 | PASS |
| D4, n = 4 | 2.282printed in the source | 2.282054644809 | 5.5e−5 | ± 0.001 | PASS |
| A2, n = 5 | 0.577printed in the source | 0.5768190730159 | −1.8e−4 | ± 0.001 | PASS |
| D3, n = 5 | 0printed in the source | 0 | 0 | ± 0.001 | PASS |
| D4, n = 5 | 2.115printed in the source | 2.114496137029 | −5.0e−4 | ± 0.001 | PASS |
| A2, n = 6 | 0.483printed in the source | 0.4832465431369 | 2.5e−4 | ± 0.001 | PASS |
| D3, n = 6 | 0printed in the source | 0 | 0 | ± 0.001 | PASS |
| D4, n = 6 | 2.004printed in the source | 2.003831266449 | −1.7e−4 | ± 0.001 | PASS |
| A2, n = 7 | 0.419printed in the source | 0.4192834604223 | 2.8e−4 | ± 0.001 | PASS |
| D3, n = 7 | 0.076printed in the source | 0.07570367850434 | −3.0e−4 | ± 0.001 | PASS |
| D4, n = 7 | 1.924printed in the source | 1.924296321496 | 3.0e−4 | ± 0.001 | PASS |
| A2, n = 8 | 0.373printed in the source | 0.3725274556687 | −4.7e−4 | ± 0.001 | PASS |
| D3, n = 8 | 0.136printed in the source | 0.136172379882 | 1.7e−4 | ± 0.001 | PASS |
| D4, n = 8 | 1.864printed in the source | 1.863827620118 | −1.7e−4 | ± 0.001 | PASS |
| A2, n = 9 | 0.337printed in the source | 0.3366969357212 | −3.0e−4 | ± 0.001 | PASS |
| D3, n = 9 | 0.184printed in the source | 0.1840183432491 | 1.8e−5 | ± 0.001 | PASS |
| D4, n = 9 | 1.816printed in the source | 1.815981656751 | −1.8e−5 | ± 0.001 | PASS |
| A2, n = 10 | 0.308printed in the source | 0.3082632706475 | 2.6e−4 | ± 0.001 | PASS |
| D3, n = 10 | 0.223printed in the source | 0.2230244580846 | 2.4e−5 | ± 0.001 | PASS |
| D4, n = 10 | 1.777printed in the source | 1.776975541915 | −2.4e−5 | ± 0.001 | PASS |
Published c4 and d2 Values
4/4 pass- Standard published tables of control-chart constants (c4, d2). The test file does not name one specific publication.
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| c4, n = 2 | 0.7979printed in the source | 0.79788 | −2.0e−5 | ± 1e−4 | PASS |
| c4, n = 5 | 0.94printed in the source | 0.93999 | −1.0e−5 | ± 1e−4 | PASS |
| c4, n = 25 | 0.9896printed in the source | 0.98964 | 4.0e−5 | ± 1e−4 | PASS |
| d2, n = 25 | 3.931printed in the source | 3.93063 | −3.7e−4 | ± 0.001 | PASS |
X̄-R and X̄-S Limits Recomputed by Hand with NIST’s Printed Constants
7/7 passThe data are a small LimitLine test set (5 subgroups of 5), not a published example. Expected values are computed in this script from the raw data with NIST’s printed A2 = 0.577, D4 = 2.115 and c4 = 0.9400, without using LimitLine’s code.
Data: xbar-r-test-5x5.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Grand mean X̿ | 10.036computed here from the source’s data and formula | 10.036 | 0 | ± 1e−12 | PASS |
| X̄ chart UCL (X̿ + 0.577 R̄) | 10.33604computed here from the source’s data and formula | 10.33594591797 | −9.4e−5 | ± 5e−4 | PASS |
| X̄ chart LCL (X̿ − 0.577 R̄) | 9.73596computed here from the source’s data and formula | 9.736054082032 | 9.4e−5 | ± 5e−4 | PASS |
| R chart UCL (2.115 R̄) | 1.0998computed here from the source’s data and formula | 1.099537991255 | −2.6e−4 | ± 5e−4 | PASS |
| R chart LCL (D3 = 0 for n = 5) | 0computed here from the source’s data and formula | 0 | none | exact match | PASS |
| X̄-S chart UCL (X̿ + 3 s̄ / (c4 √5)) | 10.32015959673computed here from the source’s data and formula | 10.32016261973 | 3.0e−6 | ± 1e−4 | PASS |
| S chart UCL (B4 from c4 = 0.9400) | 0.4158743549349computed here from the source’s data and formula | 0.4158941670148 | 2.0e−5 | ± 0.001 | PASS |
Attribute Control Charts 27/27
c, p, np, u and Laney P′ charts.
NIST c Chart: Defects on 25 Wafers
4/4 passData: nist-6331-wafer-defects.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| c̄ (center line) | 16printed in the source | 16 | 0 | ± 1e−4 | PASS |
| UCL | 28printed in the source | 28 | 0 | ± 1e−4 | PASS |
| LCL | 4printed in the source | 4 | 0 | ± 1e−4 | PASS |
| Wafers beyond the limits (1-based) | 24printed in the source | 24 | none | exact match | PASS |
NIST p Chart: 30 Wafers of 50 Chips
8/8 passNIST prints the fraction defective per wafer. Counts are fraction × 50. Limits are computed here from those data with the standard formula p̄ ± 3√(p̄(1 − p̄)/n), independently of LimitLine’s code.
Data: nist-6332-wafer-proportions.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Total defectives | 347computed here from the source’s data and formula | 347 | none | exact match | PASS |
| p̄ = 347 / 1500 | 0.2313333333333computed here from the source’s data and formula | 0.2313333333333 | 0 | ± 1e−12 | PASS |
| UCL | 0.4102391185947computed here from the source’s data and formula | 0.4102391185947 | 0 | ± 1e−12 | PASS |
| LCL | 0.05242754807193computed here from the source’s data and formula | 0.05242754807193 | 0 | ± 1e−12 | PASS |
| UCL to 4 decimals | 0.4102computed here from the source’s data and formula | 0.4102391185947 | 3.9e−5 | ± 0.001 | PASS |
| Wafers beyond the limits (1-based) | 15, 23computed here from the source’s data and formula | 15, 23 | none | exact match | PASS |
| np chart center = n·p̄ | 11.56666666667computed here from the source’s data and formula | 11.56666666667 | 0 | ± 1e−12 | PASS |
| np chart UCL | 20.51195592974computed here from the source’s data and formula | 20.51195592974 | 0 | ± 1e−12 | PASS |
Montgomery u Chart: Defects in Computers, 20 Samples of 5
3/3 passThe center line is printed. The limits are computed from the printed center line as 1.93 ± 3√(1.93/5) and compared at 2 decimals.
Data: montgomery-u-computers.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| ū (center line) | 1.93printed in the source | 1.93 | 0 | ± 1e−12 | PASS |
| LCL | 0.07computed here from the source’s data and formula | 0.06613305195891→ 0.07 | −3.9e−3 | same at 2 decimals | PASS |
| UCL | 3.79computed here from the source’s data and formula | 3.793866948041→ 3.79 | 3.9e−3 | same at 2 decimals | PASS |
Montgomery u Chart: Dyeing Rolls of Cloth, Unequal Inspection Units
5/5 passThe center line is printed. Per-roll limits are computed from ū = 153/107.5 with ū ± 3√(ū/n) and compared at 2 decimals.
Data: montgomery-u-cloth-rolls.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| ū (center line) | 1.423printed in the source | 1.423255813953 | 2.6e−4 | ± 5e−4 | PASS |
| UCL, roll 2 (8 units, widest) | 2.69computed here from the source’s data and formula | 2.688626427923→ 2.69 | −1.4e−3 | same at 2 decimals | PASS |
| UCL, roll 3 (13 units, narrowest) | 2.42computed here from the source’s data and formula | 2.41589419127→ 2.42 | −4.1e−3 | same at 2 decimals | PASS |
| LCL, roll 2 | 0.16computed here from the source’s data and formula | 0.1578851999839→ 0.16 | −2.1e−3 | same at 2 decimals | PASS |
| LCL, roll 3 | 0.43computed here from the source’s data and formula | 0.4306174366369→ 0.43 | 6.2e−4 | same at 2 decimals | PASS |
Laney P′ Chart: 16 Months, Sample Sizes 8,755 to 22,300
7/7 passData: spcforexcel-laney-p.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| p̄ | 0.4774printed in the source | 0.4773725037135 | −2.7e−5 | ± 1e−4 | PASS |
| σz = MR̄ / 1.128 (article’s MR̄ 10.4108) | 9.229432624113computed here from the source’s data and formula | 9.229389043457 | −4.4e−5 | relative ± 2e−3 | PASS |
| Jan-07 LCL (%) | 32.96printed in the source | 32.95668793931→ 32.96 | −3.3e−3 | same at 2 decimals | PASS |
| Jan-07 UCL (%) | 62.52printed in the source | 62.51781280338→ 62.52 | −2.2e−3 | same at 2 decimals | PASS |
| Jul-07 LCL (%) | 38.24printed in the source | 38.23883615181→ 38.24 | −1.2e−3 | same at 2 decimals | PASS |
| Jul-07 UCL (%) | 57.24printed in the source | 57.23566459089→ 57.24 | −4.3e−3 | same at 2 decimals | PASS |
| Months outside the limits | Jul-07printed in the source | Jul-07 | none | exact match | PASS |
Process Capability 11/11
Cp, Cpk, Pp, Ppk and the Cpk confidence interval.
NIST Capability Example: USL 20, LSL 8, Mean 16, s 2
9/9 passNIST gives summary statistics only. To drive LimitLine’s code, a 6-point dataset with mean exactly 16 and sample s exactly 2 is built, and σ within is set to 2 to mirror the handbook. In the app, Cp/Cpk use σ within = MR̄/d2, so loading this file there reproduces Pp = 1.000 and Ppk = 0.667, not Cp/Cpk.
Data: nist-616-capability-mean16-s2.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Data set mean | 16computed here from the source’s data and formula | 16 | 0 | ± 1e−4 | PASS |
| Data set s | 2computed here from the source’s data and formula | 2 | 0 | ± 1e−4 | PASS |
| Cp | 1printed in the source | 1 | 0 | ± 1e−4 | PASS |
| Cpu | 0.6667printed in the source | 0.6666666666667 | −3.3e−5 | ± 1e−4 | PASS |
| Cpl | 1.3333printed in the source | 1.333333333333 | 3.3e−5 | ± 1e−4 | PASS |
| Cpk | 0.6667printed in the source | 0.6666666666667 | −3.3e−5 | ± 1e−4 | PASS |
| Pp (overall s = 2) | 1printed in the source | 1 | 0 | ± 1e−4 | PASS |
| Ppk (overall s = 2) | 0.6667printed in the source | 0.6666666666667 | −3.3e−5 | ± 1e−4 | PASS |
| One-sided (USL only): Cpk = Cpu | 0.6667printed in the source | 0.6666666666667 | −3.3e−5 | ± 1e−4 | PASS |
Cpk 95% Confidence Interval: Cpk 1.17, 50 Points
2/2 passLimitLine uses Bissell’s approximation Cpk ± 1.96·√(1/(9n) + Cpk²/(2(n − 1))).
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Lower 95% limit | 0.92printed in the source | 0.9206097774316→ 0.92 | 6.1e−4 | same at 2 decimals | PASS |
| Upper 95% limit | 1.42printed in the source | 1.419390222568→ 1.42 | −6.1e−4 | same at 2 decimals | PASS |
Gauge R&R (Measurement Systems Analysis) 13/13
Crossed Gauge R&R by the ANOVA method. Like Minitab’s default, LimitLine removes the part × operator interaction when its p-value is above 0.05.
AIAG Gauge Study: 10 Parts × 3 Operators × 3 Trials, ANOVA Mean Squares and Variance Components
9/9 pass- Minitab output for its sample data set GAGEAIAG.MTW (same AIAG data), Gage R&R Study (Crossed), ANOVA method. Values as transcribed in LimitLine’s test file; no public link.
- SPC for Excel, “Three Methods to Analyze Gage R&R Studies” (raw data, stated there to be from the AIAG Measurement Systems Analysis manual, 4th edition; ANOVA %Contribution, Table 5)
Data: aiag-msa4-gage-rr.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| MS part | 9.81799printed in the source | 9.817992716049→ 9.81799 | 2.7e−6 | same at 5 decimals | PASS |
| MS operator | 1.58363printed in the source | 1.583631111111→ 1.58363 | 1.1e−6 | same at 5 decimals | PASS |
| MS repeatability (pooled) | 0.03997printed in the source | 0.03997327635328→ 0.03997 | 3.3e−6 | same at 5 decimals | PASS |
| Part × operator interaction p-value | 0.974printed in the source | 0.974106404329→ 0.974 | 1.1e−4 | same at 3 decimals | PASS |
| Interaction removed from the model (p > 0.05) | removedprinted in the source | removed | none | exact match | PASS |
| Variance component, operator | 0.0514553printed in the source | 0.05145526115859→ 0.0514553 | −3.9e−8 | same at 7 decimals | PASS |
| Variance component, part-to-part | 1.08645printed in the source | 1.086446604411→ 1.08645 | −3.4e−6 | same at 5 decimals | PASS |
| Variance component, total gauge R&RThe printed 0.0914253 is the sum of the rounded 0.03997 + 0.0514553; the unrounded sum is 0.0914285, so the check allows 5 parts in 100,000. | 0.0914253printed in the source | 0.09142853751187 | 3.2e−6 | relative ± 5e−5 | PASS |
| Variance component, total variation | 1.17788printed in the source | 1.177875141923→ 1.17788 | −4.9e−6 | same at 5 decimals | PASS |
AIAG Gauge Study: % Contribution (ANOVA Method)
4/4 passData: aiag-msa4-gage-rr.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| % contribution, Total gauge R&R | 7.76printed in the source | 7.762158675209→ 7.76 | 2.2e−3 | same at 2 decimals | PASS |
| % contribution, Repeatability | 3.39printed in the source | 3.393676878861→ 3.39 | 3.7e−3 | same at 2 decimals | PASS |
| % contribution, Reproducibility | 4.37printed in the source | 4.368481796348→ 4.37 | −1.5e−3 | same at 2 decimals | PASS |
| % contribution, Part-to-part | 92.24printed in the source | 92.23784132479→ 92.24 | −2.2e−3 | same at 2 decimals | PASS |
Regression and Numerical Accuracy 22/22
NIST Statistical Reference Datasets with certified values. Longley is a notoriously ill-conditioned regression; NumAcc2 and NumAcc4 test whether means and standard deviations keep their digits when values are large and close together. “Correct digits” is the log relative error, LRE = −log10(|value − certified| / |certified|).
NIST StRD Longley: 6 Predictors, 16 Observations
17/17 passData: nist-strd-longley.csv
Show All 17 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Coefficient, Intercept | −3482258.634596NIST certified value | −3482258.634596 | −6.5e−9 | relative ± 1e−8 | PASS |
| Coefficient, GNPDEFL | 15.06187227137NIST certified value | 15.06187227138 | 2.5e−12 | relative ± 1e−8 | PASS |
| Coefficient, GNP | −0.03581917929259NIST certified value | −0.03581917929259 | −6.5e−16 | relative ± 1e−8 | PASS |
| Coefficient, UNEMP | −2.020229803817NIST certified value | −2.020229803817 | −3.1e−15 | relative ± 1e−8 | PASS |
| Coefficient, ARMED | −1.033226867174NIST certified value | −1.033226867174 | −3.1e−15 | relative ± 1e−8 | PASS |
| Coefficient, POP | −0.05110410565358NIST certified value | −0.05110410565358 | 3.2e−15 | relative ± 1e−8 | PASS |
| Coefficient, YEAR | 1829.151464614NIST certified value | 1829.151464614 | 6.1e−12 | relative ± 1e−8 | PASS |
| Standard error, Intercept | 890420.3836074NIST certified value | 890420.3836075 | 9.1e−8 | relative ± 1e−8 | PASS |
| Standard error, GNPDEFL | 84.91492577477NIST certified value | 84.91492577478 | 8.9e−12 | relative ± 1e−8 | PASS |
| Standard error, GNP | 0.03349100777224NIST certified value | 0.03349100777225 | 3.4e−15 | relative ± 1e−8 | PASS |
| Standard error, UNEMP | 0.4883996816517NIST certified value | 0.4883996816517 | 5.0e−14 | relative ± 1e−8 | PASS |
| Standard error, ARMED | 0.2142741631617NIST certified value | 0.2142741631617 | 2.2e−14 | relative ± 1e−8 | PASS |
| Standard error, POP | 0.2260732000694NIST certified value | 0.2260732000694 | 2.3e−14 | relative ± 1e−8 | PASS |
| Standard error, YEAR | 455.4784991422NIST certified value | 455.4784991423 | 4.7e−11 | relative ± 1e−8 | PASS |
| Residual standard deviation | 304.854073562NIST certified value | 304.854073562 | 3.1e−11 | relative ± 1e−9 | PASS |
| R² | 0.9954790045773NIST certified value | 0.9954790045773 | −1.3e−15 | relative ± 1e−10 | PASS |
| F statistic | 330.2853392346NIST certified value | 330.2853392345 | −6.8e−11 | relative ± 1e−8 | PASS |
NIST StRD NumAcc4: 1001 Values Near 10,000,000
3/3 pass- NIST StRD Univariate Summary Statistics: NumAcc4 (certified mean 10000000.2, s 0.1, lag-1 autocorrelation −0.999). · comparison scores: Stata 17 NIST certification results
8.3 correct digits is the best possible for s here: 10000000.1 cannot be stored exactly in binary floating point. Stata 17’s published scores are shown for comparison.
Data: nist-strd-numacc4.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Mean, correct digits (Stata 17: 15.7) | 10000000.2NIST certified value | 10000000.215.7 correct digits | 1.9e−9 | ≥ 15 correct digits | PASS |
| Standard deviation, correct digits (Stata 17: 8.3) | 0.1NIST certified value | 0.10000000055888.3 correct digits | 5.6e−10 | ≥ 8 correct digits | PASS |
| Lag-1 autocorrelation, correct digits (Stata 17: 10.7) | −0.999NIST certified value | −0.998999999981410.7 correct digits | 1.9e−11 | ≥ 10.5 correct digits | PASS |
NIST StRD NumAcc2: 1001 Values Near 1.2
2/2 pass- NIST StRD Univariate Summary Statistics: NumAcc2 (certified s 0.1, lag-1 autocorrelation −0.999). · comparison scores: Stata 17 NIST certification results
The lag-1 autocorrelation is computed by the stats module but not shown in the app today.
Data: nist-strd-numacc2.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Standard deviation, correct digits (Stata 17: 15.6) | 0.1NIST certified value | 0.115.6 correct digits | −2.8e−17 | ≥ 15.5 correct digits | PASS |
| Lag-1 autocorrelation, correct digits (Stata 17: 16.0) | −0.999NIST certified value | −0.999all digits | 1.1e−16 | ≥ 15.5 correct digits | PASS |
Run Charts 97/97
Anhøj run-chart rules: a signal when the longest run on one side of the median is too long or the line crosses the median too few times.
Anhøj Limits for 10 to 30 Useful Points
21/21 passEach row feeds LimitLine a series with exactly that many points off the median and reads back its two limits.
Show All 21 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| n = 10: longest-run limit / fewest crossings | 6 / 2printed in the source | 6 / 2 | none | exact match | PASS |
| n = 11: longest-run limit / fewest crossings | 6 / 2printed in the source | 6 / 2 | none | exact match | PASS |
| n = 12: longest-run limit / fewest crossings | 7 / 3printed in the source | 7 / 3 | none | exact match | PASS |
| n = 13: longest-run limit / fewest crossings | 7 / 3printed in the source | 7 / 3 | none | exact match | PASS |
| n = 14: longest-run limit / fewest crossings | 7 / 4printed in the source | 7 / 4 | none | exact match | PASS |
| n = 15: longest-run limit / fewest crossings | 7 / 4printed in the source | 7 / 4 | none | exact match | PASS |
| n = 16: longest-run limit / fewest crossings | 7 / 4printed in the source | 7 / 4 | none | exact match | PASS |
| n = 17: longest-run limit / fewest crossings | 7 / 5printed in the source | 7 / 5 | none | exact match | PASS |
| n = 18: longest-run limit / fewest crossings | 7 / 5printed in the source | 7 / 5 | none | exact match | PASS |
| n = 19: longest-run limit / fewest crossings | 7 / 6printed in the source | 7 / 6 | none | exact match | PASS |
| n = 20: longest-run limit / fewest crossings | 7 / 6printed in the source | 7 / 6 | none | exact match | PASS |
| n = 21: longest-run limit / fewest crossings | 7 / 6printed in the source | 7 / 6 | none | exact match | PASS |
| n = 22: longest-run limit / fewest crossings | 7 / 7printed in the source | 7 / 7 | none | exact match | PASS |
| n = 23: longest-run limit / fewest crossings | 8 / 7printed in the source | 8 / 7 | none | exact match | PASS |
| n = 24: longest-run limit / fewest crossings | 8 / 8printed in the source | 8 / 8 | none | exact match | PASS |
| n = 25: longest-run limit / fewest crossings | 8 / 8printed in the source | 8 / 8 | none | exact match | PASS |
| n = 26: longest-run limit / fewest crossings | 8 / 8printed in the source | 8 / 8 | none | exact match | PASS |
| n = 27: longest-run limit / fewest crossings | 8 / 9printed in the source | 8 / 9 | none | exact match | PASS |
| n = 28: longest-run limit / fewest crossings | 8 / 9printed in the source | 8 / 9 | none | exact match | PASS |
| n = 29: longest-run limit / fewest crossings | 8 / 10printed in the source | 8 / 10 | none | exact match | PASS |
| n = 30: longest-run limit / fewest crossings | 8 / 10printed in the source | 8 / 10 | none | exact match | PASS |
Anhøj Figure 2 Case: 24 Points, 2 on the Median
6/6 passThe series reproduces the figure’s pattern (1 = above the median, −1 = below, 0 = on it), not its original values.
Data: anhoj-figure2-pattern.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Useful points (not on the median) | 22printed in the source | 22 | none | exact match | PASS |
| Longest run | 10printed in the source | 10 | none | exact match | PASS |
| Crossings | 3printed in the source | 3 | none | exact match | PASS |
| Longest-run limit | 7printed in the source | 7 | none | exact match | PASS |
| Fewest-crossings limit | 7printed in the source | 7 | none | exact match | PASS |
| Signal | yesprinted in the source | yes | none | exact match | PASS |
Anhøj Limits for 31 to 100 Useful Points
70/70 passSame method as above, against the longer table in the qicharts2 vignette.
Show All 70 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| n = 31: longest-run limit / fewest crossings | 8 / 11printed in the source | 8 / 11 | none | exact match | PASS |
| n = 32: longest-run limit / fewest crossings | 8 / 11printed in the source | 8 / 11 | none | exact match | PASS |
| n = 33: longest-run limit / fewest crossings | 8 / 11printed in the source | 8 / 11 | none | exact match | PASS |
| n = 34: longest-run limit / fewest crossings | 8 / 12printed in the source | 8 / 12 | none | exact match | PASS |
| n = 35: longest-run limit / fewest crossings | 8 / 12printed in the source | 8 / 12 | none | exact match | PASS |
| n = 36: longest-run limit / fewest crossings | 8 / 13printed in the source | 8 / 13 | none | exact match | PASS |
| n = 37: longest-run limit / fewest crossings | 8 / 13printed in the source | 8 / 13 | none | exact match | PASS |
| n = 38: longest-run limit / fewest crossings | 8 / 14printed in the source | 8 / 14 | none | exact match | PASS |
| n = 39: longest-run limit / fewest crossings | 8 / 14printed in the source | 8 / 14 | none | exact match | PASS |
| n = 40: longest-run limit / fewest crossings | 8 / 14printed in the source | 8 / 14 | none | exact match | PASS |
| n = 41: longest-run limit / fewest crossings | 8 / 15printed in the source | 8 / 15 | none | exact match | PASS |
| n = 42: longest-run limit / fewest crossings | 8 / 15printed in the source | 8 / 15 | none | exact match | PASS |
| n = 43: longest-run limit / fewest crossings | 8 / 16printed in the source | 8 / 16 | none | exact match | PASS |
| n = 44: longest-run limit / fewest crossings | 8 / 16printed in the source | 8 / 16 | none | exact match | PASS |
| n = 45: longest-run limit / fewest crossings | 8 / 17printed in the source | 8 / 17 | none | exact match | PASS |
| n = 46: longest-run limit / fewest crossings | 9 / 17printed in the source | 9 / 17 | none | exact match | PASS |
| n = 47: longest-run limit / fewest crossings | 9 / 17printed in the source | 9 / 17 | none | exact match | PASS |
| n = 48: longest-run limit / fewest crossings | 9 / 18printed in the source | 9 / 18 | none | exact match | PASS |
| n = 49: longest-run limit / fewest crossings | 9 / 18printed in the source | 9 / 18 | none | exact match | PASS |
| n = 50: longest-run limit / fewest crossings | 9 / 19printed in the source | 9 / 19 | none | exact match | PASS |
| n = 51: longest-run limit / fewest crossings | 9 / 19printed in the source | 9 / 19 | none | exact match | PASS |
| n = 52: longest-run limit / fewest crossings | 9 / 20printed in the source | 9 / 20 | none | exact match | PASS |
| n = 53: longest-run limit / fewest crossings | 9 / 20printed in the source | 9 / 20 | none | exact match | PASS |
| n = 54: longest-run limit / fewest crossings | 9 / 21printed in the source | 9 / 21 | none | exact match | PASS |
| n = 55: longest-run limit / fewest crossings | 9 / 21printed in the source | 9 / 21 | none | exact match | PASS |
| n = 56: longest-run limit / fewest crossings | 9 / 21printed in the source | 9 / 21 | none | exact match | PASS |
| n = 57: longest-run limit / fewest crossings | 9 / 22printed in the source | 9 / 22 | none | exact match | PASS |
| n = 58: longest-run limit / fewest crossings | 9 / 22printed in the source | 9 / 22 | none | exact match | PASS |
| n = 59: longest-run limit / fewest crossings | 9 / 23printed in the source | 9 / 23 | none | exact match | PASS |
| n = 60: longest-run limit / fewest crossings | 9 / 23printed in the source | 9 / 23 | none | exact match | PASS |
| n = 61: longest-run limit / fewest crossings | 9 / 24printed in the source | 9 / 24 | none | exact match | PASS |
| n = 62: longest-run limit / fewest crossings | 9 / 24printed in the source | 9 / 24 | none | exact match | PASS |
| n = 63: longest-run limit / fewest crossings | 9 / 25printed in the source | 9 / 25 | none | exact match | PASS |
| n = 64: longest-run limit / fewest crossings | 9 / 25printed in the source | 9 / 25 | none | exact match | PASS |
| n = 65: longest-run limit / fewest crossings | 9 / 25printed in the source | 9 / 25 | none | exact match | PASS |
| n = 66: longest-run limit / fewest crossings | 9 / 26printed in the source | 9 / 26 | none | exact match | PASS |
| n = 67: longest-run limit / fewest crossings | 9 / 26printed in the source | 9 / 26 | none | exact match | PASS |
| n = 68: longest-run limit / fewest crossings | 9 / 27printed in the source | 9 / 27 | none | exact match | PASS |
| n = 69: longest-run limit / fewest crossings | 9 / 27printed in the source | 9 / 27 | none | exact match | PASS |
| n = 70: longest-run limit / fewest crossings | 9 / 28printed in the source | 9 / 28 | none | exact match | PASS |
| n = 71: longest-run limit / fewest crossings | 9 / 28printed in the source | 9 / 28 | none | exact match | PASS |
| n = 72: longest-run limit / fewest crossings | 9 / 29printed in the source | 9 / 29 | none | exact match | PASS |
| n = 73: longest-run limit / fewest crossings | 9 / 29printed in the source | 9 / 29 | none | exact match | PASS |
| n = 74: longest-run limit / fewest crossings | 9 / 29printed in the source | 9 / 29 | none | exact match | PASS |
| n = 75: longest-run limit / fewest crossings | 9 / 30printed in the source | 9 / 30 | none | exact match | PASS |
| n = 76: longest-run limit / fewest crossings | 9 / 30printed in the source | 9 / 30 | none | exact match | PASS |
| n = 77: longest-run limit / fewest crossings | 9 / 31printed in the source | 9 / 31 | none | exact match | PASS |
| n = 78: longest-run limit / fewest crossings | 9 / 31printed in the source | 9 / 31 | none | exact match | PASS |
| n = 79: longest-run limit / fewest crossings | 9 / 32printed in the source | 9 / 32 | none | exact match | PASS |
| n = 80: longest-run limit / fewest crossings | 9 / 32printed in the source | 9 / 32 | none | exact match | PASS |
| n = 81: longest-run limit / fewest crossings | 9 / 33printed in the source | 9 / 33 | none | exact match | PASS |
| n = 82: longest-run limit / fewest crossings | 9 / 33printed in the source | 9 / 33 | none | exact match | PASS |
| n = 83: longest-run limit / fewest crossings | 9 / 34printed in the source | 9 / 34 | none | exact match | PASS |
| n = 84: longest-run limit / fewest crossings | 9 / 34printed in the source | 9 / 34 | none | exact match | PASS |
| n = 85: longest-run limit / fewest crossings | 9 / 34printed in the source | 9 / 34 | none | exact match | PASS |
| n = 86: longest-run limit / fewest crossings | 9 / 35printed in the source | 9 / 35 | none | exact match | PASS |
| n = 87: longest-run limit / fewest crossings | 9 / 35printed in the source | 9 / 35 | none | exact match | PASS |
| n = 88: longest-run limit / fewest crossings | 9 / 36printed in the source | 9 / 36 | none | exact match | PASS |
| n = 89: longest-run limit / fewest crossings | 9 / 36printed in the source | 9 / 36 | none | exact match | PASS |
| n = 90: longest-run limit / fewest crossings | 9 / 37printed in the source | 9 / 37 | none | exact match | PASS |
| n = 91: longest-run limit / fewest crossings | 10 / 37printed in the source | 10 / 37 | none | exact match | PASS |
| n = 92: longest-run limit / fewest crossings | 10 / 38printed in the source | 10 / 38 | none | exact match | PASS |
| n = 93: longest-run limit / fewest crossings | 10 / 38printed in the source | 10 / 38 | none | exact match | PASS |
| n = 94: longest-run limit / fewest crossings | 10 / 39printed in the source | 10 / 39 | none | exact match | PASS |
| n = 95: longest-run limit / fewest crossings | 10 / 39printed in the source | 10 / 39 | none | exact match | PASS |
| n = 96: longest-run limit / fewest crossings | 10 / 39printed in the source | 10 / 39 | none | exact match | PASS |
| n = 97: longest-run limit / fewest crossings | 10 / 40printed in the source | 10 / 40 | none | exact match | PASS |
| n = 98: longest-run limit / fewest crossings | 10 / 40printed in the source | 10 / 40 | none | exact match | PASS |
| n = 99: longest-run limit / fewest crossings | 10 / 41printed in the source | 10 / 41 | none | exact match | PASS |
| n = 100: longest-run limit / fewest crossings | 10 / 41printed in the source | 10 / 41 | none | exact match | PASS |
Time-Weighted Charts (EWMA, CUSUM) 65/65
Charts that accumulate evidence over time to catch small, sustained shifts sooner than a Shewhart chart.
NIST EWMA Chart: 20 Points, λ = 0.3
23/23 passNIST uses the long-run (asymptotic) limits. LimitLine uses the exact limits, which start narrower and widen to the same values within a few points; the last point’s limits are compared.
Data: nist-ewma.csv
Show All 23 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| EWMA at point 1 | 50.6printed in the source | 50.6→ 50.60 | 0 | same at 2 decimals | PASS |
| EWMA at point 2 | 49.52printed in the source | 49.52→ 49.52 | 0 | same at 2 decimals | PASS |
| EWMA at point 3 | 50.56printed in the source | 50.564→ 50.56 | 4.0e−3 | same at 2 decimals | PASS |
| EWMA at point 4 | 50.18printed in the source | 50.1848→ 50.18 | 4.8e−3 | same at 2 decimals | PASS |
| EWMA at point 5 | 50.16printed in the source | 50.15936→ 50.16 | −6.4e−4 | same at 2 decimals | PASS |
| EWMA at point 6 | 49.21printed in the source | 49.211552→ 49.21 | 1.6e−3 | same at 2 decimals | PASS |
| EWMA at point 7 | 49.75printed in the source | 49.7480864→ 49.75 | −1.9e−3 | same at 2 decimals | PASS |
| EWMA at point 8 | 49.85printed in the source | 49.85366048→ 49.85 | 3.7e−3 | same at 2 decimals | PASS |
| EWMA at point 9 | 50.26printed in the source | 50.257562336→ 50.26 | −2.4e−3 | same at 2 decimals | PASS |
| EWMA at point 10 | 50.33printed in the source | 50.3302936352→ 50.33 | 2.9e−4 | same at 2 decimals | PASS |
| EWMA at point 11 | 50.11printed in the source | 50.11120554464→ 50.11 | 1.2e−3 | same at 2 decimals | PASS |
| EWMA at point 12 | 49.36printed in the source | 49.35784388125→ 49.36 | −2.2e−3 | same at 2 decimals | PASS |
| EWMA at point 13 | 49.52printed in the source | 49.52049071687→ 49.52 | 4.9e−4 | same at 2 decimals | PASS |
| EWMA at point 14 | 50.05printed in the source | 50.05434350181→ 50.05 | 4.3e−3 | same at 2 decimals | PASS |
| EWMA at point 15 | 49.38printed in the source | 49.37804045127→ 49.38 | −2.0e−3 | same at 2 decimals | PASS |
| EWMA at point 16 | 49.92printed in the source | 49.92462831589→ 49.92 | 4.6e−3 | same at 2 decimals | PASS |
| EWMA at point 17 | 50.73printed in the source | 50.72723982112→ 50.73 | −2.8e−3 | same at 2 decimals | PASS |
| EWMA at point 18 | 51.23printed in the source | 51.22906787478→ 51.23 | −9.3e−4 | same at 2 decimals | PASS |
| EWMA at point 19 | 51.94printed in the source | 51.94034751235→ 51.94 | 3.5e−4 | same at 2 decimals | PASS |
| EWMA at point 20 | 51.99printed in the source | 51.98824325864→ 51.99 | −1.8e−3 | same at 2 decimals | PASS |
| UCL (long-run) | 52.5884printed in the source | 52.58843091412 | 3.1e−5 | ± 2e−4 | PASS |
| LCL (long-run) | 47.4115printed in the source | 47.41156908588 | 6.9e−5 | ± 2e−4 | PASS |
| Points beyond the limits | 0printed in the source | 0 | none | exact match | PASS |
NIST Tabular CUSUM: 20 Group Means, Target 325
42/42 passNIST gives k and h in data units (0.3175 and 4.1959, with σ of the means 0.635); LimitLine takes them in σ units, so k = 0.5 and h = 4.1959 / 0.635. NIST rounds the inputs to 2 decimals before adding, so its last sums can differ by 0.01.
Data: nist-cusum.csv
Show All 42 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Upper sum S_hi at group 1 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 2 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 3 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 4 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 5 | 0.03printed in the source | 0.03250000000002 | 2.5e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 6 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 7 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 8 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 9 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 10 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 11 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 12 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Upper sum S_hi at group 13 | 3.01printed in the source | 3.0075 | −2.5e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 14 | 4.94printed in the source | 4.94 | −1.2e−14 | ± 0.011 | PASS |
| Upper sum S_hi at group 15 | 7.45printed in the source | 7.4475 | −2.5e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 16 | 10.63printed in the source | 10.63 | −2.5e−14 | ± 0.011 | PASS |
| Upper sum S_hi at group 17 | 11.99printed in the source | 11.9875 | −2.5e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 18 | 14.44printed in the source | 14.445 | 5.0e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 19 | 16printed in the source | 16.0025 | 2.5e−3 | ± 0.011 | PASS |
| Upper sum S_hi at group 20 | 19.04printed in the source | 19.035 | −5.0e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 1 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 2 | 0.01printed in the source | 0.007499999999989 | −2.5e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 3 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 4 | 0.33printed in the source | 0.3325 | 2.5e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 5 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 6 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 7 | 0.56printed in the source | 0.5575 | −2.5e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 8 | 0.72printed in the source | 0.715 | −5.0e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 9 | 0.17printed in the source | 0.1725 | 2.5e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 10 | 0.25printed in the source | 0.255 | 5.0e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 11 | 0.31printed in the source | 0.3125 | 2.5e−3 | ± 0.011 | PASS |
| Lower sum S_lo at group 12 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 13 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 14 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 15 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 16 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 17 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 18 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 19 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| Lower sum S_lo at group 20 | 0printed in the source | 0 | 0 | ± 0.011 | PASS |
| First signal (group) | 14printed in the source | 14 | none | exact match | PASS |
| Groups signaling | 7printed in the source | 7 | none | exact match | PASS |
Lean Tools 7/7
Published worked examples for OEE, takt time and heijunka.
OEE Worked Example
4/4 pass| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Availability | 0.8881printed in the source | 0.8880952380952 | −4.8e−6 | ± 5e−5 | PASS |
| Performance | 0.8611printed in the source | 0.8610813226095 | −1.9e−5 | ± 5e−5 | PASS |
| Quality | 0.978printed in the source | 0.9780499195683 | 5.0e−5 | ± 5e−5 | PASS |
| OEE | 0.7479printed in the source | 0.7479365079365 | 3.7e−5 | ± 5e−5 | PASS |
Takt Time
1/1 pass| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Takt time, seconds (480 min ÷ 240 units) | 120printed in the source | 120 | 0 | ± 1e−12 | PASS |
Heijunka: Leveling by Volume
2/2 pass| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Level rate per day | 100printed in the source | 100 | 0 | ± 1e−12 | PASS |
| Buffer needed to cover Monday | 100computed here from the source’s data and formula | 100 | 0 | ± 1e−12 | PASS |
Power and Sample Size 2/2
How many units a t test needs to detect a given difference: the classic published table.
Cohen's Table: Two-Sample T, Power 0.80, α 0.05 Two-Sided
2/2 pass- J. Cohen (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Table 2.4.1 (n per group for a two-sample t test, power 0.80, α 0.05 two-sided)
Cohen’s medium and large effects (d = 0.5, 0.8). For the small effect (d = 0.2), Cohen’s table, built on an approximation, prints 393; the exact noncentral t (as in statsmodels and G*Power) gives 394, which LimitLine also gives, so that row isn't counted here.
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| d = 0.5: n per group | 64printed in the source | 64 | none | exact match | PASS |
| d = 0.8: n per group | 26printed in the source | 26 | none | exact match | PASS |
Cross-Checks Against Independent Statistical Software 484/484
These are not published examples. ref/ref_gen.py ran SciPy and statsmodels on seeded random data and stored their answers in ref/refs.json; LimitLine must reproduce them. They test the same calculations by a second, independent implementation, across many more inputs than textbook examples cover.
Probability Distributions
92/92 passt, F and chi-square tail areas, the normal CDF, t quantiles, and log-gamma (Python math.lgamma). The p-values for every test in LimitLine come from these functions.
Show All 92 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| t CDF (t = -3.2, df = 1) | 0.09641124797923independent software output | 0.09641124797923 | −5.6e−17 | relative ± 1e−10 | PASS |
| t CDF (t = -3.2, df = 3) | 0.02466592148135independent software output | 0.02466592148135 | 3.8e−17 | relative ± 1e−10 | PASS |
| t CDF (t = -3.2, df = 9) | 0.005415651294951independent software output | 0.005415651294951 | −8.7e−19 | relative ± 1e−10 | PASS |
| t CDF (t = -3.2, df = 30) | 0.001619300855977independent software output | 0.001619300855977 | 9.3e−18 | relative ± 1e−10 | PASS |
| t CDF (t = -3.2, df = 200) | 0.0007991768707151independent software output | 0.000799176870715 | −4.1e−17 | relative ± 1e−10 | PASS |
| t CDF (t = -1.1, df = 1) | 0.2348538278116independent software output | 0.2348538278116 | −2.8e−17 | relative ± 1e−10 | PASS |
| t CDF (t = -1.1, df = 3) | 0.1758415974673independent software output | 0.1758415974673 | −2.2e−16 | relative ± 1e−10 | PASS |
| t CDF (t = -1.1, df = 9) | 0.1499413809284independent software output | 0.1499413809284 | 3.3e−16 | relative ± 1e−10 | PASS |
| t CDF (t = -1.1, df = 30) | 0.1400404590438independent software output | 0.1400404590438 | −2.1e−15 | relative ± 1e−10 | PASS |
| t CDF (t = -1.1, df = 200) | 0.1363271860987independent software output | 0.1363271860987 | 1.6e−14 | relative ± 1e−10 | PASS |
| t CDF (t = 0.4, df = 1) | 0.6211189415908independent software output | 0.6211189415908 | 2.2e−16 | relative ± 1e−10 | PASS |
| t CDF (t = 0.4, df = 3) | 0.6420324230128independent software output | 0.6420324230128 | 1.1e−16 | relative ± 1e−10 | PASS |
| t CDF (t = 0.4, df = 9) | 0.6507579541643independent software output | 0.6507579541643 | −1.1e−16 | relative ± 1e−10 | PASS |
| t CDF (t = 0.4, df = 30) | 0.6540047417429independent software output | 0.6540047417429 | 6.7e−16 | relative ± 1e−10 | PASS |
| t CDF (t = 0.4, df = 200) | 0.655208297837independent software output | 0.6552082978369 | −1.2e−14 | relative ± 1e−10 | PASS |
| t CDF (t = 2.5, df = 1) | 0.8788810584092independent software output | 0.8788810584092 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 2.5, df = 3) | 0.956146676496independent software output | 0.956146676496 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 2.5, df = 9) | 0.9830690861585independent software output | 0.9830690861585 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 2.5, df = 30) | 0.990942175466independent software output | 0.990942175466 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 2.5, df = 200) | 0.9933884136791independent software output | 0.9933884136791 | 3.3e−16 | relative ± 1e−10 | PASS |
| t CDF (t = 6, df = 1) | 0.9474315432887independent software output | 0.9474315432887 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 6, df = 3) | 0.9953636425539independent software output | 0.9953636425539 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 6, df = 9) | 0.999898750339independent software output | 0.999898750339 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 6, df = 30) | 0.9999993028616independent software output | 0.9999993028616 | 0 | relative ± 1e−10 | PASS |
| t CDF (t = 6, df = 200) | 0.9999999954424independent software output | 0.9999999954424 | 0 | relative ± 1e−10 | PASS |
| F upper tail (F = 0.3, df 1, 5) | 0.6074354940759independent software output | 0.6074354940759 | 1.1e−16 | relative ± 1e−10 | PASS |
| F upper tail (F = 0.3, df 3, 12) | 0.8248035702258independent software output | 0.8248035702258 | 1.1e−16 | relative ± 1e−10 | PASS |
| F upper tail (F = 0.3, df 6, 9) | 0.9217416608607independent software output | 0.9217416608607 | −1.1e−16 | relative ± 1e−10 | PASS |
| F upper tail (F = 0.3, df 20, 40) | 0.9973296186574independent software output | 0.9973296186574 | 0 | relative ± 1e−10 | PASS |
| F upper tail (F = 1.2, df 1, 5) | 0.3232601902798independent software output | 0.3232601902798 | 5.0e−16 | relative ± 1e−10 | PASS |
| F upper tail (F = 1.2, df 3, 12) | 0.351512640724independent software output | 0.351512640724 | 1.6e−15 | relative ± 1e−10 | PASS |
| F upper tail (F = 1.2, df 6, 9) | 0.3865692436842independent software output | 0.3865692436842 | −5.6e−17 | relative ± 1e−10 | PASS |
| F upper tail (F = 1.2, df 20, 40) | 0.3035030346267independent software output | 0.3035030346267 | −4.4e−15 | relative ± 1e−10 | PASS |
| F upper tail (F = 4.5, df 1, 5) | 0.08735930812736independent software output | 0.08735930812736 | −5.6e−17 | relative ± 1e−10 | PASS |
| F upper tail (F = 4.5, df 3, 12) | 0.02457347884095independent software output | 0.02457347884095 | −7.3e−17 | relative ± 1e−10 | PASS |
| F upper tail (F = 4.5, df 6, 9) | 0.02214050292969independent software output | 0.02214050292969 | −2.1e−17 | relative ± 1e−10 | PASS |
| F upper tail (F = 4.5, df 20, 40) | 2.5950300463e−5independent software output | 2.5950300463e−5 | −2.3e−19 | relative ± 1e−10 | PASS |
| F upper tail (F = 30, df 1, 5) | 0.002764960301305independent software output | 0.002764960301305 | −5.2e−18 | relative ± 1e−10 | PASS |
| F upper tail (F = 30, df 3, 12) | 7.3730324758e−6independent software output | 7.3730324758e−6 | −1.2e−20 | relative ± 1e−10 | PASS |
| F upper tail (F = 30, df 6, 9) | 1.8525315594e−5independent software output | 1.8525315594e−5 | −5.8e−20 | relative ± 1e−10 | PASS |
| F upper tail (F = 30, df 20, 40) | 4.7700820249e−18independent software output | 4.7700820249e−18 | −4.3e−32 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 0.5, df = 1) | 0.479500122187independent software output | 0.479500122187 | −2.2e−16 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 0.5, df = 2) | 0.7788007830714independent software output | 0.7788007830714 | −2.2e−16 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 0.5, df = 5) | 0.9921232932326independent software output | 0.9921232932326 | 0 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 0.5, df = 17) | 0.9999999999488independent software output | 0.9999999999488 | 0 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 3, df = 1) | 0.08326451666355independent software output | 0.08326451666355 | 9.7e−17 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 3, df = 2) | 0.2231301601484independent software output | 0.2231301601484 | −5.8e−16 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 3, df = 5) | 0.6999858358786independent software output | 0.6999858358786 | 3.3e−16 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 3, df = 17) | 0.9999304982629independent software output | 0.9999304982629 | 0 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 11, df = 1) | 0.0009111188771537independent software output | 0.0009111188771537 | −8.7e−19 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 11, df = 2) | 0.004086771438464independent software output | 0.004086771438464 | 3.5e−18 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 11, df = 5) | 0.05137998348307independent software output | 0.05137998348307 | −9.0e−17 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 11, df = 17) | 0.8565639878403independent software output | 0.8565639878403 | 3.3e−16 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 40, df = 1) | 2.5396285895e−10independent software output | 2.5396285895e−10 | −5.2e−26 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 40, df = 2) | 2.0611536224e−9independent software output | 2.0611536224e−9 | 0 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 40, df = 5) | 1.4933679001e−7independent software output | 1.4933679001e−7 | −2.1e−22 | relative ± 1e−10 | PASS |
| Chi-square upper tail (x = 40, df = 17) | 0.001294198533743independent software output | 0.001294198533743 | −4.3e−18 | relative ± 1e−10 | PASS |
| Normal CDF (z = -8) | 6.2209605743e−16independent software output | 6.2209605743e−16 | 9.4e−30 | relative ± 1e−12 | PASS |
| Normal CDF (z = -5) | 2.8665157188e−7independent software output | 2.8665157188e−7 | 1.2e−21 | relative ± 1e−12 | PASS |
| Normal CDF (z = -3) | 0.00134989803163independent software output | 0.00134989803163 | 3.3e−18 | relative ± 1e−12 | PASS |
| Normal CDF (z = -1.5) | 0.06680720126886independent software output | 0.06680720126886 | −2.6e−16 | relative ± 1e−12 | PASS |
| Normal CDF (z = 0) | 0.5independent software output | 0.5 | 0 | relative ± 1e−12 | PASS |
| Normal CDF (z = 0.7) | 0.7580363477769independent software output | 0.7580363477769 | 1.1e−16 | relative ± 1e−12 | PASS |
| Normal CDF (z = 2.2) | 0.9860965524865independent software output | 0.9860965524865 | 0 | relative ± 1e−12 | PASS |
| Normal CDF (z = 4) | 0.9999683287582independent software output | 0.9999683287582 | 0 | relative ± 1e−12 | PASS |
| Normal CDF (z = 6) | 0.9999999990134independent software output | 0.9999999990134 | 0 | relative ± 1e−12 | PASS |
| t quantile (p = 0.025, df = 1) | −12.70620473617independent software output | −12.70620473617 | 9.2e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.025, df = 4) | −2.776445105198independent software output | −2.776445105198 | 0 | relative ± 1e−9 | PASS |
| t quantile (p = 0.025, df = 15) | −2.13144954556independent software output | −2.13144954556 | 4.4e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.025, df = 100) | −1.983971518524independent software output | −1.983971518523 | 7.7e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.1, df = 1) | −3.077683537175independent software output | −3.077683537175 | 1.1e−13 | relative ± 1e−9 | PASS |
| t quantile (p = 0.1, df = 4) | −1.533206274059independent software output | −1.533206274059 | −2.6e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.1, df = 15) | −1.34060560785independent software output | −1.34060560785 | −7.8e−15 | relative ± 1e−9 | PASS |
| t quantile (p = 0.1, df = 100) | −1.290074761347independent software output | −1.290074761346 | 4.2e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.9, df = 1) | 3.077683537175independent software output | 3.077683537175 | −1.1e−13 | relative ± 1e−9 | PASS |
| t quantile (p = 0.9, df = 4) | 1.533206274059independent software output | 1.533206274059 | 2.5e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.9, df = 15) | 1.34060560785independent software output | 1.34060560785 | 7.5e−15 | relative ± 1e−9 | PASS |
| t quantile (p = 0.9, df = 100) | 1.290074761347independent software output | 1.290074761346 | −4.2e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.975, df = 1) | 12.70620473617independent software output | 12.70620473617 | −8.2e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.975, df = 4) | 2.776445105198independent software output | 2.776445105198 | 8.9e−16 | relative ± 1e−9 | PASS |
| t quantile (p = 0.975, df = 15) | 2.13144954556independent software output | 2.13144954556 | −4.4e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.975, df = 100) | 1.983971518524independent software output | 1.983971518523 | −7.7e−14 | relative ± 1e−9 | PASS |
| t quantile (p = 0.995, df = 1) | 63.65674116287independent software output | 63.65674116287 | −2.0e−13 | relative ± 1e−9 | PASS |
| t quantile (p = 0.995, df = 4) | 4.60409487135independent software output | 4.60409487135 | −1.1e−13 | relative ± 1e−9 | PASS |
| t quantile (p = 0.995, df = 15) | 2.946712883475independent software output | 2.946712883475 | −1.4e−13 | relative ± 1e−9 | PASS |
| t quantile (p = 0.995, df = 100) | 2.625890521438independent software output | 2.625890521438 | −5.1e−14 | relative ± 1e−9 | PASS |
| log-gamma (0.1) | 2.252712651734independent software output | 2.252712651734 | 4.4e−16 | ± 1e−12 | PASS |
| log-gamma (0.5) | 0.5723649429247independent software output | 0.5723649429247 | −8.9e−16 | ± 1e−12 | PASS |
| log-gamma (1) | 0independent software output | −8.881784197e−16 | −8.9e−16 | ± 1e−12 | PASS |
| log-gamma (2.5) | 0.2846828704729independent software output | 0.2846828704729 | 1.3e−15 | ± 1e−12 | PASS |
| log-gamma (10) | 12.80182748008independent software output | 12.80182748008 | 7.1e−15 | ± 1e−12 | PASS |
| log-gamma (171.3) | 708.114947039independent software output | 708.114947039 | −2.3e−13 | ± 1e−12 | PASS |
Descriptive Statistics (37 Values)
7/7 passQuartiles use the (n + 1)p method that Minitab uses (numpy percentile method “weibull”). The mean and standard deviation feed every tool; skewness, kurtosis and quartiles come from the same module but the app doesn’t show them today.
Data: refs-describe.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Mean | 5.681897368676independent software output | 5.681897368676 | 0 | relative ± 1e−13 | PASS |
| Standard deviation | 4.471880036468independent software output | 4.471880036468 | 0 | relative ± 1e−13 | PASS |
| Skewness (adjusted) | 1.207487634087independent software output | 1.207487634087 | −4.4e−16 | relative ± 1e−12 | PASS |
| Kurtosis (adjusted) | 1.292806393565independent software output | 1.292806393565 | −1.8e−15 | relative ± 1e−12 | PASS |
| Q1 | 2.197822189793independent software output | 2.197822189793 | 0 | relative ± 1e−13 | PASS |
| Median | 5.170722741533independent software output | 5.170722741533 | 0 | relative ± 1e−13 | PASS |
| Q3 | 7.449728222047independent software output | 7.449728222047 | 0 | relative ± 1e−13 | PASS |
Anderson-Darling Normality Test (4 Samples)
8/8 passData: refs-anderson-darling.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Sample 1 (n = 25): AD statistic | 0.2022517255367independent software output | 0.2022517255367 | 0 | relative ± 1e−10 | PASS |
| Sample 1: p-value | 0.8634339963817independent software output | 0.8634339963817 | 0 | ± 1e−9 | PASS |
| Sample 2 (n = 40): AD statistic | 0.7966172718201independent software output | 0.7966172718201 | −7.1e−15 | relative ± 1e−10 | PASS |
| Sample 2: p-value | 0.03566547290245independent software output | 0.03566547290246 | 1.5e−15 | ± 1e−9 | PASS |
| Sample 3 (n = 200): AD statistic | 0.4711891733073independent software output | 0.4711891733073 | 2.8e−14 | relative ± 1e−10 | PASS |
| Sample 3: p-value | 0.2429460060544independent software output | 0.2429460060544 | −3.9e−14 | ± 1e−9 | PASS |
| Sample 4 (n = 15): AD statistic | 0.5809110602341independent software output | 0.5809110602341 | −3.6e−15 | relative ± 1e−10 | PASS |
| Sample 4: p-value | 0.1091912875371independent software output | 0.1091912875371 | 2.3e−15 | ± 1e−9 | PASS |
Hypothesis Tests: T Tests, ANOVA, Levene, Chi-Square, Two Proportions
25/25 pass- SciPy 1.18 (scipy.stats), run by ref/ref_gen.py on seeded random data; values stored in ref/refs.json
- statsmodels (OLS, anova_lm, normal_ad, proportions_ztest, variance_inflation_factor), run by ref/ref_gen.py on seeded random data; values stored in ref/refs.json
Data: refs-t-tests.csv · refs-anova-groups.csv
Show All 25 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| 1-sample t (μ0 = 9.5): t | 0.01935998176627independent software output | 0.01935998176627 | 0 | relative ± 1e−12 | PASS |
| 1-sample t: p | 0.984779347686independent software output | 0.984779347686 | −2.6e−14 | relative ± 1e−10 | PASS |
| 1-sample t: 95% CI low | 8.601612726835independent software output | 8.601612726835 | 1.1e−14 | relative ± 1e−11 | PASS |
| 1-sample t: 95% CI high | 10.41502743645independent software output | 10.41502743645 | −1.1e−14 | relative ± 1e−11 | PASS |
| 1-sample t: p (one-sided, greater) | 0.492389673843independent software output | 0.492389673843 | −1.3e−14 | relative ± 1e−10 | PASS |
| Welch 2-sample t: t | −0.8824736934754independent software output | −0.8824736934754 | −1.1e−16 | relative ± 1e−12 | PASS |
| Welch: p | 0.3838189220058independent software output | 0.3838189220058 | −5.0e−16 | relative ± 1e−10 | PASS |
| Welch: df | 33.44907190079independent software output | 33.44907190079 | 7.1e−15 | relative ± 1e−12 | PASS |
| Welch: 95% CI low | −2.591163598292independent software output | −2.591163598292 | 2.6e−14 | relative ± 1e−10 | PASS |
| Welch: 95% CI high | 1.02280181955independent software output | 1.02280181955 | −2.6e−14 | relative ± 1e−10 | PASS |
| Pooled 2-sample t: t | −0.8172053632849independent software output | −0.8172053632849 | −1.1e−16 | relative ± 1e−12 | PASS |
| Pooled: p | 0.4187755041075independent software output | 0.4187755041075 | −1.5e−15 | relative ± 1e−10 | PASS |
| Pooled: 95% CI low | −2.72513188634independent software output | −2.72513188634 | −4.4e−15 | relative ± 1e−10 | PASS |
| Paired t: t | −2.976068678971independent software output | −2.976068678971 | 4.4e−16 | relative ± 1e−12 | PASS |
| Paired t: p | 0.008476036680099independent software output | 0.008476036680099 | 0 | relative ± 1e−10 | PASS |
| Paired t: 95% CI high | −0.2006319767431independent software output | −0.2006319767431 | −6.3e−15 | relative ± 1e−10 | PASS |
| One-way ANOVA: F | 4.764705737524independent software output | 4.764705737524 | −1.8e−15 | relative ± 1e−12 | PASS |
| One-way ANOVA: p | 0.006885553282113independent software output | 0.006885553282113 | −4.7e−17 | relative ± 1e−10 | PASS |
| Levene (median): W | 0.6922417635841independent software output | 0.6922417635841 | −1.1e−16 | relative ± 1e−12 | PASS |
| Levene: p | 0.5629570705917independent software output | 0.5629570705917 | 4.4e−15 | relative ± 1e−10 | PASS |
| Chi-square, 3×3 table [[12,30,18],[20,25,35],[8,14,20]]: χ² | 6.468289325657independent software output | 6.468289325657 | −8.9e−16 | relative ± 1e−12 | PASS |
| Chi-square: p | 0.1667994104129independent software output | 0.1667994104129 | 0 | relative ± 1e−10 | PASS |
| Chi-square: df | 4independent software output | 4 | none | exact match | PASS |
| 2 proportions, 45/300 vs 30/280 (pooled): z | 1.537094376756independent software output | 1.537094376756 | 0 | relative ± 1e−12 | PASS |
| 2 proportions: p | 0.1242701998111independent software output | 0.1242701998111 | −2.5e−16 | relative ± 1e−10 | PASS |
Multiple Regression: 3 Predictors, 30 Observations
30/30 passData: refs-regression.csv
Show All 30 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Coefficient, intercept | 1.874689665505independent software output | 1.874689665505 | −1.8e−15 | relative ± 1e−11 | PASS |
| Coefficient, x1 | 1.361563143009independent software output | 1.361563143009 | −2.2e−16 | relative ± 1e−11 | PASS |
| Coefficient, x2 | −0.8739299498415independent software output | −0.8739299498415 | −4.4e−16 | relative ± 1e−11 | PASS |
| Coefficient, x3 | 0.1496412458402independent software output | 0.1496412458402 | 2.5e−16 | relative ± 1e−11 | PASS |
| Standard error, intercept | 0.1689927843108independent software output | 0.1689927843108 | −5.6e−17 | relative ± 1e−10 | PASS |
| Standard error, x1 | 0.1769743853737independent software output | 0.1769743853737 | −2.8e−17 | relative ± 1e−10 | PASS |
| Standard error, x2 | 0.2010811903905independent software output | 0.2010811903905 | 2.8e−17 | relative ± 1e−10 | PASS |
| Standard error, x3 | 0.2230514708441independent software output | 0.2230514708441 | −5.6e−17 | relative ± 1e−10 | PASS |
| p-value, intercept | 2.3419510988e−11independent software output | 2.3419510988e−11 | 2.1e−25 | ± 1e−8 | PASS |
| p-value, x1 | 3.6501351758e−8independent software output | 3.6501351758e−8 | 6.6e−24 | ± 1e−8 | PASS |
| p-value, x2 | 0.0001886853370603independent software output | 0.0001886853370603 | −1.2e−18 | ± 1e−8 | PASS |
| p-value, x3 | 0.5082121352493independent software output | 0.5082121352493 | 0 | ± 1e−8 | PASS |
| Adjusted R² | 0.8107432932159independent software output | 0.8107432932159 | 0 | relative ± 1e−11 | PASS |
| Model p (F test) | 3.7114932556e−10independent software output | 3.7114932556e−10 | 4.7e−24 | ± 1e−8 | PASS |
| Leverage, row 1 | 0.1062954486162independent software output | 0.1062954486162 | −6.9e−17 | relative ± 1e−10 | PASS |
| Leverage, row 2 | 0.1553432805676independent software output | 0.1553432805676 | 2.8e−17 | relative ± 1e−10 | PASS |
| Leverage, row 3 | 0.06425586121756independent software output | 0.06425586121756 | −2.8e−17 | relative ± 1e−10 | PASS |
| Leverage, row 4 | 0.1440919363799independent software output | 0.1440919363799 | −5.6e−17 | relative ± 1e−10 | PASS |
| Leverage, row 5 | 0.2929633545587independent software output | 0.2929633545587 | 1.1e−16 | relative ± 1e−10 | PASS |
| Standardized residual, row 1 | −0.1175203952685independent software output | −0.1175203952685 | 2.1e−15 | relative ± 1e−9 | PASS |
| Standardized residual, row 2 | −0.2228715870875independent software output | −0.2228715870875 | 3.1e−15 | relative ± 1e−9 | PASS |
| Standardized residual, row 3 | −0.6462074454988independent software output | −0.6462074454988 | 2.0e−15 | relative ± 1e−9 | PASS |
| Standardized residual, row 4 | −0.9651196491279independent software output | −0.9651196491279 | 2.6e−15 | relative ± 1e−9 | PASS |
| Standardized residual, row 5 | 0.297067676452independent software output | 0.297067676452 | 2.3e−15 | relative ± 1e−9 | PASS |
| VIF, x1 | 1.572894585264independent software output | 1.572894585264 | −2.2e−16 | relative ± 1e−9 | PASS |
| VIF, x2 | 1.026032907857independent software output | 1.026032907857 | 4.4e−16 | relative ± 1e−9 | PASS |
| VIF, x3 | 1.604908810459independent software output | 1.604908810459 | 0 | relative ± 1e−9 | PASS |
| Prediction at (0.5, −0.2, 0.1) | 2.745221351562independent software output | 2.745221351562 | −1.8e−15 | relative ± 1e−11 | PASS |
| 95% confidence interval, low | 2.323258410863independent software output | 2.323258410863 | −1.1e−14 | relative ± 1e−10 | PASS |
| 95% prediction interval, high | 4.651613048762independent software output | 4.651613048762 | 4.3e−14 | relative ± 1e−10 | PASS |
Non-Normal Data: Box-Cox, Lognormal and Weibull Fits
7/7 passMaximum-likelihood fits with the location fixed at 0, as with SciPy’s floc = 0. These drive LimitLine’s non-normal capability.
Data: refs-lognormal-60.csv · refs-weibull-80.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Box-Cox λ (maximum likelihood) | −0.07761589539388independent software output | −0.07761586643176 | 2.9e−8 | ± 1e−6 | PASS |
| Lognormal σ | 0.4395476313864independent software output | 0.4395476313864 | 5.6e−17 | relative ± 1e−7 | PASS |
| Lognormal μ | 0.972522082197independent software output | 0.972522082197 | −1.1e−16 | relative ± 1e−7 | PASS |
| Lognormal 99.865th percentile | 9.88634847448independent software output | 9.88634847448 | 7.1e−15 | relative ± 1e−6 | PASS |
| Weibull shape | 2.000148110365independent software output | 2.000150528074 | 2.4e−6 | relative ± 1e−5 | PASS |
| Weibull scale | 11.79568421492independent software output | 11.79571501406 | 3.1e−5 | relative ± 1e−5 | PASS |
| Weibull 0.135th percentile | 0.433653560681independent software output | 0.4336564244897 | 2.9e−6 | relative ± 1e−4 | PASS |
Gauge R&R Two-Way ANOVA (10 Parts × 3 Operators × 2 Trials)
4/4 passData: refs-gage-rr-synthetic.csv
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| SS part | 63.83469679353independent software output | 63.83469679353 | 7.1e−15 | relative ± 1e−10 | PASS |
| SS operator | 0.3980597608446independent software output | 0.3980597608447 | 9.5e−15 | relative ± 1e−10 | PASS |
| Interaction p-value (vs error) | 0.8499763059015independent software output | 0.8499763059015 | −2.7e−15 | ± 1e−9 | PASS |
| Pooled repeatability SS = interaction SS + error SS | 0.8816023027272independent software output | 0.8816023027272 | 1.8e−15 | relative ± 1e−10 | PASS |
Designed Experiment: 2³ Full Factorial, 2 Replicates, Full Model
14/14 passData: refs-doe-2x3-replicated.csv
Show All 14 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Coefficient, A | 4.139202170081independent software output | 4.139202170081 | 1.8e−15 | relative ± 1e−11 | PASS |
| p-value, A | 7.2262702212e−7independent software output | 7.2262702212e−7 | −3.0e−21 | ± 1e−8 | PASS |
| Coefficient, B | −2.607738056833independent software output | −2.607738056833 | 1.2e−14 | relative ± 1e−11 | PASS |
| p-value, B | 2.3519608957e−5independent software output | 2.3519608957e−5 | 5.4e−19 | ± 1e−8 | PASS |
| Coefficient, C | 0.4531543108019independent software output | 0.4531543108019 | 6.1e−15 | relative ± 1e−11 | PASS |
| p-value, C | 0.1685042157022independent software output | 0.1685042157022 | −5.1e−15 | ± 1e−8 | PASS |
| Coefficient, A×B | 1.597856365582independent software output | 1.597856365582 | −4.9e−15 | relative ± 1e−11 | PASS |
| p-value, A×B | 0.0006956174417653independent software output | 0.0006956174417653 | 1.4e−17 | ± 1e−8 | PASS |
| Coefficient, A×C | −0.04963003276434independent software output | −0.04963003276434 | 2.0e−15 | relative ± 1e−11 | PASS |
| p-value, A×C | 0.8724222505913independent software output | 0.8724222505913 | 9.9e−15 | ± 1e−8 | PASS |
| Coefficient, B×C | 0.1982409079729independent software output | 0.1982409079729 | −1.5e−15 | relative ± 1e−11 | PASS |
| p-value, B×C | 0.5263965727899independent software output | 0.5263965727899 | −1.9e−15 | ± 1e−8 | PASS |
| Coefficient, A×B×C | 0.1543435210803independent software output | 0.1543435210803 | 3.7e−15 | relative ± 1e−11 | PASS |
| p-value, A×B×C | 0.6200488147326independent software output | 0.6200488147326 | −7.1e−15 | ± 1e−8 | PASS |
Power and Sample Size (297 Values)
297/297 passNoncentral t and F distributions, power for t tests, two proportions and one-way ANOVA, and the sample size each needs (statsmodels’ fractional answer rounded up).
Show All 297 Checks
| Statistic | Expected | LimitLine | Difference | Tolerance | Result |
|---|---|---|---|---|---|
| Noncentral t CDF (t -2.5, df 3, δ -1) | 0.1758162407995independent software output | 0.1758162407995 | 3.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 3, δ 0) | 0.04385332350403independent software output | 0.04385332350403 | 3.5e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 3, δ 0.7) | 0.01155634567409independent software output | 0.01155634567409 | −3.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 3, δ 2.4) | 0.000105879242372independent software output | 0.0001058792423716 | −4.0e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 3, δ 5) | 8.3459984415e−10independent software output | 8.3459983369e−10 | −1.0e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 9, δ -1) | 0.1073024951327independent software output | 0.1073024951327 | 5.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 9, δ 0) | 0.01693091384149independent software output | 0.01693091384149 | −5.6e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 9, δ 0.7) | 0.003042125037528independent software output | 0.003042125037528 | −2.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 9, δ 2.4) | 9.8036877333e−6independent software output | 9.803687733e−6 | −3.6e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 9, δ 5) | 1.5059179745e−11independent software output | 1.5059176128e−11 | −3.6e−18 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 29, δ -1) | 0.079855116843independent software output | 0.079855116843 | 4.0e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 29, δ 0) | 0.009162672169213independent software output | 0.009162672169213 | −8.3e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 29, δ 0.7) | 0.001235966284947independent software output | 0.001235966284947 | −3.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 29, δ 2.4) | 1.6517464693e−6independent software output | 1.6517464692e−6 | −1.0e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 29, δ 5) | 4.6043098998e−13independent software output | 4.6040948831e−13 | −2.2e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 118, δ -1) | 0.07005319082556independent software output | 0.07005319082556 | 8.0e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 118, δ 0) | 0.006896433721524independent software output | 0.006896433721524 | −4.6e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 118, δ 0.7) | 0.0008060238929645independent software output | 0.0008060238929641 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 118, δ 2.4) | 6.7484084093e−7independent software output | 6.7484084054e−7 | −3.9e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -2.5, df 118, δ 5) | 6.8084030097e−14independent software output | 6.805667141e−14 | −2.7e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 3, δ -1) | 0.7638660923232independent software output | 0.7638660923232 | 1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 3, δ 0) | 0.39188164602independent software output | 0.39188164602 | −5.6e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 3, δ 0.7) | 0.166069540438independent software output | 0.166069540438 | −1.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 3, δ 2.4) | 0.003917166098301independent software output | 0.003917166098301 | −1.0e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 3, δ 5) | 7.8532316493e−8independent software output | 7.8532316428e−8 | −6.6e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 9, δ -1) | 0.7600603480398independent software output | 0.7600603480398 | 3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 9, δ 0) | 0.3854953518708independent software output | 0.3854953518708 | 1.7e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 9, δ 0.7) | 0.1612339539197independent software output | 0.1612339539197 | 1.7e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 9, δ 2.4) | 0.003622414571044independent software output | 0.003622414571044 | −6.0e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 9, δ 5) | 6.4811557174e−8independent software output | 6.4811557121e−8 | −5.3e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 29, δ -1) | 0.7586715083063independent software output | 0.7586715083063 | 3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 29, δ 0) | 0.3831585466645independent software output | 0.3831585466645 | 1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 29, δ 0.7) | 0.1594649776043independent software output | 0.1594649776043 | 2.5e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 29, δ 2.4) | 0.003515612456944independent software output | 0.003515612456944 | −9.5e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 29, δ 5) | 6.003394828e−8independent software output | 6.0033948257e−8 | −2.3e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 118, δ -1) | 0.7581929720233independent software output | 0.7581929720233 | 5.0e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 118, δ 0) | 0.3823524992731independent software output | 0.3823524992731 | 2.6e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 118, δ 0.7) | 0.1588549645863independent software output | 0.1588549645863 | 5.8e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 118, δ 2.4) | 0.003478954256963independent software output | 0.003478954256963 | 3.6e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t -0.3, df 118, δ 5) | 5.8423950463e−8independent software output | 5.8423950344e−8 | −1.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 3, δ -1) | 0.9517887357974independent software output | 0.9517887357974 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 3, δ 0) | 0.7589005241245independent software output | 0.7589005241245 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 3, δ 0.7) | 0.5132635944712independent software output | 0.5132635944712 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 3, δ 2.4) | 0.05651745358777independent software output | 0.05651745358777 | 2.1e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 3, δ 5) | 2.7001347972e−5independent software output | 2.7001347972e−5 | −1.6e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 9, δ -1) | 0.9598318663862independent software output | 0.9598318663862 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 9, δ 0) | 0.7778435007761independent software output | 0.7778435007761 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 9, δ 0.7) | 0.5305126123877independent software output | 0.5305126123877 | −2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 9, δ 2.4) | 0.05544663997066independent software output | 0.05544663997066 | 4.2e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 9, δ 5) | 1.686566023e−5independent software output | 1.686566023e−5 | −3.0e−18 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 29, δ -1) | 0.9627444843152independent software output | 0.9627444843152 | −6.7e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 29, δ 0) | 0.7848921657248independent software output | 0.7848921657248 | −6.7e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 29, δ 0.7) | 0.5368910355718independent software output | 0.5368910355718 | −8.9e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 29, δ 2.4) | 0.05500923283976independent software output | 0.05500923283976 | −1.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 29, δ 5) | 1.4344512887e−5independent software output | 1.4344512887e−5 | −1.3e−18 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 118, δ -1) | 0.9637433598813independent software output | 0.9637433598813 | −1.1e−14 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 118, δ 0) | 0.7873407973806independent software output | 0.7873407973806 | −1.8e−14 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 118, δ 0.7) | 0.5391023962979independent software output | 0.5391023962979 | −1.7e−14 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 118, δ 2.4) | 0.05485167280971independent software output | 0.05485167280971 | −2.2e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 0.8, df 118, δ 5) | 1.3583864124e−5independent software output | 1.3583864124e−5 | −6.3e−19 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 3, δ -1) | 0.9910765400541independent software output | 0.9910765400541 | 3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 3, δ 0) | 0.9367173987252independent software output | 0.9367173987252 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 3, δ 0.7) | 0.8302897193734independent software output | 0.8302897193734 | −4.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 3, δ 2.4) | 0.3525372488881independent software output | 0.3525372488881 | −2.8e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 3, δ 5) | 0.01104486516829independent software output | 0.01104486516829 | 1.7e−18 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 9, δ -1) | 0.9970931228641independent software output | 0.9970931228641 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 9, δ 0) | 0.9674408587939independent software output | 0.9674408587939 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 9, δ 0.7) | 0.8865478853818independent software output | 0.8865478853818 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 9, δ 2.4) | 0.3728095957465independent software output | 0.3728095957465 | 1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 9, δ 5) | 0.004178827465566independent software output | 0.004178827465566 | −6.9e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 29, δ -1) | 0.9985363596415independent software output | 0.9985363596415 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 29, δ 0) | 0.9777290209819independent software output | 0.9777290209819 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 29, δ 0.7) | 0.908732843167independent software output | 0.908732843167 | −5.6e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 29, δ 2.4) | 0.3794022612007independent software output | 0.3794022612007 | −9.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 29, δ 5) | 0.002469005914685independent software output | 0.002469005914684 | −4.1e−17 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 118, δ -1) | 0.9989200598583independent software output | 0.9989200598583 | −5.6e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 118, δ 0) | 0.9810704301108independent software output | 0.9810704301108 | 1.2e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 118, δ 0.7) | 0.9166289541005independent software output | 0.9166289541005 | −3.6e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 118, δ 2.4) | 0.3814519750588independent software output | 0.3814519750588 | −1.5e−14 | ± 1e−10 | PASS |
| Noncentral t CDF (t 2.1, df 118, δ 5) | 0.002003935692024independent software output | 0.002003935692024 | −1.4e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 3, δ -1) | 0.9982750423472independent software output | 0.9982750423472 | 2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 3, δ 0) | 0.9859957719949independent software output | 0.9859957719949 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 3, δ 0.7) | 0.9572378138883independent software output | 0.9572378138883 | −2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 3, δ 2.4) | 0.7468669436674independent software output | 0.7468669436674 | −2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 3, δ 5) | 0.2324260189244independent software output | 0.2324260189244 | 1.3e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 9, δ -1) | 0.999915978907independent software output | 0.999915978907 | 1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 9, δ 0) | 0.9984447858448independent software output | 0.9984447858448 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 9, δ 0.7) | 0.9918691016604independent software output | 0.9918691016604 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 9, δ 2.4) | 0.8631451431995independent software output | 0.8631451431995 | −2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 9, δ 5) | 0.2070182053794independent software output | 0.2070182053794 | 1.7e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 29, δ -1) | 0.999995550191independent software output | 0.999995550191 | 2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 29, δ 0) | 0.9997999680272independent software output | 0.9997999680272 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 29, δ 0.7) | 0.9981964917791independent software output | 0.9981964917791 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 29, δ 2.4) | 0.9176597240056independent software output | 0.9176597240056 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 29, δ 5) | 0.1795931465461independent software output | 0.1795931465461 | −1.3e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 118, δ -1) | 0.9999993332526independent software output | 0.9999993332526 | 2.2e−16 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 118, δ 0) | 0.9999445529635independent software output | 0.9999445529635 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 118, δ 0.7) | 0.9992815568917independent software output | 0.9992815568917 | 0 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 118, δ 2.4) | 0.9382836685405independent software output | 0.9382836685406 | 1.9e−15 | ± 1e−10 | PASS |
| Noncentral t CDF (t 4, df 118, δ 5) | 0.1645322718996independent software output | 0.1645322718995 | −1.1e−14 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 1, 10, λ 0.5) | 0.5557156520335independent software output | 0.5557156520335 | −1.2e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 1, 10, λ 4) | 0.8933063566298independent software output | 0.8933063566298 | −4.4e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 1, 10, λ 12) | 0.9962086857743independent software output | 0.9962086857743 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 2, 27, λ 0.5) | 0.6292714809071independent software output | 0.6292714809071 | −2.4e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 2, 27, λ 4) | 0.8982329788686independent software output | 0.8982329788686 | −7.8e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 2, 27, λ 12) | 0.9953805132912independent software output | 0.9953805132912 | −2.2e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 4, 45, λ 0.5) | 0.713493575503independent software output | 0.713493575503 | −1.3e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 4, 45, λ 4) | 0.9085833090931independent software output | 0.9085833090931 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 0.6, df 4, 45, λ 12) | 0.9943746929302independent software output | 0.9943746929302 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 1, 10, λ 0.5) | 0.261397607931independent software output | 0.261397607931 | −3.4e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 1, 10, λ 4) | 0.7013170488309independent software output | 0.7013170488309 | −1.4e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 1, 10, λ 12) | 0.9722017778063independent software output | 0.9722017778063 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 2, 27, λ 0.5) | 0.1918285764068independent software output | 0.1918285764068 | −3.3e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 2, 27, λ 4) | 0.5687761000603independent software output | 0.5687761000603 | −2.1e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 2, 27, λ 12) | 0.9345411935053independent software output | 0.9345411935053 | 0 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 4, 45, λ 0.5) | 0.117159817498independent software output | 0.117159817498 | 1.4e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 4, 45, λ 4) | 0.3817239425998independent software output | 0.3817239425998 | 7.2e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 2.2, df 4, 45, λ 12) | 0.8265584286474independent software output | 0.8265584286474 | 1.1e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 1, 10, λ 0.5) | 0.1347068491045independent software output | 0.1347068491045 | −4.7e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 1, 10, λ 4) | 0.5191599032642independent software output | 0.5191599032642 | −1.2e−15 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 1, 10, λ 12) | 0.9165416017982independent software output | 0.9165416017982 | −1.1e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 2, 27, λ 0.5) | 0.05513084031793independent software output | 0.05513084031793 | 1.4e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 2, 27, λ 4) | 0.2941833588881independent software output | 0.2941833588881 | −7.8e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 2, 27, λ 12) | 0.778985645366independent software output | 0.778985645366 | 3.3e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 4, 45, λ 0.5) | 0.01289858350134independent software output | 0.01289858350134 | 9.0e−17 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 4, 45, λ 4) | 0.09380426953057independent software output | 0.09380426953057 | −7.5e−16 | ± 1e−10 | PASS |
| Noncentral F upper tail (F 4, df 4, 45, λ 12) | 0.4626244427256independent software output | 0.4626244427256 | 1.1e−15 | ± 1e−10 | PASS |
| Power, 1-sample t (d 0.2, n 5, two-sided) | 0.06426957404801independent software output | 0.06426957404801 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 5, larger) | 0.1024442765543independent software output | 0.1024442765543 | 6.7e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 10, two-sided) | 0.08765715402847independent software output | 0.08765715402846 | −6.7e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 10, larger) | 0.1448180678455independent software output | 0.1448180678455 | −9.8e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 20, two-sided) | 0.1359562887674independent software output | 0.1359562887674 | 1.3e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 20, larger) | 0.2170751816719independent software output | 0.2170751816719 | −6.9e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 50, two-sided) | 0.2835924146675independent software output | 0.2835924146675 | 2.6e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 50, larger) | 0.4012215049932independent software output | 0.4012215049932 | −2.3e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 120, two-sided) | 0.5844409144364independent software output | 0.5844409144364 | −2.0e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.2, n 120, larger) | 0.7031750281047independent software output | 0.7031750281048 | 6.8e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 5, two-sided) | 0.1405166899903independent software output | 0.1405166899903 | −2.8e−17 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 5, larger) | 0.2389951709035independent software output | 0.2389951709035 | 1.3e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 10, two-sided) | 0.2931756065141independent software output | 0.2931756065141 | −1.5e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 10, larger) | 0.4272898267712independent software output | 0.4272898267712 | −1.8e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 20, two-sided) | 0.564504418439independent software output | 0.564504418439 | 2.5e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 20, larger) | 0.6951493382443independent software output | 0.6951493382443 | −8.8e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 50, two-sided) | 0.9338975812614independent software output | 0.9338975812614 | 9.8e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 50, larger) | 0.9672067458262independent software output | 0.9672067458262 | −5.3e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 120, two-sided) | 0.9997422834288independent software output | 0.9997422834288 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.5, n 120, larger) | 0.9999279233437independent software output | 0.9999279233437 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 5, two-sided) | 0.2809920635174independent software output | 0.2809920635174 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 5, larger) | 0.436359186401independent software output | 0.436359186401 | 1.7e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 10, two-sided) | 0.6162327634744independent software output | 0.6162327634744 | −1.9e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 10, larger) | 0.7544247592493independent software output | 0.7544247592493 | −1.6e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 20, two-sided) | 0.9238987553886independent software output | 0.9238987553886 | 9.4e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 20, larger) | 0.964172800606independent software output | 0.964172800606 | −2.2e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 50, two-sided) | 0.9998302862283independent software output | 0.9998302862283 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 50, larger) | 0.9999578749289independent software output | 0.9999578749289 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 120, two-sided) | 0.9999999999916independent software output | 0.9999999999916 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 0.8, n 120, larger) | 0.9999999999992independent software output | 0.9999999999992 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 5, two-sided) | 0.5302212511403independent software output | 0.5302212511403 | 3.3e−16 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 5, larger) | 0.7119106100632independent software output | 0.7119106100632 | 1.7e−14 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 10, two-sided) | 0.9203333855898independent software output | 0.9203333855898 | −8.1e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 10, larger) | 0.9674752485648independent software output | 0.9674752485648 | −4.1e−15 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 20, two-sided) | 0.9990954035492independent software output | 0.9990954035492 | 2.2e−16 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 20, larger) | 0.9997839137198independent software output | 0.9997839137198 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 50, two-sided) | 0.9999999998935independent software output | 0.9999999998935 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 50, larger) | 0.9999999999908independent software output | 0.9999999999908 | 0 | ± 1e−9 | PASS |
| Power, 1-sample t (d 1.2, n 120, larger) | 1independent software output | 1 | 0 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 5 per group, two-sided) | 0.05904263425365independent software output | 0.05904263425365 | −4.6e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 5 per group, larger) | 0.08779955011805independent software output | 0.08779955011805 | −8.7e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 10 per group, two-sided) | 0.07082134518931independent software output | 0.0708213451893 | −6.1e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 10 per group, larger) | 0.1123273468601independent software output | 0.1123273468601 | 2.1e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 20 per group, two-sided) | 0.0945673276173independent software output | 0.09456732761729 | −8.1e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 20 per group, larger) | 0.1530096672282independent software output | 0.1530096672282 | 3.3e−16 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 50 per group, two-sided) | 0.1676754864961independent software output | 0.1676754864961 | −2.6e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 50 per group, larger) | 0.2572804628117independent software output | 0.2572804628118 | 4.4e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 120 per group, two-sided) | 0.3385617478128independent software output | 0.3385617478128 | 5.4e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.2, 120 per group, larger) | 0.4601463445893independent software output | 0.4601463445893 | −2.0e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 5 per group, two-sided) | 0.1076859897976independent software output | 0.1076859897975 | −7.3e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 5 per group, larger) | 0.1788338948398independent software output | 0.1788338948398 | −1.5e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 10 per group, two-sided) | 0.1850956562912independent software output | 0.1850956562912 | −1.2e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 10 per group, larger) | 0.2847634913513independent software output | 0.2847634913513 | 3.8e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 20 per group, two-sided) | 0.337939028925independent software output | 0.337939028925 | −1.7e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 20 per group, larger) | 0.4633743492964independent software output | 0.4633743492964 | −2.2e−16 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 50 per group, two-sided) | 0.6968934055334independent software output | 0.6968934055333 | −3.7e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 50 per group, larger) | 0.7989361641502independent software output | 0.7989361641502 | 4.5e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 120 per group, two-sided) | 0.9711087800141independent software output | 0.9711087800141 | 8.1e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.5, 120 per group, larger) | 0.9866914750475independent software output | 0.9866914750475 | −2.3e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 5 per group, two-sided) | 0.200739452652independent software output | 0.200739452652 | −1.1e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 5 per group, larger) | 0.3133300246889independent software output | 0.3133300246889 | −2.1e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 10 per group, two-sided) | 0.3950692121136independent software output | 0.3950692121136 | −1.7e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 10 per group, larger) | 0.5303874685238independent software output | 0.5303874685239 | 4.9e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 20 per group, two-sided) | 0.6934041966154independent software output | 0.6934041966154 | −1.6e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 20 per group, larger) | 0.7994081780872independent software output | 0.7994081780872 | −3.3e−16 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 50 per group, two-sided) | 0.977279090654independent software output | 0.977279090654 | −5.7e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 50 per group, larger) | 0.9900258947477independent software output | 0.9900258947477 | 4.4e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 120 per group, two-sided) | 0.9999873244295independent software output | 0.9999873244295 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 2-sample t (d 0.8, 120 per group, larger) | 0.9999971089792independent software output | 0.9999971089792 | 0 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 5 per group, two-sided) | 0.3864372610122independent software output | 0.3864372610122 | −1.7e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 5 per group, larger) | 0.535662602192independent software output | 0.535662602192 | −2.6e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 10 per group, two-sided) | 0.7184053765417independent software output | 0.7184053765417 | −1.6e−14 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 10 per group, larger) | 0.8252224525487independent software output | 0.8252224525487 | 3.6e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 20 per group, two-sided) | 0.9588268143625independent software output | 0.9588268143625 | −4.4e−15 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 20 per group, larger) | 0.9812867561418independent software output | 0.9812867561418 | 0 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 50 per group, two-sided) | 0.999965601217independent software output | 0.999965601217 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 50 per group, larger) | 0.9999919537392independent software output | 0.9999919537392 | 0 | ± 1e−9 | PASS |
| Power, 2-sample t (d 1.2, 120 per group, larger) | 1independent software output | 1 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 50 per group, two-sided) | 0.1566971371917independent software output | 0.1566971371917 | −2.2e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 50 per group, larger) | 0.2423226607327independent software output | 0.2423226607327 | 8.9e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 200 per group, two-sided) | 0.4753633953952independent software output | 0.4753633953952 | −1.7e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 200 per group, larger) | 0.6004885508234independent software output | 0.6004885508234 | 1.3e−15 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 800 per group, two-sided) | 0.9674810599019independent software output | 0.9674810599019 | −1.1e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 800 per group, larger) | 0.9846730604555independent software output | 0.9846730604555 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 3000 per group, two-sided) | 0.9999999696347independent software output | 0.9999999696347 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.1 vs 0.05, 3000 per group, larger) | 0.9999999950716independent software output | 0.9999999950716 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 50 per group, two-sided) | 0.2096303634395independent software output | 0.2096303634395 | −2.2e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 50 per group, larger) | 0.3108459810388independent software output | 0.3108459810388 | 8.9e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 200 per group, two-sided) | 0.6375108199581independent software output | 0.6375108199581 | −2.2e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 200 per group, larger) | 0.7482594588413independent software output | 0.7482594588413 | 1.0e−15 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 800 per group, two-sided) | 0.9962829183347independent software output | 0.9962829183347 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 800 per group, larger) | 0.998623159011independent software output | 0.998623159011 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 3000 per group, two-sided) | 0.999999999999independent software output | 0.999999999999 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.3 vs 0.2, 3000 per group, larger) | 0.9999999999999independent software output | 0.9999999999999 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 50 per group, two-sided) | 0.1700223101303independent software output | 0.1700223101303 | −2.2e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 50 per group, larger) | 0.260088725766independent software output | 0.260088725766 | 8.9e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 200 per group, two-sided) | 0.5201179041645independent software output | 0.5201179041645 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 200 per group, larger) | 0.6432205704507independent software output | 0.6432205704507 | 1.2e−15 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 800 per group, two-sided) | 0.9808052603079independent software output | 0.9808052603079 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 800 per group, larger) | 0.9915156617608independent software output | 0.9915156617608 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 3000 per group, two-sided) | 0.9999999976102independent software output | 0.9999999976102 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.5 vs 0.4, 3000 per group, larger) | 0.9999999996615independent software output | 0.9999999996615 | 0 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 50 per group, two-sided) | 0.06939361040559independent software output | 0.06939361040559 | −3.2e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 50 per group, larger) | 0.1084983627799independent software output | 0.1084983627799 | 4.4e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 200 per group, two-sided) | 0.1301869974698independent software output | 0.1301869974698 | −1.7e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 200 per group, larger) | 0.2052938027556independent software output | 0.2052938027556 | 6.7e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 800 per group, two-sided) | 0.3765916951194independent software output | 0.3765916951194 | −1.1e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 800 per group, larger) | 0.5002102260703independent software output | 0.5002102260703 | 1.2e−15 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 3000 per group, two-sided) | 0.8901526697371independent software output | 0.8901526697371 | 1.1e−16 | ± 1e−9 | PASS |
| Power, 2 proportions (0.02 vs 0.01, 3000 per group, larger) | 0.9385502772131independent software output | 0.9385502772131 | 4.4e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 3 groups of 5) | 0.05896536542259independent software output | 0.05896536542261 | 2.2e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 3 groups of 15) | 0.0829289166563independent software output | 0.0829289166563 | −7.9e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 3 groups of 40) | 0.1479676280058independent software output | 0.1479676280058 | −2.2e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 3 groups of 100) | 0.3186429507235independent software output | 0.3186429507235 | 1.2e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 4 groups of 5) | 0.05948277737515independent software output | 0.05948277737514 | −1.3e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 4 groups of 15) | 0.08511539469272independent software output | 0.0851153946928 | 7.4e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 4 groups of 40) | 0.1575861937825independent software output | 0.1575861937826 | 9.5e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 4 groups of 100) | 0.3552591913706independent software output | 0.3552591913707 | 1.2e−13 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 6 groups of 5) | 0.06073809934242independent software output | 0.0607380993424 | −2.4e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 6 groups of 15) | 0.09028912877278independent software output | 0.09028912877279 | 5.2e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 6 groups of 40) | 0.1788523428012independent software output | 0.1788523428011 | −7.2e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.1, 6 groups of 100) | 0.4290285136216independent software output | 0.4290285136215 | −4.9e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 3 groups of 5) | 0.1095296872039independent software output | 0.109529687204 | 3.5e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 3 groups of 15) | 0.2849724443179independent software output | 0.2849724443179 | −1.5e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 3 groups of 40) | 0.6756744946708independent software output | 0.6756744946708 | 7.8e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 3 groups of 100) | 0.9780555505591independent software output | 0.9780555505591 | 5.6e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 4 groups of 5) | 0.1150101405246independent software output | 0.1150101405246 | −2.1e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 4 groups of 15) | 0.3165802409043independent software output | 0.3165802409044 | 1.7e−13 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 4 groups of 40) | 0.749404537701independent software output | 0.7494045377011 | 1.3e−13 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 4 groups of 100) | 0.9928175163418independent software output | 0.9928175163418 | 4.3e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 6 groups of 5) | 0.127476854629independent software output | 0.127476854629 | −4.2e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 6 groups of 15) | 0.381895625593independent software output | 0.381895625593 | 1.1e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 6 groups of 40) | 0.8572074314554independent software output | 0.8572074314553 | −6.3e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.25, 6 groups of 100) | 0.999336261231independent software output | 0.999336261231 | 1.1e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 3 groups of 5) | 0.2137435107015independent software output | 0.2137435107015 | 5.6e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 3 groups of 15) | 0.6350412095709independent software output | 0.6350412095709 | −1.7e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 3 groups of 40) | 0.9789277028288independent software output | 0.9789277028288 | 1.1e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 3 groups of 100) | 0.9999974799348independent software output | 0.9999974799348 | 0 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 4 groups of 5) | 0.2353286362836independent software output | 0.2353286362835 | −3.4e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 4 groups of 15) | 0.7087216415816independent software output | 0.7087216415817 | 1.6e−13 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 4 groups of 40) | 0.9932129067282independent software output | 0.9932129067282 | 7.9e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 4 groups of 100) | 0.9999999605018independent software output | 0.9999999605018 | 0 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 6 groups of 5) | 0.2819146866192independent software output | 0.2819146866192 | −6.7e−14 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 6 groups of 15) | 0.8225457871305independent software output | 0.8225457871305 | 8.4e−15 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 6 groups of 40) | 0.9993966418701independent software output | 0.9993966418701 | −5.6e−16 | ± 1e−9 | PASS |
| Power, one-way ANOVA (f 0.4, 6 groups of 100) | 0.9999999999932independent software output | 0.9999999999932 | 0 | ± 1e−9 | PASS |
| Sample size, 1-sample t (d 0.3, power 0.8, two-sided) | 90independent software output | 90 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.3, power 0.8, larger) | 71independent software output | 71 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.3, power 0.9, two-sided) | 119independent software output | 119 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.3, power 0.9, larger) | 97independent software output | 97 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.5, power 0.8, two-sided) | 34independent software output | 34 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.5, power 0.8, larger) | 27independent software output | 27 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.5, power 0.9, two-sided) | 44independent software output | 44 | none | exact match | PASS |
| Sample size, 1-sample t (d 0.5, power 0.9, larger) | 36independent software output | 36 | none | exact match | PASS |
| Sample size, 1-sample t (d 1, power 0.8, two-sided) | 10independent software output | 10 | none | exact match | PASS |
| Sample size, 1-sample t (d 1, power 0.8, larger) | 8independent software output | 8 | none | exact match | PASS |
| Sample size, 1-sample t (d 1, power 0.9, two-sided) | 13independent software output | 13 | none | exact match | PASS |
| Sample size, 1-sample t (d 1, power 0.9, larger) | 11independent software output | 11 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.3, power 0.8, two-sided) | 176independent software output | 176 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.3, power 0.8, larger) | 139independent software output | 139 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.3, power 0.9, two-sided) | 235independent software output | 235 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.3, power 0.9, larger) | 191independent software output | 191 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.5, power 0.8, two-sided) | 64independent software output | 64 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.5, power 0.8, larger) | 51independent software output | 51 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.5, power 0.9, two-sided) | 86independent software output | 86 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 0.5, power 0.9, larger) | 70independent software output | 70 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 1, power 0.8, two-sided) | 17independent software output | 17 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 1, power 0.8, larger) | 14independent software output | 14 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 1, power 0.9, two-sided) | 23independent software output | 23 | none | exact match | PASS |
| Sample size per group, 2-sample t (d 1, power 0.9, larger) | 18independent software output | 18 | none | exact match | PASS |
How to Rerun These Yourself
- Download a dataset below. Each is a plain CSV with a header row, in the order the source prints it.
- The Individuals (I-MR) tab with capability, the Pareto chart and the Run chart are on the Free plan. Subgroup and attribute charts and the other analysis tools need Pro.
- Open LimitLine and paste the file’s contents into the tool listed. LimitLine lists the header word under “Ignored non-numeric entries”, which is expected.
- Compare the result with the case’s expected values above. Or open the same file in Minitab, JMP, R or Excel and compare all three.
Settings to match when you compare with another package: I-MR sigma is MR̄/1.128 (average moving range). Gauge R&R uses the ANOVA method and removes the interaction when p > 0.05. Quartiles use the (n + 1)p method. The 2-sample t test is Welch’s (no equal-variance assumption) unless “Assume equal variances” is ticked. All results are also in machine-readable form in results.json.
| File | What it is | In LimitLine | In Minitab or elsewhere |
|---|---|---|---|
| nist-6322-flow-rate.csv10 rows | NIST §6.3.2.2 flow rate, 10 batches | Individuals (I-MR) tab: paste the column (rows are batches 1 to 10). | Minitab: Stat › Control Charts › Variables Charts for Individuals › I-MR, column flow_rate. |
| xbar-r-test-5x5.csv5 rows | 5 subgroups of 5 (LimitLine test set) | Subgroups (X̄-R / X̄-S) tab: paste the five data lines (one subgroup per line). | Minitab: Stat › Control Charts › Variables Charts for Subgroups › Xbar-R, “Observations for a subgroup are in one row of columns”. |
| nist-6331-wafer-defects.csv25 rows | NIST §6.3.3.1 defects per wafer, 25 wafers | Attributes tab, c chart: paste the 25 counts (one per line). | Minitab: Stat › Control Charts › Attributes Charts › C. |
| nist-6332-wafer-proportions.csv30 rows | NIST §6.3.3.2 defective chips, 30 wafers of 50 | Attributes tab, p chart: paste the lines “defectives, sample size” (for the np chart, also enter sample size 50). | Minitab: Stat › Control Charts › Attributes Charts › P, subgroup sizes column sample_size. |
| montgomery-u-computers.csv20 rows | Montgomery p. 318 (via IMSL): defects in 20 samples of 5 computers | Attributes tab, u chart: paste the lines “defects, units”. | Minitab: Stat › Control Charts › Attributes Charts › U, subgroup sizes column units. |
| montgomery-u-cloth-rolls.csv10 rows | Montgomery p. 321 (via IMSL): defects in 10 rolls of cloth, units of 50 m² | Attributes tab, u chart: paste the lines “defects, units”. | Minitab: Stat › Control Charts › Attributes Charts › U, subgroup sizes column units. |
| spcforexcel-laney-p.csv16 rows | SPC for Excel Laney p′ example, 16 months | Analysis tools › EWMA · CUSUM · more › Laney P′: paste the lines (the first two numbers on each line are used: defectives, sample size). | Minitab: Stat › Control Charts › Attributes Charts › Laney P′. |
| nist-616-capability-mean16-s2.csv6 rows | 6 values with mean 16 and s 2 (built to match NIST §6.1.6) | Individuals (I-MR) tab with LSL 8 and USL 20: paste the values and compare Pp and Ppk. | Minitab: Stat › Quality Tools › Capability Analysis › Normal (subgroup size 1); compare overall Pp/Ppk. |
| aiag-msa4-gage-rr.csv90 rows | AIAG MSA (4th ed.) gauge study as printed by SPC for Excel: 10 parts, operators A/B/C, 3 trials | Analysis tools › Gauge R&R: paste the lines “part, operator, measurement” (the trial column at the end is ignored). | Minitab: Stat › Quality Tools › Gage Study › Gage R&R Study (Crossed), ANOVA method. |
| nist-strd-longley.csv16 rows | NIST StRD Longley (y = TOTEMP first, then 6 predictors) | Analysis tools › Regression: paste the file with its header row (first column = Y). | Minitab: Stat › Regression › Regression › Fit Regression Model, response TOTEMP, the other six as continuous predictors. |
| nist-strd-numacc4.csv1001 rows | NIST StRD NumAcc4, 1001 values | Individuals (I-MR) tab: the center line is the mean. The app rounds what it shows, so the digit-level result is in results.json. | Minitab: Stat › Basic Statistics › Display Descriptive Statistics (increase displayed digits). |
| nist-strd-numacc2.csv1001 rows | NIST StRD NumAcc2, 1001 values | Individuals (I-MR) tab: the center line is the mean. The app rounds what it shows, so the digit-level result is in results.json. | Minitab: Stat › Basic Statistics › Display Descriptive Statistics. |
| anhoj-figure2-pattern.csv24 rows | Pattern of the qicharts “Run charts with R” Figure 2 case (above/below/on the median) | Analysis tools › Run chart: paste the column. It shows the longest run about the median (10) and flags the shift. | Any run-chart tool that applies the Anhøj rules (for example qicharts in R). |
| nist-ewma.csv20 rows | NIST §6.3.2.4 EWMA example, 20 points | Analysis tools › EWMA · CUSUM · more: Chart EWMA, Target 50, σ 2.0539, EWMA weight λ 0.3, EWMA width L 3. | Any EWMA chart with the same settings. |
| nist-cusum.csv20 rows | NIST §6.3.2.3 CUSUM spreadsheet example, 20 group means | Analysis tools › EWMA · CUSUM · more: Chart CUSUM, Target 325, σ 0.635, CUSUM k 0.5, CUSUM h 6.608. | Any tabular CUSUM with k = 0.3175 and h = 4.1959 in data units. |
| refs-describe.csv37 rows | 37 gamma-distributed values used for the descriptive-statistics cross-check | Individuals (I-MR) tab: the center line is the mean. The app rounds what it shows, so the digit-level result is in results.json. | Minitab: Stat › Basic Statistics › Display Descriptive Statistics. |
| refs-anderson-darling.csv280 rows | 4 samples (n = 25, 40, 200, 15) used for the Anderson-Darling cross-check | Analysis tools › Capability+: paste one sample’s values at a time and add any spec limit; the Anderson-Darling result is shown. | Minitab: Stat › Basic Statistics › Normality Test, Anderson-Darling. |
| refs-t-tests.csv59 rows | Samples a (n 18), b (n 23), c (n 18, paired with a) for the t-test cross-checks | Analysis tools › Hypothesis tests: paste sample a (and b, or c for the paired test). 1-sample target 9.5. | Minitab: Stat › Basic Statistics › 1-Sample t / 2-Sample t / Paired t. |
| refs-anova-groups.csv39 rows | 4 groups (n 8, 12, 10, 9) for the one-way ANOVA and Levene cross-checks | Analysis tools › Hypothesis tests › one-way ANOVA or equal variances: paste each group. | Minitab: Stat › ANOVA › One-Way; Stat › ANOVA › Test for Equal Variances (Levene). |
| refs-regression.csv30 rows | 30 observations, 3 correlated predictors, for the OLS cross-check | Analysis tools › Regression: paste with the header row; predict at 0.5, -0.2, 0.1. | Minitab: Stat › Regression › Regression › Fit Regression Model. |
| refs-lognormal-60.csv60 rows | 60 lognormal values for the Box-Cox and lognormal-fit cross-checks | Analysis tools › Capability+ (Box-Cox or Lognormal): paste the column and add a spec limit. | Minitab: Stat › Quality Tools › Individual Distribution Identification. |
| refs-weibull-80.csv80 rows | 80 Weibull values for the Weibull-fit cross-check | Analysis tools › Capability+ (Weibull): paste the column and add a spec limit. | Minitab: Stat › Quality Tools › Individual Distribution Identification (2-parameter Weibull). |
| refs-gage-rr-synthetic.csv60 rows | Simulated gauge study, 10 parts × 3 operators × 2 trials | Analysis tools › Gauge R&R: paste the lines “part, operator, measurement”; LSL 3, USL 7. | Minitab: Stat › Quality Tools › Gage Study › Gage R&R Study (Crossed), ANOVA method. |
| refs-doe-2x3-replicated.csv16 rows | 2³ factorial, 2 replicates (16 runs), coded −1/+1 | Analysis tools › DOE, step 2: paste the runs with the header row (factors A, B, C, last column = response). | Minitab: Stat › DOE › Factorial › Define Custom Factorial Design, then Analyze Factorial Design (full model). |
Sources Cited
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.2.2 Individuals Control Charts (flow-rate example, 10 batches)
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.2.1 Shewhart X-bar and R and S Control Charts (table of A2, D3, D4)
- Standard published tables of control-chart constants (c4, d2). The test file does not name one specific publication.
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.3.1 Counts Control Charts (wafer defects, 25 wafers)
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.3.2 Proportions Control Charts (30 wafers of 50 chips)
- D. C. Montgomery, Introduction to Statistical Quality Control, p. 318 (defects in computers), as reproduced in the IMSL JMSL 6.0 UChart documentation, which does not state the edition
- D. C. Montgomery, Introduction to Statistical Quality Control, p. 321 (dyeing rolls of cloth), as reproduced in the IMSL JMSL 6.0 UChart documentation (UChartUnequal), which does not state the edition
- SPC for Excel, “Laney p′ Control Chart” (health-care data, 16 months; Table 1 data, Table 2 limits)
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.1.6 What is Process Capability? (example: USL 20, LSL 8, mean 16, s 2)
- SPC for Excel, “Process Capability and Variation” (Cpk confidence-interval example: Cpk 1.17, 50 points, 95%)
- Minitab output for its sample data set GAGEAIAG.MTW (same AIAG data), Gage R&R Study (Crossed), ANOVA method. Values as transcribed in LimitLine’s test file; no public link.
- SPC for Excel, “Three Methods to Analyze Gage R&R Studies” (raw data, stated there to be from the AIAG Measurement Systems Analysis manual, 4th edition; ANOVA %Contribution, Table 5)
- NIST Statistical Reference Datasets (StRD), Linear Least Squares Regression: Longley (certified values, higher difficulty)
- NIST StRD Univariate Summary Statistics: NumAcc4 (certified mean 10000000.2, s 0.1, lag-1 autocorrelation −0.999). · comparison scores: Stata 17 NIST certification results
- NIST StRD Univariate Summary Statistics: NumAcc2 (certified s 0.1, lag-1 autocorrelation −0.999). · comparison scores: Stata 17 NIST certification results
- J. Anhøj, “Run charts with R”, vignette of the R package qicharts (CRAN): table of run-chart limits for 10–30 useful points, and Figure 2
- J. Anhøj, “Quality Improvement Charts”, vignette of the R package qicharts2 (CRAN), Appendix 2: run-chart limits for 10–100 useful points
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.2.4 EWMA Control Charts (20 points, target 50, s 2.0539, λ 0.3, k 3)
- NIST/SEMATECH e-Handbook of Statistical Methods, §6.3.2.3 Cusum Control Charts (spreadsheet example: 20 group means, target 325, k 0.3175, h 4.1959)
- OEE.com, “Calculating OEE” worked example (420 min planned, 47 min down, ideal cycle 1.0 s, 19,271 pieces, 423 rejects)
- Lean Enterprise Institute, Lean Lexicon: takt time (480 min available, 240 demanded → 2 min)
- Lean Enterprise Institute, Lean Lexicon: heijunka, leveling by volume (500 a week, 200 on Monday → 100 a day)
- J. Cohen (1988), Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Table 2.4.1 (n per group for a two-sample t test, power 0.80, α 0.05 two-sided)
- SciPy 1.18 (scipy.stats), run by ref/ref_gen.py on seeded random data; values stored in ref/refs.json
- statsmodels (OLS, anova_lm, normal_ad, proportions_ztest, variance_inflation_factor), run by ref/ref_gen.py on seeded random data; values stored in ref/refs.json
- SciPy (noncentral t and F) and statsmodels 0.15 (TTestPower, TTestIndPower, FTestAnovaPower, power_proportions_2indep), run by ref/power_gen.py; values stored in ref/power_refs.json
Code Fingerprint
SHA-256 of the calculation modules tested for this page (commit d899f86):
| File | SHA-256 |
|---|---|
spc-core.js | 3d59458120d7364cc33ee507bfee924d200a3563df251464de8b448d557c9b54 |
stats-core.js | b055813882f583a2a2509ebb062c240dd67aeb25e1e089209c1df04e2fdfd734 |
lean-core.js | b1b0ba840858a676fabe09be56f7128881d1d2d3c9aebffe31fb48fa4ed7fde8 |
Found a number that disagrees with your software? Email support@limitline.org with the dataset and both results.