Error Analysis

⚡ Key Takeaways

Error analysis is the systematic study of uncertainty in measured and computed quantities. Every measurement and every numerical calculation carries error; the discipline quantifies how these errors arise, combine, and propagate so results can be reported with honest, defensible precision. It applies across experimental science, statistics, and numerical computation. Foundational ideas trace to Gauss and Laplace (early 1800s), with the modern numerical strand consolidated by Wilkinson in the 1960s.

💡 The Core Insight

Error is not a nuisance to be ignored but a measurable, propagable quantity obeying its own algebra. If you know how much uncertainty enters at each step, you can predict — often to leading order — how much exits. The central tool is the linearization of a function about the measured point: small input errors map to output errors through derivatives. The central distinction is between the conditioning of a problem (how sensitive the true answer is to input changes) and the stability of an algorithm (how much extra error the computation introduces).

DefinitionAbsolute and Relative Error

Let $x$ be a true value and $\hat{x}$ an approximation. The absolute error is

1

\quad E_{\text{abs}} = \hat{x} - x, \qquad |E_{\text{abs}}| = |\hat{x} - x|

Derivation hint: This is definitional — the signed or unsigned difference between approximation and truth. The relative error (for $x \neq 0$ ) is

2

\quad E_{\text{rel}} = \frac{\hat{x} - x}{x}, \qquad |E_{\text{rel}}| = \frac{|\hat{x} - x|}{|x|}

Derivation hint: Normalize the absolute error by the magnitude of the true value to obtain a scale-independent, often percentage-expressed, quantity. Here $x$ is the exact value, $\hat{x}$ the estimate. Relative error is dimensionless; absolute error carries the units of $x$ .

DefinitionRandom and Systematic Error

A measurement model decomposes the observed value $\hat{x}$ as

3

\quad \hat{x} = x + \beta + \varepsilon

Derivation hint: Additively separate the deterministic bias $\beta$ (systematic) from the zero-mean fluctuation $\varepsilon$ (random); $x$ is the true value. where $\beta$ is the systematic error (bias, constant under repetition) with $\mathbb{E}[\hat{x}] = x + \beta$ , and $\varepsilon$ is the random error with $\mathbb{E}[\varepsilon]=0$ and variance $\sigma^2$ . Random error is reduced by averaging; systematic error is not.

TheoremGeneral Error Propagation Formula

Let $q = f(x_1, x_2, \dots, x_n)$ be a function of measured quantities with independent random errors of standard deviations $\sigma_{x_1}, \dots, \sigma_{x_n}$ . Then to first order the standard deviation of $q$ is

4

\quad \sigma_q = \sqrt{\sum_{i=1}^{n}\left(\frac{\partial f}{\partial x_i}\right)^2 \sigma_{x_i}^2}

Derivation hint: Take the first-order Taylor expansion of $f$ , compute the variance of the resulting linear combination, and use independence to kill cross terms. If the variables are correlated, add the covariance terms:

5

\quad \sigma_q^2 = \sum_{i=1}^{n}\left(\frac{\partial f}{\partial x_i}\right)^2 \sigma_{x_i}^2 + 2\sum_{i<j}\frac{\partial f}{\partial x_i}\frac{\partial f}{\partial x_j}\,\sigma_{x_i x_j}

Derivation hint: The variance of a sum includes covariance terms $\sigma_{x_i x_j} = \mathrm{Cov}(x_i, x_j)$ ; independence sets these to zero, recovering [4].

ProofDerivation of the Propagation Formula

Step 1 — Linearize. Expand $f$ about the true point $(x_1,\dots,x_n)$ , writing $\delta x_i = \hat{x}_i - x_i$ for small deviations:

6

\quad q \approx f(x_1,\dots,x_n) + \sum_{i=1}^{n}\frac{\partial f}{\partial x_i}\,\delta x_i

Derivation hint: First-order multivariate Taylor expansion; second-order terms are dropped assuming errors are small relative to curvature scales.

Step 2 — Subtract the mean. Since $\mathbb{E}[\delta x_i]=0$ , the deviation of $q$ from its mean is $\delta q = \sum_i \frac{\partial f}{\partial x_i}\delta x_i$ .

Step 3 — Take the variance.

7

\quad \sigma_q^2 = \mathbb{E}\!\left[\left(\sum_i \frac{\partial f}{\partial x_i}\delta x_i\right)^2\right] = \sum_i\sum_j \frac{\partial f}{\partial x_i}\frac{\partial f}{\partial x_j}\,\mathbb{E}[\delta x_i\,\delta x_j]

Derivation hint: Expand the square of the sum into a double sum; linearity of expectation moves the constant partial derivatives outside.

Step 4 — Apply independence. For independent variables $\mathbb{E}[\delta x_i\,\delta x_j] = \sigma_{x_i}^2\delta_{ij}$ , collapsing the double sum to the diagonal and giving [4]. Retaining off-diagonal terms gives [5]. $\blacksquare$

∎

CF

Carl Friedrich Gauss

1777–1855

Gauss developed the method of least squares and, crucially, the theory of the normal ("Gaussian") distribution as the law governing random observational errors, motivated by astronomical and geodetic measurements. His 1809 Theoria Motus and later work established that the arithmetic mean is the optimal estimator under normally distributed errors and laid the groundwork for the entire propagation-of-uncertainty apparatus. Independently, Laplace contributed the central limit theorem, explaining why errors so often follow the normal law.

🧠 Intuition

Think of error propagation geometrically. A function $f$ near a point is a tilted plane (its tangent); an input uncertainty is a little cloud of points. The output cloud is the input cloud passed through the tangent plane — stretched by the slope $\partial f/\partial x_i$ in each direction. A steep slope amplifies error; a flat slope suppresses it. Two rules of thumb fall out immediately:

For sums and differences, absolute errors add (in quadrature). Subtracting two nearly-equal numbers is dangerous: the absolute error stays fixed while the result shrinks, so the relative error explodes — this is catastrophic cancellation.
For products and quotients, relative errors add (in quadrature). Multiplying by an exact constant scales both value and error equally, leaving relative error unchanged.

Figure

Error propagation through a function: input uncertainty maps to output uncertainty via the local slope

ExamplePropagating Error Through a Product

Given. The area of a rectangle from $L = 12.0 \pm 0.1$ cm and $W = 5.0 \pm 0.1$ cm. Find $A = LW$ and its uncertainty.

Step 1 — Central value. $A = 12.0 \times 5.0 = 60.0\ \text{cm}^2$ .

Step 2 — Partial derivatives. $\partial A/\partial L = W = 5.0$ , $\partial A/\partial W = L = 12.0$ .

Step 3 — Apply [4].

\sigma_A = \sqrt{(5.0)^2(0.1)^2 + (12.0)^2(0.1)^2} = \sqrt{0.25 + 1.44} = \sqrt{1.69} = 1.3\ \text{cm}^2.

Step 4 — Check via relative errors. $\frac{\sigma_L}{L}=0.00833$ , $\frac{\sigma_W}{W}=0.02$ ; in quadrature $\sqrt{0.00833^2+0.02^2}=0.0217$ , times $A=60$ gives $1.3\ \text{cm}^2$ . ✓

\boxed{A = 60.0 \pm 1.3\ \text{cm}^2}

ExampleCatastrophic Cancellation in Floating Point

Given. Compute $f(x) = \dfrac{1-\cos x}{x^2}$ at $x = 10^{-4}$ in double precision ( $\approx 16$ digits). True value $\to 1/2$ .

Step 1 — Naive evaluation. $\cos(10^{-4}) \approx 0.999999995$ . Stored to 16 digits, $1-\cos x \approx 4.999\ldots\times 10^{-9}$ , but the leading 9 significant digits of $\cos x$ are lost in the subtraction — only $\sim 7$ digits survive.

Step 2 — Divide. $f \approx 4.999\times10^{-9}/10^{-8} = 0.4999\ldots$ , but with corrupted trailing digits; relative error $\sim 10^{-8}$ instead of $\sim 10^{-16}$ .

Step 3 — Stable reformulation. Use the identity $1-\cos x = 2\sin^2(x/2)$ :

f(x) = \frac{2\sin^2(x/2)}{x^2} = \frac{1}{2}\left(\frac{\sin(x/2)}{x/2}\right)^2.

No subtraction of near-equal numbers; full precision retained.

\boxed{f(10^{-4}) = 0.5000000000\ \text{(stable form, full accuracy)}}

⚠️ Errors Add Linearly — They Usually Don't

A frequent mistake is to add uncertainties directly: $\sigma_q = \sum_i |\partial f/\partial x_i|\sigma_{x_i}$ . This is the worst-case bound, correct only for fully correlated errors. For independent random errors, uncertainties combine in quadrature (sum of squares), per [4], which is smaller. Using linear addition overestimates uncertainty; using quadrature when errors are correlated underestimates it.

⚠️ More Digits ≠ More Accuracy

Reporting $A = 60.0000\ \text{cm}^2$ when the uncertainty is $\pm 1.3$ is dishonest: the trailing digits are noise. Significant figures must match the uncertainty. A computed quantity is only as accurate as its least accurate input (for products, the largest relative error dominates).

CounterexampleA Stable-Looking Algorithm That Is Unstable

Consider computing the variance of data via the "one-pass" formula $s^2 = \frac{1}{n}\sum x_i^2 - \bar{x}^2$ . Mathematically exact, but if the data are large and clustered (e.g., $x_i \approx 10^8$ with spread $\sim 1$ ), then $\frac{1}{n}\sum x_i^2$ and $\bar{x}^2$ are huge and nearly equal — their difference suffers catastrophic cancellation and can even come out negative. The mathematically identical two-pass formula $s^2 = \frac{1}{n}\sum(x_i-\bar{x})^2$ is numerically stable. Same problem, same answer in exact arithmetic, vastly different floating-point behavior.

connection

Error analysis is the meeting point of several fields. The random-error strand is pure Probability Theory and Statistics: error propagation is just the variance of a function of random variables, and the Central Limit Theorem explains the ubiquity of the normal error law. The numerical strand connects to Floating-Point Arithmetic, Numerical Linear Algebra, and the theory of Condition Number and Numerical Stability. The optimization of estimators under error links to Least Squares and Maximum Likelihood Estimation. The Taylor-expansion core ties directly to Taylor's Theorem and Differential Calculus.

🔬 Where this appears

⏳ History

- 1632 — Galileo discusses observational errors in Dialogo, noting they cluster symmetrically about the true value and that small errors are more probable than large ones.

- 1755 — Thomas Simpson formally analyzes the arithmetic mean, showing averaging reduces error — an early statistical argument.

- 1805 — Adrien-Marie Legendre publishes the method of least squares in an appendix on comet orbits.

- 1809 — Gauss publishes Theoria Motus, deriving the normal distribution as the error law and justifying least squares; sparks a priority dispute with Legendre.

- 1810–1812 — Laplace proves the central limit theorem, explaining why summed errors tend to normality.

- 1900s–1920s — Development of statistical error theory by Pearson, Student (Gosset), and Fisher; the t-distribution handles small-sample uncertainty.

- 1947 — von Neumann and Goldstine give the first rigorous rounding-error analysis of matrix inversion.

- 1961–1965 — James H. Wilkinson founds modern backward error analysis; his Rounding Errors in Algebraic Processes (1963) and The Algebraic Eigenvalue Problem (1965) become canonical.

- 1993 — Publication of the international GUM (Guide to Uncertainty in Measurement), standardizing experimental error reporting globally.

open-problem

The linear propagation formula [4] is only first-order: it fails when $f$ is strongly nonlinear over the scale of the uncertainties, or when partial derivatives vanish (e.g., near an extremum, where second-order terms dominate). Quantifying error for such cases requires higher Taylor terms, full distributional propagation, or Monte Carlo methods. Deeper open questions persist:

Tight average-case stability bounds. Worst-case backward-error bounds (e.g., for Gaussian elimination with partial pivoting) are often pessimistic by orders of magnitude; explaining the typical observed accuracy remains an active research area in random-matrix-based analysis.
Systematic error. There is no general theory for unknown systematic errors — they cannot be reduced by repetition and are fundamentally limited by understanding of the apparatus; their treatment is partly an art.
Interval and probabilistic arithmetic seek rigorous guaranteed enclosures, but interval methods often overestimate error severely (the "dependency problem"), and reconciling rigor with tightness is unsolved in general.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also