The Chain Rule and Computational Graphs

⚡ Key Takeaways

The chain rule computes the derivative of a composition of functions by multiplying the derivatives of the constituent functions. In words: to differentiate $f(g(x))$ , multiply the rate of change of $f$ (at $g(x)$ ) by the rate of change of $g$ (at $x$ ). It applies everywhere derivatives appear — single-variable calculus, multivariable analysis, differential geometry, and modern machine learning (where its computational-graph form is called backpropagation). First systematized by Leibniz (1676) via differential notation; the computational-graph perspective matured with reverse-mode automatic differentiation in the 1970s–80s.

💡 The Core Insight

A composition is a pipeline: $x \to u \to y$ . The chain rule says the overall sensitivity is the product of local sensitivities along the pipeline. On a graph with branches and merges, "product along a path" generalizes to "sum over all paths, each path being a product of edge derivatives." Reverse-mode differentiation is just an efficient order of summation that reuses shared subexpressions.

DefinitionComposite Function

Given functions $g : D \subseteq \mathbb{R} \to \mathbb{R}$ and $f : E \subseteq \mathbb{R} \to \mathbb{R}$ with $g(D) \subseteq E$ , the composite $f \circ g$ is defined by $(f \circ g)(x) = f(g(x))$ . We call $g$ the inner function and $f$ the outer function. The intermediate value $u = g(x)$ is the node variable connecting them.

TheoremChain Rule — Single Variable

Let $g$ be differentiable at $x_0$ and let $f$ be differentiable at $u_0 = g(x_0)$ . Then $f \circ g$ is differentiable at $x_0$ and

1

\quad (f \circ g)'(x_0) = f'\big(g(x_0)\big)\, g'(x_0).

Derivation hint: Differentiate $f$ at the point $u_0=g(x_0)$ , then multiply by the inner rate $g'(x_0)$ ; see the proof below using the error-form of differentiability. In Leibniz notation, with $y = f(u)$ , $u = g(x)$ :

2

\quad \frac{dy}{dx} = \frac{dy}{du}\cdot\frac{du}{dx}.

Derivation hint: Treat $dy/du$ and $du/dx$ as ratios of differentials; the cancellation of $du$ is suggestive but must be justified by the limit argument in the proof.

TheoremChain Rule — Multivariable

Let $\mathbf{g} : \mathbb{R}^n \to \mathbb{R}^m$ be differentiable at $\mathbf{x}_0$ and $\mathbf{f} : \mathbb{R}^m \to \mathbb{R}^k$ differentiable at $\mathbf{g}(\mathbf{x}_0)$ . Then the Jacobian of the composite is the matrix product of the Jacobians:

3

\quad J_{\mathbf{f}\circ\mathbf{g}}(\mathbf{x}_0) = J_{\mathbf{f}}\big(\mathbf{g}(\mathbf{x}_0)\big)\, J_{\mathbf{g}}(\mathbf{x}_0).

Derivation hint: Apply the linear-approximation definition of total differentiability to both maps; the composite's best linear approximation is the composition of the linear approximations, whose matrix is the product. Componentwise, for a scalar $z = f(u_1,\dots,u_m)$ where each $u_j = g_j(x_1,\dots,x_n)$ :

4

\quad \frac{\partial z}{\partial x_i} = \sum_{j=1}^{m} \frac{\partial z}{\partial u_j}\,\frac{\partial u_j}{\partial x_i}.

Derivation hint: Expand the matrix product in $[3]$ for a single output row and single input column; the sum runs over the shared intermediate variables $u_j$ — this is the "sum over paths through the $u_j$ layer."

DefinitionComputational Graph

A computational graph for an expression is a directed acyclic graph $G = (V, E)$ where each vertex $v$ holds a variable computed by an elementary operation $\phi_v$ of its parent vertices $\text{pa}(v)$ : $v = \phi_v\big(\{p : p \in \text{pa}(v)\}\big)$ . Source vertices are inputs; a designated sink is the output. Each edge $(p \to v)$ carries the local partial derivative $\partial v / \partial p$ , evaluated at the current values. The full derivative of the output with respect to an input is obtained by applying the chain rule across the graph.

Figure

Computational graph for

y = \sin(x_1 x_2) + x_1

with edge-local derivatives; backward pass accumulates

\partial y/\partial x_i

by summing over paths.

Why It Holds / Derivation

⚠️ Why the naïve "cancel

du

" proof fails

The seductive argument $\frac{\Delta y}{\Delta x} = \frac{\Delta y}{\Delta u}\cdot\frac{\Delta u}{\Delta x}$ followed by taking limits breaks when $\Delta u = g(x_0+h)-g(x_0) = 0$ for some sequence of $h \to 0$ (e.g. $g$ locally constant or oscillating). Division by $\Delta u$ is then illegal. The error-form proof above avoids any division by $\Delta u$ , which is exactly why it is the correct argument.

GW

Gottfried Wilhelm Leibniz

1646–1716

Leibniz, co-inventor of calculus, gave us the differential notation $dy/dx$ in which the chain rule appears as the almost-tautological cancellation $\frac{dy}{dx} = \frac{dy}{du}\frac{du}{dx}$ . His manuscripts from November 1676 already manipulate composite differentials. While the notation makes the rule memorable, it obscures the subtlety (see the warning above) — a gap rigorously closed only with the 19th-century $\varepsilon$ – $\delta$ definition of the limit by Cauchy and Weierstrass. Leibniz's deeper gift was the operational viewpoint: differentiation as a symbolic process that propagates along the structure of an expression — the conceptual seed of computational graphs three centuries later.

🧠 Intuition

Think of nested gears. If gear $A$ drives gear $B$ at ratio $3{:}1$ (B turns 3× as fast), and $B$ drives $C$ at ratio $2{:}1$ , then $A$ drives $C$ at $6{:}1$ — ratios multiply. The derivative $g'(x)$ is the local gear ratio from $x$ to $u$ ; $f'(u)$ is the ratio from $u$ to $y$ ; the composite ratio is their product. Now add branching. If $x$ influences $y$ through two independent gear trains (paths), the total effect is the sum of the two products — small pushes from each route add up. That is exactly $[4]$ : sum over paths, multiply along each. The computational graph is the literal gear diagram, and backpropagation is the disciplined way to total up all the ratios without recomputing shared gears.

ExampleSingle-Variable: a nested trig–power function

Given: $y = \big(\sin(3x)\big)^4$ . Find $\frac{dy}{dx}$ at $x = \pi/6$ .

Step 1 — Identify the layers. Outer: $f(u)=u^4$ . Middle: $u = \sin(w)$ . Inner: $w = 3x$ . Three nested functions.

Step 2 — Local derivatives.

$\dfrac{dy}{du} = 4u^3$
$\dfrac{du}{dw} = \cos w$
$\dfrac{dw}{dx} = 3$

Step 3 — Multiply (chain rule for a 3-chain).

8

\quad \frac{dy}{dx} = 4\big(\sin 3x\big)^3 \cdot \cos(3x)\cdot 3 = 12\,\sin^3(3x)\cos(3x).

Derivation hint: Apply $[2]$ twice: $\frac{dy}{dx}=\frac{dy}{du}\frac{du}{dw}\frac{dw}{dx}$ ; substitute $w=3x$ , $u=\sin 3x$ .

Step 4 — Evaluate at $x=\pi/6$ . Here $3x = \pi/2$ , so $\sin(3x)=1$ , $\cos(3x)=0$ .

\frac{dy}{dx}\Big|_{\pi/6} = 12\cdot 1^3 \cdot 0 = 0.

\boxed{\dfrac{dy}{dx}\Big|_{x=\pi/6} = 0 \ \ (\text{dimensionless})}

The zero makes sense: at $3x=\pi/2$ , $\sin(3x)$ is at its maximum, so $\sin^4(3x)$ momentarily stops changing.

ExampleMultivariable / Graph: reverse-mode pass

Given: $y = \sin(x_1 x_2) + x_1$ at $(x_1,x_2) = (2, \tfrac{\pi}{4})$ . Compute $\partial y/\partial x_1$ and $\partial y/\partial x_2$ by a forward pass then a backward pass.

Forward pass (compute node values).

$u = x_1 x_2 = 2\cdot \tfrac{\pi}{4} = \tfrac{\pi}{2} \approx 1.5708$
$v = \sin u = \sin(\tfrac{\pi}{2}) = 1$
$y = v + x_1 = 1 + 2 = 3$

Backward pass (seed $\bar y \equiv \partial y/\partial y = 1$ , propagate adjoints).

9

\quad \bar v = \bar y\,\frac{\partial y}{\partial v} = 1\cdot 1 = 1,\qquad \bar u = \bar v\,\frac{\partial v}{\partial u} = 1\cdot \cos u = \cos\tfrac{\pi}{2} = 0.

Derivation hint: Each adjoint $\bar n = \partial y/\partial n$ ; multiply the downstream adjoint by the local edge derivative.

Accumulate to inputs (sum over all incoming paths).

10

\quad \frac{\partial y}{\partial x_1} = \underbrace{\bar u\,\frac{\partial u}{\partial x_1}}_{\text{via }u} + \underbrace{\bar y\,\frac{\partial y}{\partial x_1}}_{\text{direct}} = 0\cdot x_2 + 1\cdot 1 = 1.

Derivation hint: $x_1$ reaches $y$ by two paths — through $u$ and directly via the $+x_1$ term; $[4]$ tells us to add them.

11

\quad \frac{\partial y}{\partial x_2} = \bar u\,\frac{\partial u}{\partial x_2} = 0\cdot x_1 = 0.

Derivation hint: $x_2$ reaches $y$ only through $u$ ; one path, so a single product.

\boxed{\nabla y = \Big(\tfrac{\partial y}{\partial x_1},\ \tfrac{\partial y}{\partial x_2}\Big) = (1,\ 0) \ \ (\text{dimensionless})}

Note: a single backward pass produced both partials — the efficiency that powers backpropagation.

⚠️ Three classic errors

1. Evaluating the outer derivative at $x$ instead of at $u=g(x)$ . $f'$ must be evaluated at $g(x_0)$ , not $x_0$ . Writing $(\sin(x^2))' = \cos(x)\cdot 2x$ is wrong; it is $\cos(x^2)\cdot 2x$ . 2. Forgetting the inner derivative entirely. $(\sin(x^2))' \neq \cos(x^2)$ . The factor $2x$ is mandatory. 3. Treating $dy/du$ as a genuine fraction. It is notation for a limit. The cancellation works as a mnemonic but is not a proof, and it fails to generalize cleanly to partial derivatives, where $\partial z/\partial x \ne 1/(\partial x/\partial z)$ in general.

CounterexampleA composite that is differentiable while neither factor's quotient is "tame"

Let $g(x) = x^2 \sin(1/x)$ for $x\neq 0$ , $g(0)=0$ . Then $g$ is differentiable at $0$ with $g'(0)=0$ , but $g'(x)$ oscillates wildly near $0$ and $\Delta u = g(x_0+h)-g(0)$ equals $0$ infinitely often as $h\to 0$ . With $f(u)=u$ (so $f\circ g = g$ ), the naïve "divide by $\Delta u$ " proof is invalid on those sequences — yet the chain rule still holds because the error-form proof ( $[5]$ – $[7]$ ) never divides by $\Delta u$ . This is the canonical example showing why rigor matters.

connection

The chain rule is the connective tissue of differential calculus. It powers Implicit Differentiation (differentiate $F(x,y(x))=0$ and solve for $y'$ ), Related Rates (link $dV/dt$ to $dr/dt$ through $dV/dr$ ), and The Substitution Rule for Integration (which is the chain rule read backwards). In several variables it underlies the Gradient and Directional Derivative and the Jacobian Matrix. Its graph form is precisely Backpropagation in neural networks, and the abstract version — composition of linear approximations — is the statement of the Total Derivative in Differential Geometry (the functoriality of the tangent map: $T(f\circ g) = Tf \circ Tg$ ).

🔬 Where this appears

⏳ History

- 1676 — Leibniz manipulates composite differentials in unpublished manuscripts, foreshadowing

\frac{dy}{dx}=\frac{dy}{du}\frac{du}{dx}

.

- 1684 — Leibniz publishes his first calculus paper (Nova Methodus), establishing differential notation that encodes the chain rule.

- 1690s — The Bernoullis and L'Hôpital apply chain-rule reasoning to curves and nested expressions across Europe.

- 1820s — Cauchy's Cours d'analyse (1821) introduces limit-based rigor; the chain rule gets its first careful, if still informal, treatment.

- 1860s–1870s — Weierstrass's

\varepsilon

–

\delta

framework enables the modern error-form proof, closing the "divide by

\Delta u

" gap.

- 1870s — The multivariable / Jacobian chain rule is formalized in the work of Jacobi's successors on transformations and partial differentiation.

- 1964 — Wengert publishes a method for the automatic evaluation of derivatives — an early forward-mode automatic differentiation.

- 1970 — Seppo Linnainmaa describes the reverse-mode accumulation of derivatives (the algorithmic chain rule on a graph) in his master's thesis.

- 1986 — Rumelhart, Hinton & Williams popularize backpropagation for neural networks, igniting modern deep learning.

open-problem · Boundaries of the rule

The chain rule requires differentiability; at points where the inner or outer function is merely continuous (e.g. $|x|$ at $0$ ), the composite may fail to be differentiable or require subgradients — leading to the calculus of Clarke subdifferentials and questions about which "nonsmooth chain rules" hold for compositions like $\text{ReLU}$ networks, where backpropagation is used in practice despite kink points. A related modern theme is numerical stability: repeated multiplication of edge derivatives causes vanishing or exploding gradients in deep graphs; understanding when the product of many Jacobians stays well-conditioned (via residual connections, normalization, orthogonal initialization) remains an active research area. Finally, in infinite-dimensional and stochastic settings, generalizations (Fréchet-derivative chain rule, Itô's formula as a "stochastic chain rule" with an extra second-order term) show that the simple product law is only the smooth, deterministic special case of a richer family of composition rules.

The Chain Rule and Computational Graphs

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also