Advanced Differentiation

⚡ Key Takeaways

Advanced Differentiation extends the basic derivative to multiple variables, higher orders, implicit and parametric forms, and abstract settings. Key result in words: the derivative is best understood as a linear approximation (the total derivative / Jacobian), from which gradients, directional derivatives, the chain rule, and Taylor expansions all follow. Where it applies: optimization, physics, differential geometry, numerical analysis, machine learning. Who/when: developed by Newton and Leibniz (1660s–1680s), formalized by Cauchy, Weierstrass, and Fréchet (19th–early 20th century).

💡 The Core Insight

The derivative at a point is not fundamentally a number or a slope — it is the best linear approximation to a function near that point. In one variable this linear map is multiplication by $f'(a)$ ; in $n$ variables it is multiplication by the Jacobian matrix. Every advanced rule (chain rule, implicit differentiation, Taylor expansion) is a statement about composing or refining these linear approximations.

DefinitionPartial Derivative

For $f:\mathbb{R}^n \to \mathbb{R}$ , the partial derivative with respect to $x_i$ at $\mathbf{a}$ is the ordinary derivative obtained by holding all other variables fixed:

1

\quad \frac{\partial f}{\partial x_i}(\mathbf{a}) = \lim_{h\to 0} \frac{f(\mathbf{a}+h\mathbf{e}_i)-f(\mathbf{a})}{h}

where $\mathbf{e}_i$ is the $i$ -th standard basis vector, $\mathbf{a}=(a_1,\dots,a_n)$ , and $h\in\mathbb{R}$ .

DefinitionTotal (Fréchet) Derivative

A function $f:\mathbb{R}^n\to\mathbb{R}^m$ is differentiable at $\mathbf{a}$ if there exists a linear map $Df(\mathbf{a}):\mathbb{R}^n\to\mathbb{R}^m$ such that

2

\quad \lim_{\mathbf{h}\to\mathbf{0}} \frac{\|f(\mathbf{a}+\mathbf{h})-f(\mathbf{a})-Df(\mathbf{a})\mathbf{h}\|}{\|\mathbf{h}\|}=0

Here $\mathbf{h}\in\mathbb{R}^n$ , and $\|\cdot\|$ denotes the Euclidean norm. The matrix of $Df(\mathbf{a})$ is the Jacobian.

DefinitionGradient and Jacobian

For scalar $f:\mathbb{R}^n\to\mathbb{R}$ , the gradient is the vector of partials:

3

\quad \nabla f(\mathbf{a}) = \left(\frac{\partial f}{\partial x_1}(\mathbf{a}),\dots,\frac{\partial f}{\partial x_n}(\mathbf{a})\right)

For vector-valued $f=(f_1,\dots,f_m)$ , the Jacobian matrix is

4

\quad J_f(\mathbf{a}) = \left[\frac{\partial f_i}{\partial x_j}(\mathbf{a})\right]_{1\le i\le m,\,1\le j\le n}

The total derivative acts as $Df(\mathbf{a})\mathbf{h}=J_f(\mathbf{a})\mathbf{h}$ .

TheoremDifferentiability Implies Continuity and Existence of Partials

If $f$ is differentiable at $\mathbf{a}$ , then $f$ is continuous at $\mathbf{a}$ and all partial derivatives exist there, with $Df(\mathbf{a})$ given by the Jacobian. Converse warning: existence of all partials does not imply differentiability; a sufficient condition is that the partials exist and are continuous in a neighborhood ( $f\in C^1$ ).

TheoremMultivariable Chain Rule

If $g:\mathbb{R}^n\to\mathbb{R}^m$ is differentiable at $\mathbf{a}$ and $f:\mathbb{R}^m\to\mathbb{R}^p$ is differentiable at $g(\mathbf{a})$ , then $f\circ g$ is differentiable at $\mathbf{a}$ and

5

\quad D(f\circ g)(\mathbf{a}) = Df(g(\mathbf{a}))\,Dg(\mathbf{a})

In matrix form this is a product of Jacobians.

TheoremClairaut–Schwarz: Equality of Mixed Partials

If $f$ has continuous second partial derivatives near $\mathbf{a}$ , then

6

\quad \frac{\partial^2 f}{\partial x_i\partial x_j}(\mathbf{a}) = \frac{\partial^2 f}{\partial x_j\partial x_i}(\mathbf{a})

TheoremTaylor's Theorem in Several Variables

If $f\in C^{k+1}$ near $\mathbf{a}$ , then for $\mathbf{h}$ small,

7

\quad f(\mathbf{a}+\mathbf{h}) = \sum_{|\alpha|\le k}\frac{D^\alpha f(\mathbf{a})}{\alpha!}\mathbf{h}^\alpha + R_k(\mathbf{h})

using multi-index notation $\alpha=(\alpha_1,\dots,\alpha_n)$ , $|\alpha|=\sum\alpha_i$ , $\alpha!=\prod\alpha_i!$ , $\mathbf{h}^\alpha=\prod h_i^{\alpha_i}$ , and remainder $R_k(\mathbf{h})=o(\|\mathbf{h}\|^k)$ . To second order:

8

\quad f(\mathbf{a}+\mathbf{h}) \approx f(\mathbf{a}) + \nabla f(\mathbf{a})\cdot\mathbf{h} + \tfrac{1}{2}\mathbf{h}^\top H_f(\mathbf{a})\,\mathbf{h}

where $H_f$ is the Hessian matrix of second partials.

ProofChain Rule via Linear Approximations

Step 1. Let $\mathbf{b}=g(\mathbf{a})$ . By differentiability of $g$ , write $g(\mathbf{a}+\mathbf{h}) = \mathbf{b} + Dg(\mathbf{a})\mathbf{h} + \mathbf{r}(\mathbf{h})$ where $\|\mathbf{r}(\mathbf{h})\|=o(\|\mathbf{h}\|)$ .

Step 2. By differentiability of $f$ at $\mathbf{b}$ , write $f(\mathbf{b}+\mathbf{k}) = f(\mathbf{b}) + Df(\mathbf{b})\mathbf{k} + \mathbf{s}(\mathbf{k})$ where $\|\mathbf{s}(\mathbf{k})\|=o(\|\mathbf{k}\|)$ .

Step 3. Set $\mathbf{k}=Dg(\mathbf{a})\mathbf{h}+\mathbf{r}(\mathbf{h})$ . Then

f(g(\mathbf{a}+\mathbf{h})) = f(\mathbf{b}) + Df(\mathbf{b})\big(Dg(\mathbf{a})\mathbf{h}+\mathbf{r}(\mathbf{h})\big) + \mathbf{s}(\mathbf{k}).

Step 4. Group: the linear part is $Df(\mathbf{b})Dg(\mathbf{a})\mathbf{h}$ . The remainder is $Df(\mathbf{b})\mathbf{r}(\mathbf{h}) + \mathbf{s}(\mathbf{k})$ .

Step 5. Since $Df(\mathbf{b})$ is bounded linear, $\|Df(\mathbf{b})\mathbf{r}(\mathbf{h})\|=o(\|\mathbf{h}\|)$ . Since $\|\mathbf{k}\|=O(\|\mathbf{h}\|)$ , $\|\mathbf{s}(\mathbf{k})\|=o(\|\mathbf{h}\|)$ . Both vanish faster than $\|\mathbf{h}\|$ , establishing differentiability of $f\circ g$ with derivative $Df(\mathbf{b})Dg(\mathbf{a})$ . $\blacksquare$

∎

ProofClairaut's Theorem (Sketch)

Step 1. Define the second difference $\Delta(s,t)=f(a+s,b+t)-f(a+s,b)-f(a,b+t)+f(a,b)$ .

Step 2. Fix $t$ and let $\phi(x)=f(x,b+t)-f(x,b)$ . Then $\Delta=\phi(a+s)-\phi(a)$ . By the MVT, $\Delta=s\,\phi'(a+\theta_1 s)=s\big(f_x(a+\theta_1 s,b+t)-f_x(a+\theta_1 s,b)\big)$ for some $\theta_1\in(0,1)$ .

Step 3. Apply the MVT again in $t$ to $f_x(a+\theta_1 s,\cdot)$ : $\Delta = st\,f_{xy}(a+\theta_1 s, b+\theta_2 t)$ .

Step 4. By a symmetric argument (differencing in the other order) $\Delta = st\,f_{yx}(a+\theta_3 s, b+\theta_4 t)$ .

Step 5. Divide by $st$ , let $(s,t)\to(0,0)$ , and use continuity of both mixed partials to conclude $f_{xy}(a,b)=f_{yx}(a,b)$ . $\blacksquare$

∎

AC

Augustin-Louis Cauchy

1789–1857

Cauchy gave the limit-based definition of the derivative that underpins rigorous multivariable calculus, replacing Leibniz's intuitive infinitesimals with $\varepsilon$ – $\delta$ precision. His work on functions of several variables and on convergence laid the groundwork for the modern Jacobian and the careful treatment of differentiability versus mere existence of partials. Cauchy's insistence on rigor made it possible to state precisely when the rules of advanced differentiation are valid.

🧠 Intuition

Picture standing on a hillside described by elevation $z=f(x,y)$ . The partial derivative $\partial f/\partial x$ is the slope you feel walking due east; $\partial f/\partial y$ is the slope walking due north. The gradient $\nabla f$ points in the direction of steepest ascent, and its magnitude is that maximum slope. The total derivative bundles all directions together: it says that very close to your feet, the hill looks like a flat tilted plane — the tangent plane — and that plane is the derivative. The Hessian then captures the curvature: whether you're in a valley (positive definite), on a ridge (negative definite), or at a saddle (indefinite).

Figure

Gradient and tangent plane: the gradient points uphill, perpendicular to level curves

ExampleDirectional Derivative and Steepest Ascent

Given: $f(x,y)=x^2 y + 3y^2$ at the point $\mathbf{a}=(1,2)$ . Find (a) the gradient, (b) the directional derivative in direction $\mathbf{u}=\tfrac{1}{5}(3,4)$ , (c) the direction of steepest ascent.

Step 1. Compute partials: $f_x = 2xy$ , $f_y = x^2 + 6y$ .

Step 2. Evaluate at $(1,2)$ : $f_x(1,2)=2(1)(2)=4$ , $f_y(1,2)=1+12=13$ . So $\nabla f(1,2)=(4,13)$ .

Step 3. Directional derivative $D_\mathbf{u}f = \nabla f\cdot\mathbf{u} = (4,13)\cdot\tfrac{1}{5}(3,4) = \tfrac{1}{5}(12+52)=\tfrac{64}{5}=12.8$ .

Step 4. Steepest ascent is along $\nabla f$ ; the unit direction is $\frac{(4,13)}{\sqrt{16+169}}=\frac{(4,13)}{\sqrt{185}}$ , with maximum rate $\|\nabla f\|=\sqrt{185}\approx 13.60$ .

\boxed{\nabla f=(4,13),\quad D_\mathbf{u}f = 12.8,\quad \text{max slope }=\sqrt{185}\approx 13.60}

ExampleImplicit Differentiation on a Surface

Given: The level surface $x^2 + y^2 + z^2 = 14$ defines $z$ implicitly as a function of $(x,y)$ near the point $(1,2,3)$ . Find $\partial z/\partial x$ and $\partial z/\partial y$ there.

Step 1. Let $F(x,y,z)=x^2+y^2+z^2-14=0$ . Compute $F_x=2x$ , $F_y=2y$ , $F_z=2z$ .

Step 2. By the implicit function theorem, $\dfrac{\partial z}{\partial x}=-\dfrac{F_x}{F_z}=-\dfrac{2x}{2z}=-\dfrac{x}{z}$ and $\dfrac{\partial z}{\partial y}=-\dfrac{y}{z}$ .

Step 3. Evaluate at $(1,2,3)$ : $\partial z/\partial x = -1/3$ , $\partial z/\partial y = -2/3$ .

Step 4. Check: $F_z=2(3)=6\neq 0$ , so the implicit function theorem applies (the surface is non-vertical here).

\boxed{\frac{\partial z}{\partial x}=-\frac{1}{3},\qquad \frac{\partial z}{\partial y}=-\frac{2}{3}}

⚠️ Common mistake

Existence of partial derivatives does not imply differentiability. A function can have both $f_x$ and $f_y$ at a point and still fail to have a tangent plane — it may not even be continuous there. Differentiability (equation [2]) is a stronger condition requiring a single linear map to approximate $f$ along all directions simultaneously, not just along the coordinate axes. The safe sufficient condition is $C^1$ : continuous partials in a neighborhood.

CounterexamplePartials Exist but Function Not Differentiable

Consider

f(x,y)=\begin{cases}\dfrac{xy}{x^2+y^2}, & (x,y)\neq(0,0)\\[4pt] 0,& (x,y)=(0,0).\end{cases}

At the origin, $f_x(0,0)=\lim_{h\to0}\frac{f(h,0)-0}{h}=0$ and similarly $f_y(0,0)=0$ , so both partials exist. Yet along the line $y=x$ , $f(x,x)=\frac{x^2}{2x^2}=\tfrac12$ for all $x\neq 0$ , so $f$ is not even continuous at the origin. By the theorem above, $f$ cannot be differentiable there despite having both partials.

CounterexampleMixed Partials Not Equal Without Continuity

For $f(x,y)=\dfrac{xy(x^2-y^2)}{x^2+y^2}$ (and $f(0,0)=0$ ), one computes $f_{xy}(0,0)=-1$ but $f_{yx}(0,0)=+1$ . The second mixed partials disagree because they are not continuous at the origin, violating the hypothesis of Clairaut's theorem. This shows equation [6] genuinely requires continuity.

connection

Advanced differentiation is the engine of Multivariable Optimization: critical points where $\nabla f=\mathbf{0}$ are classified by the Hessian via the second-derivative test. It underpins Lagrange Multipliers for constrained extrema, where gradients of objective and constraint must be parallel. The total derivative generalizes to Differential Forms and Differential Geometry, where it becomes the pushforward of tangent vectors. In analysis, it leads to the Implicit Function Theorem and Inverse Function Theorem, both proven via the contraction mapping principle applied to the linear approximation. The infinite-dimensional version is the Fréchet Derivative used in the calculus of variations.

🔬 Where this appears

⏳ History

- 1660s–1670s: Isaac Newton develops "fluxions," treating quantities as flowing with time and their rates of change as derivatives, for problems in mechanics.

- 1684: Gottfried Leibniz publishes the differential calculus with the

dy/dx

notation, the product rule, and the chain rule in Nova Methodus.

- 1740s: Leonhard Euler and Alexis Clairaut formalize partial derivatives and prove the symmetry of mixed partials (under regularity), pivotal for potential theory.

- 1821: Augustin-Louis Cauchy gives rigorous limit definitions of the derivative in his Cours d'analyse, separating intuition from proof.

- 1860s: Karl Weierstrass perfects the

\varepsilon

–

\delta

framework and constructs a continuous nowhere-differentiable function, sharpening the boundary of differentiability.

- 1870s: The Jacobian matrix and determinant (named for Carl Jacobi, 1830s) become central to change of variables and the implicit function theorem.

- 1900s–1910s: Maurice Fréchet introduces the abstract derivative for functions between normed spaces (the Fréchet derivative), generalizing equation [2].

- 1930s: Élie Cartan's exterior calculus recasts differentiation in terms of differential forms, unifying gradient, curl, and divergence.

open-problem

The interplay between the order of differentiation and regularity remains subtle in infinite dimensions and in nonsmooth analysis. Functions arising in optimization (e.g., the max of finitely many smooth functions) are only piecewise differentiable, prompting the theory of generalized gradients (Clarke subdifferentials) and automatic differentiation correctness for non-differentiable points. In partial differential equations, the question of when weak (distributional) derivatives coincide with classical ones — and the regularity of solutions — connects to deep open problems, including the regularity question for the 3D Navier–Stokes equations. How to define and compute derivatives robustly for stochastic and non-smooth objects is an active research area.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also