Directional Derivatives and Gradients

⚡ Key Takeaways

The directional derivative $D_{\mathbf{u}}f$ measures the instantaneous rate of change of a scalar field $f$ as you move in a specified unit direction $\mathbf{u}$ . The gradient $\nabla f$ is the vector of partial derivatives; it packages all directional rates at once, points in the direction of steepest ascent, and equals the maximum directional derivative in magnitude. Used everywhere in multivariable calculus, optimization, physics (potentials, fields), and machine learning. Formalized in the 19th century by Hamilton, Maxwell, and Gibbs.

💡 The Core Insight

The gradient is the linear map "rate of change" written as a vector. The directional derivative in any direction is just the projection of the gradient onto that direction. So the gradient is not one rate — it is the encoding of all rates simultaneously, and the geometry (steepest ascent, perpendicularity to level sets) falls out of the dot product $\nabla f\cdot\mathbf{u}=|\nabla f|\cos\theta$ .

DefinitionDirectional Derivative

Let $f:\mathbb{R}^n\to\mathbb{R}$ and let $\mathbf{u}\in\mathbb{R}^n$ be a unit vector ( $|\mathbf{u}|=1$ ). The directional derivative of $f$ at the point $\mathbf{a}$ in the direction $\mathbf{u}$ is

1

\quad D_{\mathbf{u}}f(\mathbf{a}) \;=\; \lim_{h\to 0}\frac{f(\mathbf{a}+h\mathbf{u})-f(\mathbf{a})}{h}.

Derivation hint: This is the ordinary single-variable derivative of the restriction $g(h)=f(\mathbf{a}+h\mathbf{u})$ evaluated at $h=0$ ; the path is the straight line through $\mathbf{a}$ with velocity $\mathbf{u}$ . Here $\mathbf{a}=(a_1,\dots,a_n)$ is the base point, $h$ is a scalar step size, and $\mathbf{u}=(u_1,\dots,u_n)$ specifies the direction. The result is a scalar: a rate of change of $f$ -value per unit distance traveled.

⚠️ Common mistake

The unit-length requirement on $\mathbf{u}$ is not cosmetic. If $\mathbf{u}$ is not normalized, the limit in $[1]$ scales by $|\mathbf{u}|$ and no longer measures "rate per unit distance." Some texts define $D_{\mathbf{v}}f$ for arbitrary $\mathbf{v}$ as $\nabla f\cdot\mathbf{v}$ (the "Gâteaux" convention); this is linear in $\mathbf{v}$ but only equals geometric rate-of-change when $|\mathbf{v}|=1$ . Always check the convention.

DefinitionGradient

The gradient of a differentiable function $f:\mathbb{R}^n\to\mathbb{R}$ at $\mathbf{a}$ is the vector of first-order partial derivatives:

2

\quad \nabla f(\mathbf{a}) \;=\; \left(\frac{\partial f}{\partial x_1}(\mathbf{a}),\,\frac{\partial f}{\partial x_2}(\mathbf{a}),\,\dots,\,\frac{\partial f}{\partial x_n}(\mathbf{a})\right).

Derivation hint: Each component is the directional derivative along a standard basis vector $\mathbf{e}_i$ , i.e. $\partial f/\partial x_i = D_{\mathbf{e}_i}f$ . The symbol $\nabla$ ("nabla" or "del") is the formal operator $\nabla=\left(\tfrac{\partial}{\partial x_1},\dots,\tfrac{\partial}{\partial x_n}\right)$ . The gradient is a vector field: at each point it returns a vector.

TheoremGradient Formula for the Directional Derivative

If $f$ is differentiable at $\mathbf{a}$ , then for every unit vector $\mathbf{u}$ ,

3

\quad D_{\mathbf{u}}f(\mathbf{a}) \;=\; \nabla f(\mathbf{a})\cdot\mathbf{u} \;=\; \sum_{i=1}^{n}\frac{\partial f}{\partial x_i}(\mathbf{a})\,u_i.

Derivation hint: Apply the multivariable chain rule to $g(h)=f(\mathbf{a}+h\mathbf{u})$ ; differentiability guarantees the linear approximation is exact in the limit.

TheoremSteepest Ascent & Level-Set Orthogonality

Writing $\theta$ for the angle between $\nabla f(\mathbf{a})$ and $\mathbf{u}$ ,

4

\quad D_{\mathbf{u}}f(\mathbf{a}) \;=\; |\nabla f(\mathbf{a})|\,\cos\theta.

Derivation hint: Expand the dot product in $[3]$ using $\mathbf{a}\cdot\mathbf{b}=|\mathbf{a}||\mathbf{b}|\cos\theta$ and $|\mathbf{u}|=1$ . Consequences: (i) the maximum value $|\nabla f|$ occurs at $\theta=0$ , so $\nabla f$ points in the direction of steepest ascent; (ii) the minimum $-|\nabla f|$ occurs at $\theta=\pi$ (steepest descent); (iii) $D_{\mathbf{u}}f=0$ when $\theta=\pi/2$ , so $\nabla f$ is orthogonal to the level set $\{f=c\}$ through $\mathbf{a}$ .

ProofThe Gradient Formula via the Chain Rule

We prove equation $[3]$ for $f$ differentiable at $\mathbf{a}$ .

Step 1. Define the single-variable function $g(h)=f(\mathbf{a}+h\mathbf{u})$ , a parametrization of the line through $\mathbf{a}$ with direction $\mathbf{u}$ . By $[1]$ , $D_{\mathbf{u}}f(\mathbf{a})=g'(0)$ .

Step 2. The components of the path are $x_i(h)=a_i+h u_i$ , so $\dfrac{dx_i}{dh}=u_i$ .

Step 3. Because $f$ is differentiable at $\mathbf{a}$ , the multivariable chain rule applies:

5

\quad g'(h)=\sum_{i=1}^{n}\frac{\partial f}{\partial x_i}\big(\mathbf{a}+h\mathbf{u}\big)\,\frac{dx_i}{dh}=\sum_{i=1}^{n}\frac{\partial f}{\partial x_i}\big(\mathbf{a}+h\mathbf{u}\big)\,u_i.

Derivation hint: This is the chain rule for $f\circ\boldsymbol{\gamma}$ with $\boldsymbol{\gamma}(h)=\mathbf{a}+h\mathbf{u}$ ; valid precisely because differentiability gives a single good linear approximation.

Step 4. Evaluate at $h=0$ :

6

\quad g'(0)=\sum_{i=1}^{n}\frac{\partial f}{\partial x_i}(\mathbf{a})\,u_i=\nabla f(\mathbf{a})\cdot\mathbf{u}.

Derivation hint: Set $h=0$ so $\mathbf{a}+h\mathbf{u}=\mathbf{a}$ ; recognize the sum as a dot product. This is exactly $[3]$ . $\blacksquare$

∎

⚠️ Common mistake

The proof requires differentiability, not merely existence of partial derivatives. A function can have all partial derivatives (even all directional derivatives) at a point yet fail to be differentiable, in which case formula $[3]$ can be false. See the counterexample below.

JW

Josiah Willard Gibbs

1839–1903

The American mathematical physicist J. W. Gibbs, working at Yale, developed modern vector analysis in the 1880s in his privately printed lecture notes Elements of Vector Analysis. He (alongside Oliver Heaviside) extracted the practical "del" operator and the gradient/divergence/curl trio from Hamilton's heavier quaternion machinery, giving the gradient its present notation and dot-product interpretation. His student E. B. Wilson popularized it in the textbook Vector Analysis (1901), which is essentially the form taught today.

🧠 Intuition

Picture $f$ as the elevation of a hilly landscape, with $(x,y)$ the map coordinates and $f(x,y)$ the height. Standing at a point, you can face any compass direction $\mathbf{u}$ and ask "how steep is the slope if I walk this way?" — that number is $D_{\mathbf{u}}f$ . Now the magic: there is one special direction in which the ground rises fastest. That direction is $\nabla f$ , and its length tells you how steep that fastest climb is. Walk at right angles to $\nabla f$ and you stay at the same elevation — you're walking along a contour line (a level set). Walk directly opposite to $\nabla f$ and you descend fastest. Every other direction gives a slope equal to the steepest slope times $\cos\theta$ , the cosine "shadow" of full steepness — exactly equation $[4]$ . This is why hiking trails zig-zag: walking perpendicular-ish to the gradient gives a gentle grade.

Figure

Gradient points to steepest ascent and is perpendicular to level curves

ExampleDirectional Derivative of a Paraboloid

Given: $f(x,y)=x^2+3xy+y^2$ , at the point $\mathbf{a}=(1,2)$ , in the direction toward $(4,6)$ .

Step 1 — Gradient. $\dfrac{\partial f}{\partial x}=2x+3y$ , $\dfrac{\partial f}{\partial y}=3x+2y$ . At $(1,2)$ :

7

\quad \nabla f(1,2)=(2(1)+3(2),\,3(1)+2(2))=(8,\,7).

Derivation hint: Apply $[2]$ ; differentiate treating the other variable as constant.

Step 2 — Unit direction. Displacement $(4,6)-(1,2)=(3,4)$ , length $\sqrt{3^2+4^2}=5$ , so $\mathbf{u}=(3/5,\,4/5)$ .

Step 3 — Dot product (apply $[3]$ ).

8

\quad D_{\mathbf{u}}f(1,2)=(8,7)\cdot\left(\tfrac35,\tfrac45\right)=\frac{24}{5}+\frac{28}{5}=\frac{52}{5}.

Derivation hint: Multiply componentwise and sum; this is the gradient formula in action.

Answer: $\boxed{D_{\mathbf{u}}f(1,2)=\dfrac{52}{5}=10.4 \text{ (units of } f \text{ per unit length)}}$

ExampleSteepest Ascent and Maximum Rate

Given: Temperature field $T(x,y,z)=100\,e^{-(x^2+2y^2+3z^2)}$ (°C), find the direction of fastest temperature increase at $\mathbf{a}=(1,1,0)$ and that maximum rate.

Step 1 — Gradient. Using chain rule, $\nabla T=100\,e^{-(x^2+2y^2+3z^2)}\,(-2x,\,-4y,\,-6z)$ .

Step 2 — Evaluate. At $(1,1,0)$ , the exponent is $-(1+2+0)=-3$ , so $e^{-3}\approx0.049787$ :

9

\quad \nabla T(1,1,0)=100\,e^{-3}\,(-2,\,-4,\,0)\approx(-9.957,\,-19.915,\,0)\ \text{°C/unit}.

Derivation hint: Multiply the scalar prefactor $100e^{-3}$ into each bracketed component.

Step 3 — Direction of steepest ascent is $\nabla T$ itself; normalize: $|\nabla T|=100e^{-3}\sqrt{4+16}=100e^{-3}\sqrt{20}$ .

10

\quad |\nabla T(1,1,0)|=100e^{-3}\cdot 2\sqrt5\approx 22.26\ \text{°C per unit length}.

Derivation hint: By $[4]$ , the maximum directional derivative equals the gradient's magnitude (take $\cos\theta=1$ ).

Answer: Fastest warming is in the unit direction $\boxed{\mathbf{u}=\tfrac{1}{\sqrt5}(-1,-2,0)}$ at the rate $\boxed{\approx 22.3\ \text{°C per unit length}}$ . (Physically: away from the origin's hot core, the field decays, so steepest ascent points back toward the peak.)

⚠️ Common mistake

"The gradient is the slope." No — the gradient is a vector. Its magnitude is the slope of steepest ascent; its direction tells you which way that steepest climb lies. Confusing the vector $\nabla f$ with the scalar $D_{\mathbf{u}}f$ is the single most common error.

"Partial derivatives existing means $f$ is differentiable, so $[3]$ holds." False. Differentiability is strictly stronger than partial-derivative existence. Formula $[3]$ is guaranteed only under differentiability.

"The gradient lives in the same space as $f$ -values." The gradient lives in the domain (input) space, not the output. For $f:\mathbb{R}^3\to\mathbb{R}$ , $\nabla f$ is a 3-vector tangent to the domain, never a 4-dimensional object.

connection

The directional derivative is the bridge between several structures:

It is the action of the Total Derivative (Jacobian/differential $df_{\mathbf a}$ ) on a vector: $D_{\mathbf u}f=df_{\mathbf a}(\mathbf u)$ . The gradient is just the differential expressed as a vector via the dot product.
Through $[3]$ it is built on the Multivariable Chain Rule and the <a class="wiki-link" href="/wiki/math-analysis-partial-derivatives">Partial Derivatives</a>.
The orthogonality of $\nabla f$ to level sets is the engine behind <a class="wiki-link" href="/wiki/math-analysis-lagrange-multipliers">Lagrange Multipliers</a> (gradients align at constrained optima) and Tangent Planes to Surfaces (normal vector $=\nabla F$ for a level surface $F=c$ ).
The operator $\nabla$ generalizes to Divergence and Curl and the Laplacian in vector calculus.
When $f$ has a gradient field, line integrals become path-independent — the topic of Conservative Vector Fields and the gradient theorem.

🔬 Where this appears

⏳ History

- 1740s–1760s — Euler and the development of partial derivatives in the study of functions of several variables and in the calculus of variations.

- 1820s — Cauchy formalizes limits, providing the rigorous foundation for difference quotients like

[1]

.

- 1828 — George Green's essay introduces potential theory; gradients of potential functions appear implicitly in physics.

- 1843 — William Rowan Hamilton invents quaternions and introduces the operator

\nabla

(which he called "nabla" after a Hebrew harp), the ancestor of the modern del.

- 1873 — James Clerk Maxwell, in A Treatise on Electricity and Magnetism, uses Hamilton's

\nabla

extensively, coining/naming "slope," "convergence," and "curl," cementing the gradient's role in field theory.

- 1881–1884 — J. W. Gibbs distributes Elements of Vector Analysis, stripping vector calculus from quaternions and defining gradient via the dot product, essentially the modern form.

- 1893 — Oliver Heaviside independently popularizes operational vector calculus for electromagnetism.

- 1901 — E. B. Wilson's Vector Analysis, based on Gibbs's lectures, standardizes notation and teaching of the gradient and directional derivative.

- 20th c. — Generalization to differential forms (Cartan), Riemannian geometry (where the gradient requires a metric), and the explosion of gradient methods in 20th–21st century optimization and deep learning.

open-problem

Several subtleties deserve a serious student's attention:

Gradient requires a metric. On a general manifold or with non-Euclidean inner products, the differential $df$ is a covector (1-form), and converting it to a gradient vector requires raising an index with a metric tensor $g$ : $\nabla f = g^{-1}df$ . The "obvious" identification $\nabla f \leftrightarrow$ partials is metric-dependent; in non-orthonormal coordinates the naive componentwise formula $[2]$ is wrong.
Differentiability vs. directional derivatives. As the counterexample shows, existence of all directional derivatives does not guarantee differentiability, continuity, or the validity of $[3]$ . The gap is subtle and a recurring source of error.
Nonsmooth optimization. Many practical objectives (e.g. with $\ell_1$ regularization, ReLU networks) are not differentiable everywhere. The gradient is replaced by the subgradient or Clarke generalized gradient, an active research area where the clean steepest-descent picture only partially survives.
Infinite dimensions. In Banach/Hilbert spaces the analog is the Fréchet or Gâteaux derivative; the gradient exists by the Riesz representation theorem only in Hilbert spaces, and pathologies of the finite-dimensional counterexample multiply.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also