Gradient Clipping

⚡ Key Takeaways

Gradient clipping is a technique that bounds the magnitude of gradients during the training of machine learning models to prevent destructively large parameter updates. Key result in words: when a gradient's norm exceeds a chosen threshold, it is rescaled (or its components are truncated) so that updates remain stable. Where it applies: training deep neural networks, especially recurrent networks (RNNs, LSTMs), transformers, and reinforcement learning agents. Who/when: popularized by Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio (2012–2013) as a remedy for the exploding gradient problem.

💡 The Core Insight

Optimization fails not because gradients point in the wrong direction, but because they can occasionally point in the right direction with an enormous magnitude — overshooting the minimum and landing in a worse region. Gradient clipping decouples where to step (direction) from how far to step (magnitude), capping only the latter. The direction is trusted; the size is policed.

DefinitionGradient and its norm

Let $L(\boldsymbol\theta)$ be a scalar loss function of parameters $\boldsymbol\theta \in \mathbb{R}^n$ . The gradient is

1

\quad \mathbf{g} \;=\; \nabla_{\boldsymbol\theta} L \;=\; \left(\frac{\partial L}{\partial \theta_1}, \dots, \frac{\partial L}{\partial \theta_n}\right)^{\!\top}.

Derivation hint: each component is the partial derivative of the loss with respect to one parameter; assemble into a column vector. Its Euclidean ( $\ell_2$ ) norm is

2

\quad \|\mathbf{g}\|_2 \;=\; \sqrt{\sum_{i=1}^{n} g_i^{2}}.

Derivation hint: apply the definition of the $\ell_2$ norm to the gradient's components. Here $g_i = \partial L / \partial \theta_i$ , $n$ is the number of parameters, and $\|\cdot\|_2$ measures the total step length the raw gradient would induce.

DefinitionNorm clipping (clip-by-norm)

Given a threshold $c > 0$ , the clipped gradient $\hat{\mathbf{g}}$ is

3

\quad \hat{\mathbf{g}} \;=\; \begin{cases} \mathbf{g}, & \text{if } \|\mathbf{g}\|_2 \le c,\\[4pt] \dfrac{c}{\|\mathbf{g}\|_2}\,\mathbf{g}, & \text{if } \|\mathbf{g}\|_2 > c. \end{cases}

Derivation hint: scale the vector by the ratio of allowed norm $c$ to actual norm, but only when the actual norm exceeds $c$ . A compact single-line equivalent is

4

\quad \hat{\mathbf{g}} \;=\; \mathbf{g}\cdot \min\!\left(1,\ \frac{c}{\|\mathbf{g}\|_2}\right).

Derivation hint: the $\min$ selects the scaling factor $1$ when $\|\mathbf g\|\le c$ and $c/\|\mathbf g\|$ otherwise, reproducing the piecewise definition. Symbols: $c$ is the clip threshold (a hyperparameter), $\hat{\mathbf g}$ is the gradient actually fed to the optimizer, and the scaling factor lies in $(0,1]$ .

DefinitionValue clipping (clip-by-value)

An alternative clamps each component independently to $[-v, v]$ :

5

\quad \hat{g}_i \;=\; \operatorname{clip}(g_i, -v, v) \;=\; \max\!\big(-v,\ \min(v,\ g_i)\big), \qquad i = 1,\dots,n.

Derivation hint: nest a lower bound ( $\max$ with $-v$ ) inside an upper bound ( $\min$ with $v$ ) for each coordinate. Here $v > 0$ is the per-component bound. Unlike norm clipping, this changes the direction of the gradient vector.

TheoremDirection preservation of norm clipping

Norm clipping (Eq. [3]) preserves the direction of the gradient: $\hat{\mathbf g}$ is a non-negative scalar multiple of $\mathbf g$ , i.e. $\hat{\mathbf g} = \alpha\,\mathbf g$ with $\alpha \in (0,1]$ . Consequently the cosine similarity between $\hat{\mathbf g}$ and $\mathbf g$ equals $1$ .

Proof

Step 1. If $\|\mathbf g\|_2 \le c$ , then $\hat{\mathbf g} = \mathbf g = 1\cdot \mathbf g$ , so $\alpha = 1$ .

Step 2. If $\|\mathbf g\|_2 > c$ , then by Eq. [3], $\hat{\mathbf g} = \frac{c}{\|\mathbf g\|_2}\mathbf g$ , so $\alpha = c/\|\mathbf g\|_2$ .

Step 3. Since $c > 0$ and $\|\mathbf g\|_2 > 0$ , we have $\alpha > 0$ ; since $\|\mathbf g\|_2 > c$ , we have $\alpha < 1$ . Thus $\alpha \in (0,1)$ in this case.

Step 4. Combining, $\alpha \in (0,1]$ always. The cosine similarity is

6

\quad \cos\theta = \frac{\hat{\mathbf g}\cdot \mathbf g}{\|\hat{\mathbf g}\|\,\|\mathbf g\|} = \frac{\alpha\,\|\mathbf g\|^2}{\alpha\|\mathbf g\|\cdot\|\mathbf g\|} = 1. \qquad \blacksquare

Derivation hint: substitute $\hat{\mathbf g}=\alpha\mathbf g$ , use $\hat{\mathbf g}\cdot\mathbf g = \alpha\|\mathbf g\|^2$ and $\|\hat{\mathbf g}\| = \alpha\|\mathbf g\|$ , then cancel.

∎

Figure

Norm clipping rescales an over-long gradient back to the threshold ball while preserving direction

Proof

Goal: show that norm clipping is equivalent to a per-step adaptive learning rate. Consider standard gradient descent with learning rate $\eta$ :

7

\quad \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\hat{\mathbf g}_t.

Derivation hint: substitute the clipped gradient into the ordinary update rule.

Step 1. When $\|\mathbf g\|_2 > c$ , substitute Eq. [3]:

8

\quad \boldsymbol\theta_{t+1} = \boldsymbol\theta_t - \eta\,\frac{c}{\|\mathbf g_t\|_2}\,\mathbf g_t = \boldsymbol\theta_t - \eta_{\text{eff}}\,\mathbf g_t, \qquad \eta_{\text{eff}} = \frac{\eta\,c}{\|\mathbf g_t\|_2}.

Derivation hint: group the constants multiplying the raw gradient into a single effective learning rate.

Step 2. Interpretation: the effective learning rate $\eta_{\text{eff}}$ is inversely proportional to the gradient norm. When the gradient explodes ( $\|\mathbf g_t\| \to \infty$ ), $\eta_{\text{eff}}\to 0$ , so the step length is bounded:

9

\quad \|\boldsymbol\theta_{t+1} - \boldsymbol\theta_t\| = \eta\,\|\hat{\mathbf g}_t\| \le \eta\,c.

Derivation hint: take norms of Eq. [7]; since $\|\hat{\mathbf g}\|\le c$ always, the step is bounded by $\eta c$ .

Step 3. When $\|\mathbf g\|_2 \le c$ , $\eta_{\text{eff}} = \eta$ and clipping is inert. Hence clipping is a conditional learning-rate damper that activates only on dangerous steps. $\blacksquare$

∎

RP

Razvan Pascanu, Tomas Mikolov & Yoshua Bengio

2012–2013

Working at the University of Montreal under Bengio, Pascanu and Mikolov diagnosed why recurrent neural networks failed to learn long-range dependencies. In their paper "On the difficulty of training Recurrent Neural Networks" (ICML 2013), they framed the problem geometrically: the loss surface of an RNN contains steep "cliffs" where the gradient norm spikes by orders of magnitude. They proposed gradient norm clipping as a cheap, principled defense, showing it allowed RNNs to traverse cliff edges without being flung into bad regions. Bengio, a Turing Award laureate (2018), had long championed deep architectures; this work made them trainable in practice.

🧠 Intuition

Imagine hiking downhill in thick fog, taking steps proportional to how steep the ground feels beneath your feet. On gentle slopes you take normal strides. But if you reach the edge of a cliff, the ground feels infinitely steep, and a proportional step would hurl you off the cliff into a ravine far worse than where you started. Gradient clipping is a rule: "no matter how steep it feels, never take a stride longer than $c$ ." You still step in the downhill direction — you just refuse to leap. This is why clipping rarely changes the path on smooth terrain but is a lifesaver near cliffs, which in neural networks correspond to regions where the loss curvature is extreme.

AN

A note on the "cliff" geometry

n/a

The cliffs are real consequences of the chain rule. In a recurrent network unrolled over $T$ steps, the gradient involves a product of $T$ Jacobian matrices. If the largest singular value of the recurrent weight matrix exceeds $1$ , this product grows like $\lambda^T$ — exponentially. A tiny region where parameters cross such a threshold produces a near-vertical wall in the loss landscape.

ExampleNorm clipping a 2D gradient

Given: gradient $\mathbf g = (6, 8)^\top$ , clip threshold $c = 5$ , learning rate $\eta = 0.1$ .

Step 1 — compute the norm:

10

\quad \|\mathbf g\|_2 = \sqrt{6^2 + 8^2} = \sqrt{36+64} = \sqrt{100} = 10.

Derivation hint: apply Eq. [2] to the two components.

Step 2 — check the trigger: $\|\mathbf g\|_2 = 10 > 5 = c$ , so clipping activates.

Step 3 — scale factor: $\alpha = c/\|\mathbf g\|_2 = 5/10 = 0.5$ .

Step 4 — clipped gradient: $\hat{\mathbf g} = 0.5\,(6,8)^\top = (3,4)^\top$ . Check: $\|\hat{\mathbf g}\| = \sqrt{9+16} = 5 = c$ . ✓

Step 5 — parameter step length: $\eta\,\|\hat{\mathbf g}\| = 0.1 \times 5 = 0.5$ .

\boxed{\hat{\mathbf g} = (3,\,4)^\top,\quad \text{step length} = 0.5\ \text{(parameter units)}}

ExampleValue clipping vs. norm clipping comparison

Given: gradient $\mathbf g = (10, -1)^\top$ . Compare norm clipping ( $c = 5$ ) with value clipping ( $v = 5$ ).

Norm clipping: $\|\mathbf g\| = \sqrt{100+1} = \sqrt{101} \approx 10.05 > 5$ , so $\alpha = 5/10.05 \approx 0.4975$ .

11

\quad \hat{\mathbf g}_{\text{norm}} \approx (4.975,\ -0.4975)^\top.

Derivation hint: multiply each component by $\alpha = c/\|\mathbf g\|$ . Direction unchanged: ratio $g_1/g_2 = -10$ preserved.

Value clipping ( $v=5$ ): clamp each component to $[-5,5]$ :

12

\quad \hat{\mathbf g}_{\text{value}} = (\min(5,10),\ \max(-5,-1))^\top = (5,\ -1)^\top.

Derivation hint: apply Eq. [5] componentwise; only $g_1$ exceeds the bound. Direction changed: new ratio $5/(-1) = -5 \ne -10$ .

\boxed{\text{norm: }(4.98,-0.50),\ \text{direction kept};\quad \text{value: }(5,-1),\ \text{direction altered}}

⚠️ Common mistake

Clipping is not regularization. Regularization (e.g. $\ell_2$ weight decay) penalizes large parameters and changes the objective being minimized. Clipping leaves the objective $L$ untouched; it only modifies the optimizer's step. The minimum you converge toward is the same — you simply reach it more safely.

⚠️ Common mistake

The threshold $c$ must be tuned. Too small and learning crawls (every step is throttled, effectively a tiny learning rate); too large and clipping never activates, offering no protection. A common heuristic is to track the running statistics of $\|\mathbf g\|$ during a warm-up phase and set $c$ near a high percentile (e.g. the 90th) of observed norms.

CounterexampleClipping does not fix vanishing gradients

Suppose an RNN suffers the opposite pathology: gradients shrink to $\|\mathbf g\| = 10^{-9}$ as they propagate back in time. Clipping with any $c > 10^{-9}$ is completely inert — the gradient is already below threshold, so $\hat{\mathbf g} = \mathbf g$ . Clipping addresses exploding gradients only; the vanishing gradient problem requires different remedies such as gated architectures (Long Short-Term Memory) or skip connections, not clipping.

CounterexampleValue clipping can stall descent

Let $\mathbf g = (100, 0.001)^\top$ at a point where progress requires moving mainly along $\theta_2$ . Value clipping with $v=1$ gives $\hat{\mathbf g} = (1, 0.001)^\top$ — it crushes the dominant component but leaves the tiny one almost untouched, badly distorting the direction so the step barely advances along the axis that matters. Norm clipping would have preserved the $100{:}0.001$ ratio.

connection

Gradient clipping sits at the intersection of optimization and numerical stability. It is most often paired with Gradient Descent and its variants such as Stochastic Gradient Descent and Adam Optimizer, where it is applied to the raw gradient before adaptive moment estimation. It is the standard partner of Backpropagation Through Time, the algorithm that produces the exploding gradients in the first place. Conceptually it relates to trust-region methods, which also bound step size, and to Learning Rate Scheduling, since Eq. [8] shows clipping is a data-dependent learning-rate adjustment. It complements but does not replace Batch Normalization and Long Short-Term Memory, which attack instability at the architecture level rather than the optimizer level.

🔬 Where this appears

⏳ History

- 1991 — Sepp Hochreiter characterizes the vanishing/exploding gradient problem in his diploma thesis, identifying the exponential scaling through time.

- 1994 — Bengio, Simard & Frasconi publish a formal analysis showing learning long-term dependencies with gradient descent is fundamentally difficult.

- 1997 — Hochreiter & Schmidhuber introduce LSTM, an architectural fix for vanishing gradients (not exploding).

- 2010 — Tomas Mikolov popularizes RNN language models, where exploding gradients become a practical nuisance and ad hoc clipping is used informally.

- 2012 — Mikolov's PhD thesis documents gradient norm clipping as a working trick for RNN-LM training.

- 2013 — Pascanu, Mikolov & Bengio publish "On the difficulty of training Recurrent Neural Networks" (ICML), giving the cliff-geometry analysis and formalizing norm clipping (Eq. [3]).

- 2015 — Sutskever et al. and the seq2seq era adopt clipping as a default in neural machine translation.

- 2017 — WGAN (Arjovsky et al.) uses weight clipping for Lipschitz constraints, broadening the family of clipping ideas.

- 2019–2020 — GPT-2/GPT-3 (OpenAI) entrench global-norm clipping (

c=1.0

) as standard for large-scale transformer training.

open-problem

Several open questions remain. (1) Convergence theory: clipped SGD breaks the standard assumption that the optimizer follows the true gradient, so classical convergence proofs do not directly apply; recent work (Zhang et al., 2020) gives convergence guarantees under a relaxed smoothness assumption $\|\nabla^2 L\| \le L_0 + L_1\|\nabla L\|$ , but a complete picture is still developing. (2) Threshold selection: there is no universally principled way to choose $c$ ; it remains largely empirical and architecture-dependent. (3) Per-layer vs. global clipping: whether to clip each layer's gradient separately or the concatenated global vector affects training dynamics in ways not fully understood. (4) Interaction with adaptive optimizers: how clipping composes with Adam's per-parameter scaling, and whether they sometimes work at cross purposes, is an active area. (5) Bias introduction: clipping makes the expected update a biased estimate of the true gradient direction in stochastic settings, and quantifying this bias precisely is subtle.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also