AdamW and Decoupled Weight Decay

⚡ Key Takeaways

AdamW is an optimization algorithm for training deep neural networks that decouples L2 weight decay from the adaptive gradient update, applying the decay directly to the parameters rather than through the gradient. Key result in words: for adaptive methods like Adam, the common practice of adding an L2 penalty to the loss is NOT equivalent to true weight decay, because the adaptive denominator rescales the penalty; decoupling restores the intended uniform shrinkage and dramatically improves generalization. Where it applies: training transformers, CNNs, and virtually all large-scale modern deep learning. Who/when: Ilya Loshchilov and Frank Hutter, 2017 ("Decoupled Weight Decay Regularization," ICLR 2019).

💡 The Core Insight

L2 regularization and weight decay are the same thing only for optimizers with a coordinate-uniform effective learning rate (like vanilla SGD). For adaptive optimizers, the adaptive preconditioner couples them in a way that corrupts the regularization. The fix is to decouple: never let weight decay enter the moving averages or the adaptive rescaling — apply it as an independent, multiplicative shrinkage of the weights.

DefinitionL2 Regularization vs. Weight Decay

Consider minimizing a loss $f(\theta)$ . Two distinct mechanisms are often conflated:

L2 regularization augments the objective:

1

\quad f^{\text{reg}}(\theta) = f(\theta) + \frac{\lambda'}{2}\|\theta\|_2^2,\qquad \nabla f^{\text{reg}}(\theta) = \nabla f(\theta) + \lambda'\theta.

Derivation hint: differentiate the quadratic penalty; $\frac{d}{d\theta}\tfrac12\|\theta\|^2 = \theta$ .

Weight decay modifies the update rule directly, shrinking parameters by a multiplicative factor each step:

2

\quad \theta_{t} = (1-\eta\lambda)\,\theta_{t-1} - \eta\, g_t,

Derivation hint: take the SGD step $\theta_t = \theta_{t-1} - \eta g_t$ and prepend an explicit exponential shrinkage $(1-\eta\lambda)$ on the previous parameters. where $\eta$ is the learning rate, $\lambda$ the weight-decay coefficient, $\lambda'$ the L2 coefficient, and $g_t=\nabla f(\theta_{t-1})$ the loss gradient.

TheoremEquivalence under SGD

For plain stochastic gradient descent, L2 regularization with coefficient $\lambda'$ is exactly equivalent to weight decay with coefficient $\lambda = \lambda'$ . That is, the iterates produced by [1] under SGD coincide with those of [2].

Proof

Step 1. Apply SGD to the L2-regularized objective using gradient from [1]:

3

\quad \theta_t = \theta_{t-1} - \eta\big(g_t + \lambda'\theta_{t-1}\big).

Derivation hint: substitute $\nabla f^{\text{reg}} = g_t + \lambda'\theta_{t-1}$ into the SGD step.

Step 2. Distribute the $\eta$ and collect the $\theta_{t-1}$ terms:

4

\quad \theta_t = \theta_{t-1} - \eta g_t - \eta\lambda'\theta_{t-1} = (1-\eta\lambda')\theta_{t-1} - \eta g_t.

Derivation hint: factor out $\theta_{t-1}$ from $\theta_{t-1} - \eta\lambda'\theta_{t-1}$ .

Step 3. Equation [4] is identical to the weight-decay rule [2] with $\lambda=\lambda'$ . Hence the two are interchangeable for SGD. $\blacksquare$ The equivalence breaks once the gradient is preconditioned by a non-scalar matrix $M_t$ (e.g. Adam's diagonal adaptive scaling), because the L2 term $\lambda'\theta$ is then multiplied by $M_t$ while the decay term in [2] is not. We prove the failure below.

∎

DefinitionAdam

Adam maintains exponential moving averages of the gradient (first moment $m_t$ ) and squared gradient (second moment $v_t$ ), with decay rates $\beta_1,\beta_2\in[0,1)$ :

5

\quad m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t,\qquad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2,

Derivation hint: these are standard EMA recursions; $g_t^2$ is element-wise squaring. with bias-corrected estimates and the update

6

\quad \hat m_t = \frac{m_t}{1-\beta_1^t},\quad \hat v_t = \frac{v_t}{1-\beta_2^t},\qquad \theta_t = \theta_{t-1} - \eta\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.

Derivation hint: at step $t$ an EMA initialized at zero is biased toward zero by factor $(1-\beta^t)$ ; dividing removes the bias. The final term preconditions the gradient by the inverse RMS of recent gradients. Here $\epsilon$ is a small constant (e.g. $10^{-8}$ ) for numerical stability; all operations are element-wise.

DefinitionAdamW — Decoupled Weight Decay

AdamW applies the adaptive update to the pure loss gradient (no L2 term inside $m_t,v_t$ ) and adds decoupled decay:

7

\quad \theta_t = \theta_{t-1} - \eta\left(\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon} + \lambda\,\theta_{t-1}\right),

Derivation hint: take the Adam update [6], then subtract an additional $\eta\lambda\theta_{t-1}$ that is NOT passed through the $\sqrt{\hat v_t}$ preconditioner — this is the decoupling. equivalently, with a schedule multiplier $\eta_t = \eta\,s_t$ (where $s_t$ is the learning-rate schedule):

8

\quad \theta_t = (1-\eta_t\lambda)\,\theta_{t-1} - \eta_t\,\frac{\hat m_t}{\sqrt{\hat v_t}+\epsilon}.

Derivation hint: rewrite [7] grouping the $\theta_{t-1}$ terms; in the original paper the decay is scaled by the same schedule $s_t$ as the learning rate but is independent of $\eta$ 's base magnitude. Here $m_t,v_t$ are computed from $g_t=\nabla f(\theta_{t-1})$ without the $\lambda\theta$ term.

ProofWhy L2 ≠ Weight Decay under Adam

Step 1. "Adam with L2 regularization" feeds the regularized gradient $g_t^{\text{reg}}=g_t+\lambda'\theta_{t-1}$ into the moments. The update direction becomes

9

\quad \theta_t = \theta_{t-1} - \eta\,\frac{\widehat{m}_t^{\text{reg}}}{\sqrt{\widehat v_t^{\text{reg}}}+\epsilon},

Derivation hint: substitute $g_t^{\text{reg}}$ for $g_t$ throughout [5]–[6]; both moments now contain the $\lambda'\theta$ contamination. where both $\widehat m_t^{\text{reg}}$ and $\widehat v_t^{\text{reg}}$ depend on $\lambda'\theta_{t-1}$ .

Step 2. Approximate near convergence where the contribution of the penalty to the second moment is dominated by the genuine gradient magnitude. The effective decay applied to coordinate $i$ scales as

10

\quad \text{effective decay}_i \;\approx\; \frac{\eta\,\lambda'\,\theta_{t-1,i}}{\sqrt{\hat v_{t,i}}+\epsilon}.

Derivation hint: isolate the part of [9] proportional to $\theta_{t-1,i}$ , namely $\lambda'\theta_{t-1,i}$ in the numerator divided by the per-coordinate preconditioner $\sqrt{\hat v_{t,i}}$ .

Step 3. Coordinates with large historical gradients (large $\hat v_{t,i}$ ) receive less decay; coordinates with small gradients receive more. This per-coordinate distortion means the regularization no longer shrinks all weights uniformly — the intended global $\ell_2$ ball penalty is replaced by an irregular, gradient-dependent one.

Step 4. In AdamW [7], the decay term $\lambda\theta_{t-1}$ is added outside the preconditioner, so every coordinate is shrunk by exactly $\eta\lambda$ regardless of $\hat v_{t,i}$ . This restores uniform decay and recovers the true regularizing geometry. $\blacksquare$

∎

IL

Ilya Loshchilov & Frank Hutter

2017–2019

Working at the University of Freiburg's machine learning lab, Loshchilov and Hutter were studying learning-rate schedules (their earlier SGDR / warm-restart work) when they noticed that Adam consistently generalized worse than SGD with momentum on image classification — a folklore observation across the field. Rather than accept it, they diagnosed the cause: practitioners were copying L2 regularization code from SGD into Adam, unknowingly entangling it with the adaptive denominator. Their fix was almost embarrassingly simple — move two lines of code — yet it closed much of the Adam-vs-SGD generalization gap. The paper's clarity (it includes side-by-side pseudocode highlighting the decoupling line) made AdamW an instant default. Hutter is also a central figure in AutoML and hyperparameter optimization; Loshchilov later contributed to cosine annealing schedules now used universally.

🧠 Intuition

Think of weight decay as a spring pulling every parameter toward zero with force proportional to its displacement. In SGD, every coordinate feels the same spring stiffness, so the metaphor is faithful. Adam, however, gives each coordinate its own adaptive "gear ratio" $1/\sqrt{\hat v_i}$ — coordinates seeing big gradients move in low gear (small steps), small-gradient coordinates in high gear. When you bolt the L2 spring inside the gearbox, its restoring force gets divided by the same gear ratio: the parameters that already move slowly get pulled hard toward zero, and the fast-moving ones barely feel the spring at all. That's backwards — those big-gradient weights are often the important, high-magnitude ones you'd most want to regularize. AdamW bolts the spring outside the gearbox, so it pulls uniformly on everyone, as a regularizer should.

Figure

L2-regularization inside Adam's preconditioner distorts decay per-coordinate; AdamW applies it uniformly outside

ExampleSingle-coordinate decay distortion

Given: Learning rate $\eta=10^{-3}$ , intended decay $\lambda=0.1$ , weight $\theta=0.5$ . Two coordinates with bias-corrected second moments $\hat v_A = 10^{-4}$ (quiet coordinate) and $\hat v_B = 1.0$ (active coordinate), $\epsilon\approx 0$ .

Adam + L2 (coupled), effective decay from [10], $\text{decay}_i = \dfrac{\eta\lambda\theta}{\sqrt{\hat v_i}}$ :

Coordinate A: $\sqrt{\hat v_A}=10^{-2}$ , decay $= \dfrac{10^{-3}\cdot 0.1\cdot 0.5}{10^{-2}} = \dfrac{5\times10^{-5}}{10^{-2}} = 5\times10^{-3}$ .
Coordinate B: $\sqrt{\hat v_B}=1$ , decay $= \dfrac{10^{-3}\cdot 0.1\cdot 0.5}{1} = 5\times10^{-5}$ .

Ratio A:B $= 100{:}1$ — the quiet coordinate is decayed 100× more than the active one.

AdamW (decoupled), effective decay $=\eta\lambda\theta = 10^{-3}\cdot0.1\cdot0.5 = 5\times10^{-5}$ for both coordinates.

\boxed{\text{Adam+L2: } 5\times10^{-3} \text{ vs } 5\times10^{-5};\quad \text{AdamW: } 5\times10^{-5} \text{ uniformly (dimensionless weight units)}}

ExampleMultiplicative shrinkage over an epoch

Given: AdamW with constant schedule $s_t=1$ , $\eta=3\times10^{-4}$ , $\lambda=0.05$ , and a parameter whose gradient signal is negligible (so the adaptive term ≈ 0 for that coordinate). How much does it shrink over $N=10{,}000$ steps?

Step 1. Per-step shrink factor from [8]: $1-\eta\lambda = 1 - 3\times10^{-4}\cdot0.05 = 1 - 1.5\times10^{-5}$ .

Step 2. Over $N$ steps: $\theta_N/\theta_0 = (1-1.5\times10^{-5})^{10000} \approx e^{-1.5\times10^{-5}\cdot10^4} = e^{-0.15}$ .

Step 3. $e^{-0.15} \approx 0.861$ .

\boxed{\theta_N \approx 0.861\,\theta_0 \;\;\Rightarrow\;\; \approx 13.9\%\text{ shrinkage (dimensionless)}}

Note this shrinkage is independent of the adaptive denominator — exactly the clean, predictable behavior decoupling guarantees.

⚠️ Decay is NOT scaled by the adaptive denominator

The single most common bug is writing `grad = grad + wd param` before the moment updates in an Adam implementation and calling it "weight decay." That is <strong>L2 regularization</strong>, and under Adam it is not* weight decay. The PyTorch `torch.optim.Adam(weight_decay=...)` historically did exactly this (coupled L2); only `torch.optim.AdamW` implements true decoupling. Mixing them up silently changes your effective regularization by orders of magnitude across coordinates.

⚠️ Optimal λ differs between Adam-L2 and AdamW

Because the coupled and decoupled versions apply decay differently, the optimal weight-decay coefficient is different — you cannot transfer a tuned $\lambda$ from one to the other. Loshchilov & Hutter also observed that in AdamW, the optimal $\lambda$ and optimal $\eta$ are far more decoupled (nearly orthogonal in hyperparameter space), making grid search dramatically easier — a practical bonus of the method.

CounterexampleWhere L2 = weight decay still holds

If you replace Adam's adaptive denominator with a scalar (set $\hat v_t \equiv c$ , a constant shared across all coordinates), then the preconditioner is $1/\sqrt{c}$ everywhere and divides every term — including the L2 contribution — by the same factor. In that degenerate case the distinction collapses and coupled L2 again equals weight decay (with $\lambda = \lambda'/\sqrt c$ ). The pathology is specifically the per-coordinate, non-uniform nature of $\hat v_t$ . This confirms the proof's claim: it is the coordinate-dependence of the preconditioner, not adaptivity per se, that breaks the equivalence.

connection

AdamW sits at the intersection of optimization and regularization theory. It refines Adam by correcting how regularization interacts with its preconditioner, and generalizes the SGD identity that links L2 Regularization with Weight Decay. The decoupling principle echoes Proximal Gradient Methods: weight decay is precisely the proximal operator of the $\ell_2$ penalty, $\text{prox}_{\eta\lambda\|\cdot\|^2/2}(\theta) = \theta/(1+\eta\lambda) \approx (1-\eta\lambda)\theta$ , applied after the gradient step rather than folded into it — making AdamW a quasi-proximal/forward-backward scheme. The same coordinate-distortion analysis applies to any Adaptive Gradient Methods (RMSProp, Adagrad, Adadelta), each of which benefits from decoupling. The choice of learning-rate multiplier $s_t$ for the decay term connects to Learning Rate Scheduling and the authors' own Cosine Annealing / warm-restart work.

🔬 Where this appears

⏳ History

- 1988–1991 — Weight decay popularized as a neural-net regularizer (Krogh & Hertz, A Simple Weight Decay Can Improve Generalization, NeurIPS 1991), establishing the SGD identity.

- 2011 — Adagrad (Duchi, Hazan, Singer) introduces per-coordinate adaptive learning rates, planting the seed for the decoupling problem.

- 2012 — RMSProp (Tieleman & Hinton, Coursera lecture) introduces the EMA of squared gradients used by Adam.

- 2014 — Adam introduced by Kingma & Ba (Adam: A Method for Stochastic Optimization); its paper adds L2 via the gradient, cementing the coupled convention.

- 2016 — Loshchilov & Hutter publish SGDR (warm restarts / cosine annealing), the schedule research that led them toward Adam's deficiencies.

- 2017 (Nov) — Loshchilov & Hutter post Fixing Weight Decay Regularization in Adam (arXiv:1711.05101), introducing AdamW and AdamWR.

- 2018 — Wilson et al. (The Marginal Value of Adaptive Gradient Methods) document Adam's generalization gap vs. SGD — context AdamW directly addresses.

- 2019 — The revised paper, Decoupled Weight Decay Regularization, is published at ICLR 2019; AdamW becomes the recommended default.

- 2019–2020 — AdamW adopted by major frameworks and by transformer training recipes (BERT, GPT-2), becoming ubiquitous.

open-problem

Several questions remain open or actively debated. (1) Optimal decay placement under schedules: whether the decay multiplier should track the learning-rate schedule $s_t$ (original paper) or remain fixed is empirically inconsistent; "fully decoupled" variants and AdamW with constant $\lambda$ both appear in practice. (2) Which parameters to decay: the heuristic of excluding biases, LayerNorm/BatchNorm scale-and-shift parameters from decay is folklore with limited theory; its optimality is unproven. (3) Interaction with normalization: for scale-invariant weights (those followed by a normalization layer), weight decay's true effect is an effective learning-rate modulation rather than capacity control (Zhang et al., van Laarhoven), complicating the regularization interpretation entirely. (4) Convergence guarantees: like Adam, AdamW lacks clean convergence proofs on non-convex problems matching SGD's, and Adam's original convergence proof was later found flawed (Reddi et al., AMSGrad). (5) Generalization theory: why decoupled decay generalizes better is still not rigorously characterized beyond the per-coordinate uniformity argument; the precise relationship to flat minima and implicit bias is open.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also