Adam Optimizer

⚡ Key Takeaways

Adam (Adaptive Moment Estimation) is a first-order stochastic optimization algorithm that maintains exponentially decaying running averages of both the gradient (first moment) and its element-wise square (second moment), using these to compute per-parameter adaptive learning rates. Key result: combines the benefits of momentum and RMSProp with bias-corrected moment estimates, achieving robust convergence with minimal hyperparameter tuning. Where it applies: training deep neural networks, large-scale machine learning, non-convex stochastic optimization. Who/when: Diederik P. Kingma and Jimmy Ba, 2014 (ICLR 2015).

💡 The Core Insight

Different parameters in a high-dimensional model often have gradients of vastly different scales and statistics. Adam estimates, per coordinate, both the typical direction of descent (via an exponential moving average of gradients — the first moment) and the typical magnitude of fluctuations (via an exponential moving average of squared gradients — the second moment). Dividing the smoothed gradient by the square root of the smoothed squared gradient yields a signal-to-noise normalized step: coordinates with consistent gradients move confidently, while noisy coordinates are damped automatically. Bias correction repairs the systematic underestimation that arises from initializing the moving averages at zero.

DefinitionAdam Update Rule

Let $f(\theta)$ be a stochastic objective with parameters $\theta \in \mathbb{R}^d$ , and let $g_t = \nabla_\theta f_t(\theta_{t-1})$ be the stochastic gradient at step $t$ . Adam maintains two exponential moving averages, $m_t$ (first moment) and $v_t$ (second raw moment), and updates parameters as follows. The first moment estimate:

1

\quad m_t = \beta_1 m_{t-1} + (1-\beta_1) g_t

Derivation hint: this is an exponential moving average; expand recursively to see $m_t = (1-\beta_1)\sum_{i=1}^{t}\beta_1^{t-i} g_i$ . The second raw moment estimate (element-wise square $g_t^2 = g_t \odot g_t$ ):

2

\quad v_t = \beta_2 v_{t-1} + (1-\beta_2) g_t^2

Derivation hint: identical EMA structure applied to squared gradients; $v_t = (1-\beta_2)\sum_{i=1}^{t}\beta_2^{t-i} g_i^2$ . Bias-corrected estimates:

3

\quad \hat{m}_t = \frac{m_t}{1-\beta_1^t}, \qquad \hat{v}_t = \frac{v_t}{1-\beta_2^t}

Derivation hint: take expectation of $m_t$ assuming stationary gradients; $\mathbb{E}[m_t]=(1-\beta_1^t)\mathbb{E}[g_t]$ , so divide by $(1-\beta_1^t)$ to remove bias. The parameter update:

4

\quad \theta_t = \theta_{t-1} - \alpha \, \frac{\hat{m}_t}{\sqrt{\hat{v}_t} + \epsilon}

Derivation hint: divide the bias-corrected mean step by the bias-corrected RMS magnitude; $\epsilon$ prevents division by zero.

Symbols: $\alpha$ = step size (learning rate, default $10^{-3}$ ); $\beta_1 \in [0,1)$ = first-moment decay (default $0.9$ ); $\beta_2 \in [0,1)$ = second-moment decay (default $0.999$ ); $\epsilon$ = numerical stabilizer (default $10^{-8}$ ); $\odot$ = Hadamard (element-wise) product; all divisions and square roots are element-wise. Initialization: $m_0 = 0$ , $v_0 = 0$ .

TheoremEffective Step Bound

Assuming $\epsilon = 0$ , the magnitude of the effective step taken by Adam in each coordinate satisfies a bound:

5

\quad \left| \Delta\theta_t \right| = \alpha \left| \frac{\hat{m}_t}{\sqrt{\hat{v}_t}} \right| \le \alpha \cdot \max\!\left(1, \frac{1-\beta_1}{\sqrt{1-\beta_2}}\right)

In the common case $1-\beta_1 \ge \sqrt{1-\beta_2}$ (e.g. defaults: $0.1 \ge 0.0316$ ), the bound is $|\Delta\theta_t| \le \alpha$ , establishing a trust region independent of gradient scale.

ProofBias Correction Derivation

We show that the raw moving average $v_t$ underestimates the true second moment, and that the correction in equation [3] removes this bias exactly under a stationarity assumption.

Step 1. Unroll the recursion in equation [2] from $v_0 = 0$ :

6

\quad v_t = (1-\beta_2) \sum_{i=1}^{t} \beta_2^{\,t-i}\, g_i^2

Derivation hint: substitute $v_{t-1}$ repeatedly; the geometric weights $\beta_2^{t-i}$ and the leading $(1-\beta_2)$ factor follow by induction.

Step 2. Take the expectation, and assume the second moment is approximately stationary, $\mathbb{E}[g_i^2] \approx \mathbb{E}[g_t^2]$ :

7

\quad \mathbb{E}[v_t] = (1-\beta_2)\,\mathbb{E}[g_t^2] \sum_{i=1}^{t}\beta_2^{\,t-i} + \zeta = \mathbb{E}[g_t^2]\,(1-\beta_2^t) + \zeta

where $\zeta$ is small if the stationarity assumption holds (it absorbs the drift of $\mathbb{E}[g_i^2]$ across time). Derivation hint: the finite geometric series $\sum_{i=1}^t \beta_2^{t-i} = \frac{1-\beta_2^t}{1-\beta_2}$ cancels the leading $(1-\beta_2)$ .

Step 3. The factor $(1-\beta_2^t) < 1$ is the source of bias — early in training ( $t$ small) it can be far below $1$ , severely shrinking $v_t$ . Dividing by it yields an unbiased estimate:

8

\quad \mathbb{E}[\hat v_t] = \mathbb{E}\!\left[\frac{v_t}{1-\beta_2^t}\right] \approx \mathbb{E}[g_t^2]

Derivation hint: divide equation [7] by $(1-\beta_2^t)$ ; the leading term becomes $\mathbb{E}[g_t^2]$ exactly. The identical argument applies to $\hat m_t$ . $\blacksquare$

∎

DP

Diederik P. Kingma & Jimmy Lei Ba

2014

Diederik Kingma (then a PhD student at the University of Amsterdam under Max Welling) and Jimmy Ba (then at the University of Toronto under Geoffrey Hinton) introduced Adam in a paper submitted to ICLR 2015. Kingma is also a principal architect of the variational autoencoder (VAE) and normalizing flows; Ba contributed foundational work on layer normalization and attention. The name "Adam" derives from "Adaptive Moment estimation" and is not an acronym in the conventional sense, nor a person's name. Their paper provided both the practical algorithm and an attempted regret-based convergence analysis in the online convex optimization framework of Zinkevich.

🧠 Intuition

Picture rolling a ball down a noisy, anisotropic landscape. Momentum ( $m_t$ ) gives the ball inertia: it accumulates velocity in consistent directions, smoothing over the jitter of mini-batch noise. The second moment ( $v_t$ ) acts like a per-direction shock absorber: in directions where the gradient is large and erratic, $\sqrt{v_t}$ is large, so the update is shrunk; in flat, quiet directions, $\sqrt{v_t}$ is small, so the step grows. The ratio $\hat m_t / \sqrt{\hat v_t}$ is essentially a signal-to-noise ratio. When gradients agree across mini-batches, $\hat m_t \approx \sqrt{\hat v_t}$ and the ratio is near $\pm 1$ — full-step confidence. When gradients are pure noise with zero mean, $\hat m_t \to 0$ while $\sqrt{\hat v_t}$ stays positive, so the ratio shrinks and Adam automatically slows down. This is why Adam needs so little manual tuning: it self-normalizes its step to the local geometry. Bias correction matters most in the first few dozen steps, where without it the zero-initialized averages would make the algorithm take dangerously large early steps.

Figure

Adam's adaptive step vs. SGD on an ill-conditioned valley

ExampleSingle-Step Update with Numbers

Given: a single parameter $\theta_0 = 1.000$ , first gradient $g_1 = 0.5$ , defaults $\alpha = 10^{-3}$ , $\beta_1 = 0.9$ , $\beta_2 = 0.999$ , $\epsilon = 10^{-8}$ , with $m_0 = v_0 = 0$ .

Step 1 — first moment: $m_1 = 0.9(0) + 0.1(0.5) = 0.05$ .

Step 2 — second moment: $v_1 = 0.999(0) + 0.001(0.25) = 0.00025$ .

Step 3 — bias correction ( $t=1$ ): $1-\beta_1^1 = 0.1$ , $1-\beta_2^1 = 0.001$ .

\hat m_1 = \frac{0.05}{0.1} = 0.5, \qquad \hat v_1 = \frac{0.00025}{0.001} = 0.25

Note that after correction, $\hat m_1 = g_1$ and $\hat v_1 = g_1^2$ exactly — the first step recovers the raw gradient and its square.

Step 4 — update:

\theta_1 = 1.000 - 10^{-3}\cdot\frac{0.5}{\sqrt{0.25}+10^{-8}} = 1.000 - 10^{-3}\cdot\frac{0.5}{0.5} = 1.000 - 0.001

\boxed{\theta_1 = 0.999}

The effective step is exactly $\alpha = 10^{-3}$ — the trust-region bound of equation [5] is saturated on the very first step regardless of the gradient's actual size.

ExampleNoise Suppression Over Many Steps

Given: suppose the gradient at a coordinate is zero-mean noise: it equals $+1$ on odd steps and $-1$ on even steps (so the true descent signal is zero, but the magnitude is constant). Compute the steady-state behavior with defaults.

Second moment (steady state): since $g_t^2 = 1$ always, $v_t \to 1$ , hence $\sqrt{\hat v_t} \to 1$ .

First moment (steady state): $m_t$ oscillates around $0$ . After many steps, the EMA of an alternating $\pm 1$ with $\beta_1 = 0.9$ has amplitude

|m_\infty| = \frac{(1-\beta_1)}{1+\beta_1} = \frac{0.1}{1.9} \approx 0.0526

(geometric series of alternating signs). After bias correction (factor $\approx 1$ at large $t$ ), $|\hat m_t| \approx 0.0526$ .

Effective step:

|\Delta\theta| = \alpha\,\frac{|\hat m_t|}{\sqrt{\hat v_t}} \approx 10^{-3}\cdot\frac{0.0526}{1.0} \approx 5.3\times10^{-5}

\boxed{|\Delta\theta| \approx 5.3\times10^{-5} \ll \alpha}

Compare: a consistent gradient of $+1$ would give $|\Delta\theta| \approx \alpha = 10^{-3}$ , roughly 19× larger. Adam automatically suppresses the step in the noisy direction — its signal-to-noise adaptivity in action.

⚠️ Adam Is Not Guaranteed to Converge

A widespread belief is that Adam's published convergence proof guarantees convergence on all convex problems. This is false. Reddi, Kale & Kumar (2018, "On the Convergence of Adam and Beyond") identified a flaw in the original regret analysis: the proof incorrectly assumed a positive-semidefinite quantity that can in fact be negative. The effective learning-rate matrix $\Gamma_t = \frac{\sqrt{\hat v_t}}{\alpha_t} - \frac{\sqrt{\hat v_{t-1}}}{\alpha_{t-1}}$ need not be PSD because $\hat v_t$ can decrease. They constructed simple convex problems where Adam converges to the worst possible solution.

CounterexampleA Convex Problem Where Adam Fails

Consider online linear optimization with the periodic loss sequence (period 3), $x \in [-1,1]$ :

f_t(x) = \begin{cases} C\,x & \text{if } t \bmod 3 = 1 \\ -x & \text{otherwise}\end{cases}

for large constant $C > 2$ . The optimal fixed decision is $x = -1$ (the rare large positive gradient $C$ should dominate the two small negative gradients $-1$ ). However, because Adam's $v_t$ heavily damps the rare large gradient (its squared value $C^2$ is averaged with weight $1-\beta_2$ and then divided by $\sqrt{\hat v_t}$ , which is dominated by the frequent $-1$ contributions), the effective influence of the informative gradient is reduced, and Adam drifts toward $x = +1$ — the opposite of optimal. This non-vanishing regret is impossible for plain SGD with decaying steps. The fix, AMSGrad, enforces $\hat v_t = \max(\hat v_{t-1}, v_t)$ so the denominator is non-decreasing, restoring the PSD property.

⚠️ Adam ≠ Best Generalization

Empirically, on some tasks (notably image classification with CNNs), well-tuned SGD-with-momentum generalizes better than Adam despite Adam achieving lower training loss faster. Wilson et al. (2017) argue adaptive methods can converge to flatter-but-worse minima or solutions with larger weight norm. This motivated decoupled weight decay (AdamW), where weight decay is applied directly to parameters rather than folded into the gradient, since standard L2 regularization interacts poorly with adaptive per-coordinate scaling.

connection

Adam sits at the confluence of several optimization lineages. Setting $\beta_2 \to 1$ and removing the second-moment normalization recovers classical Momentum (specifically, heavy-ball with EMA). Setting $\beta_1 = 0$ and using only the second moment gives essentially RMSProp, Hinton's unpublished method. The element-wise division by $\sqrt{\hat v_t}$ approximates a diagonal preconditioner, linking Adam to Newton's Method and quasi-Newton ideas, though Adam uses only first-order information and a crude diagonal curvature surrogate. Adam is a member of the broader family of Stochastic Gradient Descent methods and is analyzed within Online Convex Optimization via regret bounds. Its bias-correction step has a clean interpretation in terms of Exponential Moving Average estimators. The relationship to Natural Gradient Descent is partial: $\hat v_t$ resembles the diagonal of the empirical Fisher information, though this correspondence is only approximate and the subject of ongoing debate.

🔬 Where this appears

⏳ History

- 1951 — Robbins & Monro introduce the stochastic approximation framework, the theoretical bedrock of all SGD methods.

- 1964 — Polyak introduces the heavy ball momentum method, the ancestor of Adam's first-moment term.

- 2011 — Duchi, Hazan & Singer publish AdaGrad, the first widely used per-coordinate adaptive method, accumulating the full sum of squared gradients.

- 2012 — Tieleman & Hinton present RMSProp (in a Coursera lecture, never formally published), replacing AdaGrad's accumulation with an exponential moving average — directly the second-moment idea Adam reuses.

- 2012 — Zeiler publishes AdaDelta, another EMA-based adaptive method addressing AdaGrad's vanishing learning rate.

- December 2014 — Kingma & Ba post the Adam paper to arXiv; it is accepted to ICLR 2015, and rapidly becomes the most-cited optimization method in deep learning.

- 2016 — Dozat introduces Nadam, incorporating Nesterov-accelerated momentum into Adam.

- 2018 — Reddi, Kale & Kumar (ICLR) expose the convergence flaw and propose AMSGrad; Loshchilov & Hutter propose AdamW with decoupled weight decay.

- 2019–2021 — A wave of variants: RAdam (rectified variance), LAMB (layer-wise scaling for huge batches), AdaBelief, Lion; debate continues over Adam's generalization properties versus SGD.

open-problem

Several deep questions about Adam remain only partially resolved. (1) Generalization gap: there is still no complete theory explaining when and why SGD generalizes better than Adam, nor a sharp characterization of the minima each method selects in non-convex landscapes. (2) Non-convex convergence guarantees: rigorous convergence rates for Adam (and variants) on smooth non-convex objectives require additional assumptions (e.g. bounded gradients, specific $\beta$ relationships) and the constants are often vacuous; the precise role of $\epsilon$ in convergence is subtle. (3) The Fisher-information interpretation: whether $\sqrt{\hat v_t}$ should be viewed as a principled approximation to natural-gradient preconditioning, or merely an effective heuristic, is contested. (4) Hyperparameter coupling: the interaction between $\alpha$ , $\beta_2$ , weight decay, and learning-rate warmup is poorly understood theoretically, despite enormous practical importance — RAdam's analysis of early-training variance is one partial answer. (5) Optimal $\beta$ scaling with batch size and sequence length for billion-parameter models remains largely empirical.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also