Classifier-Free Guidance

Classifier-Free Guidance (CFG) is a technique for conditional generation in diffusion models that interpolates between conditional and unconditional score estimates at inference time to amplify the influence of a conditioning signal—without requiring a separately trained classifier. It is foundational to virtually every state-of-the-art text-to-image, text-to-video, and audio generation system deployed at scale, including Stable Diffusion, DALL·E 2/3, Imagen, and Sora. A counterintuitive fact: increasing guidance strength beyond a certain threshold provably degrades sample diversity and eventually fidelity, yet practitioners routinely operate in this "over-guided" regime because human raters systematically prefer sharper, more saturated outputs even when measurable quality metrics decline.

💡 The Core Insight

The key insight is that the gradient of a conditional log-likelihood can be decomposed into an unconditional score plus a term proportional to the difference between conditional and unconditional scores—and this difference can be amplified by a scalar guidance weight at inference time to steer generation toward high-probability regions of the conditional distribution. Rather than training a separate discriminative classifier (which requires noisy-image training and is fragile), CFG trains a single diffusion model that can operate in both conditional and unconditional modes by randomly dropping the conditioning signal during training. The "aha moment" is realizing that guidance is not a post-hoc trick but an exact manipulation of the score function implied by Bayes' rule, and that the guidance weight directly controls a bias–variance tradeoff between sample fidelity and diversity.

⚠️ Common Misconception

The most common mistake is treating the CFG scale ww as a simple "quality knob" with no downside—practitioners assume that higher ww always produces better images. This is wrong for two distinct reasons: first, at high ww the guided score function no longer corresponds to any valid probability distribution's score (it exits the probability simplex over data space), so samples can exhibit characteristic artifacts such as oversaturation, burnt highlights, and anatomical distortions; second, the effective temperature of the distribution is lowered, collapsing sample diversity. The correct mental model is that CFG performs implicit rejection sampling in score space, trading recall (coverage of the true conditional distribution) for precision (concentration on the highest-density modes)—analogous to reducing temperature in an autoregressive model, but applied at every denoising step rather than at token logits.

How It Works

Derivation from Bayes' Rule

Let pθ(xt∣c)p_\theta(\mathbf{x}_t | \mathbf{c}) denote the conditional noisy data distribution at diffusion timestep tt, and pθ(xt)p_\theta(\mathbf{x}_t) the marginal. By Bayes' rule:

1
log⁡p(c∣xt)=log⁡p(xt∣c)−log⁡p(xt)+const\quad \log p(\mathbf{c} \mid \mathbf{x}_t) = \log p(\mathbf{x}_t \mid \mathbf{c}) - \log p(\mathbf{x}_t) + \text{const}

Explanation: The log-posterior of the conditioning signal given a noisy image equals the difference of conditional and unconditional log-likelihoods, which is the quantity CFG implicitly manipulates. Taking gradients with respect to xt\mathbf{x}_t yields the classifier gradient:

2
∇xtlog⁡p(c∣xt)=∇xtlog⁡p(xt∣c)−∇xtlog⁡p(xt)\quad \nabla_{\mathbf{x}_t} \log p(\mathbf{c} \mid \mathbf{x}_t) = \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t \mid \mathbf{c}) - \nabla_{\mathbf{x}_t} \log p(\mathbf{x}_t)

Explanation: The score of the classifier is exactly the difference between the conditional and unconditional score functions, a relation that holds without ever training a classifier explicitly. In the Denoising Diffusion Probabilistic Models (DDPM) framework, score functions are parameterized via the predicted noise ϵθ\boldsymbol{\epsilon}_\theta. The relationship between score and noise predictor is:

3
∇xtlog⁡pθ(xt)=−ϵθ(xt,t)σt\quad \nabla_{\mathbf{x}_t} \log p_\theta(\mathbf{x}_t) = -\frac{\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t)}{\sigma_t}

Explanation: Up to a sign and the noise schedule's standard deviation σt\sigma_t, the neural network's noise prediction is a scaled score function, which is why guidance can be applied directly to noise predictions. The CFG modified noise prediction is then formed by linearly extrapolating away from the unconditional estimate toward the conditional estimate:

4
ϵ~θ(xt,t,c)=ϵθ(xt,t,∅)+w(ϵθ(xt,t,c)−ϵθ(xt,t,∅))\quad \tilde{\boldsymbol{\epsilon}}_\theta(\mathbf{x}_t, t, \mathbf{c}) = \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing) + w \bigl(\boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \mathbf{c}) - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \varnothing)\bigr)

Explanation: Here w≥0w \geq 0 is the guidance scale (often called CFG scale or guidance strength), ∅\varnothing denotes the null conditioning token, and w=1w=1 recovers ordinary conditional sampling while w=0w=0 gives unconditional sampling. This is equivalent to sampling from a sharpened distribution. Specifically, when w>1w > 1, the guided score corresponds to the score of the tilted distribution:

5
p~θ(xt∣c)∝pθ(xt∣c)1+w⋅pθ(xt)−w\quad \tilde{p}_\theta(\mathbf{x}_t \mid \mathbf{c}) \propto p_\theta(\mathbf{x}_t \mid \mathbf{c})^{1+w} \cdot p_\theta(\mathbf{x}_t)^{-w}

Explanation: This power-law tilting raises the conditional density to a power greater than one, sharpening modes but simultaneously creating negative exponents on the marginal that can yield invalid (unnormalized) densities when ww is large, explaining the artifacts observed at high guidance scales.

Training Procedure

During training, the conditioning signal c\mathbf{c} (e.g., a CLIP text embedding or a class label) is replaced by the null token ∅\varnothing with probability puncondp_{\text{uncond}} (typically 0.1–0.2). The single network ϵθ\boldsymbol{\epsilon}_\theta thereby learns both conditional and unconditional denoising simultaneously, amortizing the cost of what would otherwise require two separate networks. The training objective is the standard Evidence Lower Bound loss:

6
LCFG=Et,x0,ϵ,c′[∥ϵ−ϵθ(xt,t,c′)∥2]\quad \mathcal{L}_{\text{CFG}} = \mathbb{E}_{t, \mathbf{x}_0, \boldsymbol{\epsilon}, \mathbf{c}'}\bigl[\|\boldsymbol{\epsilon} - \boldsymbol{\epsilon}_\theta(\mathbf{x}_t, t, \mathbf{c}')\|^2\bigr]

Explanation: Here c′∈{c,∅}\mathbf{c}' \in \{\mathbf{c}, \varnothing\} is the (randomly dropped) conditioning, ϵ∼N(0,I)\boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) is the ground-truth noise, and xt=αˉtx0+1−αˉtϵ\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\boldsymbol{\epsilon} follows the forward diffusion schedule.

researcher · Jonathan Ho | b. 1993 | Inventor of Classifier-Free Guidance and DDPM lead author

Jonathan Ho completed his PhD at UC Berkeley advised by Pieter Abbeel before joining Google Brain, where he developed the DDPM framework and subsequently proposed Classifier-Free Guidance in a workshop paper at NeurIPS 2021. His CFG paper ("Classifier-Free Diffusion Guidance," 2022) is among the most practically impactful workshop papers in modern deep learning, forming the inference-time backbone of Imagen, Imagen Video, and dozens of subsequent systems. Remarkably, the CFG paper was initially rejected from the main NeurIPS track before being accepted as a workshop contribution, yet it has accumulated thousands of citations and is now considered foundational.

researcher · Tim Salimans | b. 1983 | Co-developer of improved diffusion training and guidance analysis

Tim Salimans, a research scientist at Google Brain, collaborated closely with Jonathan Ho on diffusion model training improvements and contributed foundational analysis of the precision–recall tradeoff induced by guidance. His earlier work on OpenAI's improved GAN training (2016) and progressive growing of GANs established his expertise in generative model stability. Salimans was one of the first to formally characterize the diversity collapse at high guidance weights through the lens of FID/CLIP score Pareto frontiers, giving practitioners a rigorous framework for hyperparameter selection.

Figure
Classifier-Free Guidance Inference Flow
Classifier-Free Guidance — Inference Step Noisy image x_t , timestep t Condition c (text embedding) Null token ∅ (unconditional) ε_θ(x_t,t,c) cond. noise pred ε_θ(x_t,t,∅) uncond. noise pred CFG combine ε̃ w (scale) ε̃ = ε_θ(x_t,t,∅) + w · [ ε_θ(x_t,t,c) − ε_θ(x_t,t,∅) ] w=1 → conditional sample | w>1 → amplified guidance | w=0 → unconditional
Figure
Precision–Recall Tradeoff Under Guidance
CFG Scale vs. Fidelity / Diversity Tradeoff Guidance scale w → Metric value 0 2 4 6 10 15 Fidelity (CLIP) Diversity (recall) optimal w ≈ 5–8 artifact zone

Intuition

🧠 The Mental Model

Imagine a cartographer drawing a mountain range from two photographs: one taken in dense fog (the unconditional model, which has seen countless landscapes and averages over them all) and one taken on a clear day from a specific angle specified by a caption (the conditional model). Classifier-Free Guidance instructs the cartographer to subtract the foggy photograph from the clear one and then exaggerate the difference—amplifying the peaks, deepening the valleys, and making the scene more dramatically match the caption. At low amplification (w≈1w \approx 1), the drawing is faithful to the actual landscape; at moderate amplification (w≈5w \approx 5–8), the peaks look strikingly clear and well-defined; at extreme amplification (w>15w > 15), the cartographer has extrapolated so far from reality that the mountains grow impossibly sharp, colors become lurid, and the drawing no longer represents a plausible landscape at all. The guidance scale is therefore not a quality dial but an extrapolation dial—it controls how far outside the support of the true conditional distribution you are willing to sample.

Historical Development

⏳ History
2015
Sohl-Dickstein et al. introduce non-equilibrium thermodynamics–based diffusion probabilistic models, establishing the forward/reverse process framework that CFG later exploits.
2020
Ho et al. ("Denoising Diffusion Probabilistic Models," NeurIPS 2020) reformulate diffusion models with noise-prediction objectives and demonstrate high-quality unconditional image synthesis, providing the score-function parameterization that CFG directly modifies.
2021
Dhariwal & Nichol ("Diffusion Models Beat GANs," NeurIPS 2021) introduce Classifier Guidance, training a separate classifier on noisy images and using its gradient to steer sampling—demonstrating that guidance dramatically improves FID but requiring a fragile auxiliary model and noisy-image classifier training.
2021
Jonathan Ho & Tim Salimans present "Classifier-Free Diffusion Guidance" at the NeurIPS 2021 Deep Generative Models Workshop, showing that conditioning dropout during training enables guidance without any auxiliary classifier, achieving equal or better FID/IS scores.
2022
Ramesh et al. ("Hierarchical Text-Conditional Image Generation with CLIP Latents," OpenAI) deploy CFG inside DALL·E 2, the first large-scale consumer product built on the technique; they use w≈3w \approx 3 for the prior and w≈3w \approx 3–5 for the decoder.
2022
Saharia et al. ("Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding," Google Brain) introduce Imagen with w≈7.5w \approx 7.5, establishing the typical operating range now standard in the field and introducing dynamic thresholding to mitigate saturation artifacts.
2022
Rombach et al. ("High-Resolution Image Synthesis with Latent Diffusion Models," CVPR 2022) release Stable Diffusion applying CFG in the latent space of a VQ-regularized autoencoder, making high-resolution CFG-guided generation computationally accessible on consumer GPUs.
2023
Kynkäänniemi et al. formally analyze CFG as a distribution-sharpening operation, deriving the exact precision–recall Pareto frontier it induces and proving that no guidance scale simultaneously maximizes both metrics.
2024
Research into Autoguidance (Bradley et al., 2024) and Perturbed-Attention Guidance proposes replacing the unconditional branch with corrupted or attention-degraded versions of the same model, addressing diversity collapse while retaining fidelity gains—signaling active evolution beyond standard CFG.

Why It Matters

🔬 Real-World Impact
Stable Diffusion (v1.x–SDXL)Uses CFG with w∈[5,12]w \in [5, 12] applied in the latent space of a 4× downsampled VAE; SDXL further introduces a two-stage pipeline where both base and refiner models each apply CFG independently, compounding conditioning signal.
DALL·E 2 (OpenAI, 2022)Applies CFG at two stages—a CLIP-conditioned prior that maps text embeddings to image embeddings (w≈3w \approx 3), and a pixel-space decoder (w≈3w \approx 3–5)—demonstrating that CFG composes across hierarchical diffusion pipelines.
Imagen and Imagen Video (Google Brain, 2022–2023)Introduces dynamic thresholding (clipping the predicted x0\mathbf{x}_0 at each step) on top of CFG at w=7.5w = 7.5, resolving the saturation artifact problem and enabling guidance scales that were previously infeasible; this combination became the dominant approach for video diffusion.
Sora (OpenAI, 2024)Applies CFG in a spatiotemporal diffusion transformer over video latents, with conditioning derived from T5-XXL text encodings; the unusually long video coherence is partly attributed to high guidance scales maintaining strong text alignment across hundreds of frames.
AudioLDM and MusicGen (2023)Adapts CFG to the mel-spectrogram latent space for text-conditioned audio synthesis; guidance scale must be tuned differently than for images (w≈2w \approx 2–4) because audio latent manifolds have lower intrinsic dimensionality and saturate faster.
ControlNet (Zhang et al., 2023)Combines CFG with spatially-conditioned control signals (depth maps, edge maps, pose skeletons); CFG operates on the text branch while ControlNet provides structural conditioning, demonstrating that multiple guidance mechanisms compose additively in score space.
InstructPix2Pix (Brooks et al., 2023)Extends CFG to two simultaneous conditions (image conditioning and instruction text) by maintaining three forward passes—unconditioned, image-only conditioned, and fully conditioned—and combining them with two independent guidance weights, illustrating CFG's natural composability.

Prerequisites

What This Leads To

See Also

Classifier Guidance · Denoising Diffusion Probabilistic Models · Score-Based Generative Models · Latent Diffusion Models · CLIP and Contrastive Learning

Before this