Large Language Models

⚡ Key Takeaways

Large Language Models (LLMs) are neural networks—predominantly based on the Transformer architecture—trained on vast text corpora to model the probability distribution of token sequences, enabling them to generate and understand natural language. Key result: by minimizing next-token prediction loss over billions of tokens, a single model acquires broad linguistic, factual, and reasoning capabilities that scale predictably with model size, data, and compute. Applies to: text generation, translation, question answering, code synthesis, summarization. Who/when: rooted in n-gram models (1950s–1980s), revolutionized by the Transformer (Vaswani et al., 2017) and scaled by GPT/BERT-class models (2018–present).

💡 The Core Insight

The entire enterprise reduces to a deceptively simple idea: language modeling is compression of structured knowledge into a conditional probability. If you can reliably predict the next word given everything before it, you must have implicitly modeled syntax, semantics, facts, and inference patterns. The Transformer makes this tractable by replacing recurrence with self-attention, allowing every token to directly attend to every other token in parallel, and scale makes it powerful.

DefinitionAutoregressive Language Model

A language model defines a joint probability over a token sequence $x = (x_1, \dots, x_n)$ , where each $x_i \in \mathcal{V}$ (a finite vocabulary). By the chain rule of probability:

1

\quad p_\theta(x_1, \dots, x_n) = \prod_{i=1}^{n} p_\theta(x_i \mid x_1, \dots, x_{i-1})

Derivation hint: Apply the chain rule of conditional probability iteratively; no independence assumption is made, only an ordering. Here $\theta$ denotes the model's learnable parameters, $\mathcal{V}$ is the vocabulary (typically $|\mathcal{V}| \approx 30\text{k}{-}100\text{k}$ subword tokens), and $p_\theta(x_i \mid x_{<i})$ is a categorical distribution produced by a softmax over $\mathcal{V}$ .

DefinitionTraining Objective: Cross-Entropy Loss

The model is trained by minimizing the average negative log-likelihood (cross-entropy) over a corpus $\mathcal{D}$ :

2

\quad \mathcal{L}(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(x_i \mid x_{<i})

Derivation hint: Take the negative logarithm of the likelihood [1] and average over $N$ tokens; minimizing this maximizes the data likelihood. where $N$ is the total number of training tokens. The exponentiated per-token loss defines perplexity:

3

\quad \text{PPL}(\theta) = \exp\!\left(-\frac{1}{N}\sum_{i=1}^{N} \log p_\theta(x_i \mid x_{<i})\right)

Derivation hint: Perplexity is $\exp$ of cross-entropy; it equals the effective average branching factor—the number of equally likely choices the model faces per token.

DefinitionSelf-Attention

The core computation. Given input embeddings packed as rows of $X \in \mathbb{R}^{n \times d}$ , we compute queries, keys, and values via learned projections $W_Q, W_K, W_V \in \mathbb{R}^{d \times d_k}$ :

4

\quad Q = XW_Q, \quad K = XW_K, \quad V = XW_V

Derivation hint: These are linear maps; each row of $Q, K, V$ corresponds to one token's query/key/value vector. The attention output is a weighted average of values, weighted by query–key compatibility:

5

\quad \text{Attention}(Q,K,V) = \text{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V

Derivation hint: $QK^\top$ gives pairwise dot-product similarities $(n\times n)$ ; dividing by $\sqrt{d_k}$ controls variance growth; softmax normalizes each row to a probability distribution over tokens.

TheoremUniversal Approximation of Sequence-to-Sequence Maps

Transformers with sufficient depth and width are universal approximators of continuous, permutation-equivariant sequence-to-sequence functions on compact domains (Yun et al., 2020). Adding positional encodings removes permutation equivariance, enabling approximation of arbitrary continuous sequence functions to any desired accuracy $\epsilon$ .

⚠️ Common mistake

The theorem guarantees representational capacity, not learnability. A function being approximable says nothing about whether gradient descent on finite data will find the right parameters. Most of an LLM's practical capability comes from data and optimization, not from existence proofs.

Proof

Why next-token prediction yields general capability — a sketch. 1. Factorization is exact. Equation [1] involves no approximation: any distribution over sequences factors via the chain rule. The model's only job is to approximate each conditional $p_\theta(x_i \mid x_{<i})$ . 2. Cross-entropy is a proper scoring rule. Minimizing [2] is equivalent to minimizing the KL divergence $D_{\text{KL}}(p_{\text{data}} \,\|\, p_\theta)$ up to a constant (the data entropy). Thus the global optimum is $p_\theta = p_{\text{data}}$ .

6

\quad \mathcal{L}(\theta) = H(p_{\text{data}}) + D_{\text{KL}}(p_{\text{data}} \,\|\, p_\theta)

Derivation hint: Expand cross-entropy $H(p, q) = -\sum p \log q$ as $H(p) + D_{\text{KL}}(p\|q)$ ; the first term is independent of $\theta$ . 3. Information bottleneck forces abstraction. To predict well across diverse contexts with finite parameters, the model cannot memorize every sequence; it must compress regularities—grammar, coreference, factual associations—into shared representations. 4. Attention enables long-range structure. Equation [5] lets any position aggregate information from all others in $O(1)$ sequential steps (vs. $O(n)$ for RNNs), so dependencies across long contexts are learnable without vanishing gradients. 5. Scale closes the gap. Empirically, increasing $\theta$ , data, and compute drives $\mathcal{L}$ smoothly toward $H(p_{\text{data}})$ , and the resulting better conditional models manifest as qualitatively new behaviors. $\blacksquare$

∎

Figure

Transformer decoder block and the autoregressive prediction loop

🧠 Intuition

Think of an LLM as an extraordinarily well-read autocomplete. Imagine someone who has read most of the internet and, when shown any partial text, can guess what comes next based on patterns they've absorbed. The "masked" attention enforces causality: when predicting token $i$ , the model may only look at tokens $1$ through $i-1$ (the triangular mask in [5] sets future attention weights to $-\infty$ before softmax). The key conceptual leap is that prediction requires understanding. To finish "The capital of France is ___" correctly, the model must encode a fact. To finish " $2 + 2 =$ ___" it must encode arithmetic. The richer and more varied the contexts it learns to predict, the broader the implicit knowledge it must store. Attention is the mechanism that lets the model dynamically decide which earlier words matter for the current prediction—like a reader who, encountering "it," instantly looks back to find the noun "it" refers to.

CS

Claude Shannon

1916–2001

Though not an LLM researcher, Shannon laid the foundation in his 1948 A Mathematical Theory of Communication and his 1951 paper Prediction and Entropy of Printed English. He framed language as a stochastic process, estimated the entropy of English at roughly 1 bit per character, and conducted experiments where humans guessed the next letter—the conceptual ancestor of next-token prediction. Every modern LLM minimizes precisely the cross-entropy Shannon defined, and perplexity [3] is the direct operationalization of his entropy of language.

ExampleComputing Perplexity of a Tiny Model

Given: A model assigns the following probabilities to a 3-token sequence "the cat sat":

$p(\text{"the"}) = 0.5$
$p(\text{"cat"} \mid \text{"the"}) = 0.1$
$p(\text{"sat"} \mid \text{"the cat"}) = 0.2$

Find: the sequence log-likelihood and perplexity.

Steps: 1. Sum of log-probabilities (natural log): $\log 0.5 + \log 0.1 + \log 0.2 = -0.693 - 2.303 - 1.609 = -4.605$ 2. Average negative log-likelihood (cross-entropy), $N=3$ : $\mathcal{L} = -\frac{1}{3}(-4.605) = 1.535$ nats/token 3. Perplexity via [3]: $\text{PPL} = e^{1.535} = 4.64$

Answer: $\boxed{\text{PPL} \approx 4.64 \text{ tokens}}$ — the model is, on average, as uncertain as if choosing uniformly among ~4.6 options per token.

ExampleAttention Weight for Two Tokens

Given: Two tokens with query/key dimension $d_k = 4$ . For predicting the current token, its query is $q = (1, 0, 1, 0)$ . Two candidate keys: $k_1 = (1, 0, 1, 0)$ (a matching past token) and $k_2 = (0, 1, 0, 1)$ (an unrelated one).

Find: the attention weights assigned to value 1 vs. value 2.

Steps: 1. Scaled dot products ( $\sqrt{d_k} = 2$ ): $s_1 = \frac{q \cdot k_1}{2} = \frac{1+0+1+0}{2} = 1.0$ $s_2 = \frac{q \cdot k_2}{2} = \frac{0+0+0+0}{2} = 0.0$ 2. Softmax over $(s_1, s_2) = (1.0, 0.0)$ : $\alpha_1 = \frac{e^{1.0}}{e^{1.0}+e^{0.0}} = \frac{2.718}{3.718} = 0.731$ $\alpha_2 = \frac{1}{3.718} = 0.269$

Answer: $\boxed{\alpha_1 = 0.73,\ \alpha_2 = 0.27}$ — the model places 73% of its attention on the semantically matching token, illustrating how dot-product similarity routes information.

⚠️ Common mistake

"LLMs look up answers in a database." They do not. After training, the corpus is discarded; all "knowledge" is compressed into the weights $\theta$ as continuous parameters. The model regenerates text statistically, which is precisely why it can also hallucinate—produce fluent but false statements—since there is no factual lookup, only probability maximization.

⚠️ Common mistake

"Bigger context window = better memory." The context window is the number of tokens the model can attend to at once. Beyond it, the model has no access to earlier text. Moreover, attention is $O(n^2)$ in context length $n$ , so doubling context quadruples attention cost—a hard practical limit.

Counterexample

Reversal failure (the "reversal curse"). A model trained on "Tom Cruise's mother is Mary Lee Pfeiffer" often cannot answer "Who is Mary Lee Pfeiffer's son?" This exposes that LLMs learn directional token associations, not symmetric logical relations. It refutes the naive assumption that learning a fact implies learning its logically equivalent inverse—next-token prediction optimizes [2] in one direction only.

connection

LLMs sit at the intersection of several fields. The objective [2] is grounded in Information Theory (cross-entropy, perplexity = Shannon entropy). The optimization relies on Stochastic Gradient Descent and Backpropagation to compute $\nabla_\theta \mathcal{L}$ . The architecture builds on Neural Networks and specifically the Transformer Architecture and its Attention Mechanism. The probabilistic framing connects to Maximum Likelihood Estimation and Markov Chains (n-gram models being the finite-order special case). Sampling generation strategies relate to Monte Carlo Methods.

🔬 Where this appears

⏳ History

- 1948 — Claude Shannon formalizes language as a stochastic source; introduces entropy.

- 1980s–1990s — n-gram statistical language models dominate speech recognition and translation (IBM models).

- 2003 — Bengio et al. introduce the neural probabilistic language model, learning distributed word representations.

- 2013 — Mikolov et al. release word2vec, popularizing dense word embeddings.

- 2014–2015 — Sequence-to-sequence learning (Sutskever et al.) and the attention mechanism (Bahdanau et al.) for translation.

- 2017 — Vaswani et al. publish Attention Is All You Need, introducing the Transformer—dispensing with recurrence entirely.

- 2018 — BERT (Google, bidirectional) and GPT (OpenAI, autoregressive) launch the pretrain-then-fine-tune era.

- 2020 — GPT-3 (175B parameters) demonstrates in-context/few-shot learning; Kaplan et al. publish scaling laws.

- 2022–2023 — InstructGPT and ChatGPT add RLHF; GPT-4, PaLM, LLaMA push capability and accessibility; emergent abilities and alignment become central topics.

open-problem

Scaling laws and their ceiling. Kaplan/Hoffmann scaling laws empirically predict loss as a power law in parameters $N$ and data $D$ :

7

\quad \mathcal{L}(N, D) \approx \mathcal{L}_\infty + \left(\frac{N_c}{N}\right)^{\alpha_N} + \left(\frac{D_c}{D}\right)^{\alpha_D}

Derivation hint: Fit log-loss versus log-compute on observed runs; $\alpha_N, \alpha_D \approx 0.05{-}0.5$ are empirical exponents, and $\mathcal{L}_\infty$ is the irreducible entropy floor. Open questions remain profound: Why do these power laws hold? What is the irreducible loss $\mathcal{L}_\infty$ for natural language, and how close can we get? Emergence is poorly understood—are abrupt capability jumps real phase transitions or artifacts of discontinuous metrics? Mechanistic interpretability—reverse-engineering what circuits inside $\theta$ compute—is in its infancy. Alignment: ensuring outputs match human intent and values is unsolved, as is hallucination: there is no principled guarantee a probability-maximizing model states truths. Finally, whether next-token prediction can ever yield genuine reasoning versus sophisticated pattern interpolation is fiercely debated.

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also