Lagrange Multipliers

⚡ Key Takeaways

Lagrange multipliers find the extrema of a function f(x)f(\mathbf{x}) subject to one or more constraints g(x)=0g(\mathbf{x})=0 by solving ∇f=λ∇g\nabla f = \lambda \nabla g. Key result: at a constrained extremum, the gradient of the objective is a linear combination of the constraint gradients. Applies to: optimization under equality constraints in economics, physics, machine learning, and engineering. Who/when: Joseph-Louis Lagrange, ~1788, in his Mécanique analytique.

💡 The Core Insight

At a constrained extremum, you cannot increase ff by moving along the constraint surface. This means ∇f\nabla f has no component tangent to the surface — it points entirely normal to it. Since ∇g\nabla g is also normal to the surface g=0g=0, the two gradients must be parallel: ∇f=λ∇g\nabla f = \lambda\nabla g. The multiplier λ\lambda measures how much normal force the constraint exerts.

Formal Statement

DefinitionConstrained Optimization Problem

Let f:Rn→Rf:\mathbb{R}^n\to\mathbb{R} (the objective) and g:Rn→Rg:\mathbb{R}^n\to\mathbb{R} (the constraint) be continuously differentiable. The problem is to find local extrema of f(x)f(\mathbf{x}) subject to

1
g(x)=0.\quad g(\mathbf{x}) = 0.

Derivation hint: this defines the feasible set as the level set (hypersurface) where the constraint vanishes. The feasible set is S={x∈Rn:g(x)=0}S = \{\mathbf{x}\in\mathbb{R}^n : g(\mathbf{x})=0\}, a hypersurface of dimension n−1n-1 where ∇g≠0\nabla g\neq\mathbf 0.

DefinitionLagrangian Function

The Lagrangian combines objective and constraint with a scalar multiplier λ\lambda:

2
L(x,λ)=f(x)−λ g(x).\quad \mathcal{L}(\mathbf{x},\lambda) = f(\mathbf{x}) - \lambda\, g(\mathbf{x}).

Derivation hint: subtract λ\lambda times the constraint so that stationarity in λ\lambda recovers g=0g=0. Here x∈Rn\mathbf{x}\in\mathbb{R}^n and λ∈R\lambda\in\mathbb{R} is the Lagrange multiplier, an auxiliary unknown.

TheoremLagrange Multiplier Theorem

Let f,gf,g be C1C^1 near a point x∗\mathbf{x}^* with g(x∗)=0g(\mathbf{x}^*)=0 and ∇g(x∗)≠0\nabla g(\mathbf{x}^*)\neq\mathbf{0} (the regularity or constraint qualification). If x∗\mathbf{x}^* is a local extremum of ff restricted to SS, then there exists a unique λ∗∈R\lambda^*\in\mathbb{R} such that

3
∇f(x∗)=λ∗ ∇g(x∗).\quad \nabla f(\mathbf{x}^*) = \lambda^* \,\nabla g(\mathbf{x}^*).

Derivation hint: the objective gradient must lie in the line spanned by the constraint gradient — i.e., orthogonal to every tangent direction of SS. Equivalently, (x∗,λ∗)(\mathbf{x}^*,\lambda^*) is a stationary point of L\mathcal{L}:

4
∇xL=0,∂L∂λ=−g(x∗)=0.\quad \nabla_{\mathbf{x}}\mathcal{L} = \mathbf{0},\qquad \frac{\partial\mathcal{L}}{\partial\lambda} = -g(\mathbf{x}^*) = 0.

Derivation hint: setting all partials of the Lagrangian to zero reproduces equation [3] plus the constraint [1].

For mm constraints g1=⋯=gm=0g_1=\cdots=g_m=0, the result generalizes (with linearly independent gradients):

5
∇f(x∗)=∑i=1mλi∗ ∇gi(x∗).\quad \nabla f(\mathbf{x}^*) = \sum_{i=1}^{m}\lambda_i^*\,\nabla g_i(\mathbf{x}^*).

Derivation hint: ∇f\nabla f must lie in the span of all active constraint gradients, the normal space to the intersection surface.

Why It Holds

ProofGeometric Derivation via Tangent Spaces

Step 1. Let x∗∈S\mathbf{x}^*\in S be a local extremum of f∣Sf|_S. Consider any smooth curve γ(t)\boldsymbol\gamma(t) lying in SS with γ(0)=x∗\boldsymbol\gamma(0)=\mathbf{x}^*. Since γ(t)∈S\boldsymbol\gamma(t)\in S for all tt, we have g(γ(t))=0g(\boldsymbol\gamma(t))=0 identically.

Step 2. Differentiate the constraint along the curve at t=0t=0:

6
ddtg(γ(t))∣0=∇g(x∗)⋅γ′(0)=0.\quad \frac{d}{dt}g(\boldsymbol\gamma(t))\Big|_{0} = \nabla g(\mathbf{x}^*)\cdot\boldsymbol\gamma'(0) = 0.

Derivation hint: apply the multivariable chain rule; the result says ∇g\nabla g is orthogonal to every tangent vector of SS. Thus γ′(0)\boldsymbol\gamma'(0) spans the tangent space Tx∗ST_{\mathbf{x}^*}S, the orthogonal complement of ∇g(x∗)\nabla g(\mathbf{x}^*).

Step 3. Because x∗\mathbf{x}^* is an extremum of f∘γf\circ\boldsymbol\gamma (a function of tt), its derivative vanishes:

7
ddtf(γ(t))∣0=∇f(x∗)⋅γ′(0)=0.\quad \frac{d}{dt}f(\boldsymbol\gamma(t))\Big|_{0} = \nabla f(\mathbf{x}^*)\cdot\boldsymbol\gamma'(0) = 0.

Derivation hint: a one-variable extremum has zero derivative; chain rule converts this to a dot product.

Step 4. Equation [7] holds for every tangent vector γ′(0)∈Tx∗S\boldsymbol\gamma'(0)\in T_{\mathbf{x}^*}S. Therefore ∇f(x∗)\nabla f(\mathbf{x}^*) is orthogonal to the entire tangent space — it lies in the normal line, which is spanned by ∇g(x∗)\nabla g(\mathbf{x}^*) (nonzero by regularity).

Step 5. Hence ∇f(x∗)\nabla f(\mathbf{x}^*) is a scalar multiple of ∇g(x∗)\nabla g(\mathbf{x}^*):

8
∇f(x∗)=λ∗∇g(x∗),\quad \nabla f(\mathbf{x}^*) = \lambda^*\nabla g(\mathbf{x}^*),

Derivation hint: two vectors both orthogonal to the same (n−1)(n-1)-dimensional hyperplane must be collinear in Rn\mathbb{R}^n. establishing [3]. Uniqueness of λ∗\lambda^* follows since ∇g(x∗)≠0\nabla g(\mathbf{x}^*)\neq\mathbf 0. ■\blacksquare

∎
mathematician · Joseph-Louis Lagrange | 1736–1813

Born Giuseppe Lodovico Lagrangia in Turin, Lagrange was a towering figure in analysis, number theory, and celestial mechanics. He developed the method of multipliers while reformulating mechanics in his masterwork Mécanique analytique (1788), where he famously boasted the book contained no diagrams — reducing physics to pure analysis. The multiplier technique arose from his treatment of constrained systems and the principle of virtual work, where reaction forces play the role the multipliers do in optimization. He succeeded Euler at the Berlin Academy and later moved to Paris, surviving the Revolution and becoming a senator under Napoleon.

Building Intuition

🧠 Intuition

Imagine hiking on a mountain whose elevation is f(x,y)f(x,y), but you are forced to stay on a fenced trail g(x,y)=0g(x,y)=0. You reach the highest point on the trail when the trail runs tangent to a contour line of the mountain. At that moment, walking forward along the trail neither gains nor loses elevation — the trail is level to first order. Geometrically, the contour of ff and the trail g=0g=0 are tangent, which means their normal directions ∇f\nabla f and ∇g\nabla g point the same way (up to sign and scale). The multiplier λ\lambda tells you how steeply the elevation contours are bunched relative to the trail's contour — i.e., how much higher you could climb if the fence were moved outward slightly.

Figure
At a constrained maximum, the contour of ff is tangent to the constraint curve g=0g=0, so ∇f∥∇g\nabla f \parallel \nabla g.
contours of f g(x,y)=0 x* ∇f ∇g

Worked Examples

ExampleMaximizing Area for Fixed Perimeter

Given: Maximize the area f(x,y)=xyf(x,y)=xy of a rectangle subject to fixed perimeter g(x,y)=2x+2y−P=0g(x,y)=2x+2y-P=0, with P=20P=20 m.

Step 1. Form the gradient equations from [3]:

9
∇f=(y, x),∇g=(2, 2),(y,x)=λ(2,2).\quad \nabla f = (y,\,x),\qquad \nabla g = (2,\,2),\qquad (y,x)=\lambda(2,2).

Derivation hint: compute partials of ff and gg, then set them proportional component-wise.

Step 2. From the components: y=2λy=2\lambda and x=2λx=2\lambda, so x=yx=y.

Step 3. Substitute into the constraint 2x+2y=20⇒4x=20⇒x=52x+2y=20 \Rightarrow 4x=20 \Rightarrow x=5 m, hence y=5y=5 m.

Step 4. The multiplier: λ=x/2=2.5\lambda = x/2 = 2.5 m. The optimum is a square. The maximal area is:

A=xy=25 m2\boxed{A = xy = 25\ \text{m}^2}

The multiplier λ=2.5 m\lambda = 2.5\ \text{m} predicts that increasing PP by 11 m raises max area by ≈2.5 m2\approx 2.5\ \text{m}^2 (check: d(P2/16)/dP=P/8=2.5d(P^2/16)/dP = P/8 = 2.5). ✓

ExampleClosest Point on a Plane to the Origin

Given: Minimize squared distance f(x,y,z)=x2+y2+z2f(x,y,z)=x^2+y^2+z^2 subject to the plane g(x,y,z)=x+2y+2z−9=0g(x,y,z)=x+2y+2z-9=0.

Step 1. Gradients: ∇f=(2x,2y,2z)\nabla f=(2x,2y,2z), ∇g=(1,2,2)\nabla g=(1,2,2). Set ∇f=λ∇g\nabla f=\lambda\nabla g:

10
2x=λ,2y=2λ,2z=2λ.\quad 2x=\lambda,\quad 2y=2\lambda,\quad 2z=2\lambda.

Derivation hint: each component of ∇f\nabla f equals λ\lambda times the corresponding component of ∇g\nabla g.

Step 2. Solve: x=λ/2, y=λ, z=λx=\lambda/2,\ y=\lambda,\ z=\lambda.

Step 3. Substitute into the constraint:

λ2+2λ+2λ=9  ⇒  9λ2=9  ⇒  λ=2.\tfrac{\lambda}{2}+2\lambda+2\lambda = 9 \;\Rightarrow\; \tfrac{9\lambda}{2}=9 \;\Rightarrow\; \lambda=2.

Step 4. Therefore x=1, y=2, z=2x=1,\ y=2,\ z=2. The minimal squared distance is 1+4+4=91+4+4=9, so the distance is 33.

(x,y,z)=(1,2,2),dmin⁡=3\boxed{(x,y,z)=(1,2,2),\quad d_{\min}=3}

This matches the formula ∣−9∣/12+22+22=9/3=3|{-9}|/\sqrt{1^2+2^2+2^2}=9/3=3. ✓

Common Misconceptions

⚠️ Common mistake

The condition ∇f=λ∇g\nabla f=\lambda\nabla g is necessary but not sufficient. It locates stationary (critical) points of the constrained problem — these may be maxima, minima, or saddle points. You must compare ff-values across all solutions, examine boundaries, or use the bordered Hessian second-order test to classify them. Also note: λ\lambda can be negative or zero; its sign carries meaning (the direction of the constraint's pull), not an error.

⚠️ Common mistake

The regularity condition ∇g(x∗)≠0\nabla g(\mathbf{x}^*)\neq\mathbf{0} (constraint qualification) is essential. If ∇g\nabla g vanishes at a candidate point, the theorem provides no λ\lambda, and a constrained extremum can occur there without satisfying ∇f=λ∇g\nabla f=\lambda\nabla g. Always check that constraint gradients are nonzero (and, for multiple constraints, linearly independent).

CounterexampleRegularity Failure

Minimize f(x,y)=xf(x,y)=x subject to g(x,y)=y2−x3=0g(x,y)=y^2-x^3=0 (a cusp curve). The constrained minimum of xx on this curve occurs at the cusp (0,0)(0,0), where ∇g=(−3x2, 2y)=(0,0)\nabla g = (-3x^2,\,2y)=(0,0) vanishes. Here ∇f=(1,0)\nabla f=(1,0) cannot be written as λ 0\lambda\,\mathbf{0} for any λ\lambda. The method fails precisely because regularity is violated — a vivid reminder to verify constraint qualification before trusting the multiplier equations.

Connections

connection

Lagrange multipliers sit at the crossroads of several major frameworks. They generalize to inequality constraints via the Karush-Kuhn-Tucker Conditions, where multipliers must satisfy sign and complementary-slackness conditions. The Lagrangian itself is the bridge to Convex Optimization and Duality (Optimization), where the dual function is the infimum of L\mathcal{L} over x\mathbf{x}. The geometric core — gradients orthogonal to level sets — is pure Gradient and Directional Derivatives, and the tangent/normal decomposition foreshadows Differential Geometry of submanifolds. In physics, multipliers reappear as constraint forces in the Calculus of Variations and Lagrangian mechanics.

Real-World Applications

🔬 Where this appears

Historical Development

⏳ History
- 1755: Euler and the young Lagrange correspond on the calculus of variations; Lagrange introduces the "δ\delta-calculus" of variations, the seed of constrained extremization.
- 1760s: Lagrange develops virtual-work principles for mechanics, where reaction forces anticipate multipliers.
- 1788: Publication of Mécanique analytique; the multiplier method appears systematically for constrained mechanical systems.
- 1797–1813: Lagrange's lectures at the École Polytechnique disseminate the analytic, diagram-free approach to constrained problems.
- 19th century: The method becomes standard in textbooks of advanced calculus and rational mechanics across Europe.
- 1939: William Karush, in his Master's thesis, derives optimality conditions for inequality constraints — the first form of KKT.
- 1951: Harold Kuhn and Albert Tucker independently publish the inequality-constrained generalization, founding nonlinear programming.
- 1950s–60s: Duality theory and convex analysis (Fenchel, Rockafellar) embed Lagrange multipliers in a rigorous modern framework.
- 1970s–present: Multipliers become central to numerical optimization (augmented Lagrangian, interior-point, ADMM methods) powering large-scale machine learning and engineering design.

Subtleties & Limits

open-problem

Second-order conditions and degenerate constraints. While the first-order multiplier equation is clean, classifying stationary points requires the bordered Hessian restricted to the tangent space — a subtle test that fails in degenerate cases. When constraint qualifications break down (non-smooth, redundant, or tangent constraints), multipliers may fail to exist or become non-unique, and the active research field of non-smooth and variational analysis (subdifferentials, Mordukhovich calculus) provides generalized multiplier rules. For global constrained optimization over non-convex sets, the local nature of multiplier conditions leaves open the hard problem of certifying global optimality, which connects to ongoing work in polynomial optimization and the moment/sum-of-squares hierarchies.

See Also

Gradient and Directional Derivatives · Karush-Kuhn-Tucker Conditions · Duality (Optimization) · Convex Optimization · Calculus of Variations · Bordered Hessian · Lagrangian Mechanics · Critical Points and Extrema