Lagrange Multipliers and Constrained Optimization

⚡ Key Takeaways

Lagrange multipliers turn a constrained optimization problem into an unconstrained one: to extremize $f(\mathbf{x})$ subject to $g(\mathbf{x}) = 0$ , find points where $\nabla f = \lambda \nabla g$ . The key result is that at a constrained extremum, the gradient of the objective is parallel to the gradient of the constraint. Used throughout calculus, physics (mechanics, thermodynamics), economics, machine learning, and engineering design. Introduced by Joseph-Louis Lagrange in his 1788 Mécanique analytique.

💡 The Core Insight

At a constrained extremum, you cannot improve $f$ by moving along the constraint surface. Any allowed movement is tangent to the surface $g = 0$ , and along every such tangent direction $f$ must be momentarily flat. This forces $\nabla f$ to have no component along the surface — it must point purely perpendicular to it. But $\nabla g$ is also perpendicular to the surface. Therefore $\nabla f$ and $\nabla g$ must be parallel:

1

\quad \nabla f(\mathbf{x}^*) = \lambda\, \nabla g(\mathbf{x}^*)

Derivation hint: project $\nabla f$ onto the tangent space of $\{g=0\}$ ; vanishing of the tangential component leaves only the normal direction, which is spanned by $\nabla g$ .

DefinitionConstrained Optimization Problem

Given a objective function $f:\mathbb{R}^n \to \mathbb{R}$ and constraint functions $g_i:\mathbb{R}^n \to \mathbb{R}$ for $i = 1,\dots,m$ (with $m < n$ ), the equality-constrained problem is

2

\quad \text{extremize } f(\mathbf{x}) \quad \text{subject to} \quad g_i(\mathbf{x}) = 0,\ i=1,\dots,m

Derivation hint: this is the canonical form; any constraint $h(\mathbf{x}) = c$ is rewritten as $g(\mathbf{x}) = h(\mathbf{x}) - c = 0$ . Symbols: $\mathbf{x} = (x_1,\dots,x_n)$ is the decision vector; $f$ is the quantity to maximize or minimize; each $g_i = 0$ defines a hypersurface; $m$ is the number of constraints; the feasible set is $S = \{\mathbf{x} : g_i(\mathbf{x}) = 0 \ \forall i\}$ , generically a manifold of dimension $n-m$ .

DefinitionThe Lagrangian

The Lagrangian function $\mathcal{L}:\mathbb{R}^{n+m} \to \mathbb{R}$ augments the objective with one multiplier $\lambda_i$ per constraint:

3

\quad \mathcal{L}(\mathbf{x}, \boldsymbol{\lambda}) = f(\mathbf{x}) - \sum_{i=1}^{m} \lambda_i\, g_i(\mathbf{x})

Derivation hint: subtract a weighted sum of constraints; the weights are unknowns to be solved for so that $\partial\mathcal{L}/\partial\lambda_i = 0$ recovers each constraint. Symbols: $\boldsymbol{\lambda} = (\lambda_1,\dots,\lambda_m)$ are the Lagrange multipliers. The sign convention (subtract vs. add) does not affect the located points, only the sign of $\lambda_i$ .

TheoremLagrange Multiplier Theorem

Let $f, g_1,\dots,g_m$ be continuously differentiable near a point $\mathbf{x}^* \in S$ . Suppose the constraint gradients $\nabla g_1(\mathbf{x}^*), \dots, \nabla g_m(\mathbf{x}^*)$ are linearly independent (the constraint qualification, also called regularity). If $\mathbf{x}^*$ is a local extremum of $f$ restricted to $S$ , then there exist unique scalars $\lambda_1^*,\dots,\lambda_m^*$ such that

4

\quad \nabla f(\mathbf{x}^*) = \sum_{i=1}^{m} \lambda_i^*\, \nabla g_i(\mathbf{x}^*)

Derivation hint: equivalent to $\nabla_{\mathbf{x}}\mathcal{L} = \mathbf{0}$ ; the multipliers are the coordinates of $\nabla f$ in the basis of constraint normals. Equivalently, $\mathbf{x}^*$ together with $\boldsymbol{\lambda}^*$ is a stationary point of the Lagrangian:

5

\quad \nabla_{\mathbf{x}} \mathcal{L} = \mathbf{0}, \qquad \nabla_{\boldsymbol{\lambda}} \mathcal{L} = \mathbf{0}

Derivation hint: the $\mathbf{x}$ -gradient gives equation [4]; the $\lambda$ -gradient gives $g_i = 0$ , recovering feasibility.

ProofSingle-Constraint Case via Tangent Curves

We prove $\nabla f = \lambda \nabla g$ at a constrained extremum $\mathbf{x}^*$ of $f$ on $S = \{g = 0\}$ .

Step 1 — Parametrize feasible motion. Let $\mathbf{r}(t)$ be any smooth curve lying entirely in $S$ with $\mathbf{r}(0) = \mathbf{x}^*$ . Feasibility means $g(\mathbf{r}(t)) = 0$ for all $t$ .

Step 2 — Differentiate the constraint. Applying the chain rule at $t=0$ :

6

\quad \frac{d}{dt}g(\mathbf{r}(t))\Big|_{0} = \nabla g(\mathbf{x}^*) \cdot \mathbf{r}'(0) = 0

Derivation hint: multivariable chain rule; constancy of $g$ along the curve forces its derivative to zero. This says $\nabla g(\mathbf{x}^*)$ is orthogonal to every tangent vector $\mathbf{r}'(0)$ — i.e. $\nabla g$ is normal to $S$ .

Step 3 — Use the extremum condition. Since $\mathbf{x}^*$ extremizes $f$ on $S$ , the scalar function $\phi(t) = f(\mathbf{r}(t))$ has a local extremum at $t=0$ , so $\phi'(0) = 0$ :

7

\quad \nabla f(\mathbf{x}^*) \cdot \mathbf{r}'(0) = 0

Derivation hint: stationarity of a 1-D function at an interior extremum, plus the chain rule.

Step 4 — Combine. Equation [7] holds for every feasible tangent direction $\mathbf{r}'(0)$ . Thus $\nabla f(\mathbf{x}^*)$ is also orthogonal to the entire tangent space of $S$ . The orthogonal complement of that tangent space (under regularity) is one-dimensional, spanned by $\nabla g(\mathbf{x}^*)$ . Therefore $\nabla f$ must be a scalar multiple of $\nabla g$ :

8

\quad \nabla f(\mathbf{x}^*) = \lambda\,\nabla g(\mathbf{x}^*)

Derivation hint: two vectors orthogonal to the same $(n-1)$ -dimensional subspace must be collinear.

Step 5 — Multi-constraint generalization. With $m$ regular constraints, the tangent space is the common kernel of all $\nabla g_i$ , with dimension $n-m$ . Its orthogonal complement is the $m$ -dimensional span of $\{\nabla g_i\}$ , and $\nabla f$ lying in that span gives equation [4]. $\blacksquare$

∎

mathematician · Joseph-Louis Lagrange | 1736–1813

Born Giuseppe Luigi Lagrangia in Turin, Lagrange became the dominant analyst of the late 18th century. In his monumental Mécanique analytique (1788) he reformulated all of mechanics without a single diagram, deriving the equations of motion from a variational principle and introducing the multipliers that bear his name to handle constraint forces — the tension in a string, the normal force of a surface. His genius was recognizing that constraints could be incorporated algebraically rather than by geometric elimination. He succeeded Euler at the Berlin Academy, later moved to Paris, survived the Revolution, and was made a count by Napoleon. The multiplier method, devised to keep a pendulum on its arc, now underlies optimization across all of quantitative science.

🧠 Intuition

Picture the contour map of $f$ — concentric level curves $f = c$ — and overlay the constraint curve $g = 0$ as a winding trail. As you walk along the trail, you cross higher and lower contours of $f$ . You are climbing as long as the trail cuts across contours. The highest point on your trail occurs exactly where the trail runs tangent to a contour line — there, locally, you are neither gaining nor losing altitude. At that tangency the contour of $f$ and the constraint curve share the same tangent direction, so their normals $\nabla f$ and $\nabla g$ point the same way (up to sign and length). The multiplier $\lambda$ is just the ratio of those gradient lengths. A second picture: $\lambda$ is a price. If a constraint says "spend exactly $100," then $\lambda$ tells you how much extra $f$ you would gain per extra dollar of budget. A large $\lambda$ means the constraint is biting hard; $\lambda = 0$ means the constraint is, at that point, not limiting you at all.

Figure

Tangency of an objective contour with the constraint curve at the constrained maximum

ExampleMaximize a Product Subject to a Sum

Given: Maximize $f(x,y) = xy$ subject to $g(x,y) = x + y - 10 = 0$ (a fixed perimeter-like budget). All quantities in arbitrary units; $x,y \ge 0$ .

Step 1 — Form the gradient equations. $\nabla f = (y, x)$ , $\nabla g = (1,1)$ . Setting $\nabla f = \lambda \nabla g$ :

9

\quad y = \lambda, \qquad x = \lambda

Derivation hint: component-wise equality of $\nabla f = \lambda\nabla g$ .

Step 2 — Solve. From [9], $x = y = \lambda$ . Substitute into the constraint $x + y = 10$ :

10

\quad 2\lambda = 10 \implies \lambda = 5,\quad x = y = 5

Derivation hint: plug $x=y=\lambda$ into $x+y=10$ .

Step 3 — Evaluate the objective. $f(5,5) = 25$ .

Step 4 — Interpret $\lambda$ . Since $\lambda = 5$ , relaxing the budget to $x+y = 11$ should raise the optimum by about $5$ units. Check: $f = (5.5)^2 = 30.25$ , an increase of $5.25 \approx \lambda$ . ✓

\boxed{x^* = y^* = 5,\quad f_{\max} = 25 \text{ (units}^2),\quad \lambda = 5}

ExampleClosest Point on a Plane to the Origin

Given: Find the point on the plane $2x + y + 2z = 9$ nearest the origin. Minimize squared distance $f(x,y,z) = x^2 + y^2 + z^2$ subject to $g = 2x + y + 2z - 9 = 0$ . Lengths in metres.

Step 1 — Gradient condition. $\nabla f = (2x, 2y, 2z)$ , $\nabla g = (2, 1, 2)$ . Then $\nabla f = \lambda\nabla g$ gives:

11

\quad 2x = 2\lambda,\quad 2y = \lambda,\quad 2z = 2\lambda

Derivation hint: equate components of $\nabla f = \lambda\nabla g$ .

Step 2 — Express in terms of $\lambda$ . $x = \lambda,\ y = \tfrac{\lambda}{2},\ z = \lambda$ .

Step 3 — Apply the constraint.

12

\quad 2\lambda + \tfrac{\lambda}{2} + 2\lambda = 9 \implies \tfrac{9}{2}\lambda = 9 \implies \lambda = 2

Derivation hint: substitute the $\lambda$ -expressions into $g=0$ and collect terms.

Step 4 — Recover the point. $x = 2,\ y = 1,\ z = 2$ . Distance:

13

\quad d = \sqrt{x^2+y^2+z^2} = \sqrt{4 + 1 + 4} = 3 \text{ m}

Derivation hint: evaluate the Euclidean norm at the optimal point.

\boxed{(x^*,y^*,z^*) = (2,1,2)\text{ m},\quad d_{\min} = 3\text{ m}}

This matches the known formula $d = |{-9}|/\sqrt{2^2+1^2+2^2} = 9/3 = 3$ . ✓

⚠️ Common mistake

The method finds stationary points, not maxima. Equation [4] is a necessary condition, not sufficient. It locates candidates — maxima, minima, and saddle points all satisfy it. You must afterward (a) evaluate $f$ at every candidate and compare, or (b) apply the bordered Hessian second-order test. Also: always check the constraint qualification. If $\nabla g = \mathbf{0}$ at a feasible point (a singular point of the constraint surface), the theorem can fail and an extremum may exist with no valid $\lambda$ . Finally, on unbounded feasible sets you must verify a maximum or minimum actually exists before trusting a critical point.

CounterexampleWhen No Multiplier Exists

Minimize $f(x,y) = x$ subject to $g(x,y) = y^2 - x^3 = 0$ (a cusp curve). The feasible set is a curve with a sharp cusp at the origin, where the minimum of $x$ occurs ( $x = 0$ ). At the origin $\nabla g = (-3x^2, 2y) = (0,0)$ — the constraint gradient vanishes, violating regularity. Meanwhile $\nabla f = (1,0) \ne \mathbf{0}$ . There is no scalar $\lambda$ satisfying $\nabla f = \lambda\nabla g = \mathbf{0}$ . The constrained minimum genuinely exists at the origin, yet the Lagrange equations have no solution there. The cure: the Fritz John conditions allow a multiplier $\lambda_0$ on $\nabla f$ itself, $\lambda_0\nabla f = \lambda\nabla g$ , permitting $\lambda_0 = 0$ at irregular points.

connection

Lagrange multipliers sit at the crossroads of several frameworks. They are the equality-constraint special case of the Karush-Kuhn-Tucker Conditions, which add inequality constraints $h(\mathbf{x}) \le 0$ via complementary slackness. The multiplier $\lambda$ appears as the dual variable in Convex Optimization Duality, where the Lagrangian generates the dual problem. The geometric core — gradient orthogonality to a level set — is precisely the statement of Gradient and Directional Derivative. In the calculus of variations the discrete multiplier becomes a continuous function, giving the constrained <a class="wiki-link" href="/wiki/euler-lagrange-equation">Euler-Lagrange Equation</a> for problems with integral constraints (the isoperimetric problem). And the parallel-gradient condition is the foundation of Implicit Function Theorem reasoning about how solutions move as parameters change.

🔬 Where this appears

⏳ History

- 1696: Johann Bernoulli poses the brachistochrone problem, igniting interest in optimization under constraint and seeding the calculus of variations.

- 1744: Euler publishes Methodus inveniendi, the first systematic treatment of variational problems, including isoperimetric (integral-constrained) ones.

- 1755: A 19-year-old Lagrange writes to Euler describing his purely analytic ("δ-calculus") variational method, prompting Euler to adopt and name it the calculus of variations.

- 1788: Lagrange's Mécanique analytique introduces the method of multipliers to embed constraint forces in mechanics, the technique's debut.

- 1797–1813: Lagrange's Théorie des fonctions analytiques and lectures spread the analytic optimization viewpoint through French mathematics.

- 1902: Gibbs's statistical mechanics employs multipliers (temperature, chemical potential) to derive equilibrium ensembles.

- 1939: William Karush, in an unpublished master's thesis, derives optimality conditions for inequality constraints.

- 1948: Fritz John publishes generalized conditions handling constraint-qualification failures (the counterexample above).

- 1951: Harold Kuhn and Albert Tucker independently rediscover and publish the inequality-constrained conditions, founding modern nonlinear programming as the KKT theorem.

open-problem

Several issues remain delicate even at advanced levels. Constraint qualification is a genuine gap: the standard regularity assumption (linear independence of $\nabla g_i$ ) can be weakened to Mangasarian–Fromovitz or other conditions, and characterizing the weakest sufficient qualification is intricate. Global optimization under nonconvex constraints is fundamentally hard — Lagrange conditions only localize, and verifying that a stationary point is the global optimum is NP-hard in general. Numerical conditioning plagues large problems: when constraint gradients are nearly dependent the multiplier system becomes ill-conditioned, motivating augmented-Lagrangian and interior-point methods. In infinite dimensions (control theory, PDE-constrained optimization), existence of multipliers requires functional-analytic care (Lagrange multiplier rules in Banach spaces, surjectivity of the constraint derivative). And the economic interpretation of $\lambda$ as a sensitivity holds only under second-order and regularity conditions; nondifferentiable value functions give rise to subdifferential multipliers and the modern theory of variational analysis.

Lagrange Multipliers and Constrained Optimization

Formal Statement

Why It Holds / Derivation

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also