Constraints and Lagrange Multipliers

⚡ Key Takeaways

Constraints and Lagrange Multipliers is the method for finding extrema (maxima/minima) of a function $f$ subject to one or more constraints $g = 0$ . Key result: at a constrained extremum, the gradient of the objective is a linear combination of the constraint gradients, $\nabla f = \sum_i \lambda_i \nabla g_i$ . Applies to optimization in economics, physics, engineering, machine learning, and any problem where variables are not free but tied together. Who/when: Joseph-Louis Lagrange, 1788, in Mécanique analytique.

💡 The Core Insight

At a constrained extremum, you cannot improve $f$ by moving along the constraint surface. This means the directional derivative of $f$ vanishes in every direction tangent to the surface. Equivalently, $\nabla f$ has no component along the surface — it must point entirely perpendicular to it. Since the constraint gradients $\nabla g_i$ span exactly that perpendicular (normal) space, $\nabla f$ must be a linear combination of them. The coefficients are the Lagrange multipliers.

DefinitionConstrained Optimization Problem

Given a smooth objective $f : \mathbb{R}^n \to \mathbb{R}$ and constraint functions $g_1, \dots, g_m : \mathbb{R}^n \to \mathbb{R}$ (with $m < n$ ), the problem is to find the extrema of $f(\mathbf{x})$ subject to

1

\quad g_i(\mathbf{x}) = 0, \qquad i = 1, 2, \dots, m.

Derivation hint: Each equation carves out an $(n{-}1)$ -dimensional hypersurface; their intersection is the $(n{-}m)$ -dimensional feasible set on which $f$ is optimized. Here $\mathbf{x} = (x_1,\dots,x_n)$ , $f$ is the objective function, each $g_i$ is a constraint, and the feasible set is $S = \{\mathbf{x} : g_i(\mathbf{x}) = 0 \ \forall i\}$ .

DefinitionThe Lagrangian

The Lagrangian is the auxiliary function of $n + m$ variables

2

\quad \mathcal{L}(\mathbf{x}, \boldsymbol{\lambda}) = f(\mathbf{x}) - \sum_{i=1}^{m} \lambda_i\, g_i(\mathbf{x}),

Derivation hint: Subtracting $\lambda_i g_i$ encodes the constraints; differentiating $\mathcal{L}$ in $\lambda_i$ recovers $g_i = 0$ , and in $\mathbf{x}$ recovers the gradient condition. where the scalars $\lambda_1,\dots,\lambda_m$ are the Lagrange multipliers. The sign convention ( $-$ vs $+$ ) is arbitrary and only flips the sign of $\lambda_i$ .

TheoremMethod of Lagrange Multipliers

Let $f$ and $g_1,\dots,g_m$ be continuously differentiable near a point $\mathbf{x}^*$ that is a local extremum of $f$ on $S$ . Suppose the constraint qualification holds: the gradients $\nabla g_1(\mathbf{x}^*),\dots,\nabla g_m(\mathbf{x}^*)$ are linearly independent. Then there exist unique multipliers $\lambda_1^*,\dots,\lambda_m^*$ such that

3

\quad \nabla f(\mathbf{x}^*) = \sum_{i=1}^{m} \lambda_i^*\, \nabla g_i(\mathbf{x}^*).

Derivation hint: $\nabla f$ must be orthogonal to the tangent space $T_{\mathbf{x}^*}S$ ; this orthogonal complement is spanned by the $\nabla g_i$ . Equivalently, $(\mathbf{x}^*,\boldsymbol{\lambda}^*)$ is a stationary point of the Lagrangian: $\nabla_{\mathbf{x}}\mathcal{L} = 0$ and $\nabla_{\boldsymbol{\lambda}}\mathcal{L} = 0$ , giving the system

4

\quad \frac{\partial \mathcal{L}}{\partial x_j} = 0 \ (j=1,\dots,n), \qquad \frac{\partial \mathcal{L}}{\partial \lambda_i} = -g_i = 0 \ (i=1,\dots,m).

Derivation hint: The $x_j$ -equations give [3] component-wise; the $\lambda_i$ -equations restore the original constraints [1].

ProofGradient Orthogonality at a Constrained Extremum

We prove the single-constraint case ( $m=1$ ); the general case follows by the same argument applied to the tangent space.

Step 1 — Parametrize the constraint surface. Near $\mathbf{x}^*$ , let $\mathbf{r}(t)$ be any smooth curve lying in $S$ with $\mathbf{r}(0) = \mathbf{x}^*$ . Since $\mathbf{r}(t) \in S$ , we have $g(\mathbf{r}(t)) = 0$ for all $t$ .

Step 2 — Differentiate the constraint. Apply the chain rule at $t=0$ :

5

\quad \frac{d}{dt}\, g(\mathbf{r}(t))\Big|_{0} = \nabla g(\mathbf{x}^*)\cdot \mathbf{r}'(0) = 0.

Derivation hint: $g\circ\mathbf{r}\equiv 0$ , so its derivative is zero; chain rule turns this into a dot product. Hence $\nabla g \perp$ every tangent vector. So $\nabla g(\mathbf{x}^*)$ is orthogonal to every tangent direction $\mathbf{r}'(0)$ — i.e. orthogonal to the tangent space $T_{\mathbf{x}^*}S$ .

Step 3 — Use that $\mathbf{x}^*$ is an extremum. The function $h(t) = f(\mathbf{r}(t))$ has a local extremum at $t=0$ , so $h'(0) = 0$ :

6

\quad \frac{d}{dt}\, f(\mathbf{r}(t))\Big|_{0} = \nabla f(\mathbf{x}^*)\cdot \mathbf{r}'(0) = 0.

Derivation hint: A differentiable function at an interior extremum has zero derivative; chain rule converts to the gradient dotted with the tangent. So $\nabla f(\mathbf{x}^*)$ is also orthogonal to every tangent direction.

Step 4 — Conclude collinearity. Both $\nabla f$ and $\nabla g$ lie in the one-dimensional orthogonal complement of $T_{\mathbf{x}^*}S$ (since the surface is $(n{-}1)$ -dimensional, its normal space is one-dimensional). Two vectors in a 1-D space are parallel: there exists a scalar $\lambda^*$ with

7

\quad \nabla f(\mathbf{x}^*) = \lambda^*\, \nabla g(\mathbf{x}^*),

Derivation hint: $\nabla g \neq 0$ by the constraint qualification spans the normal line; $\nabla f$ lies on that line, so it is a scalar multiple. $\square$ provided $\nabla g(\mathbf{x}^*) \neq 0$ , which is exactly the constraint qualification.

∎

JL

Joseph-Louis Lagrange

1736–1813

Born in Turin as Giuseppe Lodovico Lagrangia, Lagrange reformulated all of mechanics in Mécanique analytique (1788) with the boast that the book contained not a single diagram — everything followed from the calculus of variations and his multipliers. He introduced the multiplier technique precisely to handle constraints in mechanical systems (rigid rods, surfaces of contact) without solving for the constraint forces explicitly. The multipliers, it turned out, are the constraint forces. Lagrange succeeded Euler at the Berlin Academy and later became a founding professor at the École Polytechnique in Paris.

🧠 Intuition

Picture standing on a hillside ( $f$ = altitude) but confined to a winding path ( $g=0$ ). To find the highest point of your path, walk along it: whenever the path still climbs or descends, you are not at the top. You stop where the path is momentarily level — where the slope of the terrain has no component along the path. At that instant the steepest-ascent direction $\nabla f$ points straight across the path, perpendicular to it. Since $\nabla g$ also points perpendicular to the path (it's the normal to the constraint curve), the two arrows line up. The multiplier $\lambda$ is just the ratio of their lengths. A second picture: tangent level curves. The constrained extremum occurs where a level curve of $f$ is tangent to the constraint curve. If they crossed transversally, you could slide along the constraint into a higher level curve. Tangency forces the gradients (both normal to their respective curves) to be parallel.

Figure

At a constrained maximum, the level curve of

f

kisses the constraint curve; the gradients align.

ExampleRectangle of Maximum Area Inscribed in a Circle

Given: Maximize the area $f(x,y) = 4xy$ of a rectangle with corners $(\pm x, \pm y)$ inscribed in the circle of radius $5$ , so the constraint is $g(x,y) = x^2 + y^2 - 25 = 0$ , with $x,y > 0$ .

Step 1 — Gradients. $\nabla f = (4y,\, 4x)$ , and $\nabla g = (2x,\, 2y)$ .

Step 2 — Lagrange equations $\nabla f = \lambda \nabla g$ :

4y = \lambda(2x), \qquad 4x = \lambda(2y).

Step 3 — Eliminate $\lambda$ . From the first, $\lambda = 2y/x$ ; from the second, $\lambda = 2x/y$ . Setting equal: $2y/x = 2x/y \Rightarrow y^2 = x^2 \Rightarrow y = x$ (positive root).

Step 4 — Apply the constraint. $x^2 + x^2 = 25 \Rightarrow x^2 = 12.5 \Rightarrow x = y = \sqrt{12.5} = \tfrac{5}{\sqrt2} \approx 3.536$ .

Step 5 — Evaluate. Area $= 4xy = 4 \cdot 12.5 = 50$ . (It's a square — as expected.)

\boxed{\text{Max area} = 50 \text{ square units}, \quad x=y=\tfrac{5}{\sqrt2}, \quad \lambda = 2.}

ExampleClosest Point on a Plane to the Origin

Given: Find the point on the plane $2x + 3y + z = 14$ nearest the origin. Minimize squared distance $f(x,y,z) = x^2 + y^2 + z^2$ subject to $g = 2x + 3y + z - 14 = 0$ .

Step 1 — Gradients. $\nabla f = (2x, 2y, 2z)$ , $\nabla g = (2, 3, 1)$ .

Step 2 — Lagrange equations $\nabla f = \lambda \nabla g$ :

2x = 2\lambda,\quad 2y = 3\lambda,\quad 2z = \lambda \ \Rightarrow\ x = \lambda,\ \ y = \tfrac{3}{2}\lambda,\ \ z = \tfrac{1}{2}\lambda.

Step 3 — Substitute into the constraint.

2(\lambda) + 3\left(\tfrac32\lambda\right) + \tfrac12\lambda = 14 \ \Rightarrow\ 2\lambda + \tfrac92\lambda + \tfrac12\lambda = 14 \ \Rightarrow\ 7\lambda = 14 \ \Rightarrow\ \lambda = 2.

Step 4 — Recover the point. $x = 2,\ y = 3,\ z = 1$ .

Step 5 — Distance. $\|(2,3,1)\| = \sqrt{4+9+1} = \sqrt{14} \approx 3.742$ .

\boxed{\text{Nearest point} = (2,\,3,\,1), \quad \text{distance} = \sqrt{14}\ \text{units}.}

(Note: the answer is $14/\|\nabla g\| = 14/\sqrt{14} = \sqrt{14}$ — consistent with the point-to-plane formula.)

⚠️ Common mistake

The method finds critical points, not maxima. Equation [3] is a necessary condition for an interior extremum, not sufficient. Solving the Lagrange system gives candidates; you must then compare $f$ -values among them (and check boundaries/non-smooth points) or use a bordered Hessian second-order test to classify each as max, min, or saddle. On a compact feasible set, the extreme value theorem guarantees the global max and min occur among the critical points, so comparison alone suffices.

The sign and size of $\lambda$ are meaningful. $\lambda$ is not arbitrary slack. By the envelope theorem, $\lambda^* = \partial f^*/\partial c$ where the constraint is $g = c$ : it is the rate at which the optimal value improves as the constraint is loosened — the shadow price.

Counterexample

Failure when the constraint qualification breaks. Minimize $f(x,y) = x$ subject to $g(x,y) = y^2 - x^3 = 0$ (a cusp curve). The minimum of $x$ on the curve is at the origin $(0,0)$ , since $x = y^{2/3} \geq 0$ . But there $\nabla g = (-3x^2, 2y) = (0,0)$ — the constraint gradient vanishes. The Lagrange equation $\nabla f = \lambda \nabla g$ becomes $(1,0) = \lambda(0,0)$ , which has no solution for any $\lambda$ . The genuine constrained minimum is invisible to the method because the constraint qualification (nonvanishing, linearly independent $\nabla g_i$ ) fails at the cusp. Lesson: always check that the constraint gradients are independent at candidate points, and inspect singular points of the feasible set separately.

connection

The method is the equality-constrained special case of the Karush-Kuhn-Tucker Conditions, which add inequality constraints $h_j(\mathbf{x}) \le 0$ with sign-restricted multipliers and complementary slackness. It is the finite-dimensional shadow of the <a class="wiki-link" href="/wiki/euler-lagrange-equation">Euler-Lagrange Equation</a> in the Calculus of Variations, where one extremizes a functional subject to integral or pointwise constraints (here $\lambda$ becomes a function). Geometrically it rests on Gradient and Directional Derivative (the gradient is normal to level sets) and the Tangent Space of a manifold. In mechanics, the multipliers reappear as constraint forces in Lagrangian Mechanics, and in convex optimization they generate the Lagrangian Duality that underpins linear and quadratic programming.

🔬 Where this appears

⏳ History

- 1755 — At age 19, Lagrange writes to Euler outlining the analytic (

\delta

) calculus of variations, the seedbed of the multiplier idea.

- 1760s — Euler and Lagrange develop variational methods; constraints handled ad hoc.

- 1788 — Lagrange publishes Mécanique analytique, systematically introducing multipliers to enforce mechanical constraints, identifying them with constraint forces.

- 1797 — Lagrange's Théorie des fonctions analytiques refines the analytic foundations of the differential calculus he uses.

- 1813 — Lagrange dies in Paris; his methods are by now standard in continental mechanics.

- 1870s–1900s — The geometric reformulation (gradients normal to level surfaces, tangent spaces) is sharpened with the rise of vector analysis (Gibbs, Heaviside).

- 1939 — William Karush, in his master's thesis, derives optimality conditions with inequality constraints — generalizing Lagrange.

- 1951 — Harold Kuhn and Albert Tucker independently rediscover and popularize these conditions; the modern KKT framework is born.

- 1950s–60s — Linear/nonlinear programming and duality theory (Dantzig, von Neumann, Kuhn–Tucker) place multipliers at the center of optimization.

open-problem

When does the method guarantee a global optimum? For general nonconvex $f$ or nonconvex feasible sets, the Lagrange conditions only locate stationary points, and there may be exponentially many; certifying global optimality is NP-hard in general. Convexity rescues us: if $f$ is convex and the feasible set convex, any KKT point is a global minimizer, and strong duality holds under a constraint qualification (e.g. Slater's condition). Open and active areas include: tight duality gaps for nonconvex problems (e.g. when does the semidefinite relaxation match the true optimum?), multiplier theory on non-smooth or stratified feasible sets (where constraint qualification fails on a measure-zero singular set, as in the cusp counterexample), and robust/distributionally-robust optimization where the multipliers themselves become uncertain objects. In infinite dimensions, the existence and regularity of multipliers (Lagrange multiplier rules in Banach spaces) require subtle hypotheses and remain a fertile research domain.

Formal Statement

Why It Holds

Building Intuition

Worked Examples

Common Misconceptions

Connections

Real-World Applications

Historical Development

Subtleties & Limits

See Also