Convex Optimization · Problem 7.4

Geometric problems — Boyd & Vandenberghe

Mathematics — Optimization · estimation-covariance-mean-multivariate-normal

Problem

Estimation of covariance and mean of a multivariate normal distribution. We consider the problem of estimating the covariance matrix

R

and the mean

a

of a Gaussian probability density function

p_{R,a}(y) = (2\pi)^{-n/2} \det(R)^{-1/2} \exp(-(y-a)^T R^{-1}(y-a)/2),

based on

N

independent samples

y_1, y_2, \ldots, y_N \in \mathbf{R}^n

. (a) We first consider the estimation problem when there are no additional constraints on

R

and

a

. Let

\mu

and

Y

be the sample mean and covariance, defined as

\mu = \frac{1}{N} \sum_{k=1}^N y_k, \qquad Y = \frac{1}{N} \sum_{k=1}^N (y_k - \mu)(y_k - \mu)^T.

Show that the log-likelihood function

l(R, a) = -(Nn/2)\log(2\pi) - (N/2)\log \det R - (1/2)\sum_{k=1}^N (y_k - a)^T R^{-1}(y_k - a)

can be expressed as

l(R, a) = \frac{N}{2}\left(-n\log(2\pi) - \log \det R - \mathbf{tr}(R^{-1}Y) - (a - \mu)^T R^{-1}(a - \mu)\right).

Use this expression to show that if

Y \succ 0

, the ML estimates of

R

and

a

are unique, and given by

a_{\mathrm{ml}} = \mu, \qquad R_{\mathrm{ml}} = Y.

(b) The log-likelihood function includes a convex term (

-\log \det R

), so it is not obviously concave. Show that

l

is concave, jointly in

R

and

a

, in the region defined by

R \preceq 2Y.

This means we can use convex optimization to compute simultaneous ML estimates of

R

and

a

, subject to convex constraints, as long as the constraints include

R \preceq 2Y

, i.e., the estimate

R

must not exceed twice the unconstrained ML estimate.