Blundell Thermal Physics · Problem 15.5

Information theory

Physics — Statistical Mechanics · relative-entropy

Problem

(a) The relative entropy measures the closeness of two probability distributions

P

and

Q

and is defined by

S(P||Q) = \sum P_i \log\left(\frac{P_i}{Q_i}\right) = -S_P - \sum P_i \log Q_i, \tag{15.30}

where

S_P = -\sum P_i \log P_i

. Show that

S(P||Q) \geq 0

with equality if and only if

P_i = Q_i

for all

i

. (b) If

i

takes

N

values with probability

P_i

, then show that

S(P||Q) = -S_P + \log N, \tag{15.31}

where

Q_i = 1/N

for all

i

. Hence show that

S_P \leq \log N, \tag{15.32}

with equality if and only if

P_i

is uniformly distributed between all

N

outcomes.