Kullback divergence

Solomon Kullback, Information Theory and Statistics

If $H_{i}$ , $i = 1,2$ is the hypothesis that $X$ is from statistical measure $μ_{i}$ , where

d μ_{i} (x) = f_{i} d λ (x),

and $P (H_{i})$ is the prior probability of the respective hypothesis, $i = 1,2$ , it follows that conditional on $x$ ,

P (H_{i} | x) = \frac{P (H_{i}) f_{i} (x)}{P (H_{1}) f_{1} (x) + P (H_{2}) f_{2} (x)},

except over a zero measured set by $λ$ .

One obtains

log \frac{f_{1} (x)}{f_{2} (x)} = log \frac{P (H_{1} | x)}{P (H_{2} | x)} - log \frac{P (H_{1})}{P (H_{2})},

except over a zero measured set by $λ$ .

This logarithm of the likelihood ratio, $log [f_{1} (x) / f_{2} (x)]$ is defined as the information in $X = x$ for discrimination in favour of $H_{1}$ against $H_{2}$ .

The mean information for discrimination in favour of $H_{1}$ against $H_{2}$ over set $E$ is

I (1:2) = {\begin{matrix} \frac{1}{μ_{1} (E)} \int_{E} f_{1} (x) log \frac{f_{1} (x)}{f_{2} (x)} d λ (x) & μ_{1} (E) > 0 \\ 0 & μ_{1} (E) = 0 \end{matrix}

Over the entire space,

\begin{matrix} I (1:2) & = & \int log \frac{f_{1} (x)}{f_{2} (x)} d μ_{1} (x) \\ = & \int f_{1} (x) log \frac{f_{1} (x)}{f_{2} (x)} d λ (x) \\ = & (\int log \frac{P (H_{1} | x)}{P (H_{2} | x)} d μ_{1} (x)) - log \frac{P (H_{1})}{P (H_{2})} . \end{matrix}

It is non-negative, $0 \leq I (1:2) \leq +\infty$ . It is zero if $f_{1} = f_{2}$ except over a zero measured set by $μ_{1}$ . It attains $+\infty$ when for example, the space can be divided into two sets, on one of them $f_{1} > 0$ , $f_{2} = 0$ , and on the other $f_{1} = 0$ , $f_{2} > 0$ .

The prior information, $log \frac{P (H_{1})}{P (H_{2})}$ still remains in $I (1:2)$ .

Likewise,

\begin{matrix} I (2:1) & = & \int log \frac{f_{2} (x)}{f_{1} (x)} d μ_{2} (x) \\ = & \int f_{2} (x) log \frac{f_{2} (x)}{f_{1} (x)} d λ (x) . \end{matrix}

One may define the divergence $J (1,2)$ ,

\begin{matrix} J (1,2) & = & I (1:2) + I (2:1) \\ = & \int (f_{1} (x) - f_{2} (x)) log \frac{f_{1} (x)}{f_{2} (x)} d λ (x) \\ = & \int log \frac{P (H_{1} | x)}{P (H_{2} | x)} (d μ_{1} (x) - d μ_{2} (x)) . \end{matrix}

If we slightly rewrite it,

\begin{matrix} J (1,2) & = & I (1:2) + I (2:1) \\ = & \int (f_{1} (x) - f_{2} (x)) (log f_{1} (x) - log f_{2} (x)) d λ (x) \\ = & \int (log P (H_{1} | x) - log P (H_{2} | x)) (d μ_{1} (x) - d μ_{2} (x)) . \end{matrix}

If one swaps the position of hypothesis 1 and 2 one obtains the same quantity, $J (1,2) = J (2,1)$ . So it is symmetric, and the prior information disappeared. Kullback proposes to call $I (1:2)$ or $I (2:1)$ directed divergence.

As $0 \leq I (1:2) \leq +\infty$ , $0 \leq I (2:1) \leq +\infty$ ,

0 \leq J (1,2) \leq +\infty .

Both bounds are attainable.