Joint and Conditional Entropy

Real information sources rarely produce a single symbol; they produce correlated pairs (or longer strings). To handle two random variables $X$ and $Y$ jointly, Shannon entropy extends naturally to joint and conditional entropies, tied together by an elegant chain rule.

Joint entropy

Given a joint distribution $p(x,y)$, the joint entropy treats the pair $(X,Y)$ as a single composite variable:

It measures the total uncertainty about both outcomes together. If $X$ and $Y$ are independent, $p(x,y) = p(x)p(y)$ and the logarithm splits, giving $H(X,Y) = H(X) + H(Y)$.

Conditional entropy

The conditional entropy $H(Y\,|\,X)$ measures the average remaining uncertainty about $Y$ once $X$ is known. For a fixed value $x$, the uncertainty in $Y$ is $H(Y\,|\,X=x)$, computed from the conditional distribution $p(y\,|\,x)$. Averaging over $x$ with weights $p(x)$,

Conditioning never increases uncertainty: $0 \le H(Y\,|\,X) \le H(Y)$. Learning $X$ can only reduce (or leave unchanged) what we do not know about $Y$.

The chain rule

Using $p(x,y) = p(x)\,p(y\,|\,x)$ and $\log_2 p(x,y) = \log_2 p(x) + \log_2 p(y\,|\,x)$, the joint entropy decomposes:

In words: the uncertainty about the pair equals the uncertainty about $X$ plus the leftover uncertainty about $Y$ given $X$. Rearranged, $H(Y\,|\,X) = H(X,Y) - H(X)$ — the most convenient route in practice, since both terms on the right come straight from the joint table.

A worked example

Take the joint distribution

The joint entropy is

The marginal of $X$ is $p(X{=}0) = \tfrac34$, $p(X{=}1) = \tfrac14$, so $H(X) = h(\tfrac14) \approx 0.8113$ bits. The chain rule then gives

Try it

For the joint distribution above, compute the conditional entropy $H(Y\,|\,X)$ in bits via the chain rule $H(Y\,|\,X) = H(X,Y) - H(X)$. Return the value.

Sign in on the full site to ask questions and join the discussion.