Classical Information and Shannon Entropy

Before we can quantify quantum information, we need a sharp notion of classical information. That notion is Shannon entropy — the quantity that measures, in bits, how much we learn on average when we observe the outcome of a random source, or equivalently how uncertain we are about that outcome beforehand.

Surprise and its average

Suppose a source emits a symbol $x$ drawn from a finite alphabet with probability $p(x)$ . The surprise (or surprisal) of observing $x$ is

i(x) = \log_2 \frac{1}{p(x)} = -\log_2 p(x).

A near-certain outcome ( $p \to 1$ ) carries almost no surprise; a rare outcome ( $p \to 0$ ) is highly surprising. The base-2 logarithm fixes the unit: one bit is the surprise of a fair coin landing heads, $-\log_2 \tfrac12 = 1$ .

The Shannon entropy is the expected surprise:

H(X) = \sum_{x} p(x)\, i(x) = -\sum_{x} p(x) \log_2 p(x),

with the universal convention $0\log_2 0 = 0$ , justified because $p\log p \to 0$ as $p\to 0^+$ .

Properties

Non-negativity. $H(X) \geq 0$ , since each $-p\log_2 p \geq 0$ for $p \in [0,1]$ . It is zero iff the distribution is deterministic (one symbol has probability $1$ ).
Maximum at uniform. Over an alphabet of size $d$ , $H(X) \leq \log_2 d$ , with equality iff $p(x) = 1/d$ for all $x$ . Uncertainty is greatest when every outcome is equally likely.
Concavity. $H$ is a concave function of the probability vector, a fact that underlies many inequalities in information theory.

The binary entropy function

For a two-outcome source with $P = p$ and $1-p$ , the entropy specializes to the binary entropy function

h(p) = -p\log_2 p - (1-p)\log_2(1-p).

It is symmetric about $p = \tfrac12$ , where it attains its maximum $h(\tfrac12) = 1$ bit, and vanishes at $p = 0$ and $p = 1$ . A biased coin with $p = \tfrac14$ has

h(\tfrac14) = \tfrac14\log_2 4 + \tfrac34\log_2\tfrac43 = 0.5 + 0.75\cdot 0.41504 \approx 0.8113\text{ bits},

less than the full bit of a fair coin, because the bias makes the outcome partly predictable.

Why entropy is the right measure

Shannon's source coding theorem gives entropy its operational meaning: a source of entropy $H(X)$ bits per symbol can be compressed to, and no further than, $H(X)$ bits per symbol on average in the limit of long sequences. Entropy is not an arbitrary formula — it is exactly the number of bits you must pay to record the source's output. In the next lesson we lift this idea to quantum states, where the von Neumann entropy plays the same role for density operators.

Try it

Compute the Shannon entropy, in bits, of a biased coin with $P(\text{heads}) = \tfrac14$ and $P(\text{tails}) = \tfrac34$ , using the convention $0\log_2 0 = 0$ . Return the value as a number.

Run your code to see the quantum state.