📊 The Math Behind Entropy - Example

We will use a Healthcare Scenario: Predicting if a patient has a Fever ( $Y$ ) based on the presence of a Cough ( $X$ ).

Total Sample Space ( $n$ ) = $10$ .
For Cough ( $X$ )
- $P(X = Cough) = \frac{6}{10} = 0.6$
- $P(X = No Cough) = \frac{4}{10} = 0.4$
For Fever ( $Y$ )
- $P(Y = Fever) = \frac{5}{10} = 0.5$
- $P(Y = No Fever) = \frac{5}{10} = 0.5$
Join Probabilities
- $P (X = Cough, Y = Fever) = \frac{4}{10} = 0.4$
- $P (X = Cough, Y =No Fever) = \frac{2}{10} = 0.2$
- $P (X = No Cough, Y =Fever) = \frac{1}{10} = 0.1$
- $P (X = No Cough, Y =No Fever) = \frac{3}{10} = 0.3$

\begin{aligned} H(Fever) & H (Y) & = - \sum P (y) \log_{2} P (y) & = - [0.5 \log_{2} (0.5) + 0.5 \log_{2} (0.5)] & = 1.0 bit \end{aligned}

\begin{aligned} H(Cough) & H (X) & = - \sum P (x) \log_{2} P (x) & = - [0.4 \log_{2} (0.4) + 0.6 \log_{2} (0.6)] & = 0.971 bit \end{aligned}

Break down of metric from results

$H (Fever)$ : A value of 1.0 indicates a perfect 50/50 split—maximum uncertainty. You are essentially flipping a coin to guess if a patient has a fever.
$H (Cough)$ : A value of 0.971 indicates High uncertainty. The distribution is slightly biased (60/40), so there is slightly less "surprise" than a 50/50 split.

Joint Entropy is a measure of the uncertainty associated with a set of variables (Cough and Fever together).

\begin{aligned} H (X, Y) & = - \sum P (x, y) \log_{2} P (x, y) \\ H (X, Y) & = - [0.4 \log_{2} 0.4 + 0.2 \log_{2} 0.2 + 0.1 \log_{2} 0.1 + 0.3 \log_{2} 0.3] \\ \approx 1.846 bits \end{aligned}

Break down of metric from results

If $H (X, Y)$ is close to the sum of individual entropies ( $1.0 + 0.97 = 1.97$ ), the variables are mostly independent. At 1.846, they share some overlap, but there is still significant independent "noise" in the system.

\begin{aligned} H (Y | X) & = H (X, Y) - H (X) \\ H (Y | X) & = 1.8465 - 0.971 \\ H (Y | X) & = 0.8755 \\ Similarly H (X | Y) & \approx 0.846 bits \end{aligned}

Break down of metric from results

This is the "noise" that remains. After checking for a cough, the uncertainty about the fever dropped from 1.0 to 0.875.
Strong predictor is the value is close to 0; it is useless if it stays near 1.

\begin{aligned} I (X; Y) & = H (Y) - H (Y ∣ X) \\ = 1 - 0.8755 \\ = 0.1245 bits \end{aligned}

Interpretation of result

This is small.

Meaning:

Summary !

From the patient counts, we compute: $P (x, y)$
These define the full probabilistic structure of the system.
Everything — entropy, conditional entropy, mutual information — is derived from these probabilities.

From information theory: $H (X, Y) = H (X) + H (Y ∣ X)$
Rearranging: $H (Y ∣ X) = H (X, Y) - H (X)$
So we subtract $H (X)$ from the joint entropy because of the chain rule, not arbitrarily.

Mutual Information is The amount by which joint entropy is smaller than the sum of independent entropies.