Mutual Information Calculator

Mutual information measures how much knowing one variable reduces uncertainty about another.

Use the result as a learning aid. For classwork, still show the formula and intermediate reasoning so the final answer is not just a black-box number.

Mutual information measures how much knowing one random variable reduces uncertainty about another. For discrete variables X and Y, it compares the observed joint probability p(x,y) with the product p(x)p(y) that would occur under independence. The calculator reports the result in bits because it uses logarithms base 2.

If X and Y are independent, p(x,y)=p(x)p(y) for every pair, each log ratio is 0, and the mutual information is 0 bits. Positive mutual information means the variables share statistical dependence. The measure is symmetric: I(X;Y)=I(Y;X). It is not a signed measure, so it does not distinguish positive from negative correlation.

Counts can be used instead of already-normalized probabilities because the calculator first divides every cell by the total. Zero-probability cells contribute 0 to the sum by continuity. The table should nevertheless contain nonnegative values and consistent row lengths; a negative count or malformed table does not represent a valid joint distribution.

Interpreting the size of mutual information requires context. MI is bounded above by the smaller of H(X) and H(Y), so 0.7 bits can be very large for a low-entropy binary variable but modest for variables with many possible outcomes. A normalized comparison may therefore be useful when comparing different datasets, although several normalization conventions exist.

For a useful hand check, compute the row and column marginals first. If every joint cell equals the product of its corresponding marginals, the variables are independent and MI must be zero. At the opposite extreme, if one variable determines the other perfectly, MI equals the entropy of the determined variable. These endpoints make good tests for both data entry and software output.