Shannon Entropy and Information Gain Calculator
Calculate entropy from probabilities or class counts, and estimate decision-tree information gain after a split.
Calculate
Use commas inside a group and semicolons between child groups. Example parent: 9,5 and children: 6,2; 3,3.
What this calculator teaches
Shannon entropy measures uncertainty in a probability distribution. Information gain measures how much a split reduces uncertainty.
These ideas are used in decision trees, compression, machine learning, communications, and data science.
Formula & Symbols
| Concept | Formula | Meaning |
|---|---|---|
| Entropy | H = -Σ pᵢ log₂(pᵢ) | Measures uncertainty in bits. |
| Information gain | IG = H(parent) - Σ(w_j H(child_j)) | Reduction in weighted entropy after a split. |
| Weight | w_j = child size / parent size | Larger child groups count more. |
Step-by-Step Examples
- Counts are 9 and 5, total 14.
- Probabilities are 9/14 and 5/14.
- Compute -p log₂ p for each class and add.
- Parent entropy is computed first.
- Compute each child entropy.
- Subtract weighted child entropy from parent entropy.
Where students use this
Common Mistakes to Avoid
Shannon entropy in bits uses log base 2.
Information gain requires weighted average child entropy.
Terms with p=0 contribute 0, not undefined.
Frequently Asked Questions
Related Math Calculators
Formula Explorer connections
Interpretation: This relationship quantifies information, representation, storage, error, search or computational performance. Assumption: Use the exact encoding, data distribution, machine representation and algorithm assumptions. Real systems also include implementation and hardware overhead.