Cross Entropy Calculator

Compute cross-entropy between true probabilities and predicted probabilities. This educational calculator shows the formula, result, and step-by-step interpretation.

Computer ScienceProbabilityMachine Learning

Calculator

What this calculator teaches

Cross entropy is a common loss function in classification and information theory.

Use the result as a learning aid. For classwork, still show the formula and intermediate reasoning so the final answer is not just a black-box number.

Cross Entropy as a Penalty for Predictive Surprise

Cross entropy measures how costly predictions q are when outcomes follow a target distribution p. For discrete classes and base-2 logarithms, H(p,q) = −Σpilog2(qi). Each true-probability weight pi multiplies the information cost of assigning probability qi to that class. Predictions that place low probability on events that actually matter under p receive a large penalty.

When p is one-hot, only the true class contributes, so cross entropy becomes −log2(qtrue). Predicting the true class with probability 0.5 costs 1 bit, while probability 0.25 costs 2 bits. For a general distribution, cross entropy decomposes as H(p,q) = H(p) + DKL(p||q), so it is minimized at q = p when p is fixed.

QuantityMeaningRequirement
piTarget probability for class ipi ≥ 0 and totals 1 for a distribution
qiPredicted probability for class iqi > 0 where pi > 0
−log2(qi)Information cost in bitsGrows as assigned probability falls

The logarithm base controls the unit: base 2 gives bits and natural logarithms give nats. The ranking of predictions is unaffected by changing the base because all values are multiplied by a positive constant.

Formula & Symbols

ConceptFormula or rule
Cross entropyH(p,q) = -Σ pᵢ log₂(qᵢ)

Worked example

Example: For one-hot classification, cross entropy is -log₂(probability assigned to the true class).
Example 2: For one-hot p = [1,0,0] and q = [0.8,0.15,0.05], H = −log2(0.8) ≈ 0.32193 bits.
Example 3: For a binary one-hot target and qtrue = 0.5, the cross entropy is exactly 1 bit. Halving the assigned probability increases the information penalty.
Example 4: If p = q = [0.5,0.5], cross entropy is 1 bit. This equals the entropy of the target because the prediction matches p exactly.
Example 5: With p = [0.75,0.25] and q = [0.6,0.4], H(p,q) ≈ 0.88321 bits. Both classes contribute because the target is not one-hot.

Common mistakes

⚠️
Using the wrong input format

Keep lists comma separated, matrices as rows separated by semicolons, and modular inputs as integers.

⚠️
Ignoring assumptions

Some methods require positive probabilities, valid moduli, independent trials, or small educational input sizes.

⚠️
Using arbitrary scores instead of probabilities

Cross entropy is defined on probability distributions. Classification logits must be converted to probabilities, commonly with softmax, before applying this probability-form expression.

⚠️
Confusing cross entropy with entropy

Entropy H(p) uses p inside the logarithm. Cross entropy H(p,q) uses q inside the logarithm and measures the cost of representing p using predictions q.

FAQ

Can I use decimals?⌄
Most numerical calculators allow decimals, but modular arithmetic and coding-theory tools usually require integers or binary strings.
Is this for homework checking?⌄
Yes. The page is designed to show both the answer and the reasoning pattern.
Why does the result sometimes say approximate?⌄
Some probability, floating-point, and numerical methods naturally produce approximations.
Why is a confident wrong prediction penalized so strongly?⌄
If the true class has target weight p > 0 while its predicted probability q approaches zero, −log(q) grows without bound. Cross entropy therefore strongly penalizes assigning almost no probability to an outcome that the target says matters.
Can cross entropy be zero?⌄
For a one-hot target, cross entropy reaches zero when the predicted probability of the true class is exactly 1. For a non-degenerate target distribution with positive entropy, the minimum cross entropy is H(p), which is greater than zero.
What happens if a predicted probability is zero?⌄
If pi > 0 and qi = 0, the corresponding information cost is infinite because log(0) is undefined in the finite real numbers. In numerical machine-learning code, probabilities are often clipped away from exact zero for stability.
What is the difference between bits and nats in cross entropy?⌄
Base-2 logarithms measure information in bits; natural logarithms measure it in nats. Converting between them changes the numerical scale by a constant factor, not which prediction has lower loss.

Related calculators

🔗 Set Operations Calculator🔗 Graph Theory and Adjacency Matrix Calculator🔗 Shannon Entropy and Information Gain Calculator🔗 Numerical Error Calculator

These links will work after the calculators are registered in the final Math layout update.

Formula Explorer connections

Interpretation: This relationship quantifies information, representation, storage, error, search or computational performance. Assumption: Use the exact encoding, data distribution, machine representation and algorithm assumptions. Real systems also include implementation and hardware overhead.

Hash Table Load Factor Calculator →Mutual Information Calculator →Shannon Entropy and Information Gain Calculator →Math Formula Explorer →