Shannon Entropy Calculator

It reads the frequency table, never the sequence

The tool ships the demonstration as two presets, because the claim is exact and easy to check.

AAAABBBCCD gives 1.8464393447 bits. ABACABDCBA — the same ten symbols, shuffled — gives 1.8464393447. Every number on the page is unchanged.

Which means entropy cannot detect a pattern. ABABABABAB and AABBAABBAA and a genuinely random string with the same letter counts all score identically, and one of them is perfectly predictable.

So entropy is not a randomness test. For that you need something that looks at order — a runs test on the sequence, or a higher-order entropy that counts pairs and triples rather than single symbols.

The fix is conditional entropy. H of the next symbol GIVEN the previous one drops sharply for ABABAB and not at all for a random string, which is how compression algorithms and language models capture what single-symbol entropy misses.

Which is also why English text has an entropy of about 4.1 bits per letter by single-letter frequencies and roughly 1.1 bits once context is taken into account — the gap is entirely the order this measure ignores.

The maximum is uniform, and efficiency measures the distance

Entropy is bounded above by log₂ of the number of distinct symbols, and that bound is reached only at a uniform distribution.

Four symbols cap at 2 bits. The tool’s third preset — three of each of four symbols — hits exactly 2.0000000000, and its efficiency is exactly 1.

Efficiency is H divided by that maximum, so it runs from 0 to 1 and is comparable across alphabets of different sizes in a way raw entropy is not.

Redundancy is the leftover, log₂k − H, and it is what a compressor can exploit. A source with 0.15 bits of redundancy per symbol can be compressed by about 7.5% before any modelling of order.

Adding a symbol that almost never occurs raises the maximum more than the entropy, so efficiency falls — which is why efficiency, not entropy, is the right measure when the number of categories varies.

Perplexity turns bits into a count you can picture

Bits are hard to feel. 2 raised to the entropy is not.

Perplexity is 2ᴴ, the size of the uniform distribution carrying the same uncertainty. The first preset has entropy 1.8464 bits and a perplexity of 3.5961 — as uncertain as a fair choice among about three and a half options.

A fair coin has perplexity 2 and a fair die 6, by construction, which is what makes the number readable.

It is the standard reporting unit for language models, where “perplexity 20” means the model is as uncertain about the next word as if choosing uniformly among twenty.

The inverse Simpson index is a near-relative, 1/Σp², and is the “effective number of species” in ecology. On the first preset it is 3.3333 against a perplexity of 3.5961 — both answer “how many equally common categories is this worth?” and weight rare categories differently.

Bits, nats and bans differ by a constant and nothing else

The choice of logarithm base is a choice of unit, like metres against feet. Nothing about the distribution changes.

Bits use log₂ and are the default wherever information is stored or transmitted, because a bit is a binary decision.

Nats use the natural log and are standard in statistics and machine learning, because derivatives of ln are tidier. H_nats = H_bits × ln 2 ≈ 0.693 × H_bits.

Bans use log₁₀ and are the oldest of the three — named for Banbury, where the sheets used at Bletchley Park were printed.

The tool prints all three because papers mix them freely and a cross-entropy or KL divergence quoted without its unit is ambiguous by a factor of 1.44.

Entropy estimated from a sample is biased downward

Everything above treats the observed frequencies as the truth. When they are a sample, the plug-in estimate is systematically too small.

Symbols you did not observe contribute nothing, so a sample of 20 from a 1,000-symbol alphabet reports at most log₂20 = 4.32 bits however uniform the source is.

The bias is roughly (k−1)/(2n ln 2) bits, where k is the number of distinct symbols actually seen — the Miller-Madow correction adds it back, and it is large exactly when n is small relative to k.

Which makes entropy comparisons across samples of different sizes unsound unless corrected, in the same way that eta squared is biased upward by its own degrees of freedom.

This tool reports the plug-in estimate on the data you give it, which is the right answer when the list IS the population — a market’s shares, a text’s letters — and a downward-biased one when it is a sample from something larger.

Reporting an entropy

Four things, and the first is the one that makes a figure comparable at all.

Give the unit. Bits, nats or bans — a bare “entropy 2.4” differs by a factor of 1.44 depending on which was meant.

Give the number of distinct symbols. It sets the maximum, and without it the reader cannot tell whether 2.4 bits is nearly uniform or badly concentrated.

Prefer efficiency when alphabets differ. It is entropy divided by its own maximum, so it compares across sources with different numbers of categories.

And say whether the data is the population or a sample. The plug-in estimate is unbiased for the first and too small for the second.

Sources and methodology

References for Shannon entropy and its estimation.

• Shannon, “A Mathematical Theory of Communication”, Bell System Technical Journal 27, 1948
• Paninski, “Estimation of Entropy and Mutual Information”, Neural Computation 15(6), 2003 — on the downward bias
• Jost, “Entropy and Diversity”, Oikos 113(2), 2006 — on effective numbers
• Mutual Information Calculator — entropy shared between two variables
• Gini Coefficient Calculator — a different way to measure unevenness
• How we calculate — methodology

Method. Entropy is computed as −Σp log₂p over the observed frequencies, with the per-symbol contributions retained so the table sums to the reported total rather than being derived from it. The order-blindness claim is a preset pair rather than a sentence: the shipped sorted and shuffled sequences contain the identical multiset of symbols, and the suite asserts that every generated permutation of a sequence produces a bit-identical entropy. Nats and bans are the same quantity scaled by ln 2 and ln 2/ln 10 rather than recomputed, so the three can never disagree. Perplexity is 2 raised to the entropy and the inverse Simpson index is computed independently from the squared probabilities, which is why the two “effective number” measures differ on the same data rather than being one number twice. An empty input returns no result. That engine is verified on every change against 100 assertions. The count and the per-case breakdown are published on the formula verification page.

Related calculators

Where this goes next:

More in Math, or browse all calculators.