Claude Shannon introduced entropy in 1948 as the measure of information at the heart of communication theory. The entropy of a random outcome is the average amount of surprise it carries: an event with probability p carries −log p of information, and entropy averages that over all outcomes. A fair coin has 1 bit of entropy; a coin that always lands heads has 0 bits, because there is nothing to learn from tossing it.
Entropy is greatest when all K outcomes are equally likely, H_max = log K, and the efficiency H ÷ H_max shows how close a distribution comes to that maximum. The perplexity b^H is the “effective number” of equally likely outcomes, a measure used to evaluate language models. In data compression, entropy in bits per symbol is the lower limit for the average code length – no lossless code can beat it.
The same formula appears far beyond engineering: ecologists use it as the Shannon diversity index, geneticists for sequence variability, and machine-learning algorithms such as decision trees use it to pick the most informative split. Paste a word or a string of digits and the calculator counts the symbols for you.
How to use the Shannon Entropy Calculator
- Input
- Probabilities or counts
- Text or sequence
- Symbols are
- Logarithm base (unit)
Everything is calculated in your browser, so results appear instantly and nothing you type is sent to a server.
How it works
Enter probabilities that add to 1, raw counts (which are turned into probabilities) or a text. For text, each character – or each space-separated word – is a symbol. The calculator multiplies each probability by its logarithm in the chosen base, adds the terms and changes the sign.
Worked example
A source emits A, B, C and D with probabilities 0.5, 0.25, 0.125 and 0.125. The surprisals are 1, 2, 3 and 3 bits, so H = 0.5 × 1 + 0.25 × 2 + 0.125 × 3 + 0.125 × 3 = 1.75 bits. The maximum for four symbols is log₂ 4 = 2 bits, giving efficiency 87.5%, redundancy 12.5% and perplexity 2^1.75 = 3.36. A prefix code A = 0, B = 10, C = 110, D = 111 reaches exactly 1.75 bits per symbol on average.
Entropy of familiar sources
| Source | Entropy (bits) |
|---|---|
| Fair coin | 1 |
| Coin with 90% heads | 0.469 |
| Fair six-sided die | 2.585 |
| Random decimal digit | 3.322 |
| Random letter A–Z | 4.700 |
Common mistakes to avoid
- Mixing percentages and decimals between fields.
- Using a rule of thumb outside the range where it applies.
- Treating an estimate as an exact answer.
Formula
H = −Σ pᵢ · log_b(pᵢ) · H_max = log_b(K) · efficiency = H ÷ H_max · perplexity = b^H · b = 2 bits, e nats, 10 hartleys
Worked example
Inputs — Input: Probabilities (must add up to 1); Probabilities or counts: 0.5, 0.25, 0.125, 0.125; Logarithm base (unit): 2 – bits (shannons).
Results — Shannon entropy H = 1.75; Unit = bits; Maximum entropy log K = 2; Efficiency (normalised entropy) = 87.5 %; Redundancy = 12.5 %; Perplexity (effective number of outcomes) = 3.3636.
Key terms
- Information content (surprisal)
- −log p, the information gained when an outcome of probability p occurs.
- Entropy
- The expected surprisal over all outcomes of a distribution.
- Perplexity
- b^H, the number of equally likely outcomes that would give the same entropy.
- Redundancy
- 1 − H ÷ H_max, how far a source is from maximum entropy.
Frequently asked questions
What is Shannon entropy in simple terms?
The average number of yes/no questions (bits) needed to identify an outcome. A fair die has log₂ 6 = 2.585 bits; a fair coin exactly 1 bit.
What is the entropy of 0.5, 0.25, 0.125, 0.125?
H = 0.5·1 + 0.25·2 + 2 × 0.125·3 = 1.75 bits – less than the 2-bit maximum for four outcomes, so efficiency is 87.5%.
When is entropy zero?
When one outcome is certain (p = 1). There is no uncertainty and nothing to learn.
What units does entropy use?
Bits with log base 2, nats with natural logs and hartleys (bans) with base 10. 1 nat = 1.4427 bits and 1 hartley = 3.3219 bits.
What is the entropy of the sequence 1035830701?
Symbol frequencies 0: 3, 1: 2, 3: 2, 5, 7, 8: 1 each give H = 2.446 bits per symbol and 24.5 bits for all 10 digits.
How is Shannon entropy used in ecology?
As the Shannon–Wiener diversity index, usually with natural logs, computed from species proportions. Higher H means a more diverse community.
Do the probabilities have to add up to 1?
Yes in probability mode. If you have counts, choose "counts or frequencies" and they are normalised automatically.