Suppose we have a random variable X, which may take on n distinct values 0, 1, 2, ... n-1 with probability p_i for taking value i. If we don't care about the ordering of the values, we may as well assume they are sorted in descending order by probability: p_i > p_i+1. The Shannon entropy of this system, E = -Σ p_i log₂ p_i, gives a lower bound on the expected path length of a binary code for X, that is, a binary tree having X's values at its leaves (in any order).

But there is another formula we can use that defines an entropy based on ranks: R = Σ p_ilog₂(i+1).

( Read more... )

ETA: Why I care about this.