Gibbs' inequality for finite sums #
For weights $p$ summing to $1$ and $q$ summing to at most $1$,
Gibbs' inequality says
$\prod_i q_i^{p_i} \le \prod_i p_i^{p_i}$: the geometric mean of $q$ weighted by $p$ is greatest
at $q = p$. It is the weighted inequality of arithmetic and geometric means applied to the ratios
$q_i / p_i$, which bounds the left side by $\sum_i q_i$ times the
right with no condition on $q$. Its logarithm says
the cross-entropy of $q$ relative to $p$ is at least the entropy of $p$; mathlib has that for
measures as InformationTheory.integral_llr_add_sub_measure_univ_nonneg.
With the relative frequencies of counts $x$ as the weights, it bounds the likelihood of the counts under any $q$ by their likelihood under their relative frequencies, so the maximum-likelihood estimate of a categorical distribution is the empirical one. Equality holds only at $q = p$, so the estimate is unique.
Implementation notes #
[UPSTREAM] candidate for the AM-GM section of Mathlib/Analysis/MeanInequalities.lean. The
Each statement comes in Real, NNReal and ENNReal versions with the arguments of the
corresponding geom_mean_le_arith_mean_weighted; the ℝ≥0∞ equality case is transferred from the
real one.
References #
Gibbs' inequality, homogeneous in q: the geometric mean of q weighted by p is at
most the total of q times the geometric mean of p weighted by itself.
Gibbs' inequality: the geometric mean of q weighted by p, over q summing to at most
1, is greatest at q = p.
Gibbs' inequality, equality case: the geometric mean of q weighted by p, over q
summing to at most 1, attains its greatest value only at q = p.
Gibbs' inequality: the geometric mean of q weighted by p, over q summing to at most
1, is greatest at q = p.
Gibbs' inequality, equality case: the geometric mean of q weighted by p, over q
summing to at most 1, attains its greatest value only at q = p.
Gibbs' inequality, homogeneous in q: the geometric mean of q weighted by p is at
most the total of q times the geometric mean of p weighted by itself.
Gibbs' inequality: the geometric mean of q weighted by p, over q summing to at most
1, is greatest at q = p.
Gibbs' inequality, equality case: the geometric mean of q weighted by p, over q
summing to at most 1, attains its greatest value only at q = p.
Maximum likelihood of a categorical distribution. Under weights w summing to at most
1, counts x have likelihood at most their likelihood under their relative frequencies.
Uniqueness of the maximum-likelihood estimate. Under weights w summing to at most 1,
counts x that are not all 0 attain their greatest likelihood only at their relative
frequencies.