Entropy and normalized weights on a finite type #
On a finite type, the divergence of a probability measure q from a family of weights normalized to
a probability measure, and the entropy of q, add
up to the logarithm of the total
weight less the expected log-weight under q; a row of the softmax kernel is the
case of weights exp ∘ score. For the
weights exp ∘ f the normalized measure is the counting measure tilted by f, and the identity
reads KL(q ‖ count.tilted f) + H(q) = log ∑ exp f - ∫ f ∂q. So the expected value of f plus the entropy is at
most log ∑ exp f, with equality
exactly at the tilted measure:
this is the Gibbs variational principle with the counting measure as reference, where
InformationTheory.integral_sub_toReal_klDiv_le_log_integral_exp has a probability measure.
The divergence from normalized weights and the entropy add up to the log of the total weight less the expected log-weight.
The divergence from a row of the softmax kernel and the entropy add up to the log of the row's normalizer less the expected score.
A row of the softmax kernel is the Gibbs optimizer: no distribution on the points it gives mass to has a greater expected score plus entropy than the log of its normalizer, which the row itself attains.
The expected value of f plus the entropy, as a sum over the atoms.
The divergence from a tilt of the counting measure and the entropy add up to
log ∑ exp f - ∫ f ∂q.
The entropy of a tilt of the counting measure is log ∑ exp f less the expected value of
f.
Gibbs variational principle on a finite type: the expected value of f plus the entropy
is at most log ∑ exp f.
The expected value of f plus the entropy is log ∑ exp f exactly at the counting measure
tilted by f.