Kullback–Leibler divergence from a tilted measure #
For probability measures q and μ and a function f, the divergence of q from the tilted
measure μ.tilted f is
KL(q ‖ μ) - ∫ f ∂q + log ∫ exp f ∂μ. Since a divergence is nonnegative and vanishes only
between equal measures, ∫ f ∂q - KL(q ‖ μ)
is at most
log ∫ exp f ∂μ, with equality
exactly at
μ.tilted f. This is the Gibbs variational principle. Read as a lower bound on the divergence it
is the Donsker–Varadhan inequality, which
holds in ℝ≥0∞ with no
condition relating q to μ.
[UPSTREAM] candidate for Mathlib/InformationTheory/KullbackLeibler/Tilted.lean: the module
docstring of Mathlib/MeasureTheory/Measure/Tilted.lean names variational expressions for the
divergence as a use of tilting by a general function.
Implementation notes #
The statements with toReal assume q ≪ μ and the integrability of llr q μ, as the
toReal_klDiv lemmas of mathlib do, since toReal reads an infinite divergence as 0. μ is
a probability measure because the divergence from a finite measure carries the correction
μ.real univ - q.real univ; MeasureTheory.integral_llr_tilted_right is the statement for a
σ-finite μ.
The log-likelihood ratio of a tilted measure to its base measure is integrable against the tilted measure as soon as the tilting function is.
Tilting the reference measure does not change whether the log-likelihood ratio is integrable.
Tilting the reference measure does not change whether the divergence is finite.
The divergence from a tilted measure is
KL(q ‖ μ.tilted f) = KL(q ‖ μ) - ∫ f ∂q + log ∫ exp f ∂μ.
The divergence of a tilted measure from its base measure is
KL(μ.tilted f ‖ μ) = ∫ f ∂(μ.tilted f) - log ∫ exp f ∂μ.
Gibbs variational principle: ∫ f ∂q - KL(q ‖ μ) is at most log ∫ exp f ∂μ.
∫ f ∂q - KL(q ‖ μ) is log ∫ exp f ∂μ exactly when q is the tilted measure.
Donsker–Varadhan inequality: the divergence of q from μ is at least
∫ f ∂q - log ∫ exp f ∂μ, for every f integrable against q with exp ∘ f integrable against
μ.