Documentation

Linglib.Core.InformationTheory.KullbackLeibler.Tilted

Kullback–Leibler divergence from a tilted measure #

For probability measures q and μ and a function f, the divergence of q from the tilted measure μ.tilted f is KL(q ‖ μ) - ∫ f ∂q + log ∫ exp f ∂μ. Since a divergence is nonnegative and vanishes only between equal measures, ∫ f ∂q - KL(q ‖ μ) is at most log ∫ exp f ∂μ, with equality exactly at μ.tilted f. This is the Gibbs variational principle. Read as a lower bound on the divergence it is the Donsker–Varadhan inequality, which holds in ℝ≥0∞ with no condition relating q to μ.

[UPSTREAM] candidate for Mathlib/InformationTheory/KullbackLeibler/Tilted.lean: the module docstring of Mathlib/MeasureTheory/Measure/Tilted.lean names variational expressions for the divergence as a use of tilting by a general function.

Implementation notes #

The statements with toReal assume q ≪ μ and the integrability of llr q μ, as the toReal_klDiv lemmas of mathlib do, since toReal reads an infinite divergence as 0. μ is a probability measure because the divergence from a finite measure carries the correction μ.real univ - q.real univ; MeasureTheory.integral_llr_tilted_right is the statement for a σ-finite μ.

theorem MeasureTheory.integrable_llr_tilted_left_self {α : Type u_1} {mα : MeasurableSpace α} {μ : Measure α} {f : α → ℝ} [SigmaFinite μ] (hf : Integrable f (μ.tilted f)) (h_exp : Integrable (fun (x : α) => Real.exp (f x)) μ) :
Integrable (llr (μ.tilted f) μ) (μ.tilted f)

The log-likelihood ratio of a tilted measure to its base measure is integrable against the tilted measure as soon as the tilting function is.

theorem MeasureTheory.integrable_llr_tilted_right_iff {α : Type u_1} {mα : MeasurableSpace α} {μ q : Measure α} {f : α → ℝ} [IsFiniteMeasure q] [SigmaFinite μ] (hqμ : q.AbsolutelyContinuous μ) (hf : Integrable f q) (h_exp : Integrable (fun (x : α) => Real.exp (f x)) μ) :
Integrable (llr q (μ.tilted f)) q ↔ Integrable (llr q μ) q

Tilting the reference measure does not change whether the log-likelihood ratio is integrable.

theorem InformationTheory.klDiv_tilted_right_ne_top_iff {α : Type u_1} {mα : MeasurableSpace α} {μ q : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure q] (hf : MeasureTheory.Integrable f q) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
klDiv q (μ.tilted f) ≠ ⊤ ↔ klDiv q μ ≠ ⊤

Tilting the reference measure does not change whether the divergence is finite.

theorem InformationTheory.toReal_klDiv_tilted_right {α : Type u_1} {mα : MeasurableSpace α} {μ q : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure q] (hqμ : q.AbsolutelyContinuous μ) (h_int : MeasureTheory.Integrable (MeasureTheory.llr q μ) q) (hf : MeasureTheory.Integrable f q) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
(klDiv q (μ.tilted f)).toReal = (klDiv q μ).toReal - ∫ (x : α), f x ∂q + Real.log (∫ (x : α), Real.exp (f x) ∂μ)

The divergence from a tilted measure is KL(q ‖ μ.tilted f) = KL(q ‖ μ) - ∫ f ∂q + log ∫ exp f ∂μ.

theorem InformationTheory.toReal_klDiv_tilted_left_self {α : Type u_1} {mα : MeasurableSpace α} {μ : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] (hf : MeasureTheory.Integrable f (μ.tilted f)) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
(klDiv (μ.tilted f) μ).toReal = ∫ (x : α), f x ∂μ.tilted f - Real.log (∫ (x : α), Real.exp (f x) ∂μ)

The divergence of a tilted measure from its base measure is KL(μ.tilted f ‖ μ) = ∫ f ∂(μ.tilted f) - log ∫ exp f ∂μ.

theorem InformationTheory.integral_sub_toReal_klDiv_le_log_integral_exp {α : Type u_1} {mα : MeasurableSpace α} {μ q : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure q] (hqμ : q.AbsolutelyContinuous μ) (h_int : MeasureTheory.Integrable (MeasureTheory.llr q μ) q) (hf : MeasureTheory.Integrable f q) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
∫ (x : α), f x ∂q - (klDiv q μ).toReal ≤ Real.log (∫ (x : α), Real.exp (f x) ∂μ)

Gibbs variational principle: ∫ f ∂q - KL(q ‖ μ) is at most log ∫ exp f ∂μ.

theorem InformationTheory.integral_sub_toReal_klDiv_eq_log_integral_exp_iff {α : Type u_1} {mα : MeasurableSpace α} {μ q : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure q] (hqμ : q.AbsolutelyContinuous μ) (h_int : MeasureTheory.Integrable (MeasureTheory.llr q μ) q) (hf : MeasureTheory.Integrable f q) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
∫ (x : α), f x ∂q - (klDiv q μ).toReal = Real.log (∫ (x : α), Real.exp (f x) ∂μ) ↔ q = μ.tilted f

∫ f ∂q - KL(q ‖ μ) is log ∫ exp f ∂μ exactly when q is the tilted measure.

theorem InformationTheory.ofReal_integral_sub_log_integral_exp_le_klDiv {α : Type u_1} {mα : MeasurableSpace α} {μ q : MeasureTheory.Measure α} {f : α → ℝ} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure q] (hf : MeasureTheory.Integrable f q) (h_exp : MeasureTheory.Integrable (fun (x : α) => Real.exp (f x)) μ) :
ENNReal.ofReal (∫ (x : α), f x ∂q - Real.log (∫ (x : α), Real.exp (f x) ∂μ)) ≤ klDiv q μ

Donsker–Varadhan inequality: the divergence of q from μ is at least ∫ f ∂q - log ∫ exp f ∂μ, for every f integrable against q with exp ∘ f integrable against μ.