Documentation

Linglib.Core.InformationTheory.Entropy.Tilted

Entropy and normalized weights on a finite type #

On a finite type, the divergence of a probability measure q from a family of weights normalized to a probability measure, and the entropy of q, add up to the logarithm of the total weight less the expected log-weight under q; a row of the softmax kernel is the case of weights exp ∘ score. For the weights exp ∘ f the normalized measure is the counting measure tilted by f, and the identity reads KL(q ‖ count.tilted f) + H(q) = log ∑ exp f - ∫ f ∂q. So the expected value of f plus the entropy is at most log ∑ exp f, with equality exactly at the tilted measure: this is the Gibbs variational principle with the counting measure as reference, where InformationTheory.integral_sub_toReal_klDiv_le_log_integral_exp has a probability measure.

theorem InformationTheory.toReal_klDiv_ofWeights_add_measureEntropy {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] {q : MeasureTheory.Measure S} [MeasureTheory.IsProbabilityMeasure q] {ι : Type u_2} [MeasurableSpace ι] [Countable ι] [MeasurableSingletonClass ι] {w : ι → S → ENNReal} {i : ι} (hw : ∀ (s : S), w i s ≠ ⊤) (hq : q.AbsolutelyContinuous ((ProbabilityTheory.Kernel.ofWeights w) i)) :
(klDiv q ((ProbabilityTheory.Kernel.ofWeights w) i)).toReal + measureEntropy q = Real.log (∑ s : S, (w i s).toReal) - ∑ s : S, q.real {s} * Real.log (w i s).toReal

The divergence from normalized weights and the entropy add up to the log of the total weight less the expected log-weight.

theorem InformationTheory.toReal_klDiv_softmax_add_measureEntropy {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] {q : MeasureTheory.Measure S} [MeasureTheory.IsProbabilityMeasure q] {α : Type u_2} [MeasurableSpace α] [Countable α] [MeasurableSingletonClass α] {score : α → S → EReal} {a : α} (htop : ∀ (s : S), score a s ≠ ⊤) (hq : q.AbsolutelyContinuous ((ProbabilityTheory.Kernel.softmax score) a)) :
(klDiv q ((ProbabilityTheory.Kernel.softmax score) a)).toReal + measureEntropy q = Real.log (∑ s : S, (score a s).exp.toReal) - ∫ (s : S), (score a s).toReal ∂q

The divergence from a row of the softmax kernel and the entropy add up to the log of the row's normalizer less the expected score.

theorem InformationTheory.integral_toReal_add_measureEntropy_le_log_sum_exp {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] {q : MeasureTheory.Measure S} [MeasureTheory.IsProbabilityMeasure q] {α : Type u_2} [MeasurableSpace α] [Countable α] [MeasurableSingletonClass α] {score : α → S → EReal} {a : α} (htop : ∀ (s : S), score a s ≠ ⊤) (hq : q.AbsolutelyContinuous ((ProbabilityTheory.Kernel.softmax score) a)) :
∫ (s : S), (score a s).toReal ∂q + measureEntropy q ≤ Real.log (∑ s : S, (score a s).exp.toReal)

A row of the softmax kernel is the Gibbs optimizer: no distribution on the points it gives mass to has a greater expected score plus entropy than the log of its normalizer, which the row itself attains.

theorem InformationTheory.integral_add_measureEntropy_eq_sum {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] (q : MeasureTheory.Measure S) [MeasureTheory.IsProbabilityMeasure q] (f : S → ℝ) :
∫ (s : S), f s ∂q + measureEntropy q = ∑ s : S, q.real {s} * (f s - Real.log (q.real {s}))

The expected value of f plus the entropy, as a sum over the atoms.

theorem InformationTheory.toReal_klDiv_count_tilted_add_measureEntropy {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] (q : MeasureTheory.Measure S) [MeasureTheory.IsProbabilityMeasure q] (f : S → ℝ) :
(klDiv q (MeasureTheory.Measure.count.tilted f)).toReal + measureEntropy q = Real.log (∑ s : S, Real.exp (f s)) - ∫ (s : S), f s ∂q

The divergence from a tilt of the counting measure and the entropy add up to log ∑ exp f - ∫ f ∂q.

theorem InformationTheory.measureEntropy_count_tilted {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] (f : S → ℝ) [Nonempty S] :
measureEntropy (MeasureTheory.Measure.count.tilted f) = Real.log (∑ s : S, Real.exp (f s)) - ∫ (s : S), f s ∂MeasureTheory.Measure.count.tilted f

The entropy of a tilt of the counting measure is log ∑ exp f less the expected value of f.

theorem InformationTheory.integral_add_measureEntropy_le_log_sum_exp {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] (q : MeasureTheory.Measure S) [MeasureTheory.IsProbabilityMeasure q] (f : S → ℝ) :
∫ (s : S), f s ∂q + measureEntropy q ≤ Real.log (∑ s : S, Real.exp (f s))

Gibbs variational principle on a finite type: the expected value of f plus the entropy is at most log ∑ exp f.

theorem InformationTheory.integral_add_measureEntropy_eq_log_sum_exp_iff {S : Type u_1} [MeasurableSpace S] [MeasurableSingletonClass S] [Fintype S] (q : MeasureTheory.Measure S) [MeasureTheory.IsProbabilityMeasure q] (f : S → ℝ) :
∫ (s : S), f s ∂q + measureEntropy q = Real.log (∑ s : S, Real.exp (f s)) ↔ q = MeasureTheory.Measure.count.tilted f

The expected value of f plus the entropy is log ∑ exp f exactly at the counting measure tilted by f.