Documentation

Linglib.Core.InformationTheory.InformationLoss

The information loss of a decoder #

A state is drawn from a prior and sent through a channel, and a decoder assigns each output a distribution over states. The information the decoder loses is the expected surprisal of the state under its distribution at the output, infinite exactly when the decoder rules out a state that can produce an output. Over finite types the loss of any decoder is the conditional entropy of the state given the output plus the expected divergence of the posterior from it, so the posterior loses the least and only the posterior does.

References #

noncomputable def InformationTheory.informationLoss {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) (φ : ProbabilityTheory.Kernel 𝓧 Ω) :
ENNReal

The information the decoder φ loses about a state drawn from μ and sent through the channel κ: the expected surprisal of the state under the decoder's distribution at the output.

Equations
Instances For
    theorem InformationTheory.informationLoss_eq_sum {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) (φ : ProbabilityTheory.Kernel 𝓧 Ω) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] :
    informationLoss μ κ φ = ∑ x : 𝓧, ∑ ω : Ω, μ {ω} * (κ ω) {x} * surprisal (φ x) ω
    theorem InformationTheory.informationLoss_congr {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] {μ : MeasureTheory.Measure Ω} {κ : ProbabilityTheory.Kernel Ω 𝓧} {φ : ProbabilityTheory.Kernel 𝓧 Ω} [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] {φ' : ProbabilityTheory.Kernel 𝓧 Ω} (h : ⇑φ =ᵐ[μ.bind ⇑κ] ⇑φ') :
    informationLoss μ κ φ = informationLoss μ κ φ'

    The loss depends on the decoder only at the outputs that occur.

    theorem InformationTheory.informationLoss_eq_top_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] {μ : MeasureTheory.Measure Ω} {κ : ProbabilityTheory.Kernel Ω 𝓧} {φ : ProbabilityTheory.Kernel 𝓧 Ω} [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] :
    informationLoss μ κ φ = ⊤ ↔ ∃ (ω : Ω) (x : 𝓧), μ {ω} ≠ 0 ∧ (κ ω) {x} ≠ 0 ∧ (φ x) {ω} = 0

    The loss is infinite exactly when the decoder rules out a state that can produce an output.

    theorem InformationTheory.informationLoss_ne_top_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] {μ : MeasureTheory.Measure Ω} {κ : ProbabilityTheory.Kernel Ω 𝓧} {φ : ProbabilityTheory.Kernel 𝓧 Ω} [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] :
    informationLoss μ κ φ ≠ ⊤ ↔ ∀ (ω : Ω) (x : 𝓧), μ {ω} ≠ 0 → (κ ω) {x} ≠ 0 → (φ x) {ω} ≠ 0
    theorem InformationTheory.toReal_informationLoss {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] {μ : MeasureTheory.Measure Ω} {κ : ProbabilityTheory.Kernel Ω 𝓧} {φ : ProbabilityTheory.Kernel 𝓧 Ω} [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] [∀ (x : 𝓧), MeasureTheory.IsZeroOrProbabilityMeasure (φ x)] (h : ∀ (ω : Ω) (x : 𝓧), μ {ω} ≠ 0 → (κ ω) {x} ≠ 0 → (φ x) {ω} ≠ 0) :
    (informationLoss μ κ φ).toReal = -∑ ω : Ω, μ.real {ω} * ∫ (x : 𝓧), Real.log ((φ x).real {ω}) ∂κ ω

    A finite loss is minus the expected log-probability of the state.

    The posterior #

    theorem InformationTheory.informationLoss_posterior {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Nonempty Ω] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] :
    informationLoss μ κ (ProbabilityTheory.posterior κ μ) = ∑ x : 𝓧, (μ.bind ⇑κ) {x} * ENNReal.ofReal (measureEntropy ((ProbabilityTheory.posterior κ μ) x))

    The posterior loses the conditional entropy of the state given the output.

    theorem InformationTheory.informationLoss_eq_posterior_add {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) (φ : ProbabilityTheory.Kernel 𝓧 Ω) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Nonempty Ω] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] [∀ (x : 𝓧), MeasureTheory.IsZeroOrProbabilityMeasure (φ x)] :
    informationLoss μ κ φ = informationLoss μ κ (ProbabilityTheory.posterior κ μ) + ∑ x : 𝓧, (μ.bind ⇑κ) {x} * klDiv ((ProbabilityTheory.posterior κ μ) x) (φ x)

    Any decoder loses what the posterior loses and, besides, the expected divergence of the posterior from it over the outputs.

    theorem InformationTheory.informationLoss_posterior_le {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) (φ : ProbabilityTheory.Kernel 𝓧 Ω) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Nonempty Ω] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] [∀ (x : 𝓧), MeasureTheory.IsZeroOrProbabilityMeasure (φ x)] :
    informationLoss μ κ (ProbabilityTheory.posterior κ μ) ≤ informationLoss μ κ φ

    The posterior loses the least information.

    theorem InformationTheory.informationLoss_posterior_ne_top {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Nonempty Ω] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] :
    informationLoss μ κ (ProbabilityTheory.posterior κ μ) ≠ ⊤
    theorem InformationTheory.informationLoss_eq_posterior_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] (μ : MeasureTheory.Measure Ω) (κ : ProbabilityTheory.Kernel Ω 𝓧) (φ : ProbabilityTheory.Kernel 𝓧 Ω) [Fintype Ω] [Fintype 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Nonempty Ω] [MeasureTheory.IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] [∀ (x : 𝓧), MeasureTheory.IsZeroOrProbabilityMeasure (φ x)] :
    informationLoss μ κ φ = informationLoss μ κ (ProbabilityTheory.posterior κ μ) ↔ ⇑φ =ᵐ[μ.bind ⇑κ] ⇑(ProbabilityTheory.posterior κ μ)

    Only the posterior loses the least information: a decoder that loses no more agrees with it at every output that occurs.