The information loss of a decoder #
A state is drawn from a prior and sent through a channel, and a decoder assigns each output a distribution over states. The information the decoder loses is the expected surprisal of the state under its distribution at the output, infinite exactly when the decoder rules out a state that can produce an output. Over finite types the loss of any decoder is the conditional entropy of the state given the output plus the expected divergence of the posterior from it, so the posterior loses the least and only the posterior does.
References #
- [CT06], chapter 2.
The information the decoder φ loses about a state drawn from μ and sent through the
channel κ: the expected surprisal of the state under the decoder's distribution at the output.
Equations
- InformationTheory.informationLoss μ κ φ = ∫⁻ (ω : Ω), ∫⁻ (x : 𝓧), InformationTheory.surprisal (φ x) ω ∂κ ω ∂μ
Instances For
The loss depends on the decoder only at the outputs that occur.
The loss is infinite exactly when the decoder rules out a state that can produce an output.
A finite loss is minus the expected log-probability of the state.
The posterior #
The posterior loses the conditional entropy of the state given the output.
Any decoder loses what the posterior loses and, besides, the expected divergence of the posterior from it over the outputs.
The posterior loses the least information.
Only the posterior loses the least information: a decoder that loses no more agrees with it at every output that occurs.