Documentation

Linglib.Core.Probability.Kernel.Posterior

Exact Bayes for the posterior kernel at atoms #

Mathlib characterizes the posterior kernel κ†μ almost everywhere. On discrete spaces an ae-fact holds at every atom of positive mass, which gives Bayes' rule pointwise at any positive-mass observation, with no Radon–Nikodym derivative. Drawing a state from the posterior given an observation of a state drawn from the prior leaves the prior invariant.

Main results #

Priors carried by two atoms #

The _of_pair lemmas assume the prior's support lies in a pair {ω, ω'}, so that the observation's marginal has two terms and the comparison of the posterior with the prior at one atom is the comparison of the two likelihoods.

theorem MeasureTheory.Measure.comp_real_singleton_of_pair {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Fintype Ω] (κ : ProbabilityTheory.Kernel Ω 𝓧) (μ : Measure Ω) {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') [IsFiniteMeasure μ] [ProbabilityTheory.IsFiniteKernel κ] (x : 𝓧) :
(μ.bind ⇑κ).real {x} = μ.real {ω} * (κ ω).real {x} + μ.real {ω'} * (κ ω').real {x}
theorem MeasureTheory.measureReal_singleton_add_singleton_of_pair {Ω : Type u_1} [MeasurableSpace Ω] [MeasurableSingletonClass Ω] [Fintype Ω] (μ : Measure Ω) {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') [IsProbabilityMeasure μ] :
μ.real {ω} + μ.real {ω'} = 1
theorem MeasureTheory.Measure.comp_apply_singleton_ne_zero_of_pair {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [Fintype Ω] (κ : ProbabilityTheory.Kernel Ω 𝓧) (μ : Measure Ω) {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') [IsProbabilityMeasure μ] {x : 𝓧} (hκω : (κ ω) {x} ≠ 0) (hκω' : (κ ω') {x} ≠ 0) :
(μ.bind ⇑κ) {x} ≠ 0
theorem MeasureTheory.integral_of_pair {A : Type u_3} [Fintype A] [MeasurableSpace A] [DiscreteMeasurableSpace A] {ν : Measure A} [IsProbabilityMeasure ν] {f : A → ℝ} {a a' : A} (haa' : a ≠ a') (hall : ∀ (c : A), c = a ∨ c = a') :
∫ (c : A), f c ∂ν = ν.real {a} * f a + (1 - ν.real {a}) * f a'

On a type with two points, an integral against a probability measure weighs the value at the first by its mass and the value at the second by the rest.

theorem MeasureTheory.integral_pi_single {X : Type u_3} [Fintype X] [DecidableEq X] [MeasurableSpace X] [DiscreteMeasurableSpace X] (ν : Measure X) [IsFiniteMeasure ν] (x : X) :
∫ (y : X), Pi.single y 1 x ∂ν = ν.real {x}

The integral of the indicator of a point, as a function of the point integrated, is the mass of the point.

theorem ProbabilityTheory.posterior_apply_singleton {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (ω : Ω) :
((posterior κ μ) x) {ω} = μ {ω} * (κ ω) {x} / (μ.bind ⇑κ) {x}

At an observation of positive mass, the posterior mass of a state is given exactly by Bayes' rule.

theorem ProbabilityTheory.posterior_apply_singleton_ne_zero_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (ω : Ω) :
((posterior κ μ) x) {ω} ≠ 0 ↔ μ {ω} ≠ 0 ∧ (κ ω) {x} ≠ 0
theorem ProbabilityTheory.posterior_deterministic_eq_cond {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [Countable Ω] {f : Ω → 𝓧} (hf : Measurable f) {x : 𝓧} (hx : μ (f ⁻¹' {x}) ≠ 0) :
(posterior (Kernel.deterministic f hf) μ) x = μ[|f ⁻¹' {x}]

A deterministic observation's posterior is the prior conditioned on the observation's fibre.

theorem ProbabilityTheory.posterior_real_singleton {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (ω : Ω) :
((posterior κ μ) x).real {ω} = μ.real {ω} * (κ ω).real {x} / (μ.bind ⇑κ).real {x}
theorem ProbabilityTheory.ae_eq_posterior_of_eq_of_ne_zero {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] [Countable 𝓧] (hκ : ∀ (x : 𝓧) (ω ω' : Ω), (κ ω) {x} ≠ 0 → (κ ω') {x} ≠ 0 → (κ ω) {x} = (κ ω') {x}) {η : Kernel 𝓧 Ω} (w : Ω → ENNReal) {c : ENNReal} (hc₀ : c ≠ 0) (hc : c ≠ ⊤) (hμ : ∀ (ω : Ω), μ {ω} = c * w ω) (hη : ∀ (x : 𝓧), (μ.bind ⇑κ) {x} ≠ 0 → ∀ (ω : Ω), (η x) {ω} = (if (κ ω) {x} = 0 then 0 else w ω) / ∑ ω' : Ω, if (κ ω') {x} = 0 then 0 else w ω') :
⇑η =ᵐ[μ.bind ⇑κ] ⇑(posterior κ μ)

If each observation is equally likely at every state that can emit it, a kernel that weighs the states that can emit x by w is the posterior under any prior proportional to w, on the observations the prior predicts.

theorem ProbabilityTheory.comp_real_mul_posterior_real {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] (ω : Ω) (x : 𝓧) :
(μ.bind ⇑κ).real {x} * ((posterior κ μ) x).real {ω} = μ.real {ω} * (κ ω).real {x}

At every state and observation, the observation marginal times the posterior is the prior times the likelihood, Bayes' rule on reals with the denominator cleared.

theorem ProbabilityTheory.real_lt_posterior_real_singleton_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) {ω : Ω} (hω : μ {ω} ≠ 0) :
μ.real {ω} < ((posterior κ μ) x).real {ω} ↔ (μ.bind ⇑κ).real {x} < (κ ω).real {x}
theorem ProbabilityTheory.posterior_real_singleton_lt_iff {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) {ω : Ω} (hω : μ {ω} ≠ 0) :
((posterior κ μ) x).real {ω} < μ.real {ω} ↔ (κ ω).real {x} < (μ.bind ⇑κ).real {x}

Expectations under the posterior #

Bayes' rule reweights the prior by the likelihood, so the posterior expectation of a statistic compares with its prior expectation as the statistic's covariance with the likelihood: Chebyshev's integral inequality.

theorem ProbabilityTheory.integral_posterior {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (f : Ω → ℝ) :
∫ (ω : Ω), f ω ∂(posterior κ μ) x = (∫ (ω : Ω), f ω * (κ ω).real {x} ∂μ) / (μ.bind ⇑κ).real {x}
theorem ProbabilityTheory.integral_le_integral_posterior {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) [MeasureTheory.IsProbabilityMeasure μ] {f : Ω → ℝ} (hf : MonovaryOn f (fun (ω : Ω) => (κ ω).real {x}) {ω : Ω | μ {ω} ≠ 0}) :
∫ (ω : Ω), f ω ∂μ ≤ ∫ (ω : Ω), f ω ∂(posterior κ μ) x

Conditioning on an observation raises the expectation of a statistic that monovaries with the observation's likelihood on the states of positive prior mass.

theorem ProbabilityTheory.integral_posterior_le_integral {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) [MeasureTheory.IsProbabilityMeasure μ] {f : Ω → ℝ} (hf : AntivaryOn f (fun (ω : Ω) => (κ ω).real {x}) {ω : Ω | μ {ω} ≠ 0}) :
∫ (ω : Ω), f ω ∂(posterior κ μ) x ≤ ∫ (ω : Ω), f ω ∂μ
theorem ProbabilityTheory.posterior_real_singleton_lt_posterior_of_pair {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') {ν : MeasureTheory.Measure Ω} [MeasureTheory.IsProbabilityMeasure μ] [MeasureTheory.IsProbabilityMeasure ν] (hν : ∀ (ω'' : Ω), ν {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') {x : 𝓧} (hκω : (κ ω) {x} ≠ 0) (hκω' : (κ ω') {x} ≠ 0) (hμν : μ.real {ω} < ν.real {ω}) :
((posterior κ μ) x).real {ω} < ((posterior κ ν) x).real {ω}

Under two priors carried by the same two atoms, at an observation both atoms emit with positive likelihood, the posterior mass of an atom is strictly increasing in its prior mass, so Bayes' rule is prior-sensitive.

theorem ProbabilityTheory.real_lt_posterior_real_singleton_iff_of_pair {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') [MeasureTheory.IsProbabilityMeasure μ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (hω : μ {ω} ≠ 0) (hω' : μ {ω'} ≠ 0) :
μ.real {ω} < ((posterior κ μ) x).real {ω} ↔ (κ ω').real {x} < (κ ω).real {x}
theorem ProbabilityTheory.posterior_real_singleton_lt_iff_of_pair {Ω : Type u_1} {𝓧 : Type u_2} [MeasurableSpace Ω] [MeasurableSpace 𝓧] [MeasurableSingletonClass Ω] [MeasurableSingletonClass 𝓧] [StandardBorelSpace Ω] [Nonempty Ω] (κ : Kernel Ω 𝓧) (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] [IsFiniteKernel κ] [Fintype Ω] {ω ω' : Ω} (hne : ω ≠ ω') (hsupp : ∀ (ω'' : Ω), μ {ω''} ≠ 0 → ω'' = ω ∨ ω'' = ω') [MeasureTheory.IsProbabilityMeasure μ] {x : 𝓧} (hx : (μ.bind ⇑κ) {x} ≠ 0) (hω : μ {ω} ≠ 0) (hω' : μ {ω'} ≠ 0) :
((posterior κ μ) x).real {ω} < μ.real {ω} ↔ (κ ω).real {x} < (κ ω').real {x}

Stationarity #

A state drawn from the posterior given an observation of a state drawn from the prior is distributed as the prior. The chains that alternate the likelihood and the posterior therefore leave the prior, the prior predictive and their joint invariant; the last is the Gibbs sampler for the joint.

theorem ProbabilityTheory.invariant_comp_of_ae_eq_posterior {Ω : Type u_3} {𝓧 : Type u_4} {mΩ : MeasurableSpace Ω} {m𝓧 : MeasurableSpace 𝓧} [StandardBorelSpace Ω] [Nonempty Ω] {κ : Kernel Ω 𝓧} [IsMarkovKernel κ] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] {η : Kernel 𝓧 Ω} (hη : ⇑η =ᵐ[μ.bind ⇑κ] ⇑(posterior κ μ)) :
(η.comp κ).Invariant μ

A kernel equal to the posterior on the observations the prior predicts, composed after the likelihood, leaves the prior invariant.

theorem ProbabilityTheory.invariant_posterior_comp {Ω : Type u_3} {𝓧 : Type u_4} {mΩ : MeasurableSpace Ω} {m𝓧 : MeasurableSpace 𝓧} [StandardBorelSpace Ω] [Nonempty Ω] {κ : Kernel Ω 𝓧} [IsMarkovKernel κ] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] :
((posterior κ μ).comp κ).Invariant μ
theorem ProbabilityTheory.invariant_comp_posterior {Ω : Type u_3} {𝓧 : Type u_4} {mΩ : MeasurableSpace Ω} {m𝓧 : MeasurableSpace 𝓧} [StandardBorelSpace Ω] [Nonempty Ω] {κ : Kernel Ω 𝓧} [IsMarkovKernel κ] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] :
(κ.comp (posterior κ μ)).Invariant (μ.bind ⇑κ)
theorem ProbabilityTheory.invariant_id_prod_comp_prodMkLeft_posterior {Ω : Type u_3} {𝓧 : Type u_4} {mΩ : MeasurableSpace Ω} {m𝓧 : MeasurableSpace 𝓧} [StandardBorelSpace Ω] [Nonempty Ω] {κ : Kernel Ω 𝓧} [IsMarkovKernel κ] {μ : MeasureTheory.Measure Ω} [MeasureTheory.IsFiniteMeasure μ] :
((Kernel.id.prod κ).comp (Kernel.prodMkLeft Ω (posterior κ μ))).Invariant (μ.compProd κ)

The Gibbs sampler for the joint law of a state and its observation, which redraws the state from the posterior given the observation and then the observation from the state, leaves the joint invariant.

theorem ProbabilityTheory.dirac_ae_eq_posterior_id {Ω : Type u_3} [MeasurableSpace Ω] [MeasurableSingletonClass Ω] [StandardBorelSpace Ω] [Nonempty Ω] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsFiniteMeasure μ] :
MeasureTheory.Measure.dirac =ᵐ[μ.bind ⇑Kernel.id] ⇑(posterior Kernel.id μ)

The posterior of an observation that is the parameter itself is the point mass at it.

theorem ProbabilityTheory.posterior_const {Ω : Type u_3} {𝓧 : Type u_4} {mΩ : MeasurableSpace Ω} {m𝓧 : MeasurableSpace 𝓧} [StandardBorelSpace Ω] [Nonempty Ω] (ν : MeasureTheory.Measure 𝓧) [MeasureTheory.IsFiniteMeasure ν] (μ : MeasureTheory.Measure Ω) [MeasureTheory.IsProbabilityMeasure μ] :
⇑(posterior (Kernel.const Ω ν) μ) =ᵐ[ν] ⇑(Kernel.const 𝓧 μ)