Meister, Giulianelli and Pimentel (2024): Towards a Similarity-Adjusted Surprisal Theory

[MGP24] adapt a biodiversity index, which discounts species by their similarity to one another, into a similarity-adjusted surprisal: the negative logarithm of the expected similarity of the language model's alternatives to the observed word. It is the generalized surprisal model of [GOC24] with the negative logarithm and a similarity score (simAdjSurprisal_eq_genSurprisal). The identity similarity regards different words as completely distinct, and with it similarity-adjusted surprisal is surprisal (simAdjSurprisal_indicatorScore). The paper's theorem relates it to information value: when the similarity is one minus a distance, similarity-adjusted surprisal is the negative logarithm of one minus the information value under that distance (simAdjSurprisal_one_sub), so the two measures stand in a strictly increasing relationship (simAdjSurprisal_lt_simAdjSurprisal_iff).

Implementation notes

• The paper's similarity and distance take values in the unit interval. The theorems assume only that the distance is integrable, and the strict monotonicity that both information values are below one: at information value one the paper's similarity-adjusted surprisal is infinite, while Real.log 0 = 0 makes the formal value zero.
• The scoring functions take the alternative first, the target second and the context last, as in [GOC24]; the paper writes the similarity with the target first.
• The similarity-adjusted entropy, the appendix's cost theorem, and the reading-time regressions are not formalized.

References

• [MGP24]
• [GOC24]

noncomputable def MeisterGiulianelliPimentel2024.simAdjSurprisal {C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] (L : ProbabilityTheory.Kernel C A) (z : A → W → C → ℝ) (c : C) (w : W) : ℝ

Similarity-adjusted surprisal: the negative logarithm of the expected similarity z of the alternatives to the target.

Equations

• MeisterGiulianelliPimentel2024.simAdjSurprisal L z c w = -Real.log (∫ (a : A), z a w c ∂L c)

theorem MeisterGiulianelliPimentel2024.simAdjSurprisal_eq_genSurprisal {C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] (L : ProbabilityTheory.Kernel C A) (z : A → W → C → ℝ) (c : C) (w : W) : simAdjSurprisal L z c w = Surprisal.genSurprisal L (fun (x : ℝ) => -Real.log x) z c w

Similarity-adjusted surprisal is the generalized surprisal model with the negative logarithm and the similarity as the score.

theorem MeisterGiulianelliPimentel2024.simAdjSurprisal_indicatorScore {C : Type u_1} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace W] [MeasurableSingletonClass W] (L : ProbabilityTheory.Kernel C W) (c : C) (w : W) : simAdjSurprisal L Surprisal.indicatorScore c w = InformationTheory.surprisal (L c) w

With the identity similarity, which regards different words as completely distinct, similarity-adjusted surprisal is surprisal.

theorem MeisterGiulianelliPimentel2024.simAdjSurprisal_one_sub {C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] (L : ProbabilityTheory.Kernel C A) [ProbabilityTheory.IsMarkovKernel L] (d : A → W → C → ℝ) {c : C} {w : W} (hd : MeasureTheory.Integrable (fun (a : A) => d a w c) (L c)) : simAdjSurprisal L (fun (a : A) (w : W) (c : C) => 1 - d a w c) c w = -Real.log (1 - Surprisal.informationValue L d c w)

Under the similarity one minus a distance, similarity-adjusted surprisal is the negative logarithm of one minus the information value.

theorem MeisterGiulianelliPimentel2024.simAdjSurprisal_lt_simAdjSurprisal_iff {C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] (L : ProbabilityTheory.Kernel C A) [ProbabilityTheory.IsMarkovKernel L] (d : A → W → C → ℝ) {c c' : C} {w w' : W} (hd : MeasureTheory.Integrable (fun (a : A) => d a w c) (L c)) (hd' : MeasureTheory.Integrable (fun (a : A) => d a w' c') (L c')) (h : Surprisal.informationValue L d c w < 1) (h' : Surprisal.informationValue L d c' w' < 1) : simAdjSurprisal L (fun (a : A) (w : W) (c : C) => 1 - d a w c) c w < simAdjSurprisal L (fun (a : A) (w : W) (c : C) => 1 - d a w c) c' w' ↔ Surprisal.informationValue L d c w < Surprisal.informationValue L d c' w'

Under the similarity one minus a distance, similarity-adjusted surprisal and information value order any two target–context pairs alike: the two measures are in a strictly increasing relationship.