Docs / Studies / Studies.GiulianelliOpedalCotterell2024

Generalized Measures of Anticipation and Responsivity in Online Language Processing

Giulianelli, M., Opedal, A. & Cotterell, R. (2024). In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 11648–11669)

Giulianelli, Opedal and Cotterell define generalized surprisal as a warping of the expected score of the continuations a language model samples (Surprisal.genSurprisal), and estimate it by warping the mean score of finitely many sampled continuations (mcEstimate). The estimator is consistent when the warping is continuous: by the strong law of large numbers the mean score converges almost surely to the expected score. With the identity warping it is also unbiased.

Implementation notes

  • The alternatives are random variables on a probability space, each distributed as the language model in the context; the paper draws them by ancestral sampling. Consistency assumes them pairwise independent, the hypothesis of mathlib's strong law.
  • Consistency needs the warping continuous only at the expected score, where the paper assumes it continuous.

References

Declarations

L43–47

The Monte Carlo estimate is consistent: for pairwise independent alternatives distributed as the language model and a warping continuous at the expected score, it converges almost surely to the generalized surprisal.

theorem ae_tendsto_mcEstimateL57–73
{C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] {Ω : Type u_4} [MeasurableSpace Ω] {P : MeasureTheory.Measure Ω} {L : ProbabilityTheory.Kernel C A} {g : A → W → C → ℝ} {c : C} {w : W} {V : ℕ → Ω → A} {f : ℝ → ℝ} (hV : ∀ (n : ℕ), Measurable (V n)) (hlaw : ∀ (n : ℕ), MeasureTheory.Measure.map (V n) P = L c) (hindep : Pairwise fun (i j : ℕ) => ProbabilityTheory.IndepFun (V i) (V j) P) (hg : Measurable fun (a : A) => g a w c) (hint : MeasureTheory.Integrable (fun (a : A) => g a w c) (L c)) (hf : ContinuousAt f (∫ (a : A), g a w c ∂L c)) :
∀ᵐ (ω : Ω) ∂P, Filter.Tendsto (fun (N : ℕ) => mcEstimate f g c w (fun (n : ℕ) => V n ω) N) Filter.atTop (nhds (Surprisal.genSurprisal L f g c w))

Built from

With the identity warping, the Monte Carlo estimate from N ≠ 0 alternatives distributed as the language model is unbiased.

theorem integral_mcEstimate_idL75–85
{C : Type u_1} {A : Type u_2} {W : Type u_3} [MeasurableSpace C] [MeasurableSpace A] {Ω : Type u_4} [MeasurableSpace Ω] {P : MeasureTheory.Measure Ω} {L : ProbabilityTheory.Kernel C A} {g : A → W → C → ℝ} {c : C} {w : W} {V : ℕ → Ω → A} (hV : ∀ (n : ℕ), Measurable (V n)) (hlaw : ∀ (n : ℕ), MeasureTheory.Measure.map (V n) P = L c) (hint : MeasureTheory.Integrable (fun (a : A) => g a w c) (L c)) {N : ℕ} (hN : N ≠ 0) :
∫ (ω : Ω), mcEstimate id g c w (fun (n : ℕ) => V n ω) N ∂P = Surprisal.genSurprisal L id g c w

Built from