The Monte Carlo estimate of generalized surprisal from the first N alternatives
v, the warping f of their mean score against the target w in the context c.
- mcEstimate f g c w v N = f ((↑N)⁻¹ * ∑ n ∈ Finset.range N, g (v n) w c)
Docs / Studies / Studies.GiulianelliOpedalCotterell2024
Giulianelli, M., Opedal, A. & Cotterell, R. (2024). In Findings of the Association for Computational Linguistics: EMNLP 2024 (pp. 11648–11669)
Giulianelli, Opedal and Cotterell define generalized surprisal as a warping of the expected score
of the continuations a language model samples (Surprisal.genSurprisal), and estimate it by
warping the mean score of finitely many sampled continuations (mcEstimate). The estimator is
consistent when the warping is continuous: by the strong law of large
numbers the mean score converges almost surely to the expected score. With the identity warping it
is also unbiased.
The Monte Carlo estimate of generalized surprisal from the first N alternatives
v, the warping f of their mean score against the target w in the context c.
The Monte Carlo estimate is consistent: for pairwise independent alternatives distributed as the language model and a warping continuous at the expected score, it converges almost surely to the generalized surprisal.
Built from
With the identity warping, the Monte Carlo estimate from N ≠ 0 alternatives distributed as
the language model is unbiased.
Built from