Giulianelli, Wallbridge, Cotterell and Fernández (2026): Incremental Alternative Sampling as a Lens into the Temporal and Representational Resolution of Linguistic Prediction #
This file formalizes the incremental alternative sampling (IAS) family of [GWCF26].
A language model is the family of next-unit kernels of an autoregressive process, and a
comprehender samples continuations of the context over a forecast horizon of h units, the
partial trajectory of the process (Kernel.partialTraj); the generalised surprisal of the next
unit is the substrate's genSurprisal at that kernel. Standard surprisal is the member with the
negative logarithm and the prefix indicator at every horizon, because the alternatives' first
unit is distributed as the model's next unit (genSurprisal_prefixIndicator); it thus evaluates
alternatives by lexical identity alone, and the discrete distance shows what this conflates,
since information value with it is the probability of error (informationValue_discrete).
Incremental information value replaces the indicator by a representational distance between each
alternative and the observed unit followed by the alternatives sampled after it, the double
expectation of the paper's definition (iiv), whose horizon-one case is the information value of
the substrate (iiv_zero).
Implementation notes #
- Contexts are trajectories
Π i : Iic n, X iof the process and alternatives are trajectories to timen + h, which carry the context as their prefix; a fixed alphabet is the constant family, and an end-of-string unit belongs to it, absorbing if the model makes it so. - The alternative-set reformulation with mean, minimum and maximum summary statistics, and its reduction to the single-alternative definition under the mean, are not formalized; nor are the representation functions, which the paper takes from a Transformer's layers and which enter here only through the distance.
- The paper's regression results, which horizon and layer best predict cloze probability, the N400 and P600, and eye-tracked and self-paced reading times, are empirical fits and stay in prose: explicit predictability peaks at horizon one and lexical representations, the ERP components at horizon two, and self-paced reading of multi-sentence stimuli at longer horizons.
References #
The context x followed by the unit w.
Equations
- GiulianelliEtAl2026.snoc x w = IicProdIoc n (n + 1) (x, (MeasurableEquiv.piSingleton n) w)
Instances For
Standard surprisal at every horizon #
The prefix indicator: one when the alternative's first unit is w.
Equations
- GiulianelliEtAl2026.prefixIndicator w a = {w}.indicator 1 (a ⟨n + 1, ⋯⟩)
Instances For
The expected prefix indicator is the next-unit probability, at every horizon.
Standard surprisal is the generalised surprisal with the negative logarithm and the prefix indicator, at every horizon.
What the indicator conflates #
The discrete distance on units: an alternative is accurate only when identical to the unit.
Equations
- GiulianelliEtAl2026.discrete a w x✝ = 1 - {w}.indicator 1 a
Instances For
Information value with the discrete distance is the probability of error, 1 − p(w | c): the
alternatives' similarity to the unit counts for nothing, as under surprisal.
Incremental information value #
Incremental information value at horizon h + 1: the expected representational distance
between an alternative sampled before the unit and the unit followed by the alternatives sampled
after it, the paper's double expectation.
Equations
- One or more equations did not get rendered due to their size.
Instances For
At horizon one, incremental information value is the information value of the substrate, with the distance read on the extended contexts.