Futrell, Gibson and Levy (2020): Lossy-Context Surprisal #
This file formalizes [FGL20]'s lossy-context surprisal, on which the
difficulty of a word is its expected surprisal under a lossy memory representation of its
context, carried by LossyContext.MemoryProcess. The comprehender of section 3.3
predicts the next word from its memory representation by Bayesian inversion of the memory
kernel (bayes), and averaged over contexts its difficulty is the conditional entropy of the
word given the representation, which exceeds the difficulty of surprisal theory, the lossless
comprehender who sees the context, by exactly the predictive information memory loses
(averageDifficulty_bayes_sub_lossless); the data processing inequality makes the loss
nonnegative.
Information locality (section 5) is proved in the single-dependency configuration, where the
paper's first-order approximation is exact: under erasure noise (erasure), which keeps the
context word with probability 1 - e, difficulty is the unigram surprisal less the surviving
fraction of the pointwise mutual information, so with progressive noise, a more distant word
erased more often, difficulty grows with distance when the words are positively associated and
shrinks when they are negatively associated (locality, antilocality). No erasure is the
lossless case and certain erasure recovers the unigram prior.
Implementation notes #
- Pointwise mutual information is taken relative to the unigram distribution of the next word, the language model averaged over the context prior, and the erasure process reads an erased word as that prior.
- Structural forgetting (section 4) is a parameter-space simulation over toy grammars, with forgetting at low verb-final relative-clause rates as for English and none at the German rate; it stays in prose.
References #
The Bayes-optimal comprehender and the average form #
The joint law of the next word and its context under the language model L and the
context prior π.
Equations
- FutrellGibsonLevy2020.joint L π = MeasureTheory.Measure.map Prod.swap (π.compProd L)
Instances For
The joint law of the next word and the memory representation of its context.
Equations
- FutrellGibsonLevy2020.memJoint L π mem = (FutrellGibsonLevy2020.joint L π).bind ⇑(ProbabilityTheory.Kernel.id.parallelComp mem)
Instances For
The average difficulty of surprisal theory, a lossless comprehender who sees the context, is the conditional entropy of the next word given the context.
The Bayes-optimal comprehender: the memory kernel with the posterior predictive of the next word given the representation.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The average difficulty of the Bayes-optimal comprehender is the conditional entropy of the next word given the memory representation.
The average form of information locality: the Bayes-optimal comprehender's average difficulty exceeds surprisal theory's by exactly the predictive information memory loses.
Lossy memory cannot make comprehension easier on average.
Erasure noise and information locality #
The pointwise mutual information of the next word w with the context word y, relative to
the unigram distribution of the next word.
Equations
- FutrellGibsonLevy2020.pmi L π y w = Real.log ((L y).real {w} / (π.bind ⇑L).real {w})
Instances For
Conditional surprisal is unigram surprisal less pointwise mutual information.
The erasure-noise memory process: the context word is erased with probability e, and the
predictor reads a retained word through the language model and an erased one as the unigram
distribution.
Equations
- One or more equations did not get rendered due to their size.
Instances For
The exact single-dependency form of information locality: under erasure noise, difficulty is the unigram surprisal less the surviving fraction of the pointwise mutual information.
The excess difficulty of erasure-noise processing over plain surprisal is the erased fraction of the pointwise mutual information.
Information locality: under progressive noise, a more distant context word having a larger erasure rate, difficulty increases with distance when the words are positively associated.
Anti-locality: when the words are negatively associated, losing the context word lowers difficulty, so difficulty decreases with distance.
No erasure is the lossless comprehender who sees the context word.
No erasure recovers surprisal.
Certain erasure recovers the unigram prior: regression to prior expectations.