Hahn, Degen, and Futrell (2021): Modeling Word and Morpheme Order in Natural Language as an Efficient Trade-Off of Memory and Surprisal #

Hahn, Degen and Futrell explain word and morpheme order by the tradeoff of memory and surprisal. A listener predicts each word from a memory state updated word by word, and the paper's information locality bound says that a listener whose memory cannot hold the dependencies up to some distance pays at least the entropy rate plus the information carried at longer distances, the quantities of Core/InformationTheory/EntropyRate.lean. We prove the bound, compare two orders of a sentence by dependency length, and check the paper's morpheme templates against Bybee's relevance hierarchy.

Main statements #

information_locality_bound: if words and memory states are jointly stationary, 0 < T, and the memory entropy is at most ∑ t ∈ Icc 1 T, t * distanceInfo w μ t, then the average surprisal is at least the entropy rate plus ∑' t, distanceInfo w μ (T + t + 1).
heavyNPShift_shorter: heavy NP shift in the paper's example (2) lowers total dependency length.
japanese_violates_surveyed_relevance, sesotho_suffixes_violate_relevance: neither the Japanese suffix order nor Demuth's Sesotho order is sorted by Bybee's relevance hierarchy.

Implementation notes #

The Japanese suffix order is Japanese.Verb.slots, the order of the paper's text, its supplement and its code, with the polite suffix before the desiderative; the paper's Table 2 lists the two the other way round. The supplement's forms attest neither order, the two suffixes never sharing a form there.
japaneseCategory? and sesothoCategory? map the slots the paper labels with a category of [Byb85]'s hierarchy to it. The paper's Japanese "Voice and Mood" slot of the passive and the potential is read as voice, and its final "Tense, Aspect, Mood, and Finiteness" slot and the Sesotho "Tense/aspect" prefix as tense. The slots it labels otherwise, suru, politeness, negation, the reversive and the interrogative and relative markers, have no rank and are left out of the comparison.
The paper orders the Sesotho passive before the perfect, in its text, its Table 4 and its supplement; [Dem92]'s schema (14) and examples (28) and (44) order the perfect before the passive, and the Fragment follows Demuth. The paper's order is paperSesothoSuffixes.
The Sesotho rows are the verbs of Demuth's examples, their glosses without the noun classes, and two corpus forms from the supplement in which a tense prefix is fused with the neighbouring marker and glossed as that marker.
Time runs over ℕ from the initial memory state. Every quantity in the bound is an entropy of a finite block, so the paper's two-sided process enters through its one-sided restriction.
The initial memory state is an arbitrary random variable, so the bound holds without the paper's No Mindreading assumption, as the paper remarks.
Entropies are in nats rather than bits; the bound is homogeneous in them.
The dependency graphs of (2) follow Universal Dependencies conventions, with the comma dropped.

Hahn, Degen, and Futrell (2021): Modeling Word and Morpheme Order in Natural Language as an Efficient Trade-Off of Memory and Surprisal #

Main statements #

Implementation notes #

References #

Dependency locality #

The information locality bound #

The Japanese suffix template #

The Sesotho affix template #

Morpheme order and the relevance hierarchy #