Docs / Processing / LanguageModel / Trajectory

The trajectory of an autoregressive sequence model

import Linglib.Processing.LanguageModel.Trajectory · namespace AutoregressiveSequenceModel

The model is a prediction rule on the extended alphabet, the end of the string absorbing, and the Ionescu–Tulcea theorem gives the law of its whole trajectory, a probability measure on infinite sequences of symbols or ends. Survival is the probability that none of the first n symbols is the end, a string's probability is that of the trajectories that spell it and then end, and the model is tight exactly when the trajectory ends almost surely, which is how Du et al. 2023 define tightness.

References

  • Du et al. 2023, A Measure-Theoretic Characterization of Tight Language Models

Declarations

L45–50

The trajectory

The model is a prediction rule on the extended alphabet, with the end absorbing, and the law of its whole trajectory is the Ionescu–Tulcea trajectory measure of that rule.

The model as a prediction rule on the extended alphabet: after a history of units it draws the next unit, and after the end of the string it stays at the end.

ruleL61–65
noncomputable def AutoregressiveSequenceModel.rule {α : Type u_1} [MeasurableSpace α] [MeasurableSingletonClass α] (M : AutoregressiveSequenceModel α) [Countable α] (n : ℕ) :
ProbabilityTheory.Kernel (Fin n → Option α) (Option α)
  • M.rule n = ProbabilityTheory.Kernel.ofFunOfCountable fun (h : Fin n → Option α) => if ∀ (i : Fin n), h i ≠ none then M.next (List.ofFn h).reduceOption else MeasureTheory.Measure.dirac none

Used by instIsMarkovKernelForallFinOptionRule, rule_apply_some, seqLaw_rule_singleton_some, survival_eq_seqLaw, trajectory

L67–70
L72–77

The law of the whole trajectory of the model: the units it generates one after another, with the end of the string absorbing.

trajectoryL105–107
noncomputable def AutoregressiveSequenceModel.trajectory {α : Type u_1} [MeasurableSpace α] [MeasurableSingletonClass α] (M : AutoregressiveSequenceModel α) [Countable α] :
MeasureTheory.Measure (ℕ → Option α)

Built from

Used by instIsProbabilityMeasureForallNatOptionTrajectory, trajectory_setOf_forall_lt_ne_none, trajectory_setOf_forall_ne_none, isTight_iff_trajectory_setOf_forall_ne_none, trajectory_cylinder

L109–110
L128–139

The probability of generating exactly the string List.ofFn f is the probability that the trajectory begins with its units and then ends.

theorem trajectory_cylinderL147–167
{α : Type u_1} [MeasurableSpace α] [MeasurableSingletonClass α] (M : AutoregressiveSequenceModel α) [Countable α] {n : ℕ} (f : Fin n → α) :
M.trajectory {ω : ℕ → Option α | (∀ (i : Fin n), ω ↑i = some (f i)) ∧ ω n = none} = M.stringProb (List.ofFn f)

Built from