Docs / Processing / LanguageModel / Trajectory
The trajectory of an autoregressive sequence model
import Linglib.Processing.LanguageModel.Trajectory · namespace AutoregressiveSequenceModel
The model is a prediction rule on the extended alphabet, the end of the string
absorbing, and the Ionescu–Tulcea theorem gives the law of its whole trajectory, a
probability measure on infinite sequences of symbols or ends. Survival is the probability that
none of the first n symbols is the end, a string's
probability is that of the trajectories that spell it and then end, and
the model is tight exactly when
the trajectory ends almost surely, which is how
Du et al. 2023 define tightness.
References
- Du et al. 2023, A Measure-Theoretic Characterization of Tight Language Models
Declarations
The trajectory
The model is a prediction rule on the extended alphabet, with the end absorbing, and the law of its whole trajectory is the Ionescu–Tulcea trajectory measure of that rule.
The model as a prediction rule on the extended alphabet: after a history of units it draws the next unit, and after the end of the string it stays at the end.
- M.rule n = ProbabilityTheory.Kernel.ofFunOfCountable fun (h : Fin n → Option α) => if ∀ (i : Fin n), h i ≠ none then M.next (List.ofFn h).reduceOption else MeasureTheory.Measure.dirac none
Used by instIsMarkovKernelForallFinOptionRule, rule_apply_some, seqLaw_rule_singleton_some, survival_eq_seqLaw, trajectory
Built from
After a history of units the rule is the model's next-unit distribution.
Built from
The mass of a history of units is its prefix probability.
Built from
Survival to n is the law of the first n draws on the histories of units.
Built from
The law of the whole trajectory of the model: the units it generates one after another, with the end of the string absorbing.
Built from
Used by instIsProbabilityMeasureForallNatOptionTrajectory, trajectory_setOf_forall_lt_ne_none, trajectory_setOf_forall_ne_none, isTight_iff_trajectory_setOf_forall_ne_none, trajectory_cylinder
Built from
Survival to n is the probability that none of the first n units is the end.
Built from
The probability that the string never ends is the infimum of survival.
Built from
A model is tight exactly when its trajectory ends almost surely, Du et al. 2023's definition of tightness.
Built from
The probability of generating exactly the string List.ofFn f is the probability that the
trajectory begins with its units and then ends.
Built from