Docs / Processing / LanguageModel / Autoregressive

Autoregressive sequence models

import Linglib.Processing.LanguageModel.Autoregressive

An autoregressive sequence model over an alphabet α gives, after every prefix, a distribution over the next symbol or the end of the string, none Du et al. 2023. The probability that a string begins with a prefix is the product of the conditional probabilities along it, the probability of the string itself multiplies in the probability of ending after it, and the model is tight when the string probabilities sum to one. Conversely a language model, a probability measure on strings, has after each prefix the law of its continuation, and it is the string law of the model whose conditionals are the laws of the first symbol of the continuation. Prefixes act on both sides, on a measure by taking continuations and on a model by starting it after the prefix, and the passage from measures to models carries one action to the other.

Implementation notes

The alphabet need only carry a measurable space, and the results about string laws a countable one with measurable singletons; Du et al. 2023 take it finite. The surprisal of a string counts the end of the string, so that it is the negative logarithm of the probability of the string (surprisal_stringMeasure) rather than of its prefix probability.

References

  • Du et al. 2023, A Measure-Theoretic Characterization of Tight Language Models

Declarations

L45–51

Prefix and string probabilities

L61–65
L67
L72–75
L77–86
L88–94
L104–113
L115–116
L118
L124–126
L128–130
L132–133
L135–137
L139–140
L142–145
L147–149
L151–153

Continuations

L157–160
L162
L177–178
L194–195
L197–207
L209–219
L221–227
L229–241

The model of a language model

After each prefix of positive probability, the first symbol of the continuation of a measure on strings, none where the continuation is empty, has a law, and these laws form a model. When the measure is a probability measure, every model that agrees with them after each prefix of positive probability has the measure as its string law.

L255–265
L273–275
L277–279
L289–297
L299–307
L311–324
L327–328
L330–337
L341–351
L357–362
L365–367
L370–371
L374–379