Dirichlet priors on probabilistic context-free grammars #
A Dirichlet PCFG over G is a Dirichlet prior on the rule distribution at each nonterminal,
given by a positive pseudo-count for every rule. Integrating the rule weights out, the
probability of a corpus is a product over nonterminals of Pólya-urn likelihoods of the corpus
rule counts, so derivation trees are exchangeable but not independent: the corpus probability
does not factorise over sub-corpora as PCFG.corpusProb does. The posterior given a corpus is
again a Dirichlet PCFG, with pseudo-counts raised by the corpus rule counts, and the posterior
predictive rule distribution is the PCFG of normalised posterior pseudo-counts.
Main definitions #
DirichletPCFG G: positive pseudo-counts on the rules ofG.DirichletPCFG.lhsFactor,DirichletPCFG.corpusProb: the Pólya-urn likelihood of the corpus counts at a nonterminal, and the corpus probability.DirichletPCFG.posterior: the conjugate update by a corpus.DirichletPCFG.predictive,DirichletPCFG.predictivePCFG: the posterior predictive probability of a rule, as a real and as a PCFG.DirichletPCFG.posteriorMode: the mode of the posterior Dirichlet, defined in the interior of the simplex.
Main results #
DirichletPCFG.predictive_lt_iff_of_same_lhs,DirichletPCFG.posteriorMode_lt_iff_of_same_lhs: at a shared nonterminal both estimators order rules by posterior pseudo-count, so they agree (DirichletPCFG.predictive_lt_iff_posteriorMode_lt).DirichletPCFG.predictivePCFG_posterior: the posterior predictive PCFG of a corpus is the prior predictive PCFG of the posterior.
References #
A Dirichlet prior on the rule distributions of G: a positive pseudo-count for every rule
of the grammar.
- pseudo : ContextFreeRule T G.NT → ℝ
The Dirichlet pseudo-count of a rule.
- pseudo_pos (r : ContextFreeRule T G.NT) : r ∈ G.rules → 0 < self.pseudo r
Pseudo-counts are positive on the rules of the grammar.
Instances For
The corpus counts of the rules with left-hand side a.
Equations
- DirichletPCFG.lhsCounts a D ⟨r, property⟩ = RoseTree.corpusRuleCount r D
Instances For
The likelihood of the corpus counts at nonterminal a under the Pólya urn over the rules
with left-hand side a, weighted by M's pseudo-counts.
Equations
- M.lhsFactor a D = ProbabilityTheory.polyaUrnProb (fun (r : G.RulesWithLHS a) => M.pseudo ↑r) (DirichletPCFG.lhsCounts a D)
Instances For
The probability of a corpus with the rule weights integrated out: the product over the nonterminals the grammar expands of the Pólya-urn likelihood there.
Equations
- M.corpusProb D = ∏ a ∈ Finset.image (fun (x : ContextFreeRule T G.NT) => x.input) G.rules, M.lhsFactor a D
Instances For
Conjugate update #
The posterior given a corpus: each rule's pseudo-count grows by its corpus count.
Equations
- M.posterior D = { pseudo := fun (r : ContextFreeRule T G.NT) => M.pseudo r + ↑(RoseTree.corpusRuleCount r D), pseudo_pos := ⋯ }
Instances For
Posterior predictive rule distribution #
The posterior predictive probability of rule r at its left-hand side given a corpus, the
posterior mean of its weight: its posterior pseudo-count over the total at that nonterminal.
Equations
- M.predictive r D = (M.pseudo r + ↑(RoseTree.corpusRuleCount r D)) / ∑ r' ∈ G.rules with r'.input = r.input, (M.pseudo r' + ↑(RoseTree.corpusRuleCount r' D))
Instances For
With no data the predictive probability is the normalised prior pseudo-count.
At a shared left-hand side, predictive probabilities order rules by posterior pseudo-count.
The posterior predictive PCFG: the predictive probability of each rule of the grammar.
Equations
- One or more equations did not get rendered due to their size.
Instances For
Conditioning on a corpus and then taking the prior predictive is taking the posterior predictive.
Posterior mode #
The mode of the posterior Dirichlet at a nonterminal lies in the interior of the simplex when
every posterior pseudo-count there exceeds 1, and is then (π_r + x_r - 1) / ∑ (π_r' + x_r' - 1).
Where it is defined it orders rules exactly as the predictive probability does.
The mode of the posterior Dirichlet weight of rule r, 0 by the division convention
outside the interior regime M.PosteriorModeInterior r.input D.
Equations
- M.posteriorMode r D = (M.pseudo r + ↑(RoseTree.corpusRuleCount r D) - 1) / ∑ r' ∈ G.rules with r'.input = r.input, (M.pseudo r' + ↑(RoseTree.corpusRuleCount r' D) - 1)
Instances For
The posterior mode at nonterminal a lies in the interior of the simplex: every posterior
pseudo-count there exceeds 1.
Equations
- M.PosteriorModeInterior a D = ∀ r ∈ {x ∈ G.rules | x.input = a}, 1 < M.pseudo r + ↑(RoseTree.corpusRuleCount r D)
Instances For
Where the posterior mode is defined, it orders rules at a shared left-hand side exactly as the predictive probability does.