Documentation

Studies.PullumGazdar1982

Pullum and Gazdar (1982): Natural Languages and Context-Free Languages #

This file formalizes [PG82]'s examination of the published arguments that some natural language is not a context-free stringset, each of which the paper finds formally or empirically invalid. The formal exhibits are grammars. The finite state grammar of §1 generates unbounded long-distance number concord (concordGrammar, concord_unbounded). Grammar (9) generates exactly the nonidentity language (8) that Chomsky's comparative argument took to be beyond context-free power (xyGrammar_language), so the language is context-free (isXY_isContextFree). Grammar (29) generates exactly the Dutch verb-raising verb phrases (24) with as many object names as transitive verbs, which Huybregts's cross-serial dependencies were held to put beyond context-free power (dutchGrammar_language, isVerbPhrase_isContextFree), and the Dutch examples (25) to (31) fall out as predicted (dutch_rows). The empirical refutations are rows: the respectively judgments (18) and (19) contradict the number-matching characterization of English that the argument assumes (respectively_converse), and Mohawk classificatory and possessed incorporation, (39), (40), and (43d), break the stem-matching premise of the Mohawk argument (stem_matching_fails).

Implementation notes #

Grammars are mathlib's ContextFreeGrammar, and the language theorems go through derivation trees: soundness by induction on a licensed tree, with what each nonterminal generates spelled out (Generated), and completeness by building the derivation. The Dutch verb phrases are described as the paper does, with the categories of the lexicon (29b) as terminals; the row predictions decide membership by stripping the names and counting the transitive verbs (isVerbPhrase), which is proved equivalent to the description. Finite stateness is the right-linear form of the rules, since mathlib has no bridge from grammars to regular languages. The π argument of §3, the xx-language closure argument of §1, the ditransitive schema (32), and the Mohawk translation of Langendoen's finite state language are not formalized.

References #

§1: unbounded concord in a finite state language #

The words of the finite state grammar of §1.

Instances For
    @[instance_reducible]
    Equations
    def PullumGazdar1982.instReprWord.repr :
    Word → ℕ → Std.Format
    Equations
    Instances For

      Its three nonterminals.

      Instances For
        @[instance_reducible]
        Equations
        Equations
        • One or more equations did not get rendered due to their size.
        Instances For
          def PullumGazdar1982.concordGrammar :
          ContextFreeGrammar Word

          In the finite state grammar of §1 a singular or plural which-phrase selects the chain of she thought and you thought clauses ending in the agreeing verb.

          Equations
          • One or more equations did not get rendered due to their size.
          Instances For
            def PullumGazdar1982.IsFiniteState {T : Type u_1} (g : ContextFreeGrammar T) :

            In a finite state grammar every rule rewrites to terminals followed by at most one nonterminal.

            Equations
            Instances For
              def PullumGazdar1982.chain (ps : List Bool) :
              List Word

              A chain of she thought and you thought clauses.

              Equations
              • One or more equations did not get rendered due to their size.
              Instances For

                Concord holds over an unbounded domain in a finite state language, since for every chain of embedded clauses the plural which-phrase agrees with were and the singular with was.

                §2: the nonidentity language (8) and grammar (9) #

                The terminal vocabulary of (8) has the two letters of the differing substrings and the three markers.

                Instances For
                  @[instance_reducible]
                  Equations
                  def PullumGazdar1982.instReprSym.repr :
                  Sym → ℕ → Std.Format
                  Equations
                  Instances For
                    @[instance_reducible]
                    Equations

                    Among the nonterminals of grammar (9), S₁, S₂, A₁, B₁ are the paper's S', S'', A', B'.

                    Instances For
                      @[instance_reducible]
                      Equations
                      def PullumGazdar1982.instReprXYNT.repr :
                      XYNT → ℕ → Std.Format
                      Equations
                      Instances For
                        def PullumGazdar1982.xyGrammar :
                        ContextFreeGrammar Sym

                        Grammar (9), rules (a) to (i) in order.

                        Equations
                        • One or more equations did not get rendered due to their size.
                        Instances For
                          def PullumGazdar1982.AB (l : List Sym) :

                          A string over a and b.

                          Equations
                          Instances For

                            The language (8) has a central marker between two strings over a and b that differ.

                            Equations
                            • One or more equations did not get rendered due to their size.
                            Instances For

                              Each nonterminal of grammar (9) generates its own strings. S₁ generates a marker between strings of different lengths, S₂ a marker between strings differing at some position, A and B a specified a or b flanked by equally many characters before it and after the marker, and A₁ and B₁ a specified a or b and a tail.

                              Equations
                              Instances For
                                theorem PullumGazdar1982.xy_sound {t : RoseTree (Symbol Sym xyGrammar.NT)} (ht : RoseTree.Licensed xyGrammar.Licenses t) (N : xyGrammar.NT) :
                                t.value = Symbol.nonterminal N → Generated N t.yield

                                By soundness, a tree of grammar (9) rooted at a nonterminal yields what that nonterminal generates, which at S is a string of (8).

                                Completeness #

                                theorem PullumGazdar1982.derives_xy {w : List Sym} (hw : IsXY w) :
                                xyGrammar.Derives [Symbol.nonterminal XYNT.S] (List.map Symbol.terminal w)

                                By completeness, every string of (8) is derived.

                                theorem PullumGazdar1982.xyGrammar_language :
                                xyGrammar.language = {w : List Sym | IsXY w}

                                Grammar (9) generates exactly the language (8).

                                theorem PullumGazdar1982.isXY_isContextFree :
                                Language.IsContextFree {w : List Sym | IsXY w}

                                The nonidentity language (8) is context-free, against the formal premise of the comparative argument.

                                §4: respectively #

                                Langendoen's characterization (17) of English respectively sentences has each verb agree in number with its subject.

                                Equations
                                Instances For
                                  @[instance_reducible]
                                  Equations

                                  (18b), the one string of (18) the characterization admits, is rejected.

                                  §5: Dutch verb raising and grammar (29) #

                                  The categories of the lexicon (29b) are names, transitive infinitives, intransitive verbs, transitive and intransitive VP-complement-taking infinitives, and finite transitive and intransitive VP-complement-taking verbs.

                                  Instances For
                                    @[instance_reducible]
                                    Equations
                                    @[instance_reducible]
                                    Equations
                                    def PullumGazdar1982.instReprCat.repr :
                                    Cat → ℕ → Std.Format
                                    Equations
                                    Instances For

                                      The nonterminals of grammar (29a) are the verb phrase and its VP complement.

                                      Instances For
                                        @[instance_reducible]
                                        Equations
                                        Equations
                                        • One or more equations did not get rendered due to their size.
                                        Instances For
                                          def PullumGazdar1982.dutchGrammar :
                                          ContextFreeGrammar Cat

                                          Grammar (29a) has the rules A → B C D | C E and C → B C F | C G | B H | I.

                                          Equations
                                          • One or more equations did not get rendered due to their size.
                                          Instances For

                                            Whether a verb is subcategorized for a direct object.

                                            Equations
                                            Instances For
                                              def PullumGazdar1982.objects (vs : List Cat) :
                                              ℕ

                                              The number of direct objects a string of verbs requires.

                                              Equations
                                              Instances For
                                                @[simp]
                                                theorem PullumGazdar1982.objects_cons (c : Cat) (w : List Cat) :
                                                objects (c :: w) = (if c.transitive = true then 1 else 0) + objects w
                                                @[simp]
                                                theorem PullumGazdar1982.objects_append (v w : List Cat) :
                                                objects (v ++ w) = objects v + objects w

                                                A VP complement consists of names, a finite VP-complement-taking verb, then nonfinite VP-complement-taking verbs, with one name for each transitive verb.

                                                Equations
                                                • One or more equations did not get rendered due to their size.
                                                Instances For

                                                  The verb phrases of (24), as the paper describes them, consist of n names, a finite VP-complement-taking verb, a string of nonfinite VP-complement-taking verbs, and a final transitive or intransitive verb, with one name for each transitive verb.

                                                  Equations
                                                  • One or more equations did not get rendered due to their size.
                                                  Instances For
                                                    theorem PullumGazdar1982.dutch_sound {t : RoseTree (Symbol Cat dutchGrammar.NT)} (ht : RoseTree.Licensed dutchGrammar.Licenses t) :
                                                    (t.value = Symbol.nonterminal DutchNT.C → IsComplement t.yield) ∧ (t.value = Symbol.nonterminal DutchNT.A → IsVerbPhrase t.yield)

                                                    By soundness, a tree of grammar (29) rooted at C yields a VP complement and one rooted at A a verb phrase.

                                                    theorem PullumGazdar1982.derives_complement (x : List Cat) {n : ℕ} {h : Cat} :
                                                    h = Cat.H ∨ h = Cat.I → (∀ c ∈ x, c = Cat.F ∨ c = Cat.G) → n = objects (h :: x) → dutchGrammar.Derives [Symbol.nonterminal DutchNT.C] (List.map Symbol.terminal (List.replicate n Cat.B ++ h :: x))

                                                    By completeness for C, every VP complement is derived.

                                                    theorem PullumGazdar1982.derives_verbPhrase {w : List Cat} (hw : IsVerbPhrase w) :
                                                    dutchGrammar.Derives [Symbol.nonterminal DutchNT.A] (List.map Symbol.terminal w)

                                                    By completeness, every verb phrase is derived.

                                                    Grammar (29) generates exactly the verb phrases the paper describes.

                                                    theorem PullumGazdar1982.isVerbPhrase_isContextFree :
                                                    Language.IsContextFree {w : List Cat | IsVerbPhrase w}

                                                    Cross-serial word order with the right number of names is a context-free stringset.

                                                    The examples (25) to (31) #

                                                    def PullumGazdar1982.isVerbPhrase (w : List Cat) :
                                                    Bool

                                                    IsVerbPhrase is decided by stripping the names, reading the first and last verb, and counting.

                                                    Equations
                                                    • One or more equations did not get rendered due to their size.
                                                    Instances For
                                                      def PullumGazdar1982.categories (r : Datum) :
                                                      Option (List Cat)

                                                      A row's category string.

                                                      Equations
                                                      Instances For

                                                        A row is predicted when its grammaticality agrees with whether its category string is a verb phrase of grammar (29).

                                                        Equations
                                                        Instances For
                                                          @[instance_reducible]
                                                          Equations
                                                          • One or more equations did not get rendered due to their size.
                                                          theorem PullumGazdar1982.dutch_rows (r : Datum) :
                                                          r ∈ Examples.all → r.language = "dutc1256" → Predicted r

                                                          On (25) to (31), grammar (29) accepts the grammatical clauses and rejects the ungrammatical.

                                                          §6: Mohawk noun incorporation #

                                                          The stem-matching premise of the Mohawk argument says that the stem incorporated in the verb is the head noun stem of its external argument.

                                                          Equations
                                                          Instances For
                                                            @[instance_reducible]
                                                            Equations

                                                            Classificatory incorporation (39), (40) and possessed incorporation (43d) are grammatical with mismatched stems.