GF Resource Grammar Tutorial (2022)

Table of Contents
Introduction Contents of the course's five lessons Contents GF = Grammatical Framework Multilingual grammars The GF program The GF Resource Grammar Library Where GF is used GF run-time system Installing and using the GF system Starting the GF shell Using the GF shell: help Working with context-free grammars in GF Importing and parsing Random generation, linearization, and pipes Graphical view of abstract trees Graphical view of parse trees Abstract and concrete syntax Separating abstract and concrete syntax Functions and concatenation From context-free to GF grammars Abstract syntax, example Concrete syntax, English Making a grammar multilingual Concrete syntax, French Translation and multilingual generation Multilingual treebanks Concrete syntax, Latin Parameters in linearization Table types and tables Concrete syntax, Dutch Record types and records Concrete syntax, Hebrew Variable and inherent features, agreement Feature design Visualizing trees and word alignment From abstract trees to parse trees Generating word alignment Word alignment via trees A more involved word alignment Building applications Scaling up the grammar More to do on sentences More to do on noun phrases Exercises Contents Morphology Good start for a resource grammar What is a word? Lexical categories The main lexical categories in the resource grammar Typical feature design Module structure Example: resource module for English verb inflection Start: worst-case function Testing computation in resource modules Defining paradigms More paradigms More paradigms still What paradigm to choose Smart paradigms Pattern matching on strings Testing the smart paradigm The smart paradigm is not yet perfect The final consonant duplication paradigm Testing consonant duplication There is no waterproof solution A paradigm for irregular verbs Putting it all together An overloaded paradigm Testing the overloaded paradigm Phases of morphology implementation Other parts of speech Morphophonemic functions Building a lexicon Bootstrapping a lexicon Nonconcatenative morphology: Arabic Datastructures for Arabic Applying a pattern Encoding roots by strings Encoding patterns by strings A high-level lexicon building function Parameters for the Arabic verb type Example of Arabic verb inflection Arabic verb type: implementation An Arabic verb paradigm Applying an Arabic paradigm How we did the printing (recreational GF hacking) Exercises Contents Syntax in the resource grammar The key categories and functions The key categories The key functions Feature design Predication: building clauses Interplay between features Feature passing Predication: examples Predication: variations Complementation: building verb phrases Interplay between features Complementation: examples Complementation: variations Determination: building noun phrases Interplay between features Determination: examples Determination: variations Modification: adding adjectives to nouns Interplay between features Modification: examples Modification: variations Lexical insertion The head of a phrase What is the head of a noun phrase? Structural words A miniature resource grammar for Italian Extension vs. opening Module dependencies Producing the dependency graph The module Grammar Parameters Italian verb phrases Tense and agreement of a verb phrase, in syntax The forms of a verb, in morphology The verb phrase type Verb phrase formation Italian noun phrases Noun phrases: alternatively Determination Building determiners Adjectival modification Italian morphology Predication, at last Selection of verb form To do Ergativity in Hindi/Urdu A Hindi clause in different tenses Exercises Contents Software libraries Advantages of software libraries Grammars as software libraries Using the library: natural language output Software localization Correct number in Arabic Use cases for grammar libraries Two kinds of grammarians Two kinds of grammars Meaning-preserving translation Translation and resource grammar Domain semantics Examples of domain semantics Presenting the resource grammar Relevant part of Resource Grammar API for "Face" Concrete syntax for English Concrete syntax for Finnish Functors and interfaces The domain lexicon interface Concrete syntax functor "FaceI" An English instance of the domain lexicon Put everything together: functor instantiation Porting the grammar to Finnish Modules of a domain grammar: "Face" community Module dependency graph Porting the grammar to Italian Free variation Overview of the resource grammar API Main categories and their dependencies Categories of complex phrases Lexical categories for building predicates Functions for building predication clauses Noun phrases and common nouns Questions and interrogatives Sentence formation, tense, and polarity Utterances and imperatives More Exercises Contents The principal module structure Division of labour Roles of modules: Library API Roles of modules: Top-level grammar Roles of modules: phrase categories Type discipline and consistency Auxiliary modules Dependencies Functional programming style Functors in the Resource Grammar Library Example: DiffRomance Pros and cons of functors Suggestions about functors for new languages Effort statistics, completed languages How to start building a language, e.g. Marathi Suggested order for proceeding with a language Character encoding for non-ASCII languages Using transliteration Diagnosis methods along the way Regression testing with a treebank Sources Compiling the library Assignment: a good start

Introduction

This tutorial was given at LREC in Malta, 17 May 2010, and is an updated versions of the one used at the GF Summer School 2009. It was first presented on an on-line course in April 2009. The summer school in August 2009 had 30 participants from 20 countries. 15 new languages were started. Since that first summer school, the library has grown from 12 to over 30 languages.

The goal of this tutorial is to introduce a fast way to resource grammar writing, by explaining the practical use of GF and the linguistic concepts in the resource grammar library.

For more details, we recommend

The code examples in this tutorial are available at

https://github.com/GrammaticalFramework/gf-contrib/tree/master/lrec-tutorial.

We cannot stress enough the importance of your own work on the code examples and exercises using the GF system!

Contents of the course's five lessons

1. The GF system, simple multilingual grammars

2. Morphological paradigms and lexica

3. Building up a linguistic syntax

4. Using the Resource Grammar Library in applications

5. Developing a new resource grammar

Contents

What GF is

Installing the GF system

A grammar for John loves Mary in English, French, Latin, Dutch, Hebrew

Testing grammars and building applications

The scope of the Resource Grammar Library

Exercises

GF = Grammatical Framework

GF is a grammar formalism: a notation for writing grammars

GF is a functional programming language with types and modules

GF programs are called grammars

A grammar is a declarative program that defines

  • parsing
  • generation
  • translation

Multilingual grammars

Many languages related by a common abstract syntax

GF Resource Grammar Tutorial (1)

The GF program

Interpreter for testing grammars (the GF shell)

Compiler for converting grammars to useful formats

  • PGF, Portable Grammar Format
  • speech recognition grammars (Nuance, HTK, ...)
  • JavaScript

The GF Resource Grammar Library

Morphology and basic syntax

Common API for different languages

Currently (May 2010) 17 languages: Bulgarian, Catalan, Danish, Dutch, English, Finnish, French, German, Interlingua, Italian, Norwegian, Polish, Romanian, Russian, Spanish, Swedish, Urdu.

Under construction for at least 19 languages: Afrikaans, Amharic, Arabic, Baatonum, Esperanto, Farsi, Greek (Ancient), Hebrew, Icelandic, Japanese, Latin, Latvian, Maltese, Mongol, Portuguese, Swahili, Thai, Tswana, Turkish.

Where GF is used

Natural language interfaces: WebALT, see

webalt.math.helsinki.fi/PublicFiles/CD/Screencast/TextMathEditor%20Demo.swf

Dialogue systems: TALK, see www.youtube.com/watch?v=1bfaYHWS6zU

Translation: MOLTO, see www.molto-project.eu

GF Resource Grammar Tutorial (2)

GF run-time system

PGF grammars can be embedded in Haskell, Java, and Prolog programs

They can be used in web servers

  • fridge magnet demo: grammaticalframework.org:41296/fridge
  • translator demo: grammaticalframework.org:41296/translate

GF Resource Grammar Tutorial (3)

Installing and using the GF system

Go to the GF home page http://grammaticalframework.org and follow shortcuts to either

  • Download: download and install binaries
  • Developers: download sources, compile, and install

The Developers method is recommended for resource grammar developers:

  • latest updates and bug fixes
  • version control system

Starting the GF shell

The command gf starts the GF shell:

 $ gf * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * This is GF version 3.1.6. License: see help -license. Bug reports: http://code.google.com/p/grammatical-framework/issues/list Languages: >

Using the GF shell: help

Command h = help

 > help

gives a list of commands with short descriptions.

 > help parse

gives detailed help on the command parse.

Commands have both short (1 or 2 letters) and long names.

Working with context-free grammars in GF

These are the simplest grammars usable in GF. Example:

 Pred. S ::= NP VP ; Compl. VP ::= V2 NP ; John. NP ::= "John" ; Mary. NP ::= "Mary" ; Love. V2 ::= "loves" ;

The first item in each rule is a syntactic function, used for building trees: Pred = predication, Compl = complementation.

The second item is a category: S = Sentence, NP = Noun Phrase, VP = Verb Phrase, V2 = 2-place Verb.

Importing and parsing

Copy or write the above grammar in file zero.cf.

To use a grammar in GF: import = i

 > i zero.cf

To parse a string to a tree: parse = p

 > p "John loves Mary" Pred John (Compl Love Mary)

Parsing is, by default, in category S. This can be overridden.

Random generation, linearization, and pipes

Generate a random tree: generate_random = gr

 > gr Pred Mary (Compl Love Mary)

To linearize a tree to a string: linearize = l

 > l Pred Mary (Compl Love Mary) Mary loves Mary

To pipe a command to another one: |

 > gr | l Mary loves Mary

Graphical view of abstract trees

GF Resource Grammar Tutorial (4)

In Mac:

 > p "John loves Mary" | visualize_tree -view=open

In Ubuntu Linux:

 > p "John loves Mary" | visualize_tree -view=oeg

You need the Graphviz program to see the view.

Graphical view of parse trees

GF Resource Grammar Tutorial (5)

 > p "John loves Mary" | visualize_parse -view=open

Abstract and concrete syntax

A context-free rule

 Pred. S ::= NP VP

defines two things:

  • abstract syntax: build a tree of form Pred np vp
  • concrete syntax: this tree linearizes to a string of form np vp

The main idea of GF: separate these two things.

Separating abstract and concrete syntax

A context-free rule is converted to two judgements in GF:

  • fun, declaring a syntactic function
  • lin, giving its linearization rule
 Pred. S ::= NP VP ===> fun Pred : NP -> VP -> S lin Pred np vp = np ++ vp

Functions and concatenation

Function type: A -> B -> C, read "function from A and B to C"

Function application: f a b, read "f applied to arguments a and b"

Concatenation: x ++ y, read "string x followed by string y"

Cf. functional programming in Haskell.

Notice: in GF, ++ is between token lists and therefore "creates a space".

From context-free to GF grammars

The grammar is divided to two modules

  • an abstract module, judgement forms cat and fun
  • a concrete module, judgement forms lincat and lin
catCC is a category
fun f : Tf is a function of type T
lincat C = LC has linearization type L
lin f xs = tf xs has linearization t

Abstract syntax, example

 abstract Zero = { cat S ; NP ; VP ; V2 ; fun Pred : NP -> VP -> S ; Compl : V2 -> NP -> VP ; John, Mary : NP ; Love : V2 ; }

Concrete syntax, English

 concrete ZeroEng of Zero = { lincat S, NP, VP, V2 = Str ; lin Pred np vp = np ++ vp ; Compl v2 np = v2 ++ np ; John = "John" ; Mary = "Mary" ; Love = "loves" ; }

Notice: Str (token list, "string") as the only linearization type.

Making a grammar multilingual

One abstract + many concretes

The same system of trees can be given

  • different words
  • different word orders
  • different linearization types

Concrete syntax, French

 concrete ZeroFre of Zero = { lincat S, NP, VP, V2 = Str ; lin Pred np vp = np ++ vp ; Compl v2 np = v2 ++ np ; John = "Jean" ; Mary = "Marie" ; Love = "aime" ; }

Just use different words

Translation and multilingual generation

Import many grammars with the same abstract syntax

 > i ZeroEng.gf ZeroFre.gf Languages: ZeroEng ZeroFre

Translation: pipe linearization to parsing

 > p -lang=ZeroEng "John loves Mary" | l -lang=ZeroFre Jean aime Marie

Multilingual generation: linearize into all languages

 > gr | l Pred Mary (Compl Love Mary) Mary loves Mary Marie aime Marie

Multilingual treebanks

Treebank: show both trees and their linearizations

 > gr | l -treebank Zero: Pred Mary (Compl Love Mary) ZeroEng: Mary loves Mary ZeroFre: Marie aime Marie

Concrete syntax, Latin

 concrete ZeroLat of Zero = { lincat S, VP, V2 = Str ; NP = Case => Str ; lin Pred np vp = np ! Nom ++ vp ; Compl v2 np = np ! Acc ++ v2 ; John = table {Nom => "Ioannes" ; Acc => "Ioannem"} ; Mary = table {Nom => "Maria" ; Acc => "Mariam"} ; Love = "amat" ; param Case = Nom | Acc ; }

Different word order (SOV), different linearization type, parameters.

Parameters in linearization

Latin has cases: nominative for subject, accusative for object.

  • Ioannes Mariam amat "John-Nom loves Mary-Acc"
  • Maria Ioannem amat "Mary-Nom loves John-Acc"

Parameter type for case (just 2 of Latin's 6 cases):

 param Case = Nom | Acc

Table types and tables

The linearization type of NP is a table type: from Case to Str,

 lincat NP = Case => Str

The linearization of John is an inflection table,

 lin John = table {Nom => "Ioannes" ; Acc => "Ioannem"}

When using an NP, select (!) the appropriate case from the table,

 Pred np vp = np ! Nom ++ vp Compl v2 np = np ! Acc ++ v2

Concrete syntax, Dutch

 concrete ZeroDut of Zero = { lincat S, NP, VP = Str ; V2 = {v : Str ; p : Str} ; lin Pred np vp = np ++ vp ; Compl v2 np = v2.v ++ np ++ v2.p ; John = "Jan" ; Mary = "Marie" ; Love = {v = "heeft" ; p = "lief"} ; }

The verb heeft lief is a discontinuous constituent.

Record types and records

The linearization type of V2 is a record type with two fields

 lincat V2 = {v : Str ; p : Str}

The linearization of Love is a record

 lin Love = {v = "hat" ; p = "lieb"}

The values of fields are picked by projection (.)

 lin Compl v2 np = v2.v ++ np ++ v2.p

Concrete syntax, Hebrew

 concrete ZeroHeb of Zero = { flags coding=utf8 ; lincat S = Str ; NP = {s : Str ; g : Gender} ; VP, V2 = Gender => Str ; lin Pred np vp = np.s ++ vp ! np.g ; Compl v2 np = table {g => v2 ! g ++ "את" ++ np.s} ; John = {s = "ג'ון" ; g = Masc} ; Mary = {s = "מרי" ; g = Fem} ; Love = table {Masc => "אוהב" ; Fem => "אוהבת"} ; param Gender = Masc | Fem ; }

The verb agrees to the gender of the subject.

Variable and inherent features, agreement

NP has gender as its inherent feature - a field in the record

 lincat NP = {s : Str ; g : Gender} lin Mary = {s = "mry" ; g = Fem}

VP has gender as its variable feature - an argument of a table

 lincat VP = Gender => Str

In predication, the VP receives the gender of the NP

 lin Pred np vp = np.s ++ vp ! np.g

Feature design

Deciding on variable and inherent features is central in GF programming.

Good hint: dictionaries give forms of variable features and values of inherent ones.

Example: French nouns

  • cheval pl. chevaux masc. noun

From this we infer that French nouns have variable number and inherent gender

 lincat N = {s : Number => Str ; g : Gender}

Visualizing trees and word alignment

GF Resource Grammar Tutorial (6) GF Resource Grammar Tutorial (7) GF Resource Grammar Tutorial (8)

From abstract trees to parse trees

Link every word with its smallest spanning subtree

Replace every constructor function with its value category

Generating word alignment

In L1 and L2: link every word with its smallest spanning subtree

Delete the intervening tree, combining links directly from L1 to L2

Notice: in general, this gives phrase alignment

Notice: links can be crossing, phrases can be discontinuous

Word alignment via trees

GF Resource Grammar Tutorial (9)

 > parse "John loves Mary" | aw -view=open

A more involved word alignment

GF Resource Grammar Tutorial (10)

Building applications

Compile the grammar to PGF:

 $ gf -make ZeroEng.gf ZeroFre.gf ZeroLat.gf ZeroGer.gf ZeroHeb.gf

The resulting file Zero.pgf can be e.g. included in fridge magnets:

GF Resource Grammar Tutorial (11)

Scaling up the grammar

Zero.gf is a tiny fragment of the Resource Grammar

The current Resource Grammar has 80 categories, 200 syntactic functions, and a minimal lexicon of 500 words.

Even S, NP, VP, V2 will need richer linearization types.

More to do on sentences

The category S has to take care of

  • tenses: John has loved Mary
  • negation: John doesn't love Mary
  • word order (Dutch): als Jan Marie lief heeft, heeft Marie Jan lief

Moreover: questions, imperatives, relative clauses

More to do on noun phrases

NP also involves

  • pronouns: I, you, she, we
  • determiners: the man, every place

Moreover: common nouns, adjectives

Exercises

1. Install gf on your computer.

2. Learn and try out the commands align_words, empty, generate_random, generate_trees, help, import, linearize, parse, put_string, quit, read_file, translation_quiz, unicode_table, visualize_parse, visualize_tree, write_file.

3. Write a concrete syntax of Zero for yet another language (e.g. your summer school project language).

4. Extend the Zero grammar with ten new noun phrases and verbs.

5. Add to the Zero grammar a category A of adjectives and a function ComplA : A -> VP, which forms verb phrases like is old.

Contents

Morphology, inflection, paradigm - example: English verbs

Regular patterns and smart paradigms

Overloaded operations

Inherent features in the lexicon

Building and bootstrapping a lexicon

Nonconcatenative morphology: Arabic

Morphology

Inflectional morphology: define the different forms of words

  • English verb sing has the forms sing, sings, sang, sung, singing

Derivational morphology: tell how new words are formed from old words

  • English verb sing produces the noun singer

We could do both in GF, but concentrate now on inflectional morphology.

Good start for a resource grammar

Complete inflection system: 1-6 weeks

Comprehensive lexicon: days or weeks

Morphological analysis: up to 200,000 words per second

Export to SQL, XFST, ...

What is a word?

In abstract syntax: an object of a basic type, such as Love : V2

In concrete syntax,

  • primarily: an inflection table, the collection of all forms
  • secundarily: a string, i.e. a single form

Thus love, loves, loved are

  • distinct words as strings
  • forms of the same word as an inflection table or an abstract syntax object

Lexical categories

Part of speech = word class = lexical category

In GF, a part of speech is defined as a cat and its associated lincat.

In GF, there is no formal difference between lexical and other cats.

But in the resource grammar, we maintain a discipline of separate lexical categories.

The main lexical categories in the resource grammar

Nnounhouse
Aadjectivesmall
Vverbsleep
V2two-place verblove
Advadverbtoday

Typical feature design

Nnumber, casegender
Anumber, case, gender, degreeposition
Vtense, number, person, ...auxiliary
V2as Vcomplement case
Adv--

Module structure

Resource module with inflection functions as operations

 resource MorphoEng = {oper regV : Str -> V ; ...}

Lexicon: abstract and concrete syntax

 abstract Lex = {fun Walk : V ; ...} concrete LexEng of Lex = open MorphoEng in {lin Walk = regV "walk" ; ...}

The same resource can be used (opened) in many lexica.

Abstract and concrete are top-level - they define trees, parsing, linearization.

Resource modules and opers are not top-level - they are "thrown away" after compilation (i.e. not preserved in PGF).

Example: resource module for English verb inflection

Use the library module Prelude.

Start by defining parameter types and parts of speech.

 resource Morpho = open Prelude in { param VForm = VInf | VPres | VPast | VPastPart | VPresPart ; oper Verb : Type = {s : VForm => Str} ;

Judgement form oper: auxiliary operation.

Start: worst-case function

To save writing and to abstract over the Verbtype

 mkVerb : (_,_,_,_,_ : Str) -> Verb = \go,goes,went,gone,going -> { s = table { VInf => go ; VPres => goes ; VPast => went ; VPastPart => gone ; VPresPart => going } } ;

Testing computation in resource modules

Import with retain option

 > i -retain Morpho.gf

Use command cc = compute_concrete

 > cc mkVerb "use" "uses" "used" "used" "using" {s : Morpho.VForm => Str = table Morpho.VForm { Morpho.VInf => "use"; Morpho.VPres => "uses"; Morpho.VPast => "used"; Morpho.VPastPart => "used"; Morpho.VPresPart => "using" }}

Defining paradigms

A paradigm is an operation of type

 Str -> Verb

which takes a string and returns an inflection table.

Let's first define the paradigm for regular verbs:

 regVerb : Str -> Verb = \walk -> mkVerb walk (walk + "s") (walk + "ed") (walk + "ed") (walk + "ing") ;

This will work for walk, interest, play.

It will not work for sing, kiss, use, cry, fly, stop.

More paradigms

For verbs ending with s, x, z, ch

 s_regVerb : Str -> Verb = \kiss -> mkVerb kiss (kiss + "es") (kiss + "ed") (kiss + "ed") (kiss + "ing") ;

For verbs ending with e

 e_regVerb : Str -> Verb = \use -> let us = init use in mkVerb use (use + "s") (us + "ed") (us + "ed") (us + "ing") ;

Notice:

  • the local definition let c = d in ...
  • the operation init from Prelude, dropping the last character

More paradigms still

For verbs ending with y

 y_regVerb : Str -> Verb = \cry -> let cr = init cry in mkVerb cry (cr + "ies") (cr + "ied") (cr + "ied") (cry + "ing") ;

For verbs ending with ie

 ie_regVerb : Str -> Verb = \die -> let dy = Predef.tk 2 die + "y" in mkVerb die (die + "s") (die + "d") (die + "d") (dy + "ing") ;

What paradigm to choose

If the infinitive ends with s, x, z, ch, choose s_regRerb: munch, munches

If the infinitive ends with y, choose y_regRerb: cry, cries, cried

  • except if a vowel comes before: play, plays, played

If the infinitive ends with e, choose e_regVerb: use, used, using

  • except if an i precedes: die, dying
  • or if an e precedes: free, freeing

Smart paradigms

Let GF choose the paradigm by pattern matching on strings

 smartVerb : Str -> Verb = \v -> case v of { _ + ("s"|"z"|"x"|"ch") => s_regVerb v ; _ + "ie" => ie_regVerb v ; _ + "ee" => ee_regVerb v ; _ + "e" => e_regVerb v ; _ + ("a"|"e"|"o"|"u") + "y" => regVerb v ; _ + "y" => y_regVerb v ; _ => regVerb v } ;

Pattern matching on strings

Format: case string of { pattern => value }

Patterns:

  • _ matches any string
  • a string in quotes matches itself: "ie"
  • + splits into substrings: _ + "y"
  • | matches alternatives: "a"|"e"|"o"

Common practice: last pattern a catch-all _

Testing the smart paradigm

 > cc -all smartVerb "munch" munch munches munched munched munching > cc -all smartVerb "die" die dies died died dying > cc -all smartVerb "agree" agree agrees agreed agreed agreeing > cc -all smartVerb "deploy" deploy deploys deployed deployed deploying > cc -all smartVerb "classify" classify classifies classified classified classifying

The smart paradigm is not yet perfect

Irregular verbs are obviously not covered

 > cc -all smartVerb "sing" sing sings singed singed singing

Neither are regular verbs with consonant duplication

 > cc -all smartVerb "stop" stop stops stoped stoped stoping

The final consonant duplication paradigm

Use the Prelude function last

 dupRegVerb : Str -> Verb = \stop -> let stopp = stop + last stop in mkVerb stop (stop + "s") (stopp + "ed") (stopp + "ed") (stopp + "ing") ;

String pattern: relevant consonant preceded by a vowel

 _ + ("a"|"e"|"i"|"o"|"u") + ("b"|"d"|"g"|"m"|"n"|"p"|"r"|"s"|"t") => dupRegVerb v ;

Testing consonant duplication

Now it works

 > cc -all smartVerb "stop" stop stops stopped stopped stopping

But what about

 > cc -all smartVerb "coat" coat coats coatted coatted coatting

Solution: a prior case for diphthongs before the last char (? matches one char)

 _ + ("ea"|"ee"|"ie"|"oa"|"oo"|"ou") + ? => regVerb v ;

There is no waterproof solution

Duplication depends on stress, which is not marked in English:

  • omit [o'mit]: omitted, omitting
  • vomit ['vomit]: vomited, vomiting

This means that we occasionally have to give more forms than one.

We knew this already for irregular verbs. And we cannot write patterns for each of them either, because e.g. lie can be both lie, lied, lied or lie, lay, lain.

A paradigm for irregular verbs

Arguments: three forms instead of one.

Pattern matching done in regular verbs can be reused.

 irregVerb : (_,_,_ : Str) -> Verb = \sing,sang,sung -> let v = smartVerb sing in mkVerb sing (v.s ! VPres) sang sung (v.s ! VPresPart) ;

Putting it all together

We have three functions:

 smartVerb : Str -> Verb irregVerb : Str -> Str -> Str -> Verb mkVerb : Str -> Str -> Str -> Str -> Str -> Verb

As all types are different, we can use overloading and give them all the same name.

An overloaded paradigm

For documentation: variable names showing examples of arguments.

 mkV = overload { mkV : (cry : Str) -> Verb = smartVerb ; mkV : (sing,sang,sung : Str) -> Verb = irregVerb ; mkV : (go,goes,went,gone,going : Str) -> Verb = mkVerb ; } ;

Testing the overloaded paradigm

 > cc -all mkV "lie" lie lies lied lied lying > cc -all mkV "lie" "lay" "lain" lie lies lay lain lying > cc -all mkV "omit" omit omits omitted omitted omitting > cc -all mkV "vomit" vomit vomits vomitted vomitted vomitting > cc -all mkV "vomit" "vomited" "vomited" vomit vomits vomited vomited vomitting > cc -all mkV "vomit" "vomits" "vomited" "vomited" "vomiting" vomit vomits vomited vomited vomiting

Surely we could do better for vomit...

Phases of morphology implementation

1. Linearization type, with parametric and inherent features.

2. Worst-case function.

3. The set of paradigms, traditionally taking one argument each.

4. Smart paradigms, with relevant numbers of arguments.

5. Overloaded user function, collecting the smart paradigms.

Other parts of speech

Usually recommended order:

1. Nouns, the simplest class.

2. Adjectives, often using noun inflection, adding gender and degree.

3. Verbs, usually the most complex class, using adjectives in participles.

Morphophonemic functions

Many operations are common to different parts of speech.

Example: adding an s to an English noun or verb.

 add_s : Str -> Str = \v -> case v of { _ + ("s"|"z"|"x"|"ch") => v + "es" ; _ + ("a"|"e"|"o"|"u") + "y" => v + "s" ; cr + "y" => cr + "ies" ; _ => v + "s" } ;

This should be defined separately, not directly in verb conjunctions.

Notice: pattern variable cr matches like _ but gets bound.

Building a lexicon

Boringly, we need abstract and concrete modules even for one language.

 abstract Lex = { concrete LexEng = open Morpho in { cat V ; lincat V = Verb ; fun lin play_V : V ; play_V = mkV "play" ; sleep_V : V ; sleep_V = mkV "sleep" "slept" "slept" ; }

Fortunately, these modules can be mechnically generated from a POS-tagged word list

 V play V sleep slept slept

Bootstrapping a lexicon

Alt 1. From a morphological POS-tagged word list: trivial

 V play played played V sleep slept slept

Alt 2. From a plain word list, POS-tagged: start assuming regularity, generate, correct, and add forms by iteration

 V play ===> V play played played ===> V sleep V sleep sleeped sleeped V sleep slept slept

Example: Finnish nouns need 1.42 forms in average (to generate 26 forms).

Nonconcatenative morphology: Arabic

Semitic languages, e.g. Arabic: kataba has forms kaAtib, yaktubu, ...

Traditional analysis:

  • word = root + pattern
  • root = three consonants (radicals)
  • pattern = function from root to string (notation: string with variables F,C,L for the radicals)

Example: yaktubu = ktb + yaFCuLu

Words are datastructures rather than strings!

Datastructures for Arabic

Roots are records of strings.

 Root : Type = {F,C,L : Str} ;

Patterns are functions from roots to strings.

 Pattern : Type = Root -> Str ;

A special case is filling: a record of strings filling the four slots in a root.

 Filling : Type = {F,FC,CL,L : Str} ;

This is enough for everything except middle consonant duplication (e.g. FaCCaLa).

Applying a pattern

A pattern obtained from a filling intertwines the records:

 fill : Filling -> Pattern = \p,r -> p.F + r.F + p.FC + r.C + p.CL + r.L + p.L ;

Middle consonant duplication also uses a filling but duplicates the C consonant of the root:

 dfill : Filling -> Pattern = \p,r -> p.F + r.F + p.FC + r.C + r.C + p.CL + r.L + p.L ;

Encoding roots by strings

This is just for the ease of programming and writing lexica.

F = first letter, C = second letter, L = the rest.

 getRoot : Str -> Root = \s -> case s of { F@? + C@? + L => {F = F ; C = C ; L = L} ; _ => Predef.error ("cannot get root from" ++ s) } ;

The as-pattern x@p matches p and binds x.

The error function Predef.error stops computation and displays the string. It is a typical catch-all value.

Encoding patterns by strings

Patterns are coded by using the letters F, C, L.

 getPattern : Str -> Pattern = \s -> case s of { F + "F" + FC + "CC" + CL + "L" + L => dfill {F = F ; FC = FC ; CL = CL ; L = L} ; F + "F" + FC + "C" + CL + "L" + L => fill {F = F ; FC = FC ; CL = CL ; L = L} ; _ => Predef.error ("cannot get pattern from" ++ s) } ;

A high-level lexicon building function

Dictionary entry: root + pattern.

 getWord : Str -> Str -> Str = \r,p -> getPattern p (getRoot r) ;

Now we can try:

 > cc getWord "ktb" "yaFCuLu" "yaktubu" > cc getWord "ktb" "muFaCCiLu" "mukattibu"

Parameters for the Arabic verb type

Inflection in tense, number, person, gender.

 param Number = Sg | Dl | Pl ; Gender = Masc | Fem ; Tense = Perf | Impf ; Person = Per1 | Per2 | Per3 ;

But not in all combinations. For instance: no first person dual.

(We have omitted most tenses and moods.)

Example of Arabic verb inflection

GF Resource Grammar Tutorial (12)

Arabic verb type: implementation

We use an algebraic datatype to include only the meaningful combinations.

 param VPer = Vp3 Number Gender | Vp2Sg Gender | Vp2Dl | Vp2Pl Gender | Vp1Sg | Vp1Pl ; oper Verb : Type = {s : Tense => VPer => Str} ;

Thus 2*(3*2 + 2 + 1 + 2 + 1 + 1) = 26 forms, not 2*3*2*3 = 36.

An Arabic verb paradigm

 pattV_u : Tense -> VPer -> Pattern = \t,v -> getPattern (case t of { Perf => case v of { Vp3 Sg Masc => "FaCaLa" ; Vp3 Sg Fem => "FaCaLato" ; -- o is the no-vowel sign ("sukun") Vp3 Dl Masc => "FaCaLaA" ; -- ... } ; Impf => case v of { -- ... Vp1Sg => "A?aFoCuLu" ; Vp1Pl => "naFoCuLu" } }) ; u_Verb : Str -> Verb = \s -> { s = \\t,p => appPattern (getRoot s) (pattV_u t p) } ;

Applying an Arabic paradigm

Testing in the resource module:

 > cc -all u_Verb "ktb" kataba katabato katabaA katabataA katabuwA katabona katabota kataboti katabotumaA katabotum katabotunv2a katabotu katabonaA yakotubu takotubu yakotubaAni takotubaAni yakotubuwna yakotubna takotubu takotubiyna takotubaAni takotubuwna takotubona A?akotubu nakotubu

Building a lexicon:

 fun ktb_V : V ; lin ktb_V = u_Verb "ktb" ;

How we did the printing (recreational GF hacking)

We defined a HTML printing operation

 oper verbTable : Verb -> Str

and used it in a special category Table built by

 fun Tab : V -> Table ; lin Tab v = verbTable v ;

We then used

 > l Tab ktb_V | ps -env=quotes -to_arabic | ps -to_html | wf -file=ara.html > ! tr "\"" " " <ara.html >ar.html

Exercises

1. Learn to use the commands compute_concrete, morpho_analyse, morpho_quiz.

2. Try out some smart paradigms in the resource library files Paradigms for some languages you know (or don't know yet). Use the command cc for this.

3. Write a morphology implementation for some word class and some paradigms in your target language. Start with feature design and finish with a smart paradigm.

4. Bootstrap a GF lexicon (abstract + concrete) of 100 words in your target language.

5. (Recreational GF hacking.) Write an operation similar to verbTable for printing nice inflection tables in HTML.

Contents

The key categories and rules

Morphology-syntax interface

Examples and variations in English, Italian, French, Finnish, Swedish, German, Hindi

A miniature resource grammar: Italian

Module extension and dependency graphs

Ergativity in Hindi/Urdu

Don't worry if the details of this lecture feel difficult! Syntax is difficult and this is why resource grammars are so useful!

Syntax in the resource grammar

"Linguistic ontology": syntactic structures common to languages

80 categories, 200 functions, which have worked for all resource languages so far

Sufficient for most purposes of expressing meaning: mathematics, technical documents, dialogue systems

Must be extended by language-specific rules to permit parsing of arbitrary text (ca. 10% more in English?)

A lot of work, easy to get wrong!

The key categories and functions

The key categories

Clclauseevery young man loves Mary
VPverb phraseloves Mary
V2two-place verbloves
NPnoun phraseevery young man
CNcommon nounyoung man
Detdeterminerevery
APadjectival phraseyoung

The key functions

PredVP : NP -> VP -> Clpredicationevery man loves Mary
ComplV2 : V2 -> NP -> VPcomplementationloves Mary
DetCN : Det -> CN -> NPdeterminationevery man
AdjCN : AP -> CN -> CNmodificationyoung man

Feature design

Cltense-
VPtense, agr-
V2tense, agrcase
NPcaseagr
CNnumber, casegender
Detgender, casenumber
APgender, number, case-

agr = agreement features: gender, number, person

Predication: building clauses

Interplay between features

 param Tense, Case, Agr lincat Cl = {s : Tense => Str } lincat NP = {s : Case => Str ; a : Agr} lincat VP = {s : Tense => Agr => Str } fun PredVP : NP -> VP -> Cl lin PredVP np vp = {s = \\t => np.s ! subj ++ vp.s ! t ! np.a} oper subj : Case

Feature passing

In general, combination rules just pass features: no case analysis (table expressions) is performed.

A special notation is hence useful:

 \\p,q => t === table {p => table {q => t}}

It is similar to lambda abstraction (\x,y -> t in a function type).

Predication: examples

English

Sg Per1I sleepI sleptI will sleep
Sg Per3she sleepsshe sleptshe will sleep
Pl Per1we sleepwe sleptwe will sleep

Italian ("I am tired", "she is tired", "we are tired")

Masc Sg Per1io sono stancoio ero stancoio sarò stanco
Fem Sg Per3lei è stancalei era stancalei sarà stanca
Fem Pl Per1noi siamo stanchenoi eravamo stanchenoi saremo stanche

Predication: variations

Word order:

  • will I sleep (English), è stanca lei (Italian)

Pro-drop:

  • io sono stanco vs. sono stanco (Italian)

Ergativity:

  • ergative case of transitive verb subject; agreement to object (Hindi)

Variable subject case:

  • minä olen lapsi vs. minulla on lapsi (Finnish, "I am a child" (nominative) vs. "I have a child" (adessive))

Complementation: building verb phrases

Interplay between features

 lincat NP = {s : Case => Str ; a : Agr } lincat VP = {s : Tense => Agr => Str } lincat V2 = {s : Tense => Agr => Str ; c : Case} fun ComplV2 : V2 -> NP -> VP lin ComplV2 v2 vp = {s = \\t,a => v2.s ! t ! a ++ np.s ! v2.c}

Complementation: examples

English

Acclove me
at + Acclook at me

Finnish

Accusativetavata minut"meet me"
Partitiverakastaa minua"love me"
Elativepitää minusta"like me"
Genitive + peräänkatsoa minun perääni"look after me"

Complementation: variations

Prepositions: a two-place verb usually involves a preposition in addition case

 lincat V2 = {s : Tense => Agr => Str ; c : Case ; prep : Str} lin ComplV2 v2 vp = {s = \\t,a => v2.s ! t ! a ++ v2.prep ++ np.s ! v2.c}

Clitics: the place of the subject can vary, as in Italian:

  • Maria ama Giovanni vs. Maria mi ama ("Mary loves John" vs. "Mary loves me")

Determination: building noun phrases

Interplay between features

 lincat NP = {s : Case => Str ; a : Agr } lincat CN = {s : Number => Case => Str ; g : Gender} lincat Det = {s : Gender => Case => Str ; n : Number} fun DetCN : Det -> CN -> NP lin DetCN det cn = { s = \\c => det.s ! cn.g ! c ++ cn.s ! det.n ! c ; a = agr cn.g det.n Per3 } oper agr : Gender -> Number -> Person -> Agr

Determination: examples

English

Sgevery house
Plthese houses

Italian ("this wine", "this pizza", "those pizzas")

SgMascquesto vino
SgFemquesta pizza
PlFemquelle pizze

Finnish ("every house", "these houses")

Sgjokainen talojokaisessa talossa
Plnämä talotnäissä taloissa

Determination: variations

Systamatic number variation:

  • this-these, the-the, il-i (Italian "the-the")

"Zero" determiners:

  • talo ("a house") vs. talo ("the house") (Finnish)
  • a house vs. houses (English), une maison vs. des maisons (French)

Specificity parameter of nouns:

  • varje hus vs. det huset (Swedish, "every house" vs. "that house")

Modification: adding adjectives to nouns

Interplay between features

 lincat AP = {s : Gender => Number => Case => Str } lincat CN = {s : Number => Case => Str ; g : Gender} fun AdjCN : AP -> CN -> CN lin AdjCN ap cn = { s = \\n,c => ap.s ! cn.g ! n ! c ++ cn.s ! n ! c ; g = cn.g }

Modification: examples

English

new housenew houses

Italian ("red wine", "red house")

Mascvino rossovini rossi
Femcasa rossacase rosse

Finnish ("red house")

punainen talopunaiselta taloltapunaisina taloina

Modification: variations

The place of the adjectival phrase

  • Italian: casa rossa, vecchia casa ("red house", "old house")
  • English: old house, house similar to this

Specificity parameter of the adjective

  • German: ein rotes Haus vs. das rote Haus ("a red house" vs. "the red house")

Lexical insertion

To "get started" with each category, use words from lexicon.

There are lexical insertion functions for each lexical category:

 UseN : N -> CN UseA : A -> AP UseV : V -> VP

The linearization rules are often trivial, because the lincats match

 lin UseN n = n lin UseA a = a lin UseV v = v

However, for UseV in particular, this will usually be more complex.

The head of a phrase

The inserted word is the head of the phrases built from it:

  • house is the head of house, big house, big old house etc

As a rule with many exceptions and modifications,

  • variable features are passed from the phrase to the head
  • inherent features of the head are inherited by the noun

This works for endocentric phrases: the head has the same type as the full phrase.

What is the head of a noun phrase?

In an NP of form Det CN, is Det or CN the head?

Neither, really, because features are passed in both directions:

 lin DetCN det cn = { s = \\c => det.s ! cn.g ! c ++ cn.s ! det.n ! c ; a = agr cn.g det.n Per3 }

Moreover, this NP is exocentric: no part is of the same type as the whole.

Structural words

Structural words = function words, words with special grammatical functions

  • determiners: the, this, every
  • pronouns: I, she
  • conjunctions: and, or, but

Often members of closed classes, which means that new words are never (or seldom) introduces to them.

Linearization types are often specific and inflection are irregular.

A miniature resource grammar for Italian

We divide it to five modules - much fewer than the full resource!

 abstract Grammar -- syntactic cats and funs abstract Lang = Grammar **... -- test lexicon added to Grammar resource ResIta -- resource for Italian concrete GrammarIta of Grammar = open ResIta in... -- Italian syntax concrete LangIta of Lang = GrammarIta ** open ResIta in... -- It. lexicon

Extension vs. opening

Module extension: N = M1, M2, M3 ** {...}

  • module N inherits all judgements from M1,M2,M3

Module opening: N = open R1, R2, R3 in {...}

  • module N can use all judgements from R1,R2,R3 (but doesn't inherit them)

Module dependencies

GF Resource Grammar Tutorial (13)

rectangle = abstract, solid ellipse = concrete, dashed ellipse = resource

Producing the dependency graph

Using the command dg = dependency_graph and graphviz

 > i -retain LangIta.gf > dependency_graph wrote graph in file _gfdepgraph.dot > ! dot -Tjpg _gfdepgraph.dot >testdep.jpg

Before calling dot, removed the module Predef to save space.

The module Grammar

 abstract Grammar = { cat Cl ; NP ; VP ; AP ; CN ; Det ; N ; A ; V ; V2 ; fun PredVP : NP -> VP -> Cl ; ComplV2 : V2 -> NP -> VP ; DetCN : Det -> CN -> NP ; ModCN : CN -> AP -> CN ; UseV : V -> VP ; UseN : N -> CN ; UseA : A -> AP ; a_Det, the_Det : Det ; this_Det, these_Det : Det ; i_NP, she_NP, we_NP : NP ; }

Parameters

Parameters are defined in ResIta.gf. Just 11 of the 56 verb forms.

 Number = Sg | Pl ; Gender = Masc | Fem ; Case = Nom | Acc | Dat ; Aux = Avere | Essere ; -- the auxiliary verb of a verb Tense = Pres | Perf ; Person = Per1 | Per2 | Per3 ; Agr = Ag Gender Number Person ; VForm = VInf | VPres Number Person | VPart Gender Number ;

Italian verb phrases

Tense and agreement of a verb phrase, in syntax

Ag Masc Sg Per1arrivosono arrivato
Ag Fem Sg Per1arrivosono arrivata
Ag Masc Sg Per2arrivisei arrivato
Ag Fem Sg Per2arrivisei arrivata
Ag Masc Sg Per3arrivaè arrivato
Ag Fem Sg Per3arrivaè arrivata
Ag Masc Pl Per1arriviamosiamo arrivati
Ag Fem Pl Per1arriviamosiamo arrivate
Ag Masc Pl Per2arrivatesiete arrivati
Ag Fem Pl Per2arrivatesiete arrivate
Ag Masc Pl Per3arrivanosono arrivati
Ag Fem Pl Per3arrivanosono arrivate

The forms of a verb, in morphology

VInfarrivare
VPres Sg Per1arrivo
VPres Sg Per2arrivi
VPres Sg Per3arriva
VPres Pl Per1arriviamo
VPres Pl Per2arrivate
VPres Pl Per3arrivano
VPart Masc Sgarrivato
VPart Fem Sgarrivata
VPart Masc Plarrivati
VPart Fem Plarrivate

Inherent feature: aux is essere.

The verb phrase type

Lexical insertion maps V to VP.

Two possibilities for VP: either close to Cl,

 lincat VP = {s : Tense => Agr => Str}

or close to V, just adding a clitic and an object to verb,

 lincat VP = {v : Verb ; clit : Str ; obj : Str} ;

We choose the latter. It is more efficient in parsing.

Verb phrase formation

Lexical insertion is trivial.

 lin UseV v = {v = v ; clit, obj = []}

Complementation assumes NP has a clitic and an ordinary object part.

 lin ComplV2 = let nps = np.s ! v2.c in { v = {s = v2.s ; aux = v2.aux} ; clit = nps.clit ; obj = nps.obj }

Italian noun phrases

Being clitic depends on case

 lincat NP = {s : Case => {clit,obj : Str} ; a : Agr} ;

Examples:

 lin she_NP = { s = table { Nom => {clit = [] ; obj = "lei"} ; Acc => {clit = "la" ; obj = []} ; Dat => {clit = "le" ; obj = []} } ; a = Ag Fem Sg Per3 } lin John_NP = { s = table { Nom | Acc => {clit = [] ; obj = "Giovanni"} ; Dat => {clit = [] ; obj = "a Giovanni"} } ; a = Ag Fem Sg Per3 }

Noun phrases: alternatively

Use a feature instead of separate fields,

 lincat NP = {s : Case => {s : Str ; isClit : Bool} ; a : Agr} ;

The use of separate fields is more efficient and scales up better to multiple clitic positions.

Determination

No surprises

 lincat Det = {s : Gender => Case => Str ; n : Number} ; lin DetCN det cn = { s = \\c => {obj = det.s ! cn.g ! c ++ cn.s ! det.n ; clit = []} ; a = Ag cn.g det.n Per3 } ;

Building determiners

Often from adjectives:

 lin this_Det = adjDet (mkA "questo") Sg ; lin these_Det = adjDet (mkA "questo") Pl ; oper prepCase : Case -> Str = \c -> case c of { Dat => "a" ; _ => [] } ; oper adjDet : Adj -> Number -> Determiner = \adj,n -> { s = \\g,c => prepCase c ++ adj.s ! g ! n ; n = n } ;

Articles: see GrammarIta.gf

Adjectival modification

Recall the inherent feature for position

 lincat AP = {s : Gender => Number => Str ; isPre : Bool} ; lin ModCN cn ap = { s = \\n => preOrPost ap.isPre (ap.s ! cn.g ! n) (cn.s ! n) ; g = cn.g } ;

Obviously, separate pre- and post- parts could be used instead.

Italian morphology

Complex but mostly great fun:

 regNoun : Str -> Noun = \vino -> case vino of { fuo + c@("c"|"g") + "o" => mkNoun vino (fuo + c + "hi") Masc ; ol + "io" => mkNoun vino (ol + "i") Masc ; vin + "o" => mkNoun vino (vin + "i") Masc ; cas + "a" => mkNoun vino (cas + "e") Fem ; pan + "e" => mkNoun vino (pan + "i") Masc ; _ => mkNoun vino vino Masc } ;

See ResIta for more details.

Predication, at last

Place the object and the clitic, and select the verb form.

 lin PredVP np vp = let subj = (np.s ! Nom).obj ; obj = vp.obj ; clit = vp.clit ; verb = table { Pres => agrV vp.v np.a ; Perf => agrV (auxVerb vp.v.aux) np.a ++ agrPart vp.v np.a } in { s = \\t => subj ++ clit ++ verb ! t ++ obj } ;

Selection of verb form

We need it for the present tense

 oper agrV : Verb -> Agr -> Str = \v,a -> case a of { Ag _ n p => v.s ! VPres n p } ;

The participle agrees to the subject, if the auxiliary is essere

 oper agrPart : Verb -> Agr -> Str = \v,a -> case v.aux of { Avere => v.s ! VPart Masc Sg ; Essere => case a of { Ag g n _ => v.s ! VPart g n } } ;

To do

Full details of the core resource grammar are in ResIta (150 loc) and GrammarIta (80 loc).

One thing is not yet done correctly: agreement of participle to accusative clitic object: now it gives io la ho amato, and not io la ho amata.

This is left as an exercise!

Ergativity in Hindi/Urdu

Normally, the subject is nominative and the verb agrees to the subject.

However, in the perfective tense:

  • the subject of a transitive verb is in an ergative "case" (particle ne)
  • the verb agrees to the object

Example: "the boy/girl eats the apple/bread"

MascMascladka: seb Ka:ta: hailadke ne seb Ka:ya:
MascFemladka: roTi: Ka:ta: hailadke ne roTi: Ka:yi:
FemMascladki: seb Ka:ti: hailadki: ne seb Ka:ya:
FemFemladki: roTi: Ka:ti: hailadki: ne roTi: Ka:yi:

A Hindi clause in different tenses

GF Resource Grammar Tutorial (14)

Exercises

1. Learn the commands dependency_graph, print_grammar, system escape !, and system pipe ?.

2. Write tables of examples of the key syntactic functions for your target languages, trying to include all possible forms.

3. Implement Grammar and Lang for your target language.

4. Even if you don't know Italian, you may try this: add a parameter or something in GrammarIta to implement the rule that the participle in the perfect tense agrees in gender and number with an accusative clitic. Test this with the sentences lei la ha amata and lei ci ha amati (where the current grammar now gives amato in both cases).

5. Learn some linguistics! My favourite book is Introduction to Theoretical Linguistics by John Lyons (Cambridge 1968, at least 14 editions).

Contents

Software libraries: programmer's vs. users view

Semantic vs. syntactic grammars

Example of semantic grammar and its implementation

Interfaces and parametrized modules

Free variation

Overview of the Resource Grammar API

Software libraries

Collections of reusable functions/types/classes

API = Application Programmer's Interface

  • show enough to enable use
  • hide details

Example: maps (lookup tables, hash maps) in Haskell, C++, Java, ...

 type Map lookup : key -> Map -> val update : key -> val -> Map -> Map

Hidden: the definition of the type Map and of the functions lookup and update.

Advantages of software libraries

Programmers have

  • less code to write (e.g. how to look up)
  • less techniques to learn (e.g. efficient Map datastructures)

Improvements and bug fixes can be inherited

Grammars as software libraries

Smart paradigms as API for morphology

 mkN : (talo : Str) -> N

Abstract syntax as API for syntactic combinations

 PredVP : NP -> VP -> Cl ComplV2 : V2 -> NP -> VP NumCN : Num -> CN -> NP

Using the library: natural language output

Task: in an email program, generate phrases saying you have n message(s)

Problem: avoid you have one messages

Solution: use the library

 PredVP youSg_NP (ComplV2 have_V2 (NumCN two_Num (UseN (mkN "message")))) ===> you have two messages PredVP youSg_NP (ComplV2 have_V2 (NumCN one_Num (UseN (mkN "message")))) ===> you have one message

Software localization

Adapt the email program to Italian, Finnish, Arabic...

 PredVP youSg_NP (ComplV2 have_V2 (NumCN two_Num (UseN (mkN "messaggio")))) ===> hai due messaggi PredVP youSg_NP (ComplV2 have_V2 (NumCN two_Num (UseN (mkN "viesti")))) ===> sinulla on kaksi viestiä PredVP youSg_NP (ComplV2 have_V2 (NumCN two_Num (UseN (mkN "risaAlat.u.")))) ===> sinulla on kaksi viestiä

The new languages are more complex than English - but only internally, not on the API level!

Correct number in Arabic

GF Resource Grammar Tutorial (15)

(From "Implementation of the Arabic Numerals and their Syntax in GF" by Ali Dada, ACL workshop on Arabic, Prague 2007)

Use cases for grammar libraries

Grammars need very much very special knowledge, and a lot of work - thus an excellent topic for a software library!

Some applications where grammars have shown to be useful:

  • software localization
  • natural language generation (from formalized content)
  • technical translation
  • spoken dialogue systems

Two kinds of grammarians

Application grammarians vs. resource grammarians

expertiseapplication domainlinguistics
programming skillsprogramming in generalGF programming
language skillspractical usetheoretical knowledge

We want a division of labour.

Two kinds of grammars

Application grammars vs. resource grammars

abstract syntaxsemanticsyntactic
concrete syntaxusing resource APIparameters, tables, records
lexiconidiomatic, technicaljust for testing
sizesmall or biggerbig

A.k.a. semantic grammars vs. syntactic grammars.

Meaning-preserving translation

Translation must preserve meaning.

It need not preserve syntactic structure.

Sometimes it is even impossible:

  • John likes Mary in Italian is Maria piace a Giovanni

The abstract syntax in the semantic grammar is a logical predicate:

 fun Like : Person -> Person -> Fact lin Like x y = x ++ "likes" ++ y -- English lin Like x y = y ++ "piace" ++ "a" ++ x -- Italian

Translation and resource grammar

To get all grammatical details right, we use resource grammar and not strings

 lincat Person = NP ; Fact = Cl ; lin Like x y = PredVP x (ComplV2 like_V2 y) -- Engligh lin Like x y = PredVP y (ComplV2 piacere_V2 x) -- Italian

From syntactic point of view, we perform transfer, i.e. structure change.

GF has compile-time transfer, and uses interlingua (semantic abstrac syntax) at run time.

Domain semantics

"Semantics of English", or of any other natural language as a whole, has never been built.

It is more feasible to have semantics of fragments - of small, well-understood parts of natural language.

Such languages are called domain languages, and their semantics, domain semantics.

Domain semantics = ontology in the Semantic Web terminology.

Examples of domain semantics

Expressed in various formal languages

  • mathematics, in predicate logic
  • software functionality, in UML/OCL
  • dialogue system actions, in SISR
  • museum object descriptions, in OWL

GF abstract syntax can be used for any of these!

What messages can be expressed on the community page?

 abstract Face = { flags startcat = Message ; cat Message ; Person ; Object ; Number ; fun Have : Person -> Number -> Object -> Message ; -- p has n o's Like : Person -> Object -> Message ; -- p likes o You : Person ; Friend, Invitation : Object ; One, Two, Hundred : Number ; }

Notice the startcat flag, as the start category isn't S.

Presenting the resource grammar

In practice, the abstract syntax of Resource Grammar is inconvenient

  • too deep structures, too much code to write
  • too many names to remember

We do the same as in morphology: overloaded operations, named mkC where C is the value category.

The resource defines e.g.

 mkCl : NP -> V2 -> NP -> Cl = \subj,verb,obj -> PredVP subj (ComplV2 verb obj) mkCl : NP -> V -> Cl = \subj,verb -> PredVP subj (UseV verb)

Relevant part of Resource Grammar API for "Face"

These functions (some of which are structural words) are used.

mkCl : NP -> V2 -> NP -> ClJohn loves Mary
mkNP : Numeral -> CN -> NPfive cars
mkNP : Quant -> CN -> NPthat car
mkNP : Pron -> NPwe
mkCN : N -> CNcar
this_Quant : Quantthis, these
youSg_Pron : Pronyou (singular)
n1_Numeral, n2_Numeral : Numeralone, two
n100_Numeral : Numeralone hundred
have_V2 : V2have

Concrete syntax for English

How are messages expressed by using the library?

 concrete FaceEng of Face = open SyntaxEng, ParadigmsEng in { lincat Message = Cl ; Person = NP ; Object = CN ; Number = Numeral ; lin Have p n o = mkCl p have_V2 (mkNP n o) ; Like p o = mkCl p like_V2 (mkNP this_Quant o) ; You = mkNP youSg_Pron ; Friend = mkCN friend_N ; Invitation = mkCN invitation_N ; One = n1_Numeral ; Two = n2_Numeral ; Hundred = n100_Numeral ; oper like_V2 = mkV2 "like" ; invitation_N = mkN "invitation" ; friend_N = mkN "friend" ; }

Concrete syntax for Finnish

How are messages expressed by using the library?

 concrete FaceFin of Face = open SyntaxFin, ParadigmsFin in { lincat Message = Cl ; Person = NP ; Object = CN ; Number = Numeral ; lin Have p n o = mkCl p have_V2 (mkNP n o) ; Like p o = mkCl p like_V2 (mkNP this_Quant o) ; You = mkNP youSg_Pron ; Friend = mkCN friend_N ; Invitation = mkCN invitation_N ; One = n1_Numeral ; Two = n2_Numeral ; Hundred = n100_Numeral ; oper like_V2 = mkV2 "pitää" elative ; invitation_N = mkN "kutsu" ; friend_N = mkN "ystävä" ; }

Functors and interfaces

English and Finnish: the same combination rules, only different words!

Can we avoid repetition of the lincat and lin code? Yes!

New module type: functor, a.k.a. incomplete or parametrized module

 incomplete concrete FaceI of Face = open Syntax, LexFace in ...

A functor may open interfaces.

An interface has oper declarations with just a type, no definition.

Here, Syntax and LexFace are interfaces.

The domain lexicon interface

Syntax is the Resource Grammar interface, and gives

  • combination rules
  • structural words

Content words are not given in Syntax, but in a domain lexicon

 interface LexFace = open Syntax in { oper like_V2 : V2 ; invitation_N : N ; friend_N : N ; }

Concrete syntax functor "FaceI"

 incomplete concrete FaceI of Face = open Syntax, LexFace in { lincat Message = Cl ; Person = NP ; Object = CN ; Number = Numeral ; lin Have p n o = mkCl p have_V2 (mkNP n o) ; Like p o = mkCl p like_V2 (mkNP this_Quant o) ; You = mkNP youSg_Pron ; Friend = mkCN friend_N ; Invitation = mkCN invitation_N ; One = n1_Numeral ; Two = n2_Numeral ; Hundred = n100_Numeral ; }

An English instance of the domain lexicon

Define the domain words in English

 instance LexFaceEng of LexFace = open SyntaxEng, ParadigmsEng in { oper like_V2 = mkV2 "like" ; invitation_N = mkN "invitation" ; friend_N = mkN "friend" ; }

Put everything together: functor instantiation

Instantiate the functor FaceI by giving instances to its interfaces

 --# -path=.:present concrete FaceEng of Face = FaceI with (Syntax = SyntaxEng), (LexFace = LexFaceEng) ;

Also notice the domain search path.

Porting the grammar to Finnish

1. Domain lexicon: use Finnish paradigms and words

 instance LexFaceFin of LexFace = open SyntaxFin, ParadigmsFin in { oper like_V2 = mkV2 (mkV "pitää") elative ; invitation_N = mkN "kutsu" ; friend_N = mkN "ystävä" ; }

2. Functor instantiation: mechanically change Eng to Fin

 --# -path=.:present concrete FaceFin of Face = FaceI with (Syntax = SyntaxFin), (LexFace = LexFaceFin) ;

Modules of a domain grammar: "Face" community

1. Abstract syntax, Face

2. Parametrized concrete syntax: FaceI

3. Domain lexicon interface: LexFace

4. For each language L: domain lexicon instance LexFaceL

5. For each language L: concrete syntax instantiation FaceL

Module dependency graph

GF Resource Grammar Tutorial (16)

red = to do, orange = to do (trivial), blue = to do (once), green = library

Porting the grammar to Italian

1. Domain lexicon: use Italian paradigms and words

 instance LexFaceIta of LexFace = open SyntaxIta, ParadigmsIta in { oper like_V2 = mkV2 (mkV (piacere_64 "piacere")) dative ; invitation_N = mkN "invito" ; friend_N = mkN "amico" ; }

2. Functor instantiation: restricted inheritance, excluding Like

 concrete FaceIta of Face = FaceI - [Like] with (Syntax = SyntaxIta), (LexFace = LexFaceIta) ** open SyntaxIta in { lin Like p o = mkCl (mkNP this_Quant o) like_V2 p ; }

Free variation

There can be many ways of expressing a given semantic structure.

This can be expressed by the variant operator |.

 fun BuyTicket : City -> City -> Request lin BuyTicket x y = (("I want" ++ ((("to buy" | []) ++ ("a ticket")) | "to go")) | (("can you" | [] ) ++ "give me" ++ "a ticket") | []) ++ "from" ++ x ++ "to" ++y

The variants can of course be resource grammar expressions as well.

Overview of the resource grammar API

For the full story, see the resource grammar synopsis in

grammaticalframework.org/lib/doc/synopsis.html

Main division:

  • Syntax, common to all languages
  • ParadigmsL, specific to language L

Main categories and their dependencies

GF Resource Grammar Tutorial (17)

Categories of complex phrases

Textsequence of utterancesDoes John walk? Yes.
Uttutterancedoes John walk
Impimperativedon't walk
Ssencence (fixed tense)John wouldn't walk
QSquestion sentencewho hasn't walked
Clclause (variable tense)John walks
QClquestion clausewho doesn't walk
VPverb phraselove her
APadjectival phrasevery young
CNcommon noun phraseyoung man
Advadverbial phrasein the house

Lexical categories for building predicates

Aone-place adjective-smart
A2two-place adjectiveNPmarried (to her)
Advadverb-here
Ncommon noun-man
N2relational nounNPfriend (of John)
NPnoun phrase-the boss
Vone-place verb-sleep
V2two-place verbNPlove (her)
V3three-place verbNP, NPshow (it to me)
VSsentence-complement verbSsay (that I run)
VVverb-complement verbVPwant (to run)

Functions for building predication clauses

mkClNP -> V -> ClJohn walks
mkClNP -> V2 -> NP -> ClJohn loves her
mkClNP -> V3 -> NP -> NP -> ClJohn sends it to her
mkClNP -> VV -> VP -> ClJohn wants to walk
mkClNP -> VS -> S -> ClJohn says that it is good
mkClNP -> A -> ClJohn is old
mkClNP -> A -> NP -> ClJohn is older than Mary
mkClNP -> A2 -> NP -> ClJohn is married to her
mkClNP -> AP -> ClJohn is very old
mkClNP -> N -> ClJohn is a man
mkClNP -> CN -> ClJohn is an old man
mkClNP -> NP -> ClJohn is the man
mkClNP -> Adv -> ClJohn is here

Noun phrases and common nouns

mkNPQuant -> CN -> NPthis man
mkNPNumeral -> CN -> NPfive men
mkNPPN -> NPJohn
mkNPPron -> NPwe
mkNPQuant -> Num -> CN -> NPthese (five) man
mkCNN -> CNman
mkCNA -> N -> CNold man
mkCNAP -> CN -> CNvery old Chinese man
mkNumNumeral -> Numfive
n100_NumeralNumeralone hundred
plNumNum(plural)

Questions and interrogatives

mkQClCl -> QCldoes John walk
mkQClIP -> V -> QClwho walks
mkQClIP -> V2 -> NP -> QClwho loves her
mkQClIP -> NP -> V2 -> QClwhom does she love
mkQClIP -> AP -> QClwho is old
mkQClIP -> NP -> QClwho is the boss
mkQClIP -> Adv -> QClwho is here
mkQClIAdv -> Cl -> QClwhere does John walk
mkIPCN -> IPwhich car
who_IPIPwho
why_IAdvIAdvwhy
where_IAdvIAdvwhere

Sentence formation, tense, and polarity

mkSCl -> She walks
mkS(Tense)->(Ant)->(Pol)->Cl -> She wouldn't have walked
mkQSQCl -> QSdoes he walk
mkQS(Tense)->(Ant)->(Pol)->QCl -> QSwouldn't he have walked
conditionalTenseTense(he would walk)
futureTenseTense(he will walk)
pastTenseTense(he walked)
presentTenseTense(he walks) [default]
anteriorAntAnt(he has walked)
negativePolPol(he doesn't walk)

Utterances and imperatives

mkUttCl -> Utthe walks
mkUttS -> Utthe didn't walk
mkUttQS -> Uttwho didn't walk
mkUttImp -> Uttwalk
mkImpV -> Impwalk
mkImpV2 -> NP -> Impfind it
mkImpAP -> Impbe brave

More

Texts: Who walks? John. Where? Here!

Relative clauses: man who owns a donkey

Adverbs: in the house

Subjunction: if a man owns a donkey

Coordination: John and Mary are English or American

Exercises

1. Compile and make available the resource grammar library, latest version. Compilation is by make in GF/lib/src. Make it available by setting GF_LIB_PATH to GF/lib.

2. Compile and test the grammars face/FaceL (available in course source files).

3. Write a concrete syntax of Face for some other resource language by adding a domain lexicon and a functor instantiation.

4. Add functions to Face and write their concrete syntax for at least some language.

5. Design your own domain grammar and implement it for some languages.

Contents

Module structure

Statistics

How to start building a new language

How to test a resource grammar

The Assignment

The principal module structure

GF Resource Grammar Tutorial (18)

solid = API, dashed = internal, ellipse = abstract+concrete, rectangle = resource/instance, diamond = interface, green = given, blue = mechanical, red = to do

Division of labour

Written by the resource grammarian:

  • concrete of the row from Structural to Verb
  • concrete of Cat and Lexicon
  • Paradigms
  • abstract and concrete of Extra, Irreg

Already given or derived mechanically:

  • all abstract modules except Extra, Irreg
  • concrete of Common, Grammar, Lang, All
  • Constructors, Syntax, Try

Roles of modules: Library API

Syntax: syntactic combinations and structural words

Paradigms: morphological paradigms

Try: (almost) everything put together

Constructors: syntactic combinations only

Irreg: irregularly inflected words (mostly verbs)

Roles of modules: Top-level grammar

Lang: common syntax and lexicon

All: common grammar plus language-dependent extensions

Grammar: common syntax

Structural: lexicon of structural words

Lexicon: test lexicon of 300 content words

Cat: the common type system

Common: concrete syntax mostly common to languages

Roles of modules: phrase categories

AdjectiveadjectivesAP
Adverbadverbial phrasesAdN, Adv
ConjunctioncoordinationAdv, AP, NP, RS, S
Idiomidiomatic expressionsCl, QCl, VP, Utt
Nounnoun phrases and nounsCard, CN, Det, NP, Num, Ord
Numeralcardinals and ordinalsDigit, Numeral
Phrasesuprasentential phrasesPConj, Phr, Utt, Voc
Questionquestions and interrogativesIAdv, IComp, IDet, IP, QCl
Relativerelat. clauses and pronounsRCl, RP
Sentenceclauses and sentencesCl, Imp, QS, RS, S, SC, SSlash
Textmany-phrase textsText
Verbverb phrasesComp, VP, VPSlash

Type discipline and consistency

Producers: each phrase category module is the producer of value categories listed on previous slide.

Consumers: all modules may use any categories as argument types.

Contract: the module Cat defines the type system common for both consumers and producers.

Different grammarians may safely work on different producers.

This works even for mutual dependencies of categories:

 Sentence.UseCl : Temp -> Pol -> Cl -> S -- S uses Cl Sentence.PredVP : VP -> NP -> Cl -- uses VP Verb.ComplVS : VS -> S -> VP -- uses S

Auxiliary modules

resource modules provided by the library:

  • Prelude and Predef: string operations, booleans
  • Coordination: generic formation of list conjunctions
  • ParamX: commonly used parameter, such as Number = Sg | Pl

resource modules up to the grammarian to write:

  • Res: language specific parameter types, morphology, VP formation
  • Morpho, Phono,...: possible division of Res to more modules

Dependencies

Most phrase category modules:

 concrete VerbGer of Verb = CatGer ** open ResGer, Prelude in ...

Conjunction:

 concrete ConjunctionGer of Conjunction = CatGer ** open Coordination, ResGer, Prelude in ...

Lexicon:

 concrete LexiconGer of Lexicon = CatGer ** open ParadigmsGer, IrregGer in {

Functional programming style

The Golden Rule: Whenever you find yourself programming by copy and paste, write a function instead!

  • Repetition inside one definition: use a let expression

  • Repetition inside one module: define an oper in the same module

  • Repetition in many modules: define an oper in the Res module

  • Repetition of an entire module: write a functor

Functors in the Resource Grammar Library

Used in families of languages

  • Romance: Catalan, French, Italian, Spanish
  • Scandinavian: Danish, Norwegian, Swedish

Structure:

  • Common, a common resource for the family
  • Diff, a minimal interface extended by interface Res
  • Cat and phrase structure modules are functors over Res
  • Idiom, Structural, Lexicon, Paradigms are ordinary modules

Example: DiffRomance

Words and morphology are of course different, in ways we haven't tried to formalize.

In syntax, there are just eight parameters that fundamentally make the difference:

Prepositions that fuse with the article (Fre, Spa de, a; Ita also con, da, in, su).

 param Prepos ;

Which types of verbs exist, in terms of auxiliaries. (Fre, Ita avoir, être, and refl; Spa only haber and refl).

 param VType ;

Derivatively, if/when the participle agrees to the subject. (Fre elle est partie, Ita lei è partita, Spa not)

 oper partAgr : VType -> VPAgr ;

Whether participle agrees to foregoing clitic. (Fre je l'ai vue, Spa yo la he visto)

 oper vpAgrClit : Agr -> VPAgr ;

Whether a preposition is repeated in conjunction (Fre la somme de 3 et de 4, Ita la somma di 3 e 4).

 oper conjunctCase : NPForm -> NPForm ;

How infinitives and clitics are placed relative to each other (Fre la voir, Ita vederla). The Bool is used for indicating if there are any clitics.

 oper clitInf : Bool -> Str -> Str -> Str ;

To render pronominal arguments as clitics and/or ordinary complements. Returns True if there are any clitics.

 oper pronArg : Number -> Person -> CAgr -> CAgr -> Str * Str * Bool ;

To render imperatives (with their clitics etc).

 oper mkImperative : Bool -> Person -> VPC -> {s : Polarity => AAgr => Str} ;

Pros and cons of functors

+ intellectual satisfaction: linguistic generalizations

+ code can be shared: of syntax code, 75% in Romance and 85% in Scandinavian

+ bug fixes and maintenance can often be shared as well

+ adding a new language of the same family can be very easy

- difficult to get started with proper abstractions

- new languages may require extensions of interfaces

Workflow: don't start with a functor, but do one language normally, and refactor it to an interface, functor, and instance.

Suggestions about functors for new languages

Romance: Portuguese probably using functor, Romanian probably independent

Germanic: Dutch maybe by functor from German, Icelandic probably independent

Slavic: Bulgarian and Russian are not functors, maybe one for Western Slavic (Czech, Slovak, Polish) and Southern Slavic (Bulgarian)

Fenno-Ugric: Estonian maybe by functor from Finnish

Indo-Aryan: Hindi and Urdu most certainly via a functor

Semitic: Arabic, Hebrew, Maltese probably independent

Effort statistics, completed languages

common413--41322001
abstract729-4681197242001
Bulgarian12002329502403132008
English1025772506230362001
Finnish14711490703366462003
German1337604492243362002
Russian149236685345694182002
Romance1346--1346102003
Catalan521*9000518*1003942006
French4681789514277162002
Italian423*7423500*834632003
Spanish417*6549516*748232004
Scandinavian1293--129342005
Danish262683486143122005
Norwegian281676488144522005
Swedish280717491148842001
total12545*367006718*559631032001

Lines of source code in April 2009, rough estimates of person months. * = generated code.

How to start building a language, e.g. Marathi

1. Create a directory GF/lib/src/marathi

2. Check out the ISO 639-3 language code: Mar

3. Copy over the files from the closest related language, e.g. hindi

4. Rename files marathi/*Hin.gf to marathi/*Mar.gf

5. Change imports of Hin modules to imports of Mar modules

6. Comment out every line between header { and the final }

7. Now you can import your (empty) grammar: i marathi/LangMar.gf

Suggested order for proceeding with a language

1. ResMar: parameter types needed for nouns

2. CatMar: lincat N

3. ParadigmsMar: some regular noun paradigms

4. LexiconMar: some words that the new paradigms cover

5. (1.-4.) for V, maybe with just present tense

6. ResMar: parameter types needed for Cl, CN, Det, NP, Quant, VP

7. CatMar: lincat Cl, CN, Det, NP, Quant, VP

8. NounMar: lin DetCN, DetQuant

9. VerbMar: lin UseV

10. SentenceMar: lin PredVP

Character encoding for non-ASCII languages

GF internally: 32-bit unicode

Generated files (.gfo, .pgf): UTF-8

Source files: whatever you want, but use a flag if not isolatin-1.

UTF-8 and cp1251 (Cyrillic) are possible in strings, but not in identifiers. The module must contain

 flags coding = utf8 ; -- OR coding = cp1251

Transliterations are available for many alphabets (see help unicode_table).

Using transliteration

This is what you have to add in GF/src/GF/Text/Transliterations.hs

 transHebrew :: Transliteration transHebrew = mkTransliteration allTrans allCodes where allTrans = words $ "A b g d h w z H T y K k l M m N " ++ "n S O P p Z. Z q r s t - - - - - " ++ "w2 w3 y2 g1 g2" allCodes = [0x05d0..0x05f4]

Also edit a couple of places in GF/src/GF/Command/Commands.hs.

You can later convert the file to UTF-8 (see help put_string).

Diagnosis methods along the way

Make sure you have a compilable LangMar at all times!

Use the GF command pg -missing to check which functions are missing.

Use the GF command gr -cat=C | l -table to test category C

Regression testing with a treebank

Build and maintain a treebank: a set of trees with their linearizations:

1. Create a file test.trees with just trees, one by line.

2. Linearize each tree to all forms, possibly with English for comparison.

 > i english/LangEng.gf > i marathi/LangMar.gf > rf -lines -tree -file=test.trees | l -all -treebank | wf -file=test.treebank

3. Create a gold standard gold.treebank from test.treebank by manually correcting the Marathi linearizations.

4. Compare with the Unix command diff test.treebank gold.treebank

5. Rerun (2.) and (4.) after every change in concrete syntax; extend the tree set and the gold standard after every new implemented function.

Sources

A good grammar book

  • lots of inflection paradigms
  • reasonable chapter on syntax
  • traditional terminology for grammatical concepts

A good dictionary

  • inflection information about words
  • verb subcategorization (i.e. case and preposition of complements)

Wikipedia article on the language

Google as "gold standard": is it rucola or ruccola?

Google translation for suggestions (can't be trusted, though!)

Compiling the library

The current development library sources are in GF/lib/src.

Use make in this directory to compile the libraries.

Use runghc Make lang api langs=Mar to compile just the language Mar.

Assignment: a good start

1. Build a directory and a set of files for your target language.

2. Implement some categories, morphological paradigms, and syntax rules.

3. Give the lin rules of at least 100 entries in Lexicon.

4. Send us: your source files and a treebank of 100 trees with linearizations in English and your target language. These linearizations should be correct, and directly generated from your grammar implementation.

You might also like

Latest Posts

Article information

Author: Horacio Brakus JD

Last Updated: 10/02/2022

Views: 6601

Rating: 4 / 5 (71 voted)

Reviews: 94% of readers found this page helpful

Author information

Name: Horacio Brakus JD

Birthday: 1999-08-21

Address: Apt. 524 43384 Minnie Prairie, South Edda, MA 62804

Phone: +5931039998219

Job: Sales Strategist

Hobby: Sculling, Kitesurfing, Orienteering, Painting, Computer programming, Creative writing, Scuba diving

Introduction: My name is Horacio Brakus JD, I am a lively, splendid, jolly, vivacious, vast, cheerful, agreeable person who loves writing and wants to share my knowledge and understanding with you.