Text Preprocessing

Question 1

Corpus

Answer 1

A computer-readable collection of text or speech.

Question 2

Lemma

Answer 2

A set of lexical forms having the same stem, the same major part-of-speech, and the same word sense.

Question 3

Word-form

Answer 3

The full inflected or derived form of the word.

Question 4

Word type

Answer 4

Types are the number of distinct words in a corpus; if the set of words in the vocabulary is V, the number of types is the vocabulary size |V|.

Question 5

Work token

Answer 5

Tokens are the total number N of running words.

Question 6

Herdan’s Law

Answer 6

The larger the corpora, the more word types we find.

|V| = kN^β

The value of β depends on the corpus size and the genre, but typically ranges from .67 to .75.

A.k.a Heaps’ Law

Question 7

datasheet or data statement

Answer 7

Specifies properties of a dataset, like:

• Motivation
• Situation
• Language variety
• Speaker demographics
• Collection process
• Annotation process
• Distribution

Question 8

Datasheet properties

Motivation

Answer 8

Why was the corpus collected, by whom, and who funded it?

Question 9

Datasheet properties

Situation

Answer 9

When and in what situation was the text written / spoken?

E.g. was there a task? Was the language originally spoken conversation, edited text, social media communication, monologue vs dialogue?

Question 10

Datasheet properties

Language variety

Answer 10

What language (including dialect / region) was the corpus in?

Question 11

Datasheet properties

Speaker demographics

Answer 11

What as, e.g., age or gender of the authors of the text?

Question 12

Datasheet properties

Collection process

Answer 12

How big is the data?

If it is a subsample how was it sampled?

Was the data collected with consent?

How was the data pre-processed, and what metadata is available?

Question 13

Datasheet properties

Annotation process

Answer 13

What are the annotations, what are the demographics of the annotators, how were they trained, how was the data annotated?

Question 14

Datasheet properties

Distribution

Answer 14

Are there copyright or other intellectual property restrictions?

Question 15

3 common tasks associated with Text Normalisation

Answer 15

1. Tokenizing (segmenting) words
2. Normalizing word formats
3. Segmenting sentences

Question 16

Tokenization

Answer 16

The task of segmenting running text into words.

Question 17

Clitic contractions

Answer 17

Contractions marked by apostrophes,

e.g. what're

we're

Question 18

Clitic

Answer 18

A part of a word that can’t stand on it’s own, and can only occur when it is attached to another word.

we're, j'ai, l'homme

Question 19

Subwords

Answer 19

Subwords can be arbitrary substrings, or they can be meaning-bearing units like the morphemes -est or -er.

Question 20

Morpheme

Answer 20

The smallest meaning-bearing unit of a language.

e.g. the word unlikeliest has the morphemes un-, likely, and -est.

Question 21

2 parts of most tokenization schemes

Answer 21

A token learner and a token segmenter.

Question 22

token learner

Answer 22

takes a raw training corpus (sometimes roughly separated into words, e.g. by whitespace) and induces a vocabulary, a set of tokens.

Question 23

token segmenter

Answer 23

takes a raw test sentence and segments it into the tokens in the vocabulary.

Question 24

Byte-pair encoding algorithm

Answer 24

A token learner.

It begins with a vocabulary that is just the set of all individual characters.

It then examines the training corpus, chooses the two symbols that are most frequently adjacent (say ‘A’, ‘B’), adds a new merged symbol ‘AB’ to the vocabulary, and replaces every adjacent ‘A’ ‘B’ in the corpus with the new ‘AB’.

It continues to count and merge, creating new longer and longer character strings, until k merges have been done creating k novel tokens.

k is thus a parameter of the algorithm.

The resulting vocabulary consists of the original set of characters plus k new symbols.

Question 25

word normalisation

Answer 25

The task of putting words / tokens in a standard format, choosing a single normal form for words with multiple forms like USA and US or uh-huh and uhhuh.

Question 26

lemmatization

Answer 26

the task of determining that two words have the same root, despite their surface differences.

e.g. am, are and is have the shared lemma be.

dinner and dinners both have the lemma dinner.

Question 27

Morphology

Answer 27

The study of the way words are built up from smaller meaning-bearing units called morphemes.

Question 28

2 broad classes of morphemes

Answer 28

“stems” - the central morpheme of the word, supplying the main meaning

affixes - the “additional” meanings of various words.

Question 29

Stemming

Answer 29

A naive version of morphological analysis.

This mainly consists of chopping off word-final affixes.

Question 30

Porter Algorithm

Answer 30

Simple and efficient way to do stemming, stripping of affixes.

It does not have high accuracy but may be useful for some tasks.

Question 31

Minimum edit distance

Answer 31

The minimum edit distance between two strings is defined as the minimum number of editing operations (e.g. insertion, deletion, substitution) needed to transform one string into another.

We can also assign a weight / cost to each of these operations. Levenshtein distance is the simplest, with each of the operations having a cost of 1.