TEXT PRE PROCESSING

Question 1

Before we start text preprocessing what is it important to think about

Answer 1

What genre and domains the text is in
Genres: social media/emails/literature
Domain: Chemistry/politics/entertainment

Because we may need specific resources
Thinks like format and punctuation will change depending

Question 2

What is tokenisation

Answer 2

Breaking input into individual units of text (tokens)
Easiest way is to use whitespace

Question 3

Issue with whitespace

Answer 3

some languages do not use whitespace so it will not tokenise correctly

Question 4

What are the typical tokenisation steps

Answer 4

• Initial segmentation (White space)
  -handling abbreviations and apostrophes
  -handling hyphenations
  -dealing with other special exps (URLs)

Question 5

Traditional NLP tokenisation

Answer 5

Uses simple word like tokens

Question 6

Modern NLP tokenisation

Answer 6

Subword tokenisation

Question 7

What is normalisation

Answer 7

Process of standardizing and transforming text data to a common, consistent format
Consists of:
Lowercasing
Removing Punctuation
Removing Stop Words: removing common words (e.g., “the,” “and,” “is”)
Stemming
Lemmatization

Question 8

What is lemmatisation

Answer 8

reduction to “dictionary headword” form
(lemma)
{am, are, is} -> be

Question 9

What is morphological analysis

Answer 9

Words are not the only units of ‘meaning’
subwords = morphemes, have some meaning
(un) (happy) (ness)
(prefix) (stem) (suffix)

So morphological analysis is taking a morpheme and seeing what its stem and affixes are
This can depend on the context

Question 10

What is a ‘derivation’

Answer 10

formation of a word from its stem and suffixes and affixes
eg Un-Happi-Ness
Friend-ly

Question 11

What is inflection

Answer 11

The modification of a word to express different grammatical roles
eg come -> came
waiter -> waitress

Question 12

What is regular inflectional morphology

Answer 12

Changes that occur to express grammatical features like tense, number, gender, case
Easy and predictable

Question 13

What is derivational morphology

Answer 13

Creation of new words by adding affixes or other modifications
changes the word class
teach -> teacher

Question 14

What is stemming

Answer 14

Chops “ends of words”
removes suffixes, sometimes prefixes
quite quick
Can largely neutralise inflection and some derivation
Can yield non-words

Question 15

What is under-stemming

Answer 15

fails to conflate related terms
divide -> divid
division -> divis

Question 16

What is over-stemming

Answer 16

conflates unrelated terms
neutron, neutral -> neutr

Question 17

What is the Porter stemmer

Answer 17

One of the most common stemmers in the English language
rules based
focuses on suffix stripping

Question 18

What is character n-gram tokenisation

Answer 18

Breaks down text into “n” length substring tokens

Question 19

What is Byte-pair encoding

Answer 19

We add _ at the end of all words to know where the words are separated

Let the initial vocabulary {A,B,C…a,b,c…} be the set of individual characters in the corpus
Choose the two symbols which are the most frequently adjacent in the corpus eg ‘e’‘r’

Add a new symbol ‘er’
{…x,y,z,er}
replace all ‘e’‘r’ with ‘er’

Until k merges have been done

Question 20

What is a token learner

Answer 20

takes a raw training corpus and induces a vocabulary
(inventory of tokens)

Question 21

What is a token segmenter

Answer 21

takes a raw test i.e. input sentence and tokenizes it according to that vocabulary

Question 22

What is the result of a BPE token learner

Answer 22

Most words and subwords (affixes?) will be represented as full symbols
{ing, ed, er}
Very rare tokens (including unknown words) will be represented by their parts (subwords)
Can ‘control’ k (the number of merges) as a parameter depending on how many symbols we want in the vocabulary

Question 23

What is a BPE token segmenter

Answer 23

new data, run each merge learnt from
the training data greedily and in the order we’ve learned them
– Merge every e r to er first, then merge er to er_

Words that have not been seen before will be represented by subtokens ‘low’‘er’

BPE reduces the number of unseen tokens (OOV)