Text Tokenization

Question 1

Tokenization

Answer 1

Segmenting running text into words/tokens

Question 2

Types of Tokenization

Answer 2

1. Whitespace split
2. Penn Treebank (whitespace and punctuation)

Question 3

Word Type

Answer 3

Distinct word in a corpus

Question 4

Corpus

Answer 4

Collection of written texts

Question 5

Word Token

Answer 5

Total number of word types

Question 6

Vocabulary

Answer 6

Set of word types.
Each word is represented using distinct ID

Question 7

Word (Type) Frequency

Answer 7

Number of occurrences (tokens) of that type in a corpus

Question 8

Normalization

Answer 8

Converting words to a standard form

Question 9

Methods of Normalization

Answer 9

1. Lemmatization
2. Stemming
3. Case Folding

Question 10

Lemmatization

Answer 10

Return the word’s ‘lemma’ i.e. the root of a word despite its surface form. Strips word to an existing normalized form.

Question 11

Stemming

Answer 11

Return the word’s ‘stem’, which chops off the word affixes

Question 12

Case Folding

Answer 12

Mapping every word to its lowercase form

Question 13

Character Tokenization

Answer 13

Represents each word using its characters,
+smaller vocabulary
+can process every word
-long sequence for each word
-not intuitive for learning meaning

Question 14

Issue/Solution: Unknown Words

Answer 14

1. Special UNK Token
2. Character Tokenization
3. Subword Tokenization (Byte-Pair)

Question 15

Special UNK Token

Answer 15

Categorizes all unknown words under same token.

Question 16

Subword Tokenization

Answer 16

Splitting words into smaller sub tokens.
Ex: Byte-Pair Encoding

Question 17

Byte-Pair Encoding

Answer 17

Frequent words aren’t split.
Rare words are decomposed into smaller, meaningful subwords. (docs = doc + s).

Question 18

Byte-Pair Encoding Process

Answer 18

Represent words as characters and then keep merging frequent character pairs to form k (specified parameter) new symbols.