Words

Question 1

What are some problems with natural langauge?

Answer 1

There is lots of ambiguity from identical word forms

There is also dependency on punctuation or intonation

Question 2

What types of texts exist?

Answer 2

Formal News

Polemic News (argumentative)

Speech

Historic, Poetic, Musical

Social Media

Question 3

What is a sentence?

Answer 3

A unit of written language

Question 4

What is an utterance?

Answer 4

It is a unit of spoken language

Question 5

What is a word form?

Answer 5

It is the inflected form as it appears in the corpus

Question 6

What is a lemma?

Answer 6

It is an abstract form shared by word forms having the same stem, POS, word sense

Question 7

What are function words?

Answer 7

Indicate the grammatical relationship between terms but have little topical meaning

Question 8

What are types?

Answer 8

They are a number of distinct words in a corpus

Question 9

What are tokens?

Answer 9

It is the collection of all words

Question 10

What are some lexical analysis steps we can take?

Answer 10

Stripping punctuation, folding cases, removing function words, lemmatising and stemming text, taking an index for each of the words

Question 11

How much of text is function words?

Answer 11

They account for up to 60 percent of text

Question 12

What does repetition signal?

Answer 12

It signals intention

Question 13

What do wordclouds provide?

Answer 13

They provide visual representation of statistical summary

Question 14

What is tokenization?

Answer 14

It is the process of turning a stream of characters into a sequence of words

Question 15

What is a token?

Answer 15

A token is a lexical construct that can be assigned grammatical and semantic roles

Question 16

What is a naive solution to tokenization?

Answer 16

Break on spaces and punctuation - too simple for general case, is useful piece of information for parses, helps to indicate sentence boundaries

Question 17

What are some tokenization issues?

Answer 17

Punctuation can be internal, for example abbreviations, prices, times etc

We can have multiword expressions (New York, rock ‘n’ roll)

Clitic contractions (we’re, I’m, etc.)

Numeric expressions

Question 18

What tokenization method is better than splitting on spaces and punctuation?

Answer 18

Pattern tokenizaation

Question 19

What is pattern tokenization?

Answer 19

It is the use of regular expressions or other pattern matching styles to define tokenization rules

Question 20

What are some problems with pattern tokenization?

Answer 20

Rules will be corpus specific and probably overfitted to achieve accuracy for a very specific task

Question 21

Besides splitting on punctuation/spaces and pattern tokenization, what is another way to tokenize text?

Answer 21

We can learn common patterns from the corpus itself, or a similar training corpus

Words can be split into subword units (morphemes, significant punctuation)

Question 22

How can we split words?

Answer 22

Lemmatization is a way of determining words with different superficial forms but have the same root

Words are composed of subword units called morphemes

Word parts with recognised meanings can have an affix (stems + affixes)

Question 23

What is stemming?

Answer 23

Stemming is a lemmatizaation process that removes suffixes by applying a sequence of rewrites

Question 24

What is a popular stemmer?

Answer 24

The Porter Stemmer

Question 25

What are some issues with the Porter Stemmer?

Answer 25

It is very language specific and is very crude

Question 26

What is BPE?

Answer 26

Byte-Pair Encoding

Question 27

How does BPE work?

Answer 27

It works by starting with a vocabulary which is the set of individual characters and tries to learn new tokens. This is repeated k times examining the corpus.

Question 28

What happens each time BPE repeats?

Answer 28

It selects the two symbols that are most frequently adjacent. (A, B)

It adds the new merged symbol to the vocabulary (AB)

It replaces every adjacent A B in the corpus with the new AB

Question 29

What happens when BPE is run with thousands of merges on a very large corpus?

Answer 29

It represents most words as full symbols and only the rare words and unknown words will have to represented by their parts

Question 30

Explain what the image shows

Answer 30

It shows BPE in process. We start with the vocabulary and the word end. It then looks for the most common pair of adjacent tokens, which is ‘er’. We then create the ‘er’ token and rewrite any appearances of this token with the combined token, and then repeat this process. This continues until we have enough symbols

Question 31

What are some alternatives to BPE?

Answer 31

WordPiece - Is based on an n-gram language model using multiple-adjacent words as single tokens

SentencePiece - extends these as a simple and language independent text tokenizer

Question 32

How do we compute if words are similar?

Answer 32

Levenshtein Distance

Minimum Edit Distance

Question 33

How does Levenshtein distance work?

Answer 33

It is the shortest sequence of edits to transform one string into another. In the image, we make 3 substitutions, a delete and an insert, so we have a total of 5 edits

Question 34

How does Minimum Edit Distance work?

Answer 34

We works by writing down the set of edits needed to go from one word to another. Its applications include plagiarism analysis, alignments in parallel corpora, similarities in different generations for software updates

Question 35

How can the Levenshtein metric be calculated?

Answer 35

It can be calculated recursively