Chapter 2

Question 1

Stemming

Answer 1

Process in which you use regular expressions to combine words with similar meaning

Question 2

N-grams

Answer 2

Counts of Pairs of words (2-grams, 3-grams) etc that occur in sequence in a sentence. Help in retaining some of the meaning in the sentence as opposed to BOW.

Question 3

Challenges with stemming

Answer 3

Difficult to remove different variations of inflection (running for example), discriminating between pluralizing “s” at the end of “words” and normal “s” as in “bus” etc

Question 4

Semantic stems

Answer 4

Useful cluster of words like lemmas or synonyms

Question 5

Tokenization

Answer 5

Kind of document segmentation, which breaks up text into smaller chunks or segments with more focused info. content (but in this case into tokens, instead of paragraphs, sentences or phrases)

Question 6

Scanner or lexer

Answer 6

Tokenizer used for compiling computer languages

Question 7

Lexicon

Answer 7

Vocabulary for a computer language

Question 8

Terminal

Answer 8

Leaves in end of line for CFG grammars

Question 9

One-hot vectors

Answer 9

Numerical vector representation for each word in a sentence. Each row indicates a vector for a single word in the sentence. They are typically super-sparse (containing only one zero). It is piano paper roll. Vocab. key tells which note or word to play for each row in sequence of words or piano music.

Question 10

Disadvantage with one-hot vectors

Answer 10

Creates space explosion for long documents

Question 11

Bag of words

Answer 11

Relies on the idea of gleaning the meaning of sentence baed on words rather than order and grammar. Compress information content for each document into data structure easier to work with. (Relies on frequency). Can be indexed to indicate which words were used in which document. Note: Important to be consistent with the order of words.

Question 12

Bag of Words with dictionary

Answer 12

Saves space storing the ones and zeros instead of tuple (Pandas DataFrame series).

Question 13

Dot Product

Answer 13

Way to check for similarities between sentences by counting the number of overlapping tokens. Inner product between two matrices or inner join on two tables. (A.T * B) (row vector * column vector)

Question 14

Cross Product

Answer 14

Produces a vector as its output

Question 15

How Regex works

Answer 15

[] - character class
+ - match must contain one or more of characters inside square brackets
\s - shortcut to predefined character class

Question 16

Spacy, Stanford CoreNLP, NLTK

Answer 16

Other libraries that implement tokenizers

Question 17

Contractions

Answer 17

Important to split wasn’t into was n’t for grammar-based NLP models that use syntax trees to separate was and not to indicate contradiction.

Question 18

Problems with n-grams

Answer 18

Most of them are pretty rare. Don’t carry correlation with other words that you can use to help identify topics or themes. Filtered out if they occur too infrequently or too often.

Question 19

Stop words

Answer 19

Common words in any language that occur a high frequency but carry little information. Excluded from most NLP tasks. Caution: Might carry information Other side: retaining them might increase the length of n-grams you need to use. Including stop words allows document filters to accurately identify and ignore words and n-grams with least information content.

Question 20

Normalizing vocabulary

Answer 20

Tokens that mean similar things are combined into a single, normalized form. Reduces the number of tokens you need to retain in your vocabulary and improves association of meaning across different spellings or n-grams.

Question 21

Case folding

Answer 21

Consolidate multiple spellings of word that differ in capitalization. Helps in reducing vocabulary size and generalize the NLP pipeline. Lose some information in the process for instance doctor and “Doctor” have two different meanings. Lowercasing everything gets rid of Camel Case information as well. Better approach is to only lower case the first letter of each word. Preserves meaning of proper nouns in the middle of the sentence. Many NLP pipelines completely avoid this to preserve meaning of proper nouns. Might be different for a search engine if you want to return both capitalized and non capitalized queries.

Question 22

Stemming

Answer 22

Eliminate small differences of pluralization or possessive endings. Identify common stem (housing, houses and house share same name). Reduces size of vocabulary while limiting loss of information and meaning. Helps with dimensionality reduction. Important for search engines as you want to return searches with common stem words.