Linguistic Modules

Question 1

What are some issues with tokenization?

Answer 1

1. Hyphens: Break into multiple tokens or remain as one?
2. Optional Hyphens: Users may use different forms of the same word
3. Whitespace: Can’t be sure where to split tokens. Effect is compounded with hyphens

Question 2

What are some approaches for normalization?

Answer 2

1. Equivalence classes
2. Query expansion
3. Lemmatization

Question 3

What are equivalence classes?

Answer 3

Grouping terms by mapping them to the same base term
Ex: Deleting periods or hyphens so U.S.A. maps to USA

Question 4

What are some issues associated with normalization in other languages?

Answer 4

1. Accents: Should they be removed and mapped to the same term? Should they be kept and mapped separately?
2. Terms used in different languages mean different things

Question 5

What is case folding?

Answer 5

Reducing all letters to lower case. Better to do this since users tend to use lowercase regardless of correct captitalization

Question 6

What are stop words?

Answer 6

Extremely common words which appear to be of little value in selecting documents such as “of”, “and”, “the”, etc

Question 7

Why do most search engines index stop words even though they are of little value?

Answer 7

They are needed for phrase queries to determine proper proximity between tokens

Question 8

What is query expansion?

Answer 8

A method of normalization where you take every token the tokenizer admits and add them as separate dictionary terms. Then, when you query your word could map to several words in the index once processed

Question 9

How can we handle spelling mistakes in a query?

Answer 9

Using a soundex which forms equivalence classes based on phonetic heuristics

Question 10

What is lemmatization?

Answer 10

Reducing variant forms of a word to the base form.
Ex: am, are, is -> be
Applies linguistic rules to transform the word

Question 11

What is one of the main methods of lemmatization?

Answer 11

Stemming

Question 12

What is stemming?

Answer 12

A heuristic lemmatization process that chops off the ends of words to reduce the number of word variations

Question 13

What is the Porter Algorithm?

Answer 13

The most common algorithm for stemming english words to reduce them to their root forms by stripping prefixes and suffixes

Question 14

When can stemming improve effectiveness?

Answer 14

If stemmed words are actually close to each other in meaning it can increase recall.
Ex: It is good for tours and touring to be mapped to tour but not for operational and operative to map to oper