02 Term Vocabularies and Postings Lists

Question 1

Why Simple Boolean retrieval system in reality can be hard?

Answer 1

• different format and language
• What is the document unit for indexing? (depends on design decision)

Paragraph? Sentence? XML elements?

Question 2

What is the difinition of tokens ?

Answer 2

• Character sequence in document
• closely related to Word

Question 3

What is the difinition of type ?

Answer 3

• Equivalence class of tokens
• related to Term: (normalized type)

Question 4

I like the coffee, coffees, and the shop.

How many tokens? Types? Terms?

Answer 4

• 11 tokens
• 9 types
• 8 terms

Question 5

In June, the dog likes to chase the cat in the barn.

How many tokens? Types? Terms?

Answer 5

• 14 tokens
• 12 types
• 11 terms

Question 6

Give examples of Tokenization problems

Answer 6

• one word or two? (York University vs. New York University)
• number format
• no whitespace (Chinese, Japanese)
• ambiguous segmentation (Chinese)
• compound words (German)
• bidirectionality (Arabic)
• accents and umlauts