Introduction - Text Representation and Boolean Model

Question 1

Primary task of an IR system

Answer 1

Retrieve documents with content that is relevant to a user’s information need.

Question 2

What IR is not

Answer 2

It is not a database management systems. These store and process well-defined data. A search within them is exact and deterministic while search in an IR system is probabilistic.

Question 3

Bag of Words

Answer 3

Document is represented as consisting of words as independent units with word order ignored.

Question 4

Coordinate Matching

Answer 4

Document relevance measured by the number of query terms appearing a document. Terms provide the dimensions with the length along a dimension being either 0 or 1. Similarity measure is dot product of query and document vectors. This does not consider frequency of query terms in documents.

Question 5

Term Frequency

Answer 5

Weighting by the frequency of terms in the document.

Question 6

Inverse Document Frequency

Answer 6

Weight terms proportionally to the reciprocal of the number of documents they appear in.

Question 7

Document Length

Answer 7

Similarity measure should be normalised to prevent document getting high score simply due to length.

Question 8

Vector Space Similarity

Answer 8

Cosine of the angle between the query and document vectors.

Question 9

Document

Answer 9

An item which may satisfy the user’s information need.

Question 10

Query

Answer 10

Representation of user’s information need.

Question 11

Term

Answer 11

Any word or phrase that can serve as a link to a document.

Question 12

Inverted File

Answer 12

Keep following information for each term:

• Document ID where this term occurs.
• Frequency of occurrence of this term in each document
• Possibly: Offset of this term in document

Question 13

Tokenisation

Answer 13

Dividing a character stream into a sequence of distinct word forms (tokens). Separate on white-space, end of sentence punctuation, bracketing, hyphenation, apostrophes & slashes.

Question 14

Stop Word

Answer 14

High-frequency word which is not useful for distinguishing between documents.

Question 15

Equivalence Classes

Answer 15

Can be useful to put tokens into equivalence classes and treat a group of terms as the same term. This reduces size of index, may lead to improved retrieval and combined frequencies may better reflect content than individual frequencies.

Question 16

Stem

Answer 16

The core of a word (its main morpheme) to which inflectional and derivational morphology applies.

Question 17

Stemming

Answer 17

Attempts to remove inflectional (and some) derivational morphology.

Question 18

Lemmatisation

Answer 18

Just attempts to remove inflectional morphology.

Question 19

Porter Stemmer

Answer 19

Removes suffixes without a stem dictionary. Conflates terms and doesn’t deal with root changes.

Question 20

Porter Stemmer - Word Representation

Answer 20

C{m}[V]

C - one or more adjacent consonants
V - one or more adjacent vowels

[ ] - optionality
( ) - group operator
{x} - repetition x times
m - the “measure” measure of a word

m is calculated on word excluding suffix of rule under consideration.

Question 21

Porter Stemmer - Possible Conditions

Answer 21

Constraining the measure

Constraining the shape of the word piece e.g. stem ends with certain letter, stem contains a vowel, stem ends with a double consonant.

Boolean expressions

Question 22

Boolean Model

Answer 22

Queries consist of terms connected by AND, OR & NOT.

Assumptions:

• Terms either present or absent in a document.
• Terms are all equally informative when determining relevance.
• A document is either relevant or not.

Question 23

Boolean Model - Pros and Cons

Answer 23

Pros:

• Simple framework
• Well-defined query semantics

Cons:

• Difficult to formulate complex queries
• Difficult to control output volume
• No ranking facility