06 Scoring, term weighting, and vector space model

Question 1

Metadata

Answer 1

Digital docs generally encode, in machine-recongniziable form, certain meta- data associated with each document. Such as author(s), title, and date of publication.

Question 2

Field

Answer 2

Examples of fields: Date of creation, format, author, title. 22

Question 3

Paramtetric indexes

Answer 3

There is one parmetric index for each field. It allows us to select only the docs mathcing a data specified in the query.

Question 4

Zones

Answer 4

Similar to fields, exept the content of a zone can be arbitrary free text.

Question 5

Weighted zone scoring

Answer 5

Assings to the pair (query, doc) a score in the interval [0, 1], by computing a linear combinatin of zone scores.

Question 6

Term frequency

Answer 6

Number of occurences of a term in a doc.

Question 7

Bag of words

Answer 7

A doc representation where the exact ordering of the terms in a document is ignored. We only retain information on the number of occrrences of each term. Therefore, ÒMary is quicker than JohnÓ gives the same document representation as ÒJohn is quicker than MaryÓ.

Question 8

Document frequency

Answer 8

Defined to be the number of docs in the collection that contains a term t.

Question 9

Inverse document frequency

Answer 9

idf = log (N/df). Thus, the idf of a rare term is high.

Question 10

Tf-idf

Answer 10

tf-idf = tf x idf

Properties

1. Highest when a term occurs many times wihtin a small number of docs
2. Lower when the term occurs fewer times in a document, or occurs in many documents
3. Lowest when the term occurs in virtually all documents.

Question 11

Document vector

Answer 11

Can view each document as a vector with one component corresponding to each term in the dictionary, together with a weight for each component that is given by tf-idf.

Question 12

Vector space model

Answer 12

The representation of a set of docs as vectors in a common vector space is known as the vector space model.

Question 13

Cosine similarity

Answer 13

The standard way of quantifying the similarity between two documents or a query and a document is to compute the cosine similiarity where the number- ator represents the dot product and the denominator is the product of their Euclidean length. This measure is the cosine of the angle /Theta between two vectors.

Question 14

Term-document matrix

Answer 14

Viewing a collection of N docs as a collection of vectors leads to a natural view of a collection as term-document matrix.

Question 15

Term-at-a-time

Answer 15

The process of adding contributions one query term at a time is known as term-at-a-time scoring or accumulation, and the N elements of the array ÒScoresÓ are therfore known as accumulators.

Question 16

Document-at-a-time

Answer 16

In a postings traversal, if we compute the scores of one document at a time, its called document-at-a-time scoring. Basically, if you use the merg- ing/intersection algorithm from chaper 1, you are using document-at-a-time scoring.

Question 17

Sublinear tf-scaling

Answer 17

Use the logarithm of the term frequency, because a term that occurs twenty times is (often) not 20 times more important than a term that occurs once. Therefore: tf-idf = wf x idf where wf = 1 + log tf if tf>0 wf = 0 otherwise

Question 18

Maximum tf normalization

Answer 18

Normalize the tf weights of all terms ocuring in a document by the maximum tf in that doc.