Week 5

Question 1

Basic Idea of Vector Space Model

Answer 1

• Relevance is same as similarity

- if a document is similar to the query it has higher relevance

Question 2

Vector Space Model Framework

Answer 2

• Represent document/query by term vector
  Query vector q is query term weight
  Document vector d is document term weight

Question 3

How is relevance measured in VSM

Answer 3

based on similarity of two vectors

Question 4

What is a bit vector representation

Answer 4

1 word is present, 0 word is absent

Question 5

What is Similarity instantiation Dot Product?

Answer 5

• Common method to measure similarity
• Similarity between vector and query
• Yields a score to rank

Question 6

Dot Product example

Answer 6

Get all words in V list
create query vector with V words with 1
create document vector with words given 1/0 for each document
compute dot product

Question 7

Problem of VSM

Answer 7

• should terms have same weight
• should some terms be more important
• should term frequency be considered

Question 8

Improved VSM instantiation

Answer 8

• make specific term more important

- add in term frequency vector weighting

Question 9

Improved VSM example

Answer 9

Rank with TF weighting

Get query in vector term frequency and each document term frequency then compute dot product

Question 10

How to penalize popular terms in VSM

Answer 10

IDF weighting
function is smoothed by log IDF = log(M+1/k)
M = docs in collection
K = docs containing word

• now compare dot product my multiplying word vector value or count with IDF * count

Question 11

VSM with BM25

Answer 11

multiply vector by TF transformation of (K+1) c(w,d) / c(w,d) + k X IDF
ranking function = C(w,q) * Above

Question 12

What is the impact of document length to TR

Answer 12

• Penalize long document with long document normalizer

Question 13

Why is a document longer

Answer 13

document has more words -> meaningless, need more penalty

document has more contents -> meaningful, need less penalty

Question 14

How to normalize/penalize doc length

Answer 14

Use pivot length normalizer - Normalizer = 1 if document length is average
Otherwise Normalizer = 1 - b + b (document length/ average document length)