05 Term weighting

Question 1

what is zipf’s distribution

Answer 1

a few words occur frequently

medium words occur mediumly (most useful and descriptive)

many words occur only once

Question 2

what are the contrasting assumptions for Best match approach or a co-ordinated search

Answer 2

• term is present in document or not is binary
• it does not consider the degree of association between the term and the document
  “relevancy”

Question 3

what is the basis

Answer 3

determine word’s semantic utility based on its statistical properties

• count word frequency in document / all document in the collection

Question 4

what is zipf’s law

Answer 4

rank (R) of a word * frequency is approximately a constant (K)

rank (R) * probability of the word occurrence is approximately empirical constant (A)

quite accurate except for very high rank or very low rank

Question 5

what are the consequences of zipf’s law

Answer 5

some frequent words are not good discriminators called stop words
remove them to reduce inverted index storage cost

Question 6

what is heaps law

Answer 6

V = size of vocabulary(number of unique words)
n = number of words

v = K * n**beta

typically
k 10 to 100
beta 0.5

Question 7

what is resolving power

Answer 7

measure of relevancy
2 critical factors
- word frequency within a document
- word frequency in collection

Question 8

what is term weighting

Answer 8

effective approach to score documents
- how many query terms it contains
- how discriminative those terms are

not all terms are useful so we give them a weight

output W = weight of kth word in document
input
f = number of occurrence of kth word in document (term frequency)
N = number of documents in collection
D = number of documents containing the kth word

inverse document frequency idf =
log(N/D)

calculate TF-IDF