Information Retrieval

Question 1

What are the components of information retrieval?

Answer 1

Documents
Index
Query
Matching

Question 2

What is the formula for Zipf’s law?

Answer 2

F(r)=C/r^α

log(F(r)) = log(C) - αlog(r)

Question 3

What are the two parts of text pre-processing?

Answer 3

Stop word removal

Stemming

Question 4

What is stop word removal?

Answer 4

The removal of common ‘noise words’ from text (e.g. ‘the’, ‘and’)

Question 5

What is stemming?

Answer 5

Removing irrelevant differences from different ‘versions’ of the same word
This reduces the number of unique words in a corpus but increases the number of instances of each word

Question 6

What is the formula for the inverse document frequency?

Answer 6

IDF(t)=log(ND/ND_t )

Question 7

What is the formula for the term frequency - inverse document frequency weight?

Answer 7

w_td=f_td.IDF(t)

Question 8

What is the formula for the similarity between a document and a query?

Answer 8

sim(q,d)=[sum of all terms in q and d(w_td.w_tq)]/(||q||.||d||)

Question 9

What’s the formula for document length?

Answer 9

||d||= √(∑w_td^2 )

Question 10

What is the formula for recall?

Answer 10

recall=|retrieved ∩relevant|/|relevant|

Question 11

What is the formula for precision?

Answer 11

precision=|retrieved ∩relevant|/|retrieved|

Question 12

What is query expansion?

Answer 12

Adding terms to a query in order to increase the overlap between the query and relevant documents

Question 13

What is term reweighting?

Answer 13

Increasing the weight of query terms that appear in relevant documents and decreasing the weights of terms that don’t appear in relevant documents

Question 14

What is a hyponym?

Answer 14

Subset of a word

Question 15

What is a hypernym?

Answer 15

Superset of a word

Question 16

What does the vector representation of a document contain?

Answer 16

The TF-IDF weight for each term in the corpus

Question 17

What does latent semantic analysis do?

Answer 17

Discovers relationships between words automatically from the data

Question 18

What is the formula for the word-document matrix?

Answer 18

A=USV^T
U and V are orthogonal
S is a diagonal matrix

S is the strength of the most significant correlation
V is the direction of the most significant correlation

Question 19

What is the formula for usefulness?

Answer 19

U(t) = P(t|T)log(P(t|T)/P(t))

Question 20

What is the formula for salience?

Answer 20

S(t) = P(T|t)log(P(T|t)/P(T))

Or

S(t) = P(T)*U(t)/P(t)

Question 21

What are the steps of Latent Dirichlet Allocation?

Answer 21

1. Make an initial estimate of N topics
2. Decompose each document into its component topics
3. Use the decomposition to re-estimate the topic word probabilities

Question 22

What is the recursive page rank formula?

Answer 22

pr_(n+1) (d)= ∑pr_n (e).w_ed

Question 23

What is the Markov chain formula for page rank?

Answer 23

pr_(n+1) = W^T.pr_n

Question 24

What is the damping factor?

Answer 24

The probability that a user with exit at any given page, denoted by delta

Question 25

What is the formula for page rank including damping?

Answer 25

pr(d)=(1-δ)/N + δ∑pr_n (e).w_ed

N = number of documents