C9: neural IR

Question 1

why would we want dense retrieval?

Answer 1

sometimes we need exact matching, but often we also want in-exact matching of documents to queries: if we use exact matching in 1st stage we might miss relevant documents

Question 2

what is dense retrieval?

Answer 2

neural first-stage retrieval, so using embeddings
- bi-encoder architecture: encoding the query and document independently, then computing the relevance

Question 3

bi-encoder architecture: 3 steps

Answer 3

1. generate a representation of the query that captures the information need
2. generate a representation of the document that captures the information contained
3. match the query and the document representations to estimate their mutual relevance

Question 4

how do we measure the relevance between query and document?

Answer 4

use a function to compute the similarity between the query and the document representation vectors

Question 5

what are the 4 differences between cross-encoders and bi-encoders?

Answer 5

1. cross: one encoder for q and d
   bi: separate encoders for q and d
2. cross: full interaction between words in q and d
   bi: no interaction between words in q and d
3. cross: higher quality ranker than bi
4. cross: only possible in re-ranking
   bi: highly efficient (also in 1st stage)

Question 6

Sentence-BERT

Answer 6

commonly used bi-encoder, originally designed for sentence similarity but can be used for q,d pairs

it is a pointwise model, because we only take one d into account per learning item. At inference we measure the similarity between q and each d and then sort the docs by this similarity

Question 7

what is the goal of training bi-encoders?

Answer 7

the similarity between the 2 vectors is maximized for docs relevant to q and minimized for non-relevant docs to q, given the similarity function

Question 8

why are bi-encoders less effective than cross-encoders?

Answer 8

cross-encoders can learn relevance signals from attention between the query and candidate texts at each transformer encoder layer

Question 9

ColBERT

Answer 9

proposed as a model that has the effectiveness of cross-encoders and the efficiency of bi-encoders
- compatible with nearest neighbour search techniques

Question 10

what is nearest-neighbour search?

Answer 10

finding which document embeddings vector are most similary to the query embeddings vector

computing the similarity for each d,q pair is not scalable => approximate nearest-neighbour (ANN) search

Question 11

similarity in ColBERT

Answer 11

similarity between d and q is the sum of maximum cosine similarities between each query term and the best matching term in d: compute the similarity of each query term to every document term and sum the maximum similarities

Question 12

ColBERT: query time

Answer 12

1. query term embedding
2. 1st stage: the top-k texts from the corpus are retrieved for each query embedding
3. 2nd stage: k candidate texts are scored using all query token representations according to the MaxSim operator and then ranked

Question 13

ColBERT loss function

Answer 13

L(q,d+, d-) = - log exp(s_q,d+) / ( exp(s_q,d+) + exp(s_q,d-))

Question 14

challenges of long documents

Answer 14

• memory burden of reading the whole document in the encoder
• mixture of many topics, query matches may be spread
• neural model must aggregate the relevant matches from different parts

Question 15

challenges of short documents

Answer 15

• fewer query matches
• but neural model is more robust towards the vocabulary mismatch problem than term-based matching models

Question 16

what is the long-tail problem?

Answer 16

a good IR method must be able to retrieve infrequently searched-for documents and perform reasonably well on queries with rare terms