Week 3 - Information Retrieval

Question 1

What is information retrieval

Answer 1

It is finding material (usually documents) of an unstructured nature that satisfied an information need from within large collections

Question 2

What is the definition of a search

Answer 2

It is a conversation between user and search engine

Question 3

When your search result is not good, what does this tell you?

Answer 3

You have entered an ambiguous query or the query is irrelevant

Question 4

Why is user intent important

Answer 4

Billions of users, everyone has a different idea of what they are looking for based on the same search term. Search engines conduct localisation to get better at INTENT

Question 5

How can we abstract the IR of a search problem?

Answer 5

Input —> Process —> Output
Input is a query in a string of characters
output is a list of characters (Article name, link, article id)

Question 6

Why is a search problem considered ad hoc retrieval

Answer 6

Because we cannot anticipate all the queries upfront

Question 7

Why is relevance also ad hoc?

Answer 7

Because it is tied to the information need and the need is very specific

Question 8

Why is it called a TERM-DOCUMENT matrix

Answer 8

Terms do not always correspond with words. It can be numbers, symbols, abbreviations etc.

Question 9

A passage from Hamlet has been converted into 1s and 0s, what sort of IR can you carry out

Answer 9

Simple queries, like searching for the presence of a word

Question 10

Boolean retrieval

Answer 10

Query is posed in the form of a Boolean expression of terms

Question 11

What is tokenization

Answer 11

Tokenization is the process of splitting the sentence into words

Question 12

What is normalisation

Answer 12

So that people can search for plural,singular versions of the word, ways of spelling (US/UK), sentence capitalisation etc.

Question 13

What is stemming

Answer 13

Chopping off the word: “jumping” —> “jump”

Question 14

What is lemmatisation?

Answer 14

You maintain a dictionary. Dictionary matching: “run” from the word “ran”

Question 15

What is the purpose behind pre-processing?

Answer 15

To make the query easier for the user. If not the user must search for the EXACT word form

Question 16

When does “double quotes” feature?

Answer 16

When you are looking for a specific variance. For example, “RUN”

Question 17

Which is better? Stemming or lemmatisation?

Answer 17

Lemmatisation is often slower than stemming. Because you minded to maintain a dictionary. And if new words come up, you need to update your lemmatisation dictionary

Question 18

What are stop word?

Answer 18

Filters out the commonly used words (a, an, by, or). They go beyond conjunctions and filter out words that have no real meaning. They are being removed from the query first.

Question 19

What does TFIDF mean

Answer 19

Term frequency - inverse document frequency

Question 20

How does TFIDF work?

Answer 20

It prioritises rare terms in the collection, across documents. If your document has a high frequency, it has high TF-IDF

Question 21

Vector space model, how is it used in IR?

Answer 21

Each vector represents the document. Given that, which vectors are the most similar. Calculate the vectors similarity. the query will be represented as a vector as well.

Question 22

Precision definition

Answer 22

What fraction of the returned results are relevant to the information needed

Question 23

Recall definition

Answer 23

What fraction of the relevant documents in the collection were returned by the system

Question 24

When should you focus on precision or recall?

Answer 24

Recall: if you want to do an exhaustive search, covering all relevant ground.
Precision: when you are more focused on results

Question 25

There is always a trade off between precision and recall. What is the trade off?

Answer 25

The trade off involves threshold and cut off in scoring. This might cause some terms to slip through

Question 26

When do you have perfect precision and recall

Answer 26

If you perfectly capture unwed intent

Question 27

Search engine data structures. What is an index?

Answer 27

An index has all your key terms and pages in which the terms occur

Question 28

what is an inverted index?

Answer 28

For each term t, store list of docIDs (unique) that contain t

Question 29

What is a term dictionary

Answer 29

Set of terms in the dictionary forms the vocabulary

Question 30

How do you have an inverted index?

Answer 30

Dictionary + postings = inverted index