Introduction to IR

Question 1

What is data mining?

Answer 1

Extracting knowledge from large amounts of data

Question 2

What are the 4 main parts of information retrieval?

Answer 2

1. The corpus
2. An information need
3. A metric of relevance
4. A query

Question 3

What is a corpus?

Answer 3

A large repository of documents

Question 4

What is an information need?

Answer 4

The topic about which you desire to get information

Question 5

What is relevance?

Answer 5

Measures if a document contains information satisfying the information need

Question 6

What is a query?

Answer 6

How the information need is expressed to the computer

Question 7

What is structured data?

Answer 7

Data that conforms to a predefined schema. Tends to refer to information in tables with clear structure

Question 8

What is unstructured data?

Answer 8

Any data without a clear structure

Question 9

What type of systems do each type of data require?

Answer 9

Structured: database systems
Unstructured: Information retrieval systems

Question 10

What is semi-structured data?

Answer 10

Data that has some sort of structure but not a strict one. Almost no data is truly unstructured
Ex: A document has a title, subtitle, references, etc

Question 11

What is information retrieval?

Answer 11

Finding material of an unstructured nature that satisfies an information need from within large collections

Question 12

What is the goal of information retrieval?

Answer 12

To retrieve documents with information relevant to the user’s information need and helps the user complete a task

Question 13

What are 2 metrics to measure the relevance of retrieved documents?

Answer 13

Precision and recall

Question 14

What is precision?

Answer 14

The fraction of retrieved docs relevant to the user’s information need
TP/(TP + FP)
Number of good ones out of all ones retrieved

Question 15

What is recall?

Answer 15

Fraction of relevant docs in the collection that are retrieved
TP/(TP + FN)
Number of good ones retrieved out of all good ones

Question 16

Why is a linear scan of documents to perform a term search not a good solution?

Answer 16

1. It has to be repeated for every query
2. Only works for a small corpus, way too slow otherwise
3. Operations like proximity searching are not possible
4. Doesn’t allow for ranked retrieval

Question 17

What is a term-document incidence matrix?

Answer 17

A way to speed up query processing by preprocessing the corpus in advance.
It creates a matrix of terms and documents. When a term appears in a document, the corresponding matrix entry is set to 1

Question 18

How is a compound boolean query performed on a term-document incidence matrix?

Answer 18

Perform bitwise operations on term bit strings and combine results

Question 19

What is the issue with a term-document incidence matrix?

Answer 19

For very large corpuses with many terms, storing the matrix takes up a lot of memory. Most of the matrix is sparse

Question 20

How do we solve the issue with a term-document incidence matrix?

Answer 20

Only record the places in the matrix where 1’s would appear. Use an inverted index to store the doc IDs where each term appears in a dictionary structure.

Question 21

What are the methods for storing a postings list in an inverted index?

Answer 21

Linked lists and variable length arrays

Question 22

How do you perform a boolean retrieval query on an inverted index?

Answer 22

Retrieve postings lists for each term and merge them using set operations