Text Mining

Question 1

What methods do Text Mining use?

Answer 1

Information Retrieval. Pre-processing of text documnets

Question 2

What tasks do text mining do?

Answer 2

Text Classification, Text Clustering or Text Summarization

Question 3

What is an issue with text mining vs traditional data mining?

Answer 3

Traditional data mining is structured. Text often has no real structure.

Question 4

What is a Vector Space Model?

Answer 4

A document is represented as a “bag” of words.

Question 5

What is a problem with Vector Space Model?

Answer 5

There are many words in the English language.

Question 6

How do you fix the limitations of the Vector Space Model?

Answer 6

Removing the stop words (“A, the, this, that …”)

Stemming (e.g combine the similar verbs (past/present tense)

Question 7

How do you assign the weight (importance) of a term in text-mining?

Answer 7

Use TF-IDF

Weight = TF * IDF

TF = Term Frequency (how many times)

IDF = Inverse Document Frequency = log (total documents / document frequency)

Question 8

What are the steps involved in text mining?

Answer 8

1. Get the text
2. Remove the stop words
3. Convert all the words to lowercase (optional step)
4. Stem the commonly associated word (interesting-> interested)
5. Count the term frequency
6. Create an index file, which has all the terms and all their frequency. Sort it alphabetically.
7. Create Vector Space Model: For each occurence, put a 1 in its vector space, occurs 3, put 3).
8. Compute the IDF. How many documents did this word appear in? / How many documents there are.
9. Compute the weight (tf * idf)
10. Normalize to less than 1. For each term, the weight is divided by the square root of the sum of all the weights squared

Question 9

How to measure the similarity between two documents?

Answer 9

Use cosine distance.