Foundations of Text Classification

Question 1

What is classification?

Answer 1

Classification is assigning tags / labeling some input. Example is classifying if email is a spam or not spam

Question 2

What are steps in supervised ML (algos such as classification)?

Answer 2

1. Training: Taking a large amount of labeled data (manually labeled usually) and giving it to the ML model (classification model).
2. Prediction: Taking an un-labeled text and feeding it to the model and getting its prediction.

Question 3

Describe Naive Bayes approach.

Answer 3

In reality, when trying to get probability of some outcome given multiple evidences, the math gets complicated. To reduce the complexity, we can assume that every evidence is independent of any other possible set of evidences. In reality, this is not the case since since they are dependent.

Look at slides 22 and 23 in Lecture 2 for examples. VERY IMPORTANT FOR THE EXAM

Question 4

What is sequence labeling?

Answer 4

• Each token in a sequence is assigned a label. They are dependent on the labels of other neighbour-tokens.
• We use this because the standard classification assumes indepdendence which is usually not the case

There are many ways:
- Classifying a token based on the n-1 and n+1 tokens, using sliding window
- Classifying a token based on the n-1 (or more) output tag and n+1 token
- Since in some cases we need to know future tags, we can use n+1 (or more) future tags and classify backwards
-

Question 5

What is HMM?

Answer 5

HMM (Hidden Markov Model) is a statistical model of hidden, stochastic state transitions with observable, stochastic output.

It has:
- Fixed set of states (at each time, model is in exactly one of these states: happy, neutral, angry Dart Vader)
- State transition probabilities (probabilities for each transition between all states, starting state can be either fixed or probabilistic)
- A fixed set of possible outputs (destroys a planet or not)
- For each state, a distribution of probabilities for every possible ourput (emission probabilities).

See slide 79 in Lecture 2 for an example

Question 6

In HMM, how to determine most likely state sequence given an output sequence?

Answer 6

• Brute force: not good
• Viterbi algorithm by saving the progress which means we don’t have to calculate everything all the time (dynamic programming)