Supervised Classification 7

Question 1

Give examples of Text Classification. Three or more.

Think about how you would classify a piece of text or a book

Answer 1

• Assigning subject categories, topics or genres
• Spam detection
• Authorship identification
• Age/gender identification
• Language identification
• Sentiment analysis

Question 2

What is Rule-Based Classification?

Answer 2

Rules based on combinations of words or other features
• Spam: whether mentions my name, whether mentions money, feature
phrases like ‘you are selected’, ‘this is not a spam’
• POS tagging: prefixes (inconvenient, irregular), suffixes (friendly, quickly),
upper case letters, 35-year
• Sentiment analysis: ‘rock’, ‘appreciation’, ‘marvel’, ‘masterpiece’
• Gender identification from names: number of vowels, the ending letter
• Accuracy can be high
• If the features are well designed and selected by experts
• However, building and maintaining these rules is expensive!

Question 3

What is Text Classification?

Answer 3

Input
• A document d
• A fixed set of classes C = {C1, C2, … , Cm}
• Output
• The predicted class c ∈ C for d

Question 4

What is Supervised Classification?

Answer 4

Input
• A document d
• A fixed set of classes C = {C1, C2, … , Cm}
• A training set of N hand-labeled documents {(d1 , c1), … , (dN , cN)}
Output
• A classifier such that for any document, it predicts its class

Question 5

When is a classifier supervised?

Answer 5

• A classifier is called supervised if it is built based on training corpora containing the correct label for each input

Question 6

What is a dev-test set for?

Answer 6

analyze errors, select
features, optimize
hyper-parameters

Question 7

What is a test set for?

Answer 7

test on held-out data
(model should not be
optimized to this data!)

Question 8

What is the training model for?

Answer 8

Train the model

Question 9

What is a multinomial logistic regression?

Answer 9

It’s a classification algorithm

• Predicts the probability of the input falling into a category

Question 10

How do you calculate the similarity between the prediction at the truth labels in multinomial logistic regression?

Answer 10

Loss function
e.g. cross entropy loss
CE(P1,P2) = − ∑P1(xi) log P2(xi)
Cross entropy loss is not symmetric! 
CE(P1,P2) = -∑I P1(xi) log P2(xi)

See slide 7 pg. 31

Question 11

What does more features in ML algorithms do?

Answer 11

• More features usually allows for more flexibility, hence better performance at training time
• But using more features also usually increase the chance of overfitting

Question 12

What are the issues with vocabulary size in TF and TF-IDF?

Answer 12

Why it harms 
• Over-parameterization, overfitting 
• Increases both computational and
representational expenses
• Introduces many ‘noisy features’, which
may harm the performance (especially
when raw tf/idf values are used)

Question 13

What are some methods to reduce the vocabulary size in TF(-IDF)?

Answer 13

• Remove extremely common words, e.g. stop words and punctuations
• Remove extremely uncommon words, i.e. words that only appear in very few documents
Among the rest, you may:
• Select the top TF words, because they are more representative
• Select the top IDF words, because they are more informative
• Select the top TF-IDF words, to strike a balance

Question 14

Why do you decide your vocabulary at training time, and keep it fixed at test time!?

Answer 14

Because your model does not understand what each feature means; it relies on the position of each feature to learn the importance/weight of each feature

Question 15

How do you calculate accuracy in a model performance evaluation?

Answer 15

Accuracy
• Of all predictions, how many are correct
• Acc=(TP+TN)/(TP+FP+FN+TN)

Question 16

When the label distribution is highly unbalanced, you can be easily fooled by the accuracy. How do you avoid being fooled?

Answer 16

• Report the accuracy of the ‘simple
majority baseline
• Check the label distribution of the
training data

Question 17

How do you calculate precision in a model performance evaluation?

Answer 17

Precision:
• % of labelled items that are correct
• TP/(TP+FP)
If precision is 0 it is N/A not 0

Question 18

How do you calculate recall in a model performance evaluation?

Answer 18

Recall:
• % of correct items that have
been labelled
• TP/(TP+FN)

Question 19

What are the qualities of an aggressive classifier?

Answer 19

• Tend to label more items
• High-recall low-precision
• When you don’t want to miss
any spam; suitable for first
round filtering (shortlist)

Question 20

What are the qualities of a conservative classifier?

Answer 20

• Tend to label fewer items; only
label the very certain ones
• High-precision low-recall
• When you don’t want any false
alarms; suitable for second
round selection

Question 21

What is F1 Score/F Measure and how is it calculated?

Answer 21

It is the combined weighted precision and weighted recall.

1/2 (equally weight P and R): F1=2PR/(P+R)

Question 22

What are ways to deal with imbalanced data?

Answer 22

• class weights: assign higher weights
to the minority class, i.e., use higher
loss function if a minority item is
misclassified as majority

Down-sampling: sample the majority
class to make their frequencies closer
to the rarest class. Use the sampled
subsets of data to train a model
• Pros: easy to implement; allows many
different sampling methods
• Cons: smaller training data size;
sometimes poor performance on real
data (with real class distributions)

• Up-sampling: the minority
class is resampled to increase
the corresponding frequencies
• In NLP, it means you need to
create some new text of the
minority class. This is also
known as data augmentation.

Question 23

To select the most likely class for a given document what do you do?

Answer 23

Given an input document, a classifier assigns a probability to each
class, P(c|d), and selects the most likely one: c = arg max(ci) P(d | ci)P(ci)

Question 24

Go over chapter 7 slide 19 to 24

Answer 24

Go over chapter 7 slide 19 to 24

Question 25

watch logistic regression video chapter 4 vid 3

Answer 25

watch logistic regression video chapter 4 vid 3

Question 26

What is F1 score?

Answer 26

The F-score, also called the F1-score, is a measure of a model’s accuracy on a dataset.

The F-score is a way of combining the precision and recall of the model, and it is defined as the harmonic mean of the model’s precision and recall.

Question 27

What tool in python is used for f1, accuracy, precision and recall?

Answer 27

sklearn