CHAP 7 : Logistic Regression

Question 1

What is logistic regression?

Answer 1

It is a supervised learning algorithm. It is a classfication algorithm that assigns data to a discrete set of classes

Question 2

Give example(s) of classification problems.

Answer 2

1. Email classification : spam or not spam
2. Financial data analysis : fraud / not fraud
3. Credit analysis : approve or deny credit
4. Marketing : will buy or wont buy

• basically a binary classificaion (only 2 classes)

Question 3

What is the logistic function for logisitic regression? (analogous to best fit line eqn of linear regression)?

Answer 3

y hat = g(W.X^T),
g(X) = 1/(1+e^-z),

thus y hat = 1/(1+e^-(W.X^T))

Question 4

What is the name for the logistic function?

Answer 4

Sigmoid function

Question 5

From the values generated by the sigmoid function, how do the values get classified into class 0 or 1 by the classifier?

Answer 5

if the value < 0.5, the class value = 0. If the value >= 0.5, class value = 1.

Question 6

What is the error function given by in logistic regression?

Answer 6

E(W) = 1/2N (summation (y(i) - yhat(i)^2) – refer to notes

Question 7

Why cant we use the same error function (average MSE) as linear regression for logisitic regression?

Answer 7

There will be many local minima and the algorithm may be stuck in a local minima.

Question 8

What is the cost function for logistic regression?

Answer 8

cost (yhat(x), y) =
-log(yhat(x)) if y = 1;
-log(1-yhat(x)) if y = 0.

See notes [we can rerite error function using the cost function]

Question 9

How does the gradient descent algorithm work for logistic regression?

Answer 9

1. initialise W with random values or zeros
2. Loop till convergence
   for each W(j) in W do :
   w(j) = w(j) + L . 1/N summation (y(i) - y hat (xi))x(j)(i)) , where j –> jth col, ith col

see notes for equation.

Question 10

What is a confusion matrix?

Answer 10

A confusion matrix is a performance measurement for machine learning classification

It presents a table layout of the different outcomes of the prediction and results of a classification problem and helps visualize its outcomes.

Values : True positive, true negative, false positive, false negative.

Question 11

What is the difference between the training dataset and the validation dataset?

Answer 11

Training dataset is a set of examples used for learning, that is to fit the parameters of the classifier. A validation dataset contains different samples to evaluate trained ML models.

[The validation dataset is useful when it comes to hyper-parameter tuning and model selection. The validation examples included in this set will be used to find the optimal values for the hyper-parameters of the model under consideration.]

Question 12

From the confusion matrix, there are 4 other metrics to evaluate classification output. What are they?

Answer 12

1. Precision
2. Recall (sensitivity)
3. F1 score
4. Support

Question 13

What is precision, how is it calculated?

Answer 13

Precision is the ratio of correctly predicted positive observations to the total predicted positive observations. (High precision relates to the low false positive rate. )

Precision = TP/(TP+FP)

TP: true positive ; FP : False positive

Question 14

What is recall and how is it calculated?

Answer 14

It is the ratio of correctly predicted positive observations to the all observations in actual class .

Recall = TP / TP+FN

FN: False negatives

Question 15

What is F1 score and how is it calculated?

Answer 15

F1 Score is the weighted average of Precision and Recall

• F1 Score = 2*(Recall * Precision) / (Recall + Precision)

Question 16

What is support?

Answer 16

Support is the number of actual occurrences of the class in the specified dataset, i.e., the support is the number of occurrences of each class in original y value from the dataset.

Question 17

You are training a logistic regression model and you find that your training error is close to 0, but the testing error is very high. What can be done to improve this situation? Note: This situation is applicable to all the machine learning problems and not specific to logistic regression.

Answer 17

1. Increase the training data size
2. Train on a combination of your training data and your test data but test only on your test data.

(: Since we are facing overfitting we can increase the training data size to combat this. Also, if we train on our test data our test loss will definitely improve dramatically, but you should never do this in practice because it defeats the purpose of testing, and will make performance worse when the model is deployed and used on new data.)

Question 18

What are 2 kinds of validation methods used?

Answer 18

1. K-fold cross validation
2. leave-one-out cross validation

Question 19

How does k fold cross validation work?

Answer 19

It splits the data into k folds, then trains the data on k-1 folds and test on the one fold that was left out. It does this for all combinations and averages the result on each instance.

Question 20

What is the advantage of k fold cross validation?

Answer 20

The advantage is that all observations are used for both training and validation, and each observation is used once for validation.

Question 21

What is leave-one-out cross validaton?

Answer 21

A variant of k-Fold CV is Leave-one-out Cross-Validation (LOOCV). LOOCV uses each sample in the data as a separate test set while all remaining samples form the training set. This variant is identical to k-fold CV when k = n (number of observations).