Logistic Regression

Question 1

What’s the equation for mean squared error? (multiple dimensions)

Answer 1

Question 2

What’s the equation for the prediction of logistic regression?

Answer 2

sigmoid(theta^T*x) (I think)

Question 3

What does logistic regression output?

Answer 3

Calculates the probability of each class, and takes the class with the highest probability. The prediction is based on the values of a set of independent variables.

Question 4

What is this?

Answer 4

The output of logistic regression

Question 5

What are some important characteristics to remember about logistic regression? (2)

Answer 5

• easily interpretable
• gives the probability of an event occurring, not just the predicted classification.

Question 6

Can you apply linear regression to a classification problem?

Answer 6

Usually it’s a bad idea

Question 7

What is the output of logistic regression?

Answer 7

The argmax of probabilities (between 0 and 1)

Question 8

What is this?

Answer 8

The hypothesis of linear regression

Question 9

What is the hypothesis of logistic regression in:

• words
• equation form

Answer 9

The hypothesis of linear regression fed into the sigmoid function

Question 10

What does the graph of logistic regression look like?

Answer 10

Sigmoid function

Question 11

• What is this?
• How do you interpret it?

Answer 11

• The probability expression of logistic regression’s output (before the argmax)
• Probability that y=1, given x, parametrized by theta

Question 12

Do the outputs of logistic regression add up to exactly 1?

Question 13

How should you think of the prediction of binary logistic regression?

Answer 13

Predict 1 when θ^Tx >= 0.5. Otherwise, 0

Question 14

How can you solve for the line of the decision boundary for binary logistic regression?

Answer 14

Essentially, setting theta transpose x (the hypothesis of linear regression) equal to 0 is the equation for the decision boundary.

Steps:

1. Try to get theta transpose * x.
2. Plug intercept value into theta transpose * x
3. Set that equal to 0
4. treat x2 as y and x1 as x and solve for the equation of the line
5. If the line is over the origin, then the half-space that doesn’t contain the origin predicts 1. If it’s under the origin, the half-space with the origin predicts 1

Question 15

What is important to remember about the decision boundary of binary logistic regression?

Answer 15

h(x) = 0.5

Question 16

Can logistic regression take on a nonlinear decision boundary? If so, how?

Answer 16

Yes, by adding higher-order polynomial term features

Question 17

Can binary logistic regression have a decision boundary that is a circle?

Answer 17

• Yes, if you use higher order polynomial features

Question 18

For logistic regression, can we use the same cost function that linear regression uses?

Answer 18

No. Because plugging the sigmoid (which is a nonlinear function) into the MSE equation makes for a nonconvex function

Question 19

• What’s the cost function for logistic regression?
• What does the graph look like?
• What’s the intuition?

Answer 19

Intuition:

• For y=1, as the h approaches 0, the penalty goes to infinity. Same idea for 0, except the graph of the cost function is flipped horizontally

Question 21

What do we know about the cost function for logistic regression? (3)

Answer 21

• It’s derived from the principle of MLE
• It’s convex
• No closed-form solution for logistic regression because of the nonlinearity of the sigmoid

Question 22

What does learning in logistic regression do? Why?

Answer 22

• We minimize the negative log conditional likelihood.
• We can’t maximize likelihood (as in Naïve Bayes) because we don’t have a joint model p(x,y)

Question 23

What’s the cost function for logistic regression in compact form?

Answer 23

Question 24

What do we know about the negative average conditional log likelihood for logistic regression?

Answer 24

It’s convex

Question 26

What’s the softmax function?

Question 27

What’s the relation between the softmax function and the sigmoid function?

Answer 27

• The sigmoid function is used for the two-class logistic regression, whereas the softmax function is used for the multiclass logistic regression
• When num classes = 2, the softmax function reduces to the sigmoid function used for binary logistic regression. So in some sense, they’re the same

Question 28

What happens if theta^T * x = 0 during training for binary logistic regression?

Answer 28

Assuming no bias term, No change because the cost function penalizes based on the predicted probability and if that’s 0, the probability = 0.5.