2 Linear Classifiers

Question 1

What is one-hot encoding?

Answer 1

In classification we use this instead of a salar y.

K-dim vector per disered putput
Vector of 0 with one 1 corresponding to the class the indx represents

Ex: three class problem
[1,0,0] [0,1,0][0,0,1]

Question 2

WHat is a perceptron?

Answer 2

We apply a non-linear activation of top of the linear transfrorm:
f(x) = g(w^Tx)

Question 3

What does the perceptron define?

Answer 3

A step function

g(a) = { 1 if a >= 0
{ -1 otherwise

Question 4

What does the linear least squares classifier do?

Answer 4

It seperates two or more classes by finding a hyperplane that maximizes the margin between the classes.

Simple case: defines a line that seperates the two classes.

Question 5

What is the solution to LS classifier?

Answer 5

W = x^t y (t is pseudoinverse)

We “learn” the predictor by solveing for W, which are the weights

Differentiate Loss ( L ) with reference to w and solve loss by setting to zero

Question 6

How does W look in least squares classifier?

Answer 6

It becomes a matrix
A value for every possible outcome, for every time we do it.

Question 7

How do you calculate loss in least squares classifier?

Answer 7

We need to solve for a matrix of coefficients ( one model per column)

L(f(x),y) = 1/2 * sum(m,i=1) (y i - w^T xi)^2
= 1/2||y - X w||^2

Optimizing, we want to minimize loss

Question 8

How do you determine class density in least squares classifier?

Answer 8

each is approximated by its own regression model:

p(Ck|x) ~~ fk(x) = wk^T x

Question 9

What is the perceptron criterion?

Answer 9

The loss in perceptron:
L(f(x),y) = - sum(m,i=1) w^T xi yi

Question 10

How is learning done in perceptrons

Answer 10

By stockastic gradient descent

Question 11

What is stockastic

Answer 11

Select training examples one by one in random order

Question 12

What is gradient descent?

Answer 12

Use negatve of the gradient to update the weights
w <- w - DeltaL
w <- w + xi yi

Question 13

How do you calculate the gradiant of the loss

Answer 13

Delta L = -xi yi

Question 14

What is logistic regression?

Answer 14

Used for binary classification tasks.
Uses the sigmoid funtion to model the probability of an event occuring.

p(C1|x) = sigmoid(w^T x)
p(C2|x) = 1 - p(C1|x)

Question 15

What is the sigmoid funcion

Answer 15

Used for logistic regression

sigmoid (a) = 1/(1+e^-a)

Question 16

What is the output of logistic regression

Answer 16

Either 0 or 1, which defines a bernoulli trail

Question 17

How do you find w in logistic regression

Answer 17

Through maximum likelihood

Question 18

What is the log likelyhood for linear regression?

Most important

Answer 18

As usual in ML estimation, its easier to take the logarithm

log Fnzy L(w) = sum(m,i=1) [yi log sigmoid(ai) + (1-yi) log(1- sigma(ai))]

Question 19

How does the Bernoulli trail look?

Answer 19

p(C1 | x)^y (1-p(C1|x))^1-y

Question 20

What is the loss function for linear regression?
(MSE) (logistic regression)

Answer 20

L(f(x),y) = 1/2 sum(m,i=1) (yi - w^T xi)^2

Question 21

What is the equation for linear regression?

Answer 21

y = f(x) + epsilon = w^T x + epsilon
~N(0,sigmoid^2)

where
p(y|x) Ñ(f(x), sigmoid^2)

Question 22

What is the gradiant of the log likelihood? (Logistic regression)

Answer 22

Delta log fnzy L = sum(m,i=1)(yi - sigmoid(w^T xi)) xi

Question 23

K-NN vs LLS classifier

Answer 23

K-NN:
- Best when data is high dim
- More robust with small number of training examples
- Better choise when decision boundry is non-linear
- more computational expensive, since it needs comparinginput to each training example
- Generalize better with small dataset

LLS:
- best when data is low dim
- requires larger number of training examples
- Better choice when decision boundry is linear
- Closed form low comlpexity
- Generalize better with large dataset

Question 24

How do we do multiclass logistic regression?

Answer 24

• K classes instead of 2
• K weight vectors
• for each class we model the density by the soft max function:

p(C | x) = exp(ak)/(sum(k,i=1)exp(ai))
where
ak = wk^T x

Question 25

What is the log-likelihood for MLR

Answer 25

log fnzy L(w1, …, wk) = sum(m,i=1) sum(K,k=1) yi,k log yhati,k

We use one shot encoding
Our estimated output Yhat follows a discrete distribution

Question 26

What is the gradient with reference to w for MLR

Answer 26

Delta wk Fnzy L(w1, …, wk) = sum(m,i=1) (yi,k-yhati,k)xi

Question 27

What does linear least squares do?

Answer 27

Predict the scalar response y from an input vector x.
Fit a and b in y =ax+b, to make the line fit the points.

Question 28

What is classification?

Answer 28

From the input data, output the label/category which the input belongs to.

Question 29

What are the classification problems?

Answer 29

Predict K categopries (2 or more)

Question 30

what is regression?

Answer 30

Predict a continuous output from the input.

Question 31

What is least squares classifier?

Answer 31

• change y from a continous value to a discrete value
• instead of scalar y, we use one-hot encoding.

Question 32

Is KNN linear?

Answer 32

The overall strugture is not linear, but it has n
linear parts

Question 33

What are options for encoding?

Answer 33

Integer labels
One-hot encoding