lecture 4 - classification

Question 1

How do linear models handle classification tasks?

Answer 1

Linear models for classification take an input vector x and map it onto one of K discrete classes by using a separable hyperplane in the input space.

Question 2

How are linear models separated in a D-dimensional input space?

Answer 2

They are separated by (D−1)-dimensional hyperplanes.

Question 3

How are linear models represented in regression?

Answer 3

y(x) = w^T x + w_0

Question 4

What is the role of the activation function in classification with linear models?

Answer 4

• The activation function f(⋅) maps the output of the linear model to discrete classes, converting the continuous output into a class label.
• This makes the model nonlinear in its outputs while the underlying equation remains linear.

Question 5

How can a step function be used for classification?

Answer 5

A step function can assign

1. y(x)>0 to “class 1”
2. y(x)≤0 to “class 2”

Question 6

What is a discriminant function in classification tasks?

Answer 6

A discriminant function is a mathematical function used to separate data points into distinct classes by mapping input features to a decision boundary.

Question 7

What is the simplest form of a discriminant function for a 2-class classification problem?

Answer 7

y(x) = w^T x + w_0

Question 8

How is the decision boundary defined in classification?

Answer 8

The decision boundary is the set of all points x that satisfy:

• y(x) = w^T x + w_0 = 0

Question 9

How are classes assigned based on the discriminant function?

Answer 9

1. y(x)>0 to “class 1”
2. y(x)<0 to “class 2”
   - y(x)=0 is the decision boundary

Question 10

If points x_a and x_b lie on the decision surface, then:

Answer 10

1. y(x_a) = y(x_b) = 0

therefore

2. w^T (x_a - x_b) = 0 (dot product of the two vectors is zero)

this indicates that w is orthogonal to every vector lying within the decision surface

Question 11

How is the decision boundary equation interpreted in terms of projection and bias?

Answer 11

1. The left side represents the projection of x onto w and determines how far x is from the boundary
2. The right side represents the displacement or location of the decision boundary relative to the origin (w_0)

Question 12

Why is using multiple 2-class classifiers for K-class classification not ideal?

Answer 12

It can lead to ambiguous regions where boundaries overlap, making it unclear which class a point belongs to.

Question 13

What is the solution for well-defined decision boundaries in K-class classification?

Answer 13

Use a unified K-class classifier where each class C_k has its own discriminant linear function of the form y_k(x) = w_k^T x + w_k0

Question 14

What is the decision rule for assigning a point to a class in K-class classification?

Answer 14

A point x belongs to class C_k if y_k(x) > y_j(x) for all j =/= k

Question 15

How is the decision boundary between two classes defined in K-class classification?

Answer 15

• The boundary between classes C_k and C_j occurs when their scores are equal
• y_k(x) = y_j(x)
• this results in a (D-1) dimensional hyperplane

Question 16

What is the general form of the hyperplane between two classes C_k and C_j?

Answer 16

(w_k - w_j)^T x + (w_k0 - w_j0) = 0

Question 17

What is a property of the decision boundary in K-class classification?

Answer 17

Linearity of the discriminant functions makes the decision boundary in a K-class classifier singly connected and convex

Question 18

What are the steps for assigning a class using a K-class classifier?

Answer 18

1. Define the linear discriminants for each class.
2. Assign weights and biases for each class.
3. Calculate the discriminant scores for a given point.
4. Assign the point to the class with the highest score.

Question 19

How are weights and biases assigned for each class in K-class classification?

Answer 19

Each class C_k is assigned a weight vector w_k and a bias w_k0 which define its discriminant function

Question 20

What does the discriminant score represent in a K-class classifier?

Answer 20

The discriminant score represents how strongly a data point is associated with a specific class.

Question 21

What is a perceptron

Answer 21

• The perceptron is the first model that could learn.
• It is a linear model with a step activation function, classifying inputs into two distinct categories.

Question 22

How does the step activation function in a perceptron work?

Answer 22

• y(x) = f(w^t ϕ(x))
• f(a)
• if a is positive or zero, it outputs +1, indicating one class.
• if a is negative, it outputs -1, indicating another class.

Question 23

What criterion is used for training a perceptron?

Answer 23

Training is done using the perceptron criterion, which focuses on minimizing the total error function E_p

Question 24

What does the total error function E_p represent in perceptron training?

Answer 24

• E_p is a score that tells how “wrong” the perceptron is on the points it misclassifies, based on the sum of predicted output multiplied by target output for the misclassified points.
• focuses only on the n misclassified points in M
• computes the sum of the terms (w^⊤ϕ_n​ t_n​), which is the weight vectr * predicted output * target output

Question 25

Why is direct misclassification using the total number of misclassified patterns not effective in perceptron training?

Answer 25

• The step activation function f(a) is non-linear, making the number of misclassified points not differentiable.
• gradient-based methods like the perceptron learning rule require a differentiable error function
• E_p indirectly aproximates a differentiable error function

Question 26

Why is the error function E_p negative for misclassified points?

Answer 26

The negative sign ensures that E_p increases when the perceptron misclassifies points, guiding the algorithm to reduce misclassification by updating the weights.

Question 27

How does the perceptron update weights to reduce misclassification?

Answer 27

The perceptron updates its weights to minimize E_p, moving closer to correctly classifying the misclassified points.

Question 28

How is the perceptron criterion applied step by step?

Answer 28

1. The perceptron checks the predicted score before applying the step function.
2. For misclassified points, it multiplies the predicted score by the true label.
   ​
3. E_p accumulates the negative product for all misclassified points.

Question 29

Why is it difficult to calculate the gradient over the total error function E_p directly?

Answer 29

The total error function E_p is piecewise linear and not smooth because it sums over all misclassified points, causing abrupt changes when the set of misclassified points changes.

Question 30

How does stochastic gradient descent (SGD) handle the error function?

Answer 30

SGD takes just one misclassified point at a time, making the perceptron criterion for a single point linear and smooth, allowing direct gradient computation.

Question 31

What is the update rule for stochastic gradient descent in perceptron training?

Answer 31

• w^{t+1} = w^{t} + η ϕ_n t_n
• η is the learning rate.
• ϕ_n is the feature vector of the misclassified point.
• t_n is the target label of the misclassified point (+1 or -1)

Question 32

Why does the update rule not directly depend on the full weight vector w?

Answer 32

• The update is not a function of w
• The update adjusts weights only based on the misclassified point being considered, allowing the learning rate η to be set to 1 without affecting performance.

Question 33

How does SGD nudge the weights for each misclassified point?

Answer 33

• If a point is positive but classified as negative, the algorithm increases the weights to better capture the positive point.
• The weights are updated in the correct direction to improve classification.

Question 34

What does the perceptron convergence theorem state?

Answer 34

If there exists an exact solution (i.e., the problem is linearly separable), perceptron learning will find a solution in a finite number of steps.

Question 35

How did Minsky and Papert’s criticism of the perceptron convergence theorem impact perceptron research?

Answer 35

They pointed out that the perceptron cannot solve non-linearly separable problems, such as the XOR problem, causing a significant slowdown in neural computation research for nearly a decade.

Question 36

For which type of perceptrons was Minsky and Papert’s criticism valid?

Answer 36

Their criticism was only valid for single-layer perceptrons, as they cannot model non-linear decision boundaries.

Question 37

What is the first step in the perceptron algorithm visualization?

Answer 37

Select one of the misclassified points.

Question 38

How is the weight vector updated in the perceptron algorithm visualization?

Answer 38

The arrow from the boundary to the misclassified point is added to the current weight vector, forming a new weight vector.

Question 39

How does a change in the weight vector affect the decision boundary?

Answer 39

Since the decision boundary is perpendicular to the weight vector, changing the weight vector shifts the decision boundary.

Question 40

When does the perceptron algorithm stop updating the decision boundary?

Answer 40

1. When all points are correctly classified.
2. After a maximum number of iterations, if the data is not linearly separable.

Question 41

What are the key takeaways from the perceptron algorithm visualization?

Answer 41

1. The algorithm updates the decision boundary iteratively by shifting the weight vector based on misclassified points.
2. Learning stops when all points are correctly classified or after a set number of iterations.
3. The weight vector w directly influences the orientation and position of the decision boundary.

Question 42

What is the objective of probabilistic generative models for classification?

Answer 42

To classify an input x by calculating the probability that it belongs to a certain class C_k, denoted as P(C_k|x)

Question 43

How do probabilistic generative models differ from models that directly draw decision boundaries?

Answer 43

Instead of directly drawing decision boundaries, they model the probability distributions of data in each class and use Bayes’ Rule to make decisions.

Question 44

How is the posterior probability of class C_1 calculated?

Answer 44

• P(C_1 | x)

==

• (P(x|C_1)P(C_1)) / (P(x|C_1)P(C_1)) + (P(x|C_2)P(C_2))

==

• 1/(1+exp(-a))

==

• σ(a)
• where a is the log odds, and σ is the logistic sigmoid function

Question 45

What are the log odds

Answer 45

• for posterior probability P(C_1|x) = σ(a)
• a = (P(x|C_1)P(C_1)) / (P(x|C_2)P(C_2))
• so calculating p(C_1|x) requires the calculation of the sigmoid function of the log odds a

Question 46

What is the logit function, and how is it related to the sigmoid function?

Answer 46

The logit function is the inverse of the sigmoid function

• a = ln (σ/(1-σ))

Question 47

Why does calculating p(C_1∣x) require the sigmoid function?

Answer 47

Because the probability is expressed as the sigmoid of the log odds
𝑎.

Question 48

How is classification generalized to multiple classes?

Answer 48

• For multiple classes, the softmax function is used.
• It divides the exponentials of confidence scores by their total to produce proper probabilities.

Question 49

What is the formula for the softmax function?

Answer 49

• p(C_k∣x)= [exp(a_k)] / [∑_j exp(a_j)]
  ​

Where a_k = ln(p(x∣C_k)p(C_k))

Question 50

How do the sigmoid and softmax functions differ in their use?

Answer 50

1. The sigmoid function is used for binary classification.
2. The softmax function is used for K-class classification.

Question 51

How do probabilistic generative models extend to continuous inputs?

Answer 51

By assuming that

1. x follows a Gaussian (normal) distribution within each class C_k
2. that all classes share the same covariance matrix.

Question 52

What is the formula for the posterior probability p(C_1∣x) in continuous input models?

binary classification

Answer 52

• p(C_1∣x)=σ(w^⊤ x + w_0)
• where w = Σ^−1 (μ_1−μ_2)
• where w_0 = -(1/2) μ_1^⊤ Σ^−1 μ_1 + (1/2) μ_2^⊤ Σ^−1 μ_2 + ln[p(C_2)/p(C_1)]

Question 53

What parameters are used in modeling the Gaussian distribution for each class?

Answer 53

1. μ_k : Mean vector for class C_k
2. Σ: Covariance matrix (same for all classes).
3. D: Number of features (dimensionality of x)

Question 54

What are the key takeaways from modeling with continuous inputs?

Answer 54

1. Data is modeled using a normal distribution.
2. Posterior probability is calculated using Bayes’ Rule.
3. The decision boundary is linear due to the vanishing quadratic term from the normal distribution.
4. Parameters of the sigmoid function are determined by the parameters of the normal distribution.
5. the priors (p(C_k)) only enter via the bias parameters

Question 55

How is the model for continuous input generalized for multiple classes?

Answer 55

For K classes, the discriminant function is:

• p(C_1∣x) = [exp(a_k)] / [∑_J exp(a_j)]

= a_k(x) = (w_k^⊤ x + w_k0)

• where w = Σ^−1 (μ_k)
• where w_k0 = -(1/2) μ_k^⊤ Σ^−1 μ_k + ln[p(C_k)]

Question 56

What simplifications occur when covariances are shared?

Answer 56

1. Shared covariances result in a linear boundary (Linear Discriminant Analysis, LDA).
2. Unlinked covariances result in a quadratic boundary (Quadratic Discriminant Analysis, QDA).

Question 57

What is the purpose of using maximum likelihood in probabilistic generative models?

Answer 57

It helps determine the values of the μ parameters and priors by maximizing the probability that the data is described by the given parameters.

Question 58

How is the parameter q estimated in maximum likelihood?

Answer 58

1. differentiating the log-likelihood with respect to q
2. setting the derivative to zero, and solving for q
3. this results as q being the fraction of points in class 1.

• q = N_1/(N_1+N_1)

Question 59

How is the parameter μ_1 (mean vector for class 1) estimated in maximum likelihood?

Answer 59

1. differentiating the log-likelihood with respect to μ_1
2. setting the derivative to zero, and solving for μ_1
3. this results as μ_1 being the average of points in class 1.

• μ_1 = 1/(N_1) ∑t_n*x_n

Question 60

How does logistic regression differ from models that use normal distributions?

Answer 60

Logistic regression directly provides a model for the probability of class membership using a sigmoid function, making it more compact than Gaussian models.

Question 61

Why is logistic regression considered more compact than Gaussian models?

Answer 61

• Logistic regression requires only M parameters for weights and bias
• Gaussian models require
  2M parameters for means and M(M+1)/2 additional parameters for shared covariance.

Question 62

What function is used in logistic regression to model the probability of a class?

Answer 62

• The sigmoid function is used, which maps a linear combination of the input features to a probability.
• p(C_1|x) = y(ϕ) = σ(w^T ϕ)

Question 63

What is the maximum likelihood function in logistic regression

Answer 63

p(t|w) = prod (y_n^{t_n}) (1-y_n)^{1-t_N}

Question 64

What loss function does logistic regression minimize?

Answer 64

• Logistic regression minimizes the negative log-likelihood, which yields the cross-entropy loss function.

Question 65

What is the cross entropy loss function

Answer 65

• the negative log likelihood
• -SUM {t_n ln(y_n) + (1-t_n) ln(1-y_n)}

Question 66

What does the cross-entropy loss measure in logistic regression?

Answer 66

Cross-entropy measures the difference between the predicted probability distribution and the true distribution, penalizing incorrect predictions more harshly when the model is confident but wrong.

Question 67

How does the gradient of the logistic regression loss resemble that of linear regression?

Answer 67

The gradient for logistic regression is similar to the gradient for the sum of squared errors in linear regression, showing a close relationship between the two models.

Question 68

Why does gradient descent work well for logistic regression?

Answer 68

Since the sigmoid function is nonlinear but the curvature in weight space is convex, gradient descent converges to the optimal solution efficiently.

Question 69

What key points should be highlighted about logistic regression?

Answer 69

1. Logistic regression models probabilities using the sigmoid function.
2. It is more compact than Gaussian models, especially in high-dimensional data.
3. The negative log-likelihood forms the basis of optimization, leading to cross-entropy loss.

Question 70

derivative of the sigmoid function

Answer 70

• i.e., the derivative of the sigmoid function (y(ϕ) = σ(w^T ϕ) = σ(a) with respect to its inside
• dσ(a) / da = σ(a) (1-σ(a))
• = y(1-y)

Question 71

What is the Newton-Raphson update rule used in IRLS?

Answer 71

It updates the weights by subtracting the gradient of the error function multiplied by the inverse of the Hessian matrix.

Question 72

Why does IRLS for linear regression converge in one step?

Answer 72

For linear regression, the error function is quadratic, and the curvature (Hessian matrix) is constant, allowing optimization to converge in a single step.

Question 73

How does IRLS differ when applied to logistic regression compared to linear regression?

Answer 73

For logistic regression, IRLS requires iterative updates because the cross-entropy error function is not quadratic, and the weights depend on the current predictions.

Question 74

Why does IRLS involve reweighting in logistic regression?

Answer 74

IRLS reweights the basis functions based on the current predictions, requiring iterative updates to reach the optimal solution

Question 75

What are the key differences between gradient descent and IRLS?

Answer 75

1. Gradient descent updates weights using the gradient iteratively, while IRLS solves weighted least squares problems iteratively.
2. IRLS uses second-order information (Hessian matrix) and converges faster.
3. Gradient descent requires a learning rate parameter, whereas IRLS does not.

Question 76

Why is IRLS considered an efficient optimization method for logistic regression?

Answer 76

IRLS leverages the Newton-Raphson method, which uses second-order information, leading to faster convergence compared to gradient descent.