Error-Based Learning

Question 1

What defines a parameterized model?

Answer 1

A parameterized model is defined by a collection of parameters, with specific parameter combinations yielding optimal model performance.

Question 2

What is the difference between parameters and hyper-parameters in a model?

Answer 2

Hyper-parameters are set before training begins. Parameters are learned during training.

Question 3

How are model parameters adapted or fit?

Answer 3

Model parameters are adapted based on the feature distribution in a training set.

Question 4

What guides the model fitting process?

Answer 4

Model fitting is guided by an error function that evaluates the model’s error rate against the ground truth.

Question 5

What indicates model convergence in the context of an error function?

Answer 5

Model convergence is indicated by an error rate approaching a lower limit, ideally close to zero.

Question 6

Why does the L2 function (sum of squared error) use the sum of squared errors rather than just the sum of errors?

Answer 6

To ensure that positive and negative errors don’t cancel each other out.

Question 7

What does each combination of weights w[0] and w[1] correspond to on an error surface?

Answer 7

Each combination corresponds to a sum of squared errors value, defining a point on the error surface above the x-y plane in weight space.

Question 8

What is weight space in the context of modelling error surfaces?

Answer 8

Weight space is the x-y plane defined by possible combinations of weights w[0] and w[1].

Question 9

What does the error surface represent in a model’s weight space?

Answer 9

The error surface represents the sum of squared errors for each combination of weights, with height indicating the error value.

Question 10

Where is the model that best fits the training data located on the error surface?

Answer 10

It’s located at the lowest point on the error surface, which corresponds to the minimum sum of squared errors.

Question 11

What two key properties of error surfaces help in finding the optimal combination of weights?

Answer 11

Error surface are convex (bowl-shaped) and have a global minimum, making it easier to locate the optimal weights.

Question 12

Why are error surfaces for linear models typically convex with a global minimum?

Answer 12

The convex shape is determined by the linearity of the model, not by the properties of the data.

Question 13

What is the method called for finding the best set of weights by minimizing the error surface?

Answer 13

Least squares optimization.

Question 14

What is gradient descent?

Answer 14

It is an algorithm that uses a guided search from a random starting position to iteratively move toward the global minimum of the error surface.

Question 15

How does gradient descent work?

Answer 15

Gradient descent uses the slope of the error surface to take small steps in the direction that reduces error, moving closer to the global minimum with each step.

Question 16

What is the learning rate (alpha) in gradient descent?

Answer 16

It is a parameter that determines the size of the adjustments made to weights at each iteration of the algorithm.

Question 17

What does the error delta function do in gradient descent?

Answer 17

It calculates the adjustment (delta value) for each weight based on the gradient of the error surface, ensuring movement toward the global minimum.

Question 18

What is batch gradient descent?

Answer 18

It is a form of gradient descent where each weight adjustment is made based on the sum of squared errors across the entire training set.

Question 19

What is the inductive bias of batch gradient descent in multivariable linear regression?

Answer 19

It includes a preference for models that minimize the sum of squared errors and restricts to linear combinations of descriptive features.

Question 20

Why is a single random starting point used in gradient descent?

Answer 20

It allows the algorithm to explore the error surface and find the global minimum without the need to try multiple starting points due to the convex nature of the surface.

Question 21

How do the learning rate and initial weights impact the gradient descent algorithm?

Answer 21

They influence the speed and accuracy of convergence.

Question 22

What happens if the learning rate is:
a) too small?
b) too large?

Answer 22

a) Gradient descent converges very slowly (tiny changes to weights at each iteration).
b) It can cause large jumps across the error surface, potentially missing the global minimum and causing instability.

Question 23

What is the ideal learning rate behavior in gradient descent?

Answer 23

A well-chosen learning rate converges quickly to the global minimum without overshooting or instability.

Question 24

How does normalization of features affect the selection of initial weights?

Answer 24

Normalization makes it easier to select initial weights, as the range for weights with normalized features is better defined.

Question 25

What are four differences between linear regression and logistic regression?

Answer 25

1) OUTPUT TYPE
Linear regression outputs a continuous numeric value.
Logistic regression outputs a probability value (between 0 and 1), which is then thresholded to classify the input as one of two classes.

2) FUNCTION TYPE
Linear regression uses a linear equation directly as the output.
Logistic regression applies a sigmoid (logistic) function to the linear equation’s output, mapping it to a probability range.

3) ERROR TYPE
Linear regression uses MSE as loss function.
Logistic function uses binary cross-entropy as loss function.

4) GRADIENT DESCENT APPLICABILITY
Linear Regression can directly use gradient descent on the error surface defined by the MSE.
Logistic regression uses gradient descent on the log loss function, thanks to the differentiable sigmoid function. This wouldn’t be possible with a step function due to its non-differentiability.

Question 26

What are three advantages of using basis functions?

Answer 26

1) Allows more flexible modeling by enabling more complex relationships, not restricted to linear models.

2) Can capture non-linear patterns.

3) Changes the inductive bias of the gradient descent algorithm, allowing for more complex types of regression models.

Question 27

What are four disadvantages of basis functions?

Answer 27

1) The analyst must design the set of basis functions to use, which can be challenging.

2) As the number of basis functions increases, the model complexity grows, making it harder to optimize.

3) Gradient descent must search through a more complex weight space when more basis functions are added, potentially slowing training.

4) Risk of overfitting as more basis functions are added, especially with limited training data.

Question 28

What is a multinomial logistic regression model? And what does it mean when the model is one-versus-all?

Answer 28

A multinomial logistic regression model is an extension that handles categorical target features with more than two levels.

A one-versus-all model distinguishes between one level of the target feature and all the others.

Question 29

What is the difference between a multiclass problem and a multilabel problem?

Answer 29

In a multiclass problem, there are multiple classes, but each instance belongs only to one class.
In a multilabel problem, each instance can belong to multiple classes simultaneously.