Week 2: Regression & Classification (Linear & Nonlinear Models)

Question 1

Perceptron Training Rule

Answer 1

Linear classification models can draw decision boundaries between regions, with each region representing its own class.

Question 2

Error Rate for Classification Models

Answer 2

Error rate = 1 - \frac{1}{m}\sum_{i=1}^m score, with score = 0 for misclassifications and 1 for correct classifications

Question 3

Error Rate for Regression Models

Answer 3

Error rate = \sum_{i=1}^m (y - y_i)^2

Question 4

Training Models

Answer 4

This applies to both classification and regression models.

Training:
1. Select the training set
2. Initialise model parameters
3. Apply the model to all training set instances
4. Computer the error rate
5. Adjust the parameters to obtain a model w/ lower error
6. Repeat from step 3 until desirable error rate reached
7. Output the training error

Evaluation:
1. Select the test set
2. Apply the model to all test set instances
3. Compute the error rate
4. Output the evaluation error

Question 5

Cross-validation

Answer 5

1. Split the dataset in N approximately equal-sized folds.
2. Perform N repetitions where one fold is used for testing and the remaining folds are used in training.
3. Compute the error rate N times after each repetition and average the results to yield the overall error rate.

Question 6

Spurious Correlations

Answer 6

Just because correlation exists, doesn’t mean there’s a causal relationship between the variables.

Question 7

Linear Regression

Answer 7

\hat{y}i = x_i w = w_0 + \sum{j=1}^n w_j x_i,j

Question 8

Nonlinear Regression

Answer 8

These can include interaction terms and polynomial terms. Ex. \hat{y}_i = w_0 + w_1 \cdot x_i + w_2 \cdot x_i^2

Question 9

Underfitting

Answer 9

When the model doesn’t predict the training data well.

Question 10

Overfitting

Answer 10

When the model fits the training data relatively well, but fails to generalise to unseen data.

Question 11

Mean of Squared Errors

Answer 11

S(w) = \frac{1}{m} \sum_{i=1}^m (y_i - \hat{i})^2. Linear Regression models try to minimise Mean of Squared Errors.

Question 12

Gradient Descent

Answer 12

Initialise weights to 0 or to random values.

Until convergence is achieved:
for i \in {1,…,m}
for j \in {1,…,n}
w_j \leftarrow w_j + \alpha(y_i - \hat{y}i)x{i,j}

Termination criteria: \left\lvert S(w^k) - S(w^{k+1}) \right\rvert < \epsilon

Question 13

Single-scan/On-line Algorithm

Answer 13

for i \in {1,…,m}:
repeat:
for j \in {1,…,n}:
w_j \leftarrow w_j + \alpha(y_i - \hat{y}i) x{i,j}
until S(w) isn’t significantly changed

This method updates after each individual example. Other names include online approximation, stochastic gradient descent.

Question 14

Logistic Regression

Answer 14

This method assumes binary classification. If y \le 0.5, then predict 0. EIse y > 0.5, then predict 1. \hat{y} = \frac{1}{1 + exp[-(w_) + w_1 x_1 + … + w_n x_N)]}. Essentially if w^T x > 0, return 1. If w^T x \le 0, return 0.

Question 15

Perceptrons

Answer 15

This method learns a hyperplane separating two classes. Perceptrons form the building blocks of neural networks, such as single-layer feed-forward neural networks. They use the perceptron learning rule.

Question 16

Squared Error

Answer 16

The perceptron learning rule uses this metric. squared_error = \sum_{i=1}^m (y_i - \hat{y}_i)^2

Question 17

Perceptron Learning Rule

Answer 17

It utilises gradient descent. For each i \in {1,…,m} and j \in {1,…,n}, w_j \leftarrow w_j + \alpha (y_i - \hat{y}i)x{i,j}

Question 18

Feed-Forward Multilayer Neural Network

Answer 18

These have inputs, hidden units, and outputs. The network only goes in one direction towards the output.

Question 19

Neural Network Unit

Answer 19

Each node has an input fed into an input function, an activation function, and an output function. Each node has input links and output links.

Question 20

Sigmoid Activation Function

Answer 20

f(x) = \frac{1}{1+e^{-x}}

Question 21

Hyperbolic Tangent (Tanh) Function

Answer 21

tanh(x) = \frac{2}{1 + e^{-2x}} - 1

Question 22

Rectified Linear Unit (ReLU)

Answer 22

f(x) = 0 for x < 0, x for x \ge 0

Question 23

Activation Functions

Answer 23

Ideal properties include nonlinear (can generalise well), differentiable (can update weights during training), and monotonic (for fast convergence)

Question 24

Neural Networks

Answer 24

Generally, there’s a specific cost function that is minimised when training neural networks.

Important factors to consider include:

1. Number of layers
2. Number of nodes per layer
3. Number of incoming links per node
4. Activation Function

Pros:
1. Great at nonlinear transformations of input.
2. Highly parameterised and can model even small function irregularities.
3. Even a small number of hidden layers can sufficiently model any continuous function.
4. Can be optimised to reduce overfitting.

Cons:
1. Explainibility is challenging, making it difficult to infer causal relationships.
2. Computationally expensive
3. Needs a very large dataset to work properly.

Question 25

Recurrent Network

Answer 25

This type of neural network feeds outputs back to its own inputs. Network activation levels form a dynamic system that may reach a steady state or show oscillations and potentially chaotic behaviour.

Question 26

Backpropagation

Answer 26

Given a specific weight w,

w \leftarrow w + \Delta w = w - \eta \frac{\partial J}{\partial w}

Question 27

Lasso Regression / L1 Regularisation

Answer 27

It uses a diamond-shaped region to eliminate irrelevant variables.

Question 28

Ridge Regression / L2 Regularisation

Answer 28

It uses a disk-shaped region to reduce the coefficients of irrelevant predictors, which approach 0 but doesn’t equal 0.