Supervised Non-Linear Models

Question 1

What do Non-Linear algorithms assume?

Answer 1

Non-Linear Algorithms assume a non-linear relationship between x and y.

Question 2

What are the 4 common Non-Linear models?

Answer 2

• K-Nearest Neighbour
• Kernel SVM
• Decision Trees
• Neural Networks

Question 3

How does K-Nearest Neighbour work?

Answer 3

1. Decide on the value of K, which determines the amount of neighbours in consideration.
2. Calculate the Euclidean distance between the primary point you’re analysing, and the K nearest neighbours
3. Sort all neighbours by distance and class
4. Select the K nearest Neighbours
5. Pick the majority class

Question 4

How do you calculate the Euclidean distance between two points?

Answer 4

distance = square root of ((x1 - x2)^2 + (y1 - y2)^2)

Question 5

What are the advantages of using the K-nearest Neighbour algorithm?

Answer 5

Doesn’t require any prior training, just the storage of data
Can be used for both classification and regression

Question 6

What are some disadvantages of K Nearest Neighbour?

Answer 6

• Computationally Expensive - Finding distances to all training points can be slow
• Sensitive to K and Distance
• Curse of Dimensionality - Distance becomes less meaningful with higher dimensions

Question 7

What are the steps involved in using a Hard Margin SVM?

Answer 7

1. Define the Hyperplane with the equation w^Tx + b = 0, where w is the weight vector, b is the bias term, and x is the feature vector
2. Maximise the margin, where the distance between the hyperplane and the closest data points from either class is greatest.
3. Apply constraints for Hard Margin
4. Minimise the norm of the weight vector, w, by maximising the margin

Question 8

What are the advantages of Hard Margin SVMs?

Answer 8

Theoretical Guarantee - Finds the hyperplane with the maximum margin, leading to good generalisation

Deterministic - There is always a unique solution for linearly separable data.

Question 9

What are some disadvantages of Hard Margin SVMs?

Answer 9

Assumes the data is perfectly linearly separable, otherwise the algorithm fails.

Sensitive to Outliers - A single outlier can drastically affect the hyperplane.

Question 10

What is the main difference between Soft Margin SVMs and Hard Margin SVMs?

Answer 10

Soft Margin SVMs are designed to handle cases where the data is not perfectly linearly separable, whereas Hard Margin SVMs can’t handle them as well.

Question 11

What are the effects of increasing the value of C within Soft Margin SVMs?

Answer 11

A higher C value will result in a larger penalisation for margin violations, thereby leading to smaller margins and fewer miscalculations.

Question 12

How does a Kernel SVM differ from that of a Hard or Soft Margin SVM algorithm?

Answer 12

Kernel SVMs are able to solve problems where the data is not linearly separable in the original feature space, by mapping the data to a higher-dimensional space.

Question 13

What are the advantages of using a Soft Margin SVM over a Hard Margin SVM?

Answer 13

Soft Margin SVMs are more robust to outliers, and they can be customised more thoroughly to better cater for the problem it’s attempting to solve.

Question 14

What is the step-by-step operation of a Kernel SVM?

Answer 14

1. Discover if the data is linearly separable or not in the standard feature space
2. If not, then map the data to a higher dimensional space
3. Apply the ‘Kernel Trick’, which uses a kernel function to map the data to a higher-dimensional space
4. Solve the SVM optimisation problem by using the kernel function
5. Create the Decision Boundary, based on the terms of the kernel

Question 15

What are the advantages of a Kernel SVM?

Answer 15

Able to handle non-linear data
Very powerful tool for high-dimensional and non-linear datasets
Multiple kernel options allow adaptation to different types of data.

Question 16

What are the disadvantages of a Kernel SVM algorithm?

Answer 16

Kernel computation can be slow for large datasets
It requires careful selection of the kernel function and its hyperparameters
Struggles with very large datasets due to quadratic complexity

Question 17

When would you use a Soft Margin SVM compared to standard Linear Regression?

Answer 17

Soft Margin SVM - Designed for Classification problems, especially when the data is not linearly separable
Linear Regression - Designed for Regression problems to pick continuous target values

Question 18

How does a Decision Tree work?

Answer 18

1. Select the feature that best divides the dataset, where the ‘best’ feature is chosen using a splitting criterion
2. Divide the dataset into two or more subsets based on the selected feature’s values
3. Repeat steps 1 and 2 at each new ‘node’ that you generate
4. Stop the process above when you meet the stopping criterion e.g. max number of depths
5. At each node, assign either labels of the most frequent class as the predicted label (classification), or the average value of samples in the node (regression)

Question 19

What kind of Splitting Criterion is used to select the best features in a Classification Problem?

Answer 19

Gini Impurity
Entropy (Information Gain)

Question 20

What kind of Splitting Criterion is used to select the best features in a Regression problem?

Answer 20

Use Mean Squared Error (MSE)
Variance Reduction

Question 21

What are the advantages of using Decision Trees?

Answer 21

• Robust to features with different scales - Doesn’t require feature normalisation
• Embedded Feature Selection - Redundant features won’t be selected at decision node
• Scalable to handle a large dataset
• Could handle missing feature values by ignoring it
• High interpretability

Question 22

What are the disadvantages of using Decision Trees?

Answer 22

Greedy search at each node, which is computationally expensive

Overfitting as the tree goes deeper

Question 23

What is a Random Forest?

Answer 23

A Random Forest is an ensemble learning method that combines multiple decision trees to improve performance.

Question 24

What are the advantages of using Random Forest?

Answer 24

Reduces overfitting
Handles non-linear data well
Can be used to evaluate importance of features
Left out data can be used to estimate the error without needing a separate validation set

Question 25

What are the disadvantages of using Random Forest?

Answer 25

Not easily interpretable
Computationally intensive
Making predictions requires aggregating the results of many trees, so it can be slower compared to individual models.

Question 26

What are the hyperparameters for Random Forest?

Answer 26

Number of trees in the forest
Maximum depth of each tree
Minimum number of samples required to split a node
Minimum number of samples required for a leaf node
Maximum number of features to consider when looking for the best split.
The impurity function to use