Final Exam Prep

Question 1

What is density estimation?

Answer 1

Making an estimate by trying to fit a probability density function to some data.

Question 2

What is occam’s razor?

Answer 2

The simplest explanation is usually right.

Question 3

Overfitting will give a low training set error but a high test set error. True or false?

Answer 3

True.

Question 4

Underfitting will give a high training set error and a high test set error. True or False?

Answer 4

True.

Question 5

Does a underfitted model have high or low bias?

Answer 5

High bias.

Question 6

What is generalization?

Answer 6

Generalization is the ability of a trained model to classify new data.

Question 7

What is overfitting?

Answer 7

Overfitting is when the model fits the data too closely and doesn’t generalize to new data. It also captures noise.
Avoid overfitting by cross-validation, early stopping, pruning - further training may hinder generalisation.

Question 8

What is underfitting?

Answer 8

Underfitting is when the model doesn’t fit the data. It doesn’t capture the underlying pattern.

Question 9

What is regularlisation?

Answer 9

• A technique used to penalise overfitting.

- Applies a penalty to the cost function to take into account any outliers that have made the model more complex

Question 10

What is linear regression?

Answer 10

A line fitted to data points. Line is placed to limit the SSE. Used for continuous problems with an independent and dependent variables.

Question 11

What is logistic regression?

Answer 11

A classification regression model. It takes inputs into a log function and squashes it. And outputs it as 0 or 1 etc.

Question 12

What is optimisation in machine learning?

Answer 12

Finding the right parameters in order to minimise a cos t function. A popular one is gradient descent which looks for local minima.

Question 13

What is the learning rate?

Answer 13

How steep the parameters are changing in gradient descent.

Question 14

What is the curse of dimensionality?

Answer 14

The more dimensions (features) you have, the more data you need to build good models (generalize). More dimensions (features) doesn’t exactly mean better classification. There is an optimum number, beyond this you can get overfitting.

Question 15

What is the bias/variance tradeoff?

Answer 15

Hard for a model to be complex enough and simple enough at the same time. (underfitting/overfitting balance).
Simple model = high bias and low variance
Complex model = low bias and high variance

Question 16

What is backpropogation?

Answer 16

Basically, adjust all the weights and the biases on the way back after forward propagating. You do this until you find weights and biases that minimise a cost function.

Question 17

What is feature/subset selection?

Answer 17

Using/Choosing only a number of features from the dataset to construct a model.

Question 18

What is feature extraction?

Answer 18

Feature extraction is a general term for methods of constructing combinations of features to provide insight into a dataset.

Question 19

What is MLE?

Answer 19

A method for determining the parameters of a model.

Done by checking that the parameters most likely match the observed data points.

Question 20

What is Bias in ML?

Answer 20

Bias is the difference between the average prediction of our model and the correct value which we are trying to predict

Question 21

What is Variance in ML?

Answer 21

Variance is the variability of model prediction for a given data point or a value which tells us spread of our data.

Question 22

How is Bayes Rule used in ML?

Answer 22

A probabilistic method for classifying new data based on previous examples.

Question 23

What are discriminant functions?

Answer 23

Basically, classifying based on a function.

Question 24

What are decision rules?

Answer 24

Basically, a lot if - then statements for classifying.

Question 25

What is non-parametric classification?

Answer 25

Doesn’t make assumptions about the underlying model. Good when you have no prior knowledge and don’t want to choose the right features

Question 26

What is NN-classification?

Answer 26

Set a k-value and classify based on the k-nearest neighbours. If the majority of them are red, then it’s classified as red.

Question 27

What’s the difference between discriminative and generative learning?

Answer 27

A Discriminative model ‌models the decision boundary between the classes and learns the conditional probability distribution p(y|x).

A Generative Model ‌explicitly models the actual distribution of each class and learns the joint probability distribution p(x,y). It predicts the conditional probability with the help of Bayes Theorem [1]

Question 28

What are kernel density estimators?

Answer 28

Smoothing function used to estimate distributions
Histogram bins are sharp, they don’t take into account which side of the histogram column they belong to. KDEs are smooth, take into account this.

Question 29

What is a mixture model?

Answer 29

Clustering method that uses Gaussian distribution to best fit unclassified data.
You have multiple peaks in the data
You use when you don’t have perfectly circular data
Uses the EM algorithm to move the parameters of the Gaussian
Soft clustering vs k-means hard clustering

Question 30

What are autoencoders?

Answer 30

Autoencoders are a type of neural network that attempts to output it’s own input i.e. learn the identity function.

It does this in two parts, encoding and decoding. In the encoding phase it tries to compress the features. And the decoding phase it tries to uncompress.

Question 31

Why are autoencoders used?

Answer 31

• The most common is finding a more suitable representation of the input data which we can use in a neural network for classification for example.
• Outlier detection

Question 32

What is LDA?

Answer 32

• Focuses on maximising separability among features.
• Find new axis and project data onto new axis in a way that maximises separation
  => Maximise distance between means of all classes
  => Minimise variation of each class

Question 33

What is k-Nearest Neighbour Classification?

Answer 33

- Make prediction by finding the most common class that corresponds to the k nearest training points 
 => Hyper Parameters: Distance metric and k (number of neighbours)

Question 34

What is Model Selection?

Answer 34

• Indication of overfitting if the test error is much greater than the training error.
• Simple model => poor predictive power on the test set
• Complex model = > too specific to the training set, therefore have high error on test set