Final Exam Past Exams Flashcards
How is Occam’s Razor applied to Machine Learning?
If you have two comparable machine learning models, the simplest is the better.
How many parameters does this model have?
d + 1
What is the difference between feature selection and feature extraction?
Feature selection is the selection of a subset of the features for building a model.
Feature extraction creates new features from functions of the original features, whereas feature selection returns a subset of the features
Describe PCA
PCA is the projection of the principle components onto a lower dimensional space. These principle components are the features that have the highest share of the variance.
What dimensionality reduction technique works the best?
LDA (red) works the best because it has the best separability.
How does the k-means clustering algorithm work and what is the “solution” that it produces?
k-means works by initialising randomly k-centroids. Each data point is assigned to the nearest centroid. At each stage of the algorithm, the centroid locations are updated to be the mean of all the data points. Data points are assigned to their local centroid. This continues, until their is convergence.
How would you assess if k-means clustering has worked properly?
If the centroids don’t move.
How would you assess if k-means has converged?
If all the data points are assigned to the same cluster in successive iterations, there’s convergence
How do you decide how many base learners when using bagging?
The number that can reduce the variance is an optimum number.
What is the misclassification error of this dataset?
All non-diagonal elements divided by the total
Explain these models in terms of overfitting/underfitting.
Top left - degree 1
Top right - degree 2
Bottom left - degree 10
Bottom right - degree 25
The top left model is underfitting because no matter how much training data is added, it’s performance isn’t increasing.
The bottom models are overfitting because the test error is significantly higher than the training error.
What’s the purpose of the validation set?
The purpose of the validation set is to use a set of examples used to tune the parameters of a classifier
One commonly used learning algorithm for linear discriminant models and MLP is Gradient Descent. What’s the basic idea behind gradient descent?
Find function parameters (coefficients) that minimize a cost function as far as possible.
In MLP, why are sigmoid functions used instead of hard-step functions?
Hard-step aren’t continuous, sigmoid are.
It is especially useful for models where we have to predict the probability as an output. Since probability of anything exists only between the range of 0 and 1, sigmoid is used instead of hard-step functions.
In MLP what is the role of weight and bias?
A weight represent the strength of the connection between units. Decides how much influence the input node has on the output.
A bias ensures there’s always an activation in a node, even if the weight is zero. Makes a more flexible mlp model.
Bayesian inference is a general alternative to maximum likelihood estimation that can be used to train a variety of models given data. Explain the main idea of Bayesian inference and compare with MLE. Your answer should mention the prior and posterior distributions over model parameters.
Bayesian estimation takes into account prior probability when assigning the likely parameters of a model.
MLE just tries to estimate the parameter which maximizes the likelihood function.
If d= 10, how many parameters would a six degree polynomial have compared to the linear model?
61
What is a hyper parameter in the context of Bayesian inference? Give an example.
The prior and the likelihoods, as well as the parameters of the prior distribution are all hyper parameters in the context of Bayesian Inference.
In machine learning, what is known as “generalization”?
Generalization is how well a trained model accurately classifies new data. An overfit model, doesn’t generalize well.
You are given a 5 dimensional dataset. After doing PCA, you discover that the 4th and 5th features have zero eignenvalues. What should you do?
They can be removed as they don’t contribute to the variance.
What’s an expression for the percentage of the variance captured by the first principle component, where the eigenvalues and the covariance matrix of the data are lambda1 and lambda2?
lamda1 / lambda1 + lambda2
What is the total number of data points in this training set? How?
Sum all rows and columns.
24
How many data points do we have in each class?
Sum the rows:
A: 5
B: 4
C: 15
What is the sum of the diagonal values in a confusion matrix?
The accurately predicted data points.
What is overfitting?
Overfitting is where the model is too complex and fits the training data too well. This means its captured the data too well (noise/outliers) and can’t generalize well.
MLPs are trained using backpropogation. Discuss in detail some practical challenges to do with this.
Backpropogation uses gradient descent to optimise weights. The gradient descent algorithm is generally very slow because it requires small learning rates for stable learning.
Explain briefly what weight decay is. You can write it in terms of an equation or in words.
A number multiplied to the weight (<1) to stop the weight growing too large in a neural network. This is also prevent the neural network overfitting.
The training error is higher with weight decay than without weight decay. Does this matter?
Test error is the highest priority as it defines how well the model generalizes
Discuss how weight decay works and what effect it can have on a model’s prediction (discriminant).
Weight decay is a regularization factor that penalizes a model overfitting. This leads to more simple disciminant functions for categorising data.
Explain briefly how support vector machines utilise the concept of the margin in producing a classifier with good intended generalization performance?
Margin is the maximum distance between support vectors. By maximising this distance and using it to classify data, the probability that a new data point is classified correctly is increased.
What is the general principle behind bayesian inference in machine learning.
Prior probability can be used to determine posterior probability and update a hypothesis.
What does the shaded region represent in both of these pictures
Figure 9(a) represents the prior distribution, which is our belief of what the model may be before any observations are made.
Figure 9(b) represents the posterior distribution, so our belief of what the model may be based on the prior and the observed data. In figure 9(b) the shaded area shrinks at each observed data point since we know the actual value of the model at those points, but as we move away from the observed values the possible model values grows and so the distribution (shaded area) also grows.
What is the difference between classification and regression?
Regression is used for predicting continuous variables.
Classification is used for predicting categorical variables.
What’s the difference between likelihood and discriminant approaches to classification?
Likelihood approaches make assumptions regarding the distribution of the data. The goal is to use bayes rule and model the posterior distribution of the classes given the training data. A test data point is assigned a class based on the highest posterior probability.
Discriminant approaches make no assumptions about the data and instead try and separate the classes of the data with a boundary(the discriminant). This is accomplished using some distance measure and by placing a hyperplane between the classes which allows classification if the point is within the boundary.
What information does a confusion matrix give?
The FP, TP, FN and TN of a dataset.
What is the forward selection algorithm?
Build a model by adding features that have highest significance first. If the E value improves with adding the next feature, the next most significant feature is added next.
What are the steps of forward select algorithm in this example?
Check each column to see which row has two zeroes and a 1.
Look for the smallest value of these.
It is x2 in this case.
Check the combinations with x2, so (x1, x2) and (x2, x3).
If lower than the current e value. It’s the new case.
In this example, (x1, x2) < x2 -> new case.
Then check the next combination (x1, x2, x3) does it have a lower value than (x1, x2). No, so x1 and x2 are the features that would be selected in the algorithm.
What’s a limitation to Forward Selection?
It’s a greedy algorithm. It doesn’t consider the best subset overall.
If (x2, x3) are final with forward selection, it could have ignored (x1, x3).
What are the symbols represent in a GMM?
Gi the components/groups/clusters
P(Gi) priors
P( x | Gi) the component densities
P(x) the mixture model of the given data
k is the number of clusters
What are the parameters of the model?
The parameters of a GMM are the
k parameter,
the covariance matrix,
initial conditions and the regularization parameter.
The k parameter is the number of clusters
What are the E and M steps in the EM algorithm?
E: Compute the expected value of your “hidden variables”, based on the current values of the parameters
M: Recompute the most likely value of your parameters based on the value of the hidden values and the observed data.
Why might you get GMM that have different covariant matrices?
As the EM is started on random initial conditions there is a chance it will converge to a local optima. This is what can be seen in the results as the solution is non-convex and does not guarantee a global optimum
What is a kernel density estimator? How does it use the data and what are the parameters?
A kernel density estimator is used to estimate an unknown probability distribution. Similar to a histogram. You place a kernel function on each data point and you sum them together. This gives a smooth distribution rather than a histogram model which is dependent on bin size.
Explain how Bayes rule is used to produce a classifier from probability density estimation models given supervised data. From this, explain what is shown in Figure 2 (right).
The RHS is the posterior estimate which is the discriminant.
The PDE is used as the prior in Bayes Rule.
How many weights in the model?
For each layer add 1 and multiply by the next layer to the right.
So [784+1] * 50, then [50+1] *50, [50+1]*20 … until the last layer.
What do the results in Figure y show regarding
(i) the effect of weight decay;
(ii) the effect of varying the number of hidden units?
i) Weight decay stops the model overfitting and increasing complexity. As more hidden layers are added, the weight decay reduces the weights of each node.
ii)
As the number of hidden units grow the weights are decreased having less and less impact using the second term in the equation.
Figure 4 shows results that examine more closely the effect of the weight decay value on the test error. Explain the relationship between the trend of the results in Figure 4, under/over-fitting and model complexity
The model is overfitting when there is no weight decay, as weight decay increases this test error reduces. By using weight decay it is finding a balance between bias and variance which can be seen as the variance and bias reduces to find the optimal solution. As this parameter is increased further the test error increases again as the system is being limited to a simple system and mildly under fits the data.
Explain what parts of the Adaboost need to be removed to make it a bagging algorithm.
Bagging is parallel, Boosting is iterative, using misclassified elements from a previous bag to improve the model. Remove anything except sampling and training.
In the test pseudocode, anything below “calculate” until “test”.
How do they compute the cross-validation results?
The data is split into 10 subsets where each fold is run with 10 - 1 (9) subsets of the data. Each time the data is rotated until all subsets of data have been left out once.
During each run the error values are recorded and the mean and standard deviation is calculated from these outputs to be shown on the figure.
How many experiments?
Experiments =5*5*5*5*2=1250
5 for each parameter and 2 for the last.
Draw a diagram
In Figure 3, what symbol is used to denote the objective function for the optimisation problem?
C. C for cost
Using the notation from Figure 3, how would you indicate the dataset in the reduced (i.e. low dimensional) space after 100 iterations?
y^(100)
Explain what the curve and shaded region in Figure 6 represent. What does the model give as an output for a test input?
Shaded region is how confident the model is around a particular input, x. This is the posterior output where every subset of x is represented by multivariate gaussian distribution.
GP is a method of regression given a test input the model will output a multivariate gaussian distribution for that point which will give insight into if that point.
Explain why the curve and shaded region in Figure 6 have this shape by referring to the data and the covariance function of the Gaussian Process
Before figure 6 the model is represented over the function as a prior and following the observation of some data figure 6 is generated which is the posterior.
As more data is seen the confidence of that point is increased which results in a narrower shaded region at that point. The covariance function varies how the function space responds to the length of the characteristic length-scale. I.e. it is a smoother function or a more noisy function.
The pseudocode algorithms in this question is taken from the Alpaydin textbook.
(a) In the pseudocode, what does N represent?
The number of data points
(b) In the pseudocode, what does d represent?
The number of features
What is the learning rate parameter in these algorithms?
If the learning rate is too large, what effects might be observed during training?
The n variable.
if the learning rate is too high then the algorithm may not converge to the local optima.
Figure 11.11 shows pseudocode for backpropagation in a multi-layer perceptron with a single hidden layer. What symbol represents the weights connecting the inputs to hidden units?
The wth value.
If we modify the code in Figure 11.11 (as the caption suggests) to perform two-class classification, which lines need to be modified? Write out the modifications
Change 6,7,8 to y_i = sigmoid(v^T_i z)
Does Figure 11.11 describe online or batch backpropagation? Explain your answer by referring to the pseudocode
online (stochastic because random order), the update in weight of output and hidden layer is done for each datapoint instead of whole iteration of the dataset.
(i) What do the “ReLU” rows in this output represent?
Using the Rectified Linear Unit Activation function
(ii) Which layers contain adaptive weights that are modified during training?
Convolution layers => adaptive filters, Fully Connected => adaptive weights, this is all.
(iii) What happens in the “Dropout” layers?
iii) Randomly discard a percentage number of nodes.
