Xijia

Question 1

What is a kernel function and the role in algorithm?

Answer 1

A function that’s used when non-linear data is projected into higher dimensions. ( X => ØX ) The kernel function is used to make this computation at the same time.

Given x1 and x2 returns < Øx1 , Øx2 > without calculating ØX.

One is therefore able to transform the data without the need for excessive computer power.

Question 2

What is the basic property of kernel function? (Gram matrix)

Answer 2

The kernel function holds one central part. The Gram-matrix. Which is made up from the:

• kernel
• training set

The gram-matrix contains the evaluation of the kernel function on al pairs of data-points. All the information to the algorithm must pass through this.

Question 3

Polynomial Kernel

Answer 3

F(x, xj) = (x (Dot-product) xj+1)^d

Generalized version of linear kernel and not preferred.

+ more powerful than linear kernel
+ strong physical control
- more hyperparameters
- high polynomial degree

Question 4

RBF kernel

Answer 4

F(x, xj) = exp(- ||x - xj||^2)/(2pi^2)
pi => kontrolls the complexity of the model

It is one of the most preferred and used kernel functions. It is usually chosen for non-linear data. It helps to make proper separation when there is no prior knowledge of data.

+ only one hyperparameter
+ less computional
- powerful and flexible

Question 5

What is the basic idea of kernel methods?

Answer 5

Functions that given x1 and x2 return without calculating Ø(x).

Transform future-vectors to infinite dimensions without extra computational burden.

Question 6

Why is the kernel method viewed as the memory-based method?

Answer 6

Memory-based methods keep the training samples and use them during the prediction phase.

Kernel functions store everything in the gram-matrix (kernel information and training-set)

Hence they are memory based

Question 7

What is the basic idea of ensemble methods?

Answer 7

Use multiple weak methods together to make a better predictive performance.

ex:
{random forest & adabosting}

{boosting & adaboosting}

Question 8

What is the difference between kernel methods and Ensemble methods?

Answer 8

Kernel methods main goal is to help classification by adding dimensions to the data.

Ensemble methods use multiple methods and combine them to one classification.

Question 9

Why does bagging (bootstrap aggregation) can help us to improve the predictions?

Answer 9

Bagging is when we divide the train-dataset into many subparts (randomly) and train many models and take the average:

+ Raises stability of the model
+ Reduces overfitting

Question 10

What is the difference between bagging and adaboost?

Answer 10

Both are ensemble methods with random sampling from the test data. But boosting redistributes it’s weights after each training-step.

Question 11

Understand how does Perceptron algorithm work?

Answer 11

• Needs linearly separable data
• Input: w = (w0,w1,w2)^T {Dimension +1} {w0 = bias}

SUM [w0x0 + w1x1 + w2*x2 ] = (+) or (-)

If missclassification:
wi = wi + Ndxi
new = old + lear. rate

N = learning rate ( how fast are we gonna step towards the line )
d = {1 if miss should be above the line} {-1 if miss should be below line}

Features?

• {supervised learning}
• {optimal weight coefficients are automatically learned}

Question 12

Features of Perceptron algorithm?

Answer 12

• {supervised learning}

* {optimal weight coefficients are automatically learned}

Question 13

Understand how to kernelize the linear PCA method.

Answer 13

• First axis is of highest important ( x1 is not to scale of x2)
• PCA’s covariance matrix scales with the number of input dimensions
• Kernel PCA’s kernel matrix scales with the number of datapoints.

Question 14

Be able to implement kernel PCA from scratch in R

Answer 14

1. Pick kernel function
2. Calculate the kernel matrix
3. Center the kernel matrix
4. Solve the Eigenproblem
5. Project the data to each other

Question 15

Understand how to kernelize the ridge regression method.

Answer 15

Ridge regression is regression where the cost function is altered by adding a penalty equation to the magnitude of the coefficient.

With kernel alteration xi –> (IO)(Xi) we can analyze infinity higher dimensions

Question 16

Understand the mean of hyper parameters in KRR.

Answer 16

• Kernel Parameter

* Regularization constant

Question 17

formulation of maximum margin classifier (MMC)

Answer 17

Maximum margin classifier
min {b,w} 1/2 w^T*w

S.T. yi(w^Txi+b)>=1 för alla i=1,…,N

Question 18

formulation of soft margin classifier (SMC)

Answer 18

• Give up some high noise cases
• Introduce slackness parameter

min {b,w,epsi} 1/2 w^t*w + C sum of epsi{i} i = 1 to N

S.T. yi(w^T*xi+b) >= 1- epsi{i} and epsi{i} >= 0 for every I

Hyper parameter: Large C, less noice tolerance, high cost

Question 19

Understand the difference and connections between MMC, SMC, and SVM

(SVM is refered as a kernelized SMC)

Answer 19

SVM = SMC + kernel trick

SMC = MMC + penalty on slackness parameter

Question 20

formulation of SVM

Answer 20

• Works like logistic regression or perceptron

* multiple hyperplanes are working

Question 21

Why SVM is called a sparse kernel method?

Answer 21

Only the “outliers” matter for our model. We can add new data-points behind the line and it won’t change. That’s why its called a sparse model.

Question 22

What are the hyper-parameters in RBF-kernel SVM? What is the meaning of them?

Answer 22

• Gamma defines how far the influence of a single training example reaches .
• C - trades correct classification of training examples against maximization of the decision margin.

Question 23

Understand the outputs of the function ’ksvm’ from ’kernlab’ package

Answer 23

alpha - resulting suport vectors 
alphaindex - index of resulting suport vectors
coef - corresponding coefficients
 b - negative intercept
nSV - the number of support vectors
obj - value of objective function
error - training error 
cross - cross validation
prob.model - width of the laplacian fitted

Question 24

Know how to train a random forest

Answer 24

• Random Sampling with replacement when choosing data n-times.
• Random select features and build a decision tree for each dataset (n).
• Use majority voting of each decision tree
• bootstrapping + aggregation = bagging

Kolla upp den här så att du förstår den

Question 25

Know how to train a decision tree

Answer 25

• kolla upp detta bättre så att du kan beskriva

Question 26

Connection between decision tree and RF?

Answer 26

A random forest is made up of decision threes that use voting to classify the sample.

Question 27

What are the hyper-parameters in RF? What is the meaning for them?

Answer 27

mTRM - Amount of variables at a node-split (bara två variabler?)
noDESIZE - Amount of observations in the leaves
nTREE - Number of trees to grow.

Question 28

Why Random forest is self-validated?

Answer 28

Random forest is built on bagging. in each round of bootstrap 2/3 of the samples will be included for training. So the decision three can be evaluated on the remaining 1/3.

Question 29

What is OOB error? (Random forest)

Answer 29

out of bag error - is a method of measuring the prediction error of random forests, boosted decision trees, and other machine learning models utilizing bootstrap aggregating (bagging).

Question 30

How to apply Random forest for features selection?

Answer 30

Random forest is good as features selection since they rank naturally by how well they improve the purity of the node. This is called mini impurity. Nodes with the greatest decrease in impurity is at the top of the tree.

Question 31

Understand the outputs from ’randomForest’ package

Answer 31

???????

Question 32

Understand the basic idea of the Adaboost algorithm.

Answer 32

• Combines a lot of weak learners
• Some stumps/trees get more say in classification
• Each stump is made by taking the previous stumps misstakes