Topic 1: Loss functions

Question 1

What is a target

Answer 1

The output y of a function

Question 2

What is the basic model of the function of sample

Answer 2

y = ftrue(x) + ϵ

ϵ = noise, modelled by a gaussian distribution

Question 3

What is k-nn regression

Answer 3

When a new x value is observed, to calculate y
Take the k nearest neighbours to x and average their y values

Question 4

What is an instance-based algorithm

Answer 4

Not good at generalising beyond the current scenario
i.e. k-nn

Question 5

What is “fine-tuned” the same as

Answer 5

overfitting = complex

Question 6

What are the 6 main types of function approximations

Answer 6

linear/polynomial regression
support vector machines
neural networks (CNN and logistic regression)
naive bayes (probabilistic models)
decision trees (for both regression and classification)
ensemble models

Question 7

How do the 6 main types of function approximations work

Answer 7

By minimising loss functions

Question 8

What are the properites of overfitting

Answer 8

high accuracy on training data
captures noise
high testing errors

Question 9

What are the properties of underfitting

Answer 9

too simple
high training and testing errors

Question 10

What does x ∈ R^d mean

Answer 10

x is a real-valued feature vector of length d

Question 11

What is the common loss function used for classification problems

Answer 11

Cross entropy loss

Question 12

What is the formula for binary CE loss

Answer 12

l(y, f(x)) = -[ylnf(x) + (1-y)ln(1-f(x))]

where y ∈ {0,1} and f(x) ∈ (0,1)

Question 13

What is the common loss function used for regression problems

Answer 13

Squared loss function

Question 14

What is the formula for squared loss function

Answer 14

l(y, f(x)) = (y - f(x))^2

Question 15

Where does the true label y always go

Answer 15

First before the function label f(x)
eg ( y - f(x))

Question 16

What is the formula for squared loss training error

Answer 16

ltrain(f) = 1/n Σ (yi - f(xi))^2

Question 17

What does asterisk denote

Answer 17

Optimal solution or best value

Question 18

What is the general equation for w*

Answer 18

argmin(w) [ltrain(f)]

Question 19

What is a probability simplex

Answer 19

A geometric object that represents all possible probability distributions over a finite set of outcomes
Eg for classification problem with 3 classes
a traingle where each vertex corresponds to a class

Question 20

What does ∈ (0,1) mean

Answer 20

The variable can take any value in this interval but not 0 and 1 themselves

Question 21

what does ∈ {0,1} mean

Answer 21

The variable can take any value in this interval including 0 and 1 themselves

Question 22

What is the purpose of gradient descent

Answer 22

To optimise complex models

Question 23

Theoretically what parameter would we like to adjust to achieve global minimum

Answer 23

wj
too computationally expensive to manually plot
instead we use iterative methdods

Question 24

What is the key principle of gradient descent when gradient is negative

Answer 24

increase the parameter wj

Question 25

What is the key principle of gradient descent when gradient is positive

Answer 25

decrease the parameter wj

Question 26

What does nabla ∇ denote

Answer 26

gradient

Question 27

What is the update rule

Answer 27

w ← w - η . ∇l(y, f(x))

Question 28

What is η in the update rule

Answer 28

The learning rate (step size of algorithm)

Question 29

What is full batch gradient descent

Answer 29

The entire dataset is used to compute the loss function gradient
w ← w - η . 1/n Σ [∇l(yi, f(xi))]

Question 30

What is mini batch gradient descent

Answer 30

Randomly picks a sample S of m datapoints
w ← w - η . 1/m Σ [∇l(yi, f(xi))]

Question 31

What is stochastic gradient descent

Answer 31

At the extreme of mini batch where m=1
(sometimes referred to with values m>1)

Question 32

What does SGD help with

Answer 32

Tends to help with escaping local minima

Question 33

What is sgd, gd and mini batch gd all examples of

Answer 33

first order gradient descent algorithms

Question 34

What are first order vs second order gd algorithms

Answer 34

second order use the gradient but also information about the curvature of the loss function (second derivative)

Question 35

When and why are decision trees useful

Answer 35

Fast to train and deploy
good for tabular data (anything that fits a spreadsheet)
not images, speech, videos

Question 36

How do decision trees work

Answer 36

Recursively splits the data into subsets based on values of input features
Then fits each subsets to a simple model (constant label/ linear regression)
When a new label x is observed, traverse the tree to find the right prediction

Question 37

How do classification and regresssion trees compare

Answer 37

Use the same branching
eg X1 > 0.83 (yes or no)
then X4 < 0.3 … etc

Except regression trees predict a continuous value at the final node - usually the mean of the target variable within that node
for classification the node is a class eg class 3

Question 38

What is the main parameter of decision trees

Answer 38

Depth
similar to increasing k for knn
increased depth -> increased complexity

Question 39

what is thew update rule updating/optimising

Answer 39

parameters w
Determines how the parameters should be adjusted in order to minimise the loss function