Basics

Question 1

Give examples of self-supervised learning

Answer 1

Images: patches shuffle, inpainting. NLP: auto-regressive LLMs

Question 2

Explain Bias variance trade-off.

Answer 2

Bias: expected In case of MSE: generalisation error is irreducible + bias^2 + variance. Bias is E[f(n)-f*(n)]. Variance is E[E[f(n)]-f(n)]. Expectation is taken with respect to current training dataset of finite size.

Question 3

What is empirical risk minimisation principle?

Answer 3

Ideal goal: find algorithm which minimises expected (over samples) error. In practice we minimise error on training dataset. The principle says that we should use the algorithm which has the lowest error on the training set.

Question 4

What’s VC dimensionality? What vc dimensionality of logistic regression?

Answer 4

It’s the largest dataset size for which any labelling can be achieved by the classifier (shattered). For linear classifier it’s three: all labelling of “triangles” of three points can be achieved though “square” with zeros on one diagonal, ones on another diagonal is not possible with a straight line.