Adaptive basis function models

Question 1

What is a basis function and give an example of when we can use domain knowledge to employ one to a model?

Answer 1

A map from the input space to the reals. E.g. gravitation. If we were trying to learn Force from a dataset of masses and radii, we could construct a basis function m1m1/r which would be a better feature.

Question 2

Explain the aim of an adaptive basis function model

Answer 2

We want to learn basis functions from the data. Our prediction functions will be linear combinations of our basis functions.

Question 3

In an adaptive basis function model, what is the empirical risk we wish to minimise?

Answer 3

The sum of losses for a function in the hypothesis space.

Question 4

Can we use gradient descent to minimise this risk in general?

Answer 4

In general, no. Only if each basis function is parameterised by weights and the objective function is differentiable with respect to those weights.

Question 5

Describe the algorithm of forward stagewise additive modelling.

Answer 5

Start with some set of weak/base learners
- Initialise prediction function
- Update with the linear combination of basis functions which minimise the loss
continue

Question 6

What are some possible weak learners?

Answer 6

Shallow trees

Question 7

Explain gradient boosting

Answer 7

The update step can be learned by noting that the objective function only depends on the prediction at the training points. We can take the gradient of J with respect to f_i for each point, and this would give the set of points that our update sum of basis functions should be closest to. Then we just minimise the difference over our hypothesis space and use this as the update step.