Linear Models for Regression

Question 1

What is regression?

Answer 1

The problem in which we have a set of points from a function and we want to aproximate that function without knowing the original function.

Question 2

What is the relationship between regression and classification?

Answer 2

Any regression problem can be blassed as a classification one. Each point for regression lies on the decision boundary (we just need to string them together)

Question 3

What are the 3 basis for finding a function?

Answer 3

> Polynomial basis

> “Gaussian” basis

> Sigmoid basis

Question 4

What is polynomial basis?

Answer 4

ϕ_i(x) = xⁱ

y(x,w) = w₀ + w₁x + w₂x² + … + w_Mx^M

Set ϕ₀ = 1

y(x,w) = w₀ + ∑ (w_iϕ_i(x)) = w^Tϕ(x)

Question 5

What is Gaussian basis?

Answer 5

ϕ_i = e^(-(x - μ_i )² / (2s² ))

Question 6

What is the sigmoid basis?

Answer 6

ϕ_i(x) = σ((x - μ) / s) = σ_(u,s)

σ(a) = 1 / (1 + e^-a)

Question 7

What is the vector equation that we would use to calculate the exact solution to a regression problem and what is required for this?

Answer 7

w = Φ^-1 t

This requires that matrix, Φ, be square so we can only pick a square number of points.

Question 8

What does overdetermined mean? and what impact does this have?

Answer 8

This is when there are more equations than varaibles and there is no exact solution possible. Instead we can define an error to minise this.

Question 9

What is the equation for error?

Answer 9

E = 0.5 ∑(ϕ_i^Tw - t_i )²

Question 10

Derrive the equations for the least squares solution

Answer 10

E = 0.5 ∑ (ϕ_i^Tw - t_i )²

∇E = ∑ ϕ_i(ϕ_i^Tw - t_i)

This will be minimum when the gradient ∇E = 0

ϕ^T(ϕw - t) = 0

ϕ^Tϕw = ϕ^Tt

w = (ϕ^Tϕw)^-1 ϕ^Tt

ϕ_p = (ϕ^Tϕw)^-1 ϕ^T

w = ϕ_pt

Question 11

What is the data set and equation form and what are the steps for sum of squares solution?

Answer 11

Data set:

• (x, t)
• (x_value, target)
• Equation:
• Example: y = w₀ + w₁f(x)

Basis:

• Polynomial
• Gaussian
• Sigmoid

Step 1: Calculate ϕ using the basis and bias

ϕ = [bias, basis]

[… , …]

Step 2: Compute ϕ^Tϕ

Step 3: Compute (ϕ^Tϕ)^-1

Step 4: Compute ϕ_p = (ϕ^Tϕ)^-1ϕ^T

Step 5: Compute the result using the targets w = ϕ_pt

Question 12

How can sequential learning be applied to the sum of squares solution? Why do we want to do this? What is this process called?

Answer 12

With lots of points, this is a computationally expensive process so we can consider one point at a time and use gradient descent. This is the least-mean-squares algorithm

Question 13

What is the equation for the least mean squares algorithm?

Answer 13

w_t+1 = w^t - ηϕ_n^T(ϕ_nw - t_n)

Question 14

Why is the sum of squares error equation ideal?

Answer 14

Because it is convex and has one global minimum