Support Vector Machines

Question 1

Define a linear classifier margin

Answer 1

the width that the boundary could be increased by before touching a datapoint

Question 2

What is the margin of the Linear SVM

Answer 2

The maximum margin

Question 3

What are support vectors

Answer 3

The datapoints that the margin pushes up against

Question 4

Advantages of Maximum Margin

Answer 4

• if the boundary is marginally misplaced, this gives us the least chance of misclassification
• Empirically, this works very well
• model is immune to removal of any non-support-vector data points

Question 5

The hyperplane: wx + b = 0, is fully determined by ?

Answer 5

(w, b)

w = Weight Vector, b = bias term

Question 6

w is perpendicular to the plus and minus planes, how does this help us calculate the margin width?

Answer 6

We know that the distance between two points on opposite planes will be w multiplied by a constant

w . x⁺ + b = +1
w . x^- + b = -1
x^- = x⁺ + λw
|x⁺ - x^-| = M
M = 2 / ||w||

Question 7

Problems with Maximum Margin

Answer 7

• The solution can change drastically if there is an outlier
• no solution if the classes are not linearly separable

Question 8

What is the general idea of the Soft Margin SVM

Answer 8

• “Relax” the formulation to allow points to be on the “wrong” side.
• Penalize points according to how far they are on the wrong side

Question 9

How well does Soft Margin SVM do on unseen data?

Answer 9

• Depends on the training error and the number of support vectors
• When the number of support vectors is small, we can be sure that the generalization error is not much higher than the training error

Question 10

What is a vector

Answer 10

an object that has both a magnitude and a direction

Question 11

what is a vector’s norm

Answer 11

the magnitude, or length, of a vector.
Denoted ||x||

Question 12

What is a vector norm and How is it computed

Answer 12

It is the magintude, or length, of a vector.
Calculated using Euclidean norm formula;
Square root of the sum of the squared points

Question 13

Give the vector that denotes the direction of a vector

Answer 13

W = (cos(θ), cos(α))

Question 14

Dot product of n-dimensional vectors (formula)

Answer 14

x ⋅ y = SUM(i=0, n) x_iy_i

Question 15

What separates data in a) one dimension b) two dimensions c) three dimensions

Answer 15

point
line
plane

Question 16

Describe the kernel trick used in SVM and what is the mathematical reasoning that makes it work?

Answer 16

Applying a transformation ∅ to all data points before running the algorithm to find non-linear patterns
A function K(⋅, ⋅) is a kernel if there exists a function ∅(⋅) s.t.
K(x_i, x_j) = ∅(x_i) ⋅ ∅(x_j)

Question 17

Why is the kernel trick important?

Answer 17

We get the advantage of using a lot of non-linear features without the computational price

Question 18

What do SVMs learn and what approach is this called

Answer 18

SVMs learn the discrimination boundary
They are called discriminatory approaches

Question 19

What are the 3 key ideas of SVMs

Answer 19

1. Use optimisation to find solution with few errors
2. seek large margin separator to improve generalisation
3. use kernel trick to make large feature spaces efficiently (computationally)