Classification: Linear Models II

Question 1

What can the fisher discriminant be used for?

Answer 1

Dimensionality reduction: automatically find low-dimensional projections that preserve most of the information/structure in the data

Different rojections preserve different amounts of the data structure

Question 2

What are interesting comparisons with the fisher discriminant, least squares and the perceptron?

Answer 2

The most interesting things are shown in this figure

Question 3

What are the basic principles of the fisher criterion?

Answer 3

Principle: find 1-dimensional projection w, such that clusters defined by class labels are well separated and cohesive

Question 4

Explain the gaussian mixture

Answer 4

Quick note: what is a guassian?

https://gyazo.com/c20b13f2eaaeeeef7ba81aaae246e10a

1. a guassian normal distribution,
2. a mean (myg)
3. an variance (co-variance). Which is used for the shape of the guassian.

Now a guassian mixture is a collection of guassians including some mixing coefficient (lambda) (which is the size of the gaussian)

https://gyazo.com/08d8525a3a9a663ce4ddeea0689ec73f

We can now look at the data and place clusters we think is there, and draw lines to understand the data.

Place a reasonable amount of clusters, and use the EM algorithm for mixed guassians to find the best fit for these clusters

Example:

https://youtu.be/qMTuMa86NzU?t=309

Question 5

What can we say if the covariance matrices for all clusters are the same, in a mixed gussian model?

Answer 5

That means the decision boundries are linear!

Question 6

What are the basics and use cases of logistic regression?

Answer 6

https://gyazo.com/59b5e9b6a703f6c9fbfc2a9c2cf1efd7

one or more independent input variables that can be distrece or nondiscrete.

Always one binary output variable

The goal is to estimate p for a linear combination of independent variables. In order to tie together our linear combination of variables, we use the sigmoid function which spans from 0 to 1

The regression coefficients are calculated using maximum likelihood estimation

Question 7

What are the properties of logistic regression?

Answer 7

See figure

Question 8

Explain how we can use data transformations and how they are usefull

Answer 8

By using data transformations we can change the data from being non linear seperable to being linear seperable,

this does however often have a computing cost

An example is shown in figure where we map (x, y) to (|x|, y)