Regression

Question 1

How can Linear Regression be extended?

Answer 1

With regularisers (L2 Regularisation)

Question 2

What is regression even for?

Answer 2

For predicting continuous classes; where classification fails

Question 3

What is Linear Regression?

Answer 3

Linear Regression is an attempt to build a linear model to predict the target values, by finding a weight for each attribute.

Captures a relationship between two variables or attributes
It makes the assumption that there is a linear relationship between the two variables

x = W0 + sum of WiAi

x = class
w* are the weights
a* are the attribute values

Question 4

How to choose the best line for linear regression?

Answer 4

option 1) Finding the line that minimises the distance between all points and the line
- Euclidean distance: d(a,b) = sqrt(sumof(ai - bi)^2))

option 2) Least squares estimation: finding the line that minimises the sum of the squares of the vertical distances between approximated/predicted and observed
- minimise the Residual Sum of Squares (RSS) -> aka Sum of Square Errors (SSE):
RSS(Beta) = Sumof(yi - betaxi)^2

Question 5

Which metric to use for linear regression to find the best line?

Answer 5

• Actual choice of metric isn’t that important, they’re all pretty stable

Just use either

• Root mean-squared error
• Root relative squared error
• Correlation coefficient

Question 6

WTF is a Regression Tree?

Answer 6

Extension of Decision Trees, where the “class” (value) at each leaf is calculated by averaging over the values of all instances at that node

Question 7

WTF is a Model Tree?

Answer 7

Generalised regression trees where the class at each leaf is calculated via linear regression over training instances at that node

Basically partitioning our data set and applying linear regression to each partition

As you work down the tree, the result at each leaf node is which linear regression model to use on our data set

Question 8

Regression vs Model trees

Answer 8

Model trees have advantages over regression trees in both compactness and prediction accuracy, because model trees can exploit local linearity in the data

Regression trees will never give a predicted value lying outside the range observed in the training cases, whereas model trees can extrapolate

Question 9

How to translate a regression task into a simple classification task?

Answer 9

Can map a continuous class onto discrete classes via DISCRETISATION

• Set range of continuous variables that corresponds to each discrete class

Question 10

How to translate a classification task into a suite of regression task?

Answer 10

MULTI-RESPONSE LINEAR REGRESSION

• Perform one regression per discrete class
  
  • With all positive instances set to 1 and all negative instances set to 0
• Classify a given test instance by estimating its value relative to each class, and selecting the class with the highest value
• Approximates a numeric membership function for each class

Question 11

WTF is Maximum Likelihood Estimation?

Answer 11

Goal is to search for a value of Beta so that the probability P(y = 1 | x) = hbeta(X) is large when x belongs to the “1” class and small when x belongs to the “0” class (so that P(y = 0 | x) is large)

Question 12

Linear Regression uses gradient descent, what does Logistic Regression use?

Answer 12

Tries to maximise so uses gradient ascent

Question 13

Can Logistic Regression be applied to multi-class classification?

Answer 13

By default, no, only for binary classification.

However, can extend to multi-classification by assuming a multinomial distribution.
- Mutlinomial Logistic Regression
Applies softmax - a generalisation of the logistic function to J dimensions.
- Results in a J-Dimensional vector of real values in the range (0,1) that add up to 1

Question 14

Logistic Regression Pros and Cons

Answer 14

PROS

• Simple yet low-bias classifier
• Unlike Naive Bayes not confounded by diverse, correlated features

CONS

• Slow to train
• Some feature scaling issues
• Often needs a lot of data to work well
• Choosing a regularisation a nuisance but important since OVERFITTING is a problem - adds constraints on the parameter space

Question 15

What is Regression? How is it similar to Classification, and how is it different?

Answer 15

Regression is used for when the target attribute (class) is numeric (continuous).
Consequently, we can't assess the likelihood of each class like we can in Classification.

Question 16

How do we build a linear regression model? What is RSS and what advantage does it have over some alternatives?

Answer 16

1) Learn the weights using Gradient Descent
2) This assumes that our error function is convex; one popular alternative is to consider the sum of squared differences between our predicted values and the actual target values from the training data

Question 17

How can we use a Decision Tree to do Regression?

Answer 17

We can have a single prediction value at each leaf of the tree (this is a Regression tree)

Alternatively, we can apply a linear regression over the instances at each leaf of the tree (a Model tree)

Question 18

What is Logistic Regression?

Answer 18

Logistic Regression is an attempt to build a model where the target is close to “1” for positive instances of the class, and close to “0” for the negative instances of the class

Question 19

How is Logistic Regression similar to Naïve Bayes and how is it different?

Answer 19

Both Naïve Bayes and Logistic Regression are attempting to find the class c for a test instance T, by maximising P(c | T)

In Naïve Bayes, we make some simplifying assumptions, most notably, that the attributes are conditionally independent of the class labels - hence the product of all the attributes

In Logistic Regression, we attempt to model this directly, without the simplifying assumptions of independence. This is possible because we don’t attempt to generate class possibilities; only attempt to discriminate amongst the various classes

Question 20

What is “Logistic”? What are we “Regressing”?

Answer 20

Logistic function = 1 / (1 + e^m)

Regression output = (beta dotprod x)

Question 21

How do we train a Logistic Regression model? In particular, what is the significance of the following argmaxBeta(sumof(yiloghbeta(xi) + (1 - yi)log(1-hbeta(xi)))

Answer 21

To train, use the logistic function y = 1 / (1 + exp^(-betax))

• This means we want the output of the linear regression to be positive when the target class is 1, and negative when the target class is 0

Question 22

How can Nearest Neighbour/Prototype classifiers be applied to regression tasks?

Answer 22

K-NN models can be applied directly through simply weighted combination of the continuous labels associated with the nearest neighbour.

However, it is less clear for Nearest Prototype models.

Question 23

How to do Naïve Bayes for continuous-valued features?

Answer 23

Probability Density Functions

Question 24

Can Naïve Bayes be applied to regression?

Answer 24

Yes, but it’s bad

Question 25

How to deal with continuous features in Decision Trees?

Answer 25

Information-Based Supervised Discretisation

Question 26

Can Decision Trees be applied to regression?

Answer 26

Yes, Regression Trees and Model Trees

Question 27

How to deal with continuous features in SVMs?

Answer 27

Like NN/NP, SVMs natively handle continuous-valued features.

Question 28

Can SVMs be applied to regression?

Answer 28

Yes, Support Vector Regressors

Modelling a tube/street to pick up all the training data, don’t want to pick up any of the support vectors? (I think)

Question 29

How to deal with continuous features in Logistic Regression?

Answer 29

Similar to Naïve Bayes, logistic regression models are usually defined in terms of discrete features, but, kernel density functions can be used to handle continuous features.

Question 30

Why is Logistic Regression considered a classification model?

Answer 30

Because of the use of a binary decision rule, however it very much is a regression model.