Session 9- Logistic Regression

Question 1

What is the basic principle of penalised methods?

Answer 1

Improve prediction accuracy by reducing the variability of the regression estimates at the cost of increased bias

Question 2

Penalised/ Regularized Regression allows what?

Answer 2

Automatic variable selection by shrinkage:

The coefficients of the weaker predictors are shrunk towards zero.
- useful for high dimensional data if the number of variables is close or larger than sample size

• deals effectively with multi-collinearity

Question 3

Variance bias trade off due to shrinkage usually results what?

Answer 3

A model which predicts better unseen case than normal regression

Question 4

What is a big advantage of regularized regression methods especially if using lasso or elastic net?

Answer 4

They also perform automatic variable selection

Question 5

What is
popular if have small number of variables but some of them are highly correlated to get a more stable result/model?

Answer 5

Ridge

Question 6

What can be easily extended from linear regression for continuous outcomes to a larger class of models, including:
Logistic, Multinomial, Ordinal, Poisson and Cox regressions?

Answer 6

Regularized Logistic Regression

Question 7

Regularized Logistic Regression is usually referred to as what?

Answer 7

Generalized Linear Models

Question 8

What does Logistic regression allow modelling of?

Answer 8

Relationship between binary outcomes, such as recurrence of psychoses (yes/no), as a non-linear function of our predictor variables.

Other models can be fitted in a similar way, for example time to event data using Cox-regression

Question 9

What are problems with linear regression?

Answer 9

Does not make sense for one binary outcomes

We rather assume a nonlinear relationship

Output variable is limited to [0,1], but with linear regression some of our observations are outside this range

Our goal is to separate best the two groups not to minimize MSE. Linear regression would be highly sensitive to influential cases

Assumptions of linear regression are violated (esp. homogeneity of variances) and hence inference is not valid

Question 10

What is an s-shaped/ sigmoid relationship?

Answer 10

A non-linear relationship

Question 11

How can a sigmoid relationship be modelled by the logistic function?

Answer 11

f(x)= 𝒆𝒂+𝒙)/(𝒆𝒂+𝒙+𝟏)

Question 12

How is Logistic function flexible?

Answer 12

If vary alpha will have different logistic functions

If vary x can also obtain very different functions - If change x/Beta e.g., our age relationship can vary

Question 13

The simple linear regression describes the linear relationship between the outcome y and the predictor variable x.
𝜖 describes the random component (error) and is assumed to be normal distributed.

In a logistic regression we….

Answer 13

Relate x and y by way of a function, known as link function g():
g(𝑦)=𝛽0+𝛽1 𝑥1+𝜖

so that we can model a linear relationship between the left and right hand side of the equation.

Question 14

We use the so-called logistic function to produce our models output:

P(Y=1|X)= 𝑒𝛽0+𝛽1 𝑥1 /𝑒𝛽0+𝛽1 𝑥1 +1)

Here, we are modelling the probability that our outcome belongs to class 1 given the input feature variable 𝑋1.

1. What is the underlying probability distribution of this logistic model?
2. How can we rearrange this function? and what is the term on the left side known as?

Answer 14

1. The Bernoulli distribution.
2. 𝑙𝑛⌊(𝑃(𝑌=1|𝑋)/1−(𝑃(𝑌=1|𝑋)⌋=𝛽0+𝛽1 𝑥1

Term on the left side logit-link function!

Question 15

We use the so-called logistic function to produce our models output:

P(Y=1|X)= 𝑒𝛽0+𝛽1 𝑥1 /𝑒𝛽0+𝛽1 𝑥1 +1)

Here, we are modelling the probability that our outcome belongs to class 1 given the input feature variable 𝑋1.

1. What is the underlying probability distribution of this logistic model?
2. How can we rearrange this function? and what is the term on the left side known as?

Answer 15

1. The Bernoulli distribution.
2. 𝑙𝑛⌊(𝑃(𝑌=1|𝑋)/1−(𝑃(𝑌=1|𝑋)⌋=𝛽0+𝛽1 𝑥1

Term on the left side logit-link function!

Question 16

In linear regression we estimated the coefficients 𝛽𝑖 by minimizing the sum of the squared error terms.

How can we estimate parameters in logistic regression?

Answer 16

Maximizing the likelihood of the data!

Likelihood of an observation is the probability of seeing that observation under a particular model

In logistic regression, the likelihood of seeing an observation for class 1 is P(Y|X), for seeing an observation in class 0 is 1-P(Y|X).

Question 17

What does the Maximum Likelihood estimation choose?

Answer 17

Parameter that maximizes the likelihood of the model given the data.

It is computational easier to work with the natural log of likelihoods (Log Likelihood, LL) rather than the likelihoods themselves.