Regression

Question 1

Is it possible to create models in Machine learning?

Answer 1

Machine learning is mainly about creating mathematical relations between features of a dataset. These mathematical relations are called models.

Question 2

How was linear regression done prior to machine learning?

Answer 2

Long long ago, it was done by mathematicians with very few datasets. In the availability of computers, we are now enabled with high computational powers and hence we can prepare linear regression models with any amount of data in today’s world.

Question 3

What was fearless about ML?

Answer 3

It’s a reference that ML practitioners usually don’t spend too much time evaluating the statistical validity of a method; they prefer to just create a model and evaluate its performance, rather than thinking about the statistical validity of the process.

Question 4

If we don’t understand how a model’s intricacies work, how can we be 100% sure that it’s a successful model?

Answer 4

Interpretation of how the model is working mathematically is not always possible. While preparing the algorithm, its mathematical steps are derived. While applying the model it is not always possible and needed to know the steps. It is about the performance of the model on unseen data that decides the success of the model.

Question 5

Do more data points bring new insights?

Answer 5

Yes, with larger data the insights will be supported with more data and hence they will be more trustworthy. The same will be the quality of model training. With more data, models train better.

Question 6

Why is the model interpretation important?

Answer 6

Model interpretation is important because it gives an understanding of how the model reached the final results/predictions. It gives some clarity and indication about if some changes have to be done to make the model perform better.

Question 7

What are the features of the machine learning model?

Answer 7

Features are the attributes or variables that are used in a machine learning model. Generally, there are independent and dependent features. Independent features are the input features of the model while the dependent feature is the output variable.

Question 8

Is risk similar to variance?

Answer 8

For a machine learning model, the higher the error more will be chance of its failure over the unseen data and it will be a measure of the risk of the model. So risk can be interpreted as the error associated with the model and hence minimizing risk is equivalent to minimizing error or variance.

Question 9

What are overfitting and underfitting?

Answer 9

Overfitting is a case when a machine learning model follows the noises of the data too much. It becomes too complex to make good predictions on the unseen data. Underfitting is the opposite case where the model is simple enough to capture the patterns of the data and hence it is also unable to generalize over the unseen data.

Question 10

What does interpolation mean?

Answer 10

Interpolation is a statistical method by which related known values are used to estimate an unknown value. It is useful in treating missing values in machine learning.

Question 11

Why do we square the residual?

Answer 11

It is for mathematical purposes. If we don’t square them then the positive values and negative values cancel out each other and we will still get the error as zero or close to zero which is wrong.

Question 12

What is the limit of the error made by a regression model?

Answer 12

There is no existing defined limit on errors by a linear regression model. The only condition is that the sum squared of error terms should be minimum.

Question 13

Can we use either normalization or standardization?

Answer 13

We use either standardization or normalization as it brings all the variables to a uniform scale. If we don’t do that there is a chance a less important variable will be given more priority.

Question 14

Is it acceptable if the slope of a variable is very small in the model?

Answer 14

The slope of any variable tells about how the dependent variable would change for a unit increase in the independent variable while keeping other variables constant. The larger the value of slope higher the change and Vice versa. A small slope will indicate that a certain feature is not influencing the target variable and that can be removed.

Question 15

Is that always good to add more variables that could potentially have an influence on the outcome?

Answer 15

Yes, it is good to add more variables that are influencing the outcome. It will make the predictions more generic and reliable over unseen data.

Question 16

Is there a way to validate the independence of error terms?

Answer 16

Yes, to do that hypothesis testing is done.

Question 17

What do we mean by “linear predictor”?

Answer 17

A linear predictor means a linear relationship between the output and the input variables. The term linear is associated with the coefficients/parameters of the model. Every equation that is not following the linearity in parameters is a non-linear relation.

Question 18

Don’t we divide the TSS by ‘n’ so we have something somehow ‘independent’ from the number of samples we take?

Answer 18

Dividing the total sum of squares with n will not make any sense because the residual sum of squares will also be divided by n. As we are considering the ratio between them we do not need to divide TSS with n.

Question 19

Can we interpret R² as a measure to understand how linear our dataset is?

Answer 19

No, R² is used to explain how much our model can capture the variance in the data. The higher the value better is the performance of the model.

Question 20

How to choose between Normalization and Standardization?

Answer 20

Normalization, typically means that the range of values is “normalized to be from 0 to 1” while Standardization typically means that the range of values is standardized to measure how many standard deviations the value is from its mean.

Question 21

What is an acceptable R-squared in the real world?

Answer 21

Higher the value of R-square better is the fit of the model over the given data. This is because a higher r-squared value implies a lower residual sum of squares.

Question 22

Do we call it R squared because the square is mathematically useful somewhere else?

Answer 22

R is the correlation between the predicted values and the observed values of Y. R square is the square of this coefficient and indicates the percentage of variation explained by your regression line out of the total variation.

Question 23

Can we discriminate the features before the training? like finding the correlation between features against labels?

Answer 23

Yes, we can find the correlation between the variables and if any 2 variables are highly correlated then we drop one of them as it is redundant to our model and adds no new information to the model we develop. Along with that by doing exploratory data analysis we can remove variables based on many criteria.

Question 24

Do we need to test for multicollinearity in case of more variables?

Answer 24

Yes, we do need to check for multicollinearity before we develop a model. We drop the redundant variables. If we have a VIF value for a variable greater than 5 we generally drop those variables.

Question 25

Is Linear Regression a good base model to compare other methods?

Answer 25

Yes. It is always good to develop a base model as Linear regression for continuous variables and then we can move to more complicated algorithms such as Boosting, neural nets, etc.

Question 26

How to estimate the parameters in the model?

Answer 26

To estimate the parameters in the model, we minimize the sum square of errors in which the parameters are treated as variables. By applying optimization algorithms we can estimate them.

Question 27

Do we iterate to reduce the distance from the true value?

Answer 27

Yes, if a model is not performing well in the first go, it means it is not close enough to the true line. In such a case we do iterate by making changes in the variables of the model, in the data, or hyper-parameters of the model.

Question 28

How reliable is the prediction if the user provides a clean dataset to run the training?

Answer 28

Having clean data is good for the machine learning model, but it is not the only factor that decides the reliability of the model. Along with clean data, the features selected in the model, its performance over unseen data are things that make the model reliable.

Question 29

How do we know if there is bias in our estimation and how to measure it?

Answer 29

Bias can be interpreted as the training error in the model. If the training error is very high, then the model is biased and simple enough to perform reliably on the unseen data.

Question 30

Is there a book or website that you recommend on today’s concepts?

Answer 30

You can refer to ISLR (An Introduction to Statistical Learning) by Gareth James and co-authors.

Question 31

Are confidence interval and confidence band the same?

Answer 31

They are two different representations of the same thing. A confidence band is the lines on a probability plot or fitted line plot that depict the upper and lower confidence bounds for all points on a fitted line within the range of data. On a fitted line plot, the confidence interval for the mean response of a specified predictor value is the points on the confidence bands directly above and below the predictor value.

Question 32

Can you describe a little bit about the difference between MLE and OLS? Especially under what kind of context we might favor one over the other?

Answer 32

The ordinary least squares, or OLS, can also be called the linear least squares. This is a method for approximately determining the unknown parameters located in a linear regression model. Maximum likelihood estimation, or MLE, is a method used in estimating the parameters of a statistical model and for fitting a statistical model to data.

Question 33

Does R-square error increase with the addition of new features?

Answer 33

Yes, the r-square value increases with the addition of new features.

Question 34

Can we use logarithmic regression in machine learning?

Answer 34

Yes, if the data is fitting better over a logarithmic equation then it can be fit on that. Machine learning bothers about what relation is fitting the best to the given data.

Question 35

Is standardization the same as scaling?

Answer 35

To standardize data is about how many standard deviations the data point is away from the mean of the data. Standardization is just a technique to perform feature scaling.

Question 36

How do we know that linear regression is the wrong choice for a problem?

Answer 36

If the trained linear regression model is not doing good over the unseen data after all the possible modifications then it is not a good choice for that specific problems and one should prefer to choose from other algorithms suitable for regression.

Question 37

PCA gives us coefficients to multiply with the variables (X) to find the Y, is it some sort of regression?

Answer 37

Even though PCA does that, it is not a supervised learning method and hence it does not do any sort of regression. The computations of PCA are different from that of linear regression.

Question 38

Can you use PCA to reduce dimension for your dataset for regression?

Answer 38

The main purpose of PCA is to reduce the dimension (number of features) from the given dataset. It is applied mainly when the number of features is very high. It is not used for regression.

Question 39

Is ML always a black box?

Answer 39

It’s not always a black box. It’s generally a scale of interpretability, some ML algorithms like Linear Regression and Decision Trees are highly interpretable, others like Neural Networks are not very interpretable.

Question 40

How is ML different from a model?

Answer 40

Machine learning algorithms are procedures that are implemented in code and are run on data. Machine learning models are output by algorithms and are comprised of model data and a prediction algorithm.

Question 41

What is the difference between linear and non-linear regression?

Answer 41

Nonlinear regression is a form of regression analysis where we model data in a nonlinear combination of model parameters. But in linear regression, we have a linear combination of parameters.

Question 42

What does the common notion that “Theory doesn’t always explain the success” mean?

Answer 42

There can be instances where the model can give very good performance on some data but might not be suitable for the data as per the theory of the algorithms. For example, deep neural networks work very nicely on large datasets but it is not very obvious ‘why’ it worked on the basis of mathematical theory.

Question 43

When you were comparing building a model to just making a prediction, were you referring to having interpretable coefficients?

Answer 43

No, if the purpose of the model is simply to make predictions and provide insights on the problem, then it is not important for the coefficients to be interpretable. On the other hand, if we want the model to be interpretable, then it is of utmost importance we know what parameters are we using and they must be interpretable to us.

Question 44

How accurate does an ML algorithm need to be to be considered a correct model?

Answer 44

A data scientist’s job is to strive for as high accuracy as possible, but there is no pre-stated benchmark to consider a model ‘correct’. It depends on the problem at hand and business requirements.

Question 45

What is the difference between Vector data records and vector components?

Answer 45

A vector data record is an input array where particular values at particular positions are signifying particular coefficients. For example, if I want to express length and breadth in an array with the format [L, B], then [5, 2] is a vector data record. Whereas vector data components refer to each value within that vector, for example, 5 and 2 are two components in the previous example.

Question 46

Does the time and situations where advertisements were done matter?

Answer 46

Yes, it might be the case but in our scenario, we are more concerned with the overall effects of each kind of advertisement we have had so far.

Question 47

What’s the idea behind degrees of freedom?

Answer 47

In machine learning, the degrees of freedom refer to the number of parameters in the model. In linear regression, it refers to the number of coefficients. If there are more degrees of freedom (model parameters) in machine learning, then the model is expected to overfit the training dataset.

Question 48

What is the degree of freedom for regression?

Answer 48

For a simple linear regression with one variable, there are two degrees of freedom, the slope, and the intercept. If there are multiple variables, the number of degrees of freedom increases as the number of slope parameters used to describe the line in the n-dimensional space increases. So, if there are n variables, then there are n+1 degrees of freedom (intercept + slopes).

Question 49

Are Theta and X_i two vector components that give X at given I?

Answer 49

Theta is the vector denoting the slope parameters, and X stands for our data. So the product of these two matrices should ideally give us the predicted value of y (the target variable) at given x.

Question 50

Is residual vertical? or orthogonal to the line?

Answer 50

Residual is the difference between the actual value of the target variable and the predicted values. Hence, it is always vertical.

Question 51

Does theta represent a different slope?

Answer 51

Theta is the vector denoting the slope parameters. Theta has components - Theta0, Theta1 … Theta-m

Question 52

Why did we add the 1’s into the X vector? Was that to help with some of the math we’re doing here somehow?

Answer 52

We need to add 1 to the vector of X’s because that way, on multiplying the theta vector with the X vector we automatically get the intercept.

Question 53

What happens when the relationship is nonlinear?

Answer 53

Linear Regression only tries to find the “line of best fit” i.e. the best possible line that explains the pattern. We are assuming that there is a linear relationship between the independent and dependent variables. If a non-linear pattern is better suited, then more complex models can be built.

Question 54

How often do we actually have linear relationships?

Answer 54

It depends on the problem on hand. Linear relationships are more interpretable than any other model, but as the complexity of the problem increases, the linear relationship between independent and dependent variables is observed less often.

Question 55

From the estimation - would that mean if there is no advertising on TV, radio, or news, there is already a sale of 2.94?

Answer 55

Yes, the intercept of the model implies that there’s already a sales of 2.94 even if there is no advertising on TV, radio, and /or news.

Question 56

Do we need to perform a hypothesis test for theta_not?

Answer 56

Yes, it is advised we find confidence intervals and perform a hypothesis test on each coefficient to check whether that variable is significant for our model or not.

Question 57

If the newspaper does not help in estimating sales, can we remove it?

Answer 57

Yes, the evaluation metric Adjusted R-squared gives good intuition about the usefulness of each feature contributing to the final prediction.

Question 58

There may be 10 independent variables to a target variable. How do we choose the right dependent variable to a target variable?

Answer 58

We need to see if all ten of these target variables are significantly contributing enough to the target variables. If we want to reduce the dimension of feature space, we can do PCA to do dimensionality reduction.

Question 59

Will more samples will result in the slopes of the lines converging?

Answer 59

Yes, the more the sample data, the closer we are to a perfect prediction graph. So the slopes for our line start to converge towards their optimal value.

Question 60

What is W_i in the structural model?

Answer 60

W_i stands for noise which is normally distributed and independent.

Question 61

Do we only need to assume W_i is Gaussian or we should assume all variables are jointly Gaussian?

Answer 61

We only assume W_i as normal. There is no such restriction for the independent and dependent variables.

Question 62

What is Wi and what if it’s not normal?

Answer 62

Wi is the error of regression. It should be normally distributed. This is the basic assumption which should not be violated. Maximum likelihood estimation does not work if this assumption doesn’t hold true, in which case we should not use linear regression.

Question 63

Can we use prior information? For example, maybe we know a coefficient must be positive in our model

Answer 63

Yes, it is usable. We can do by using bayesian model with the prior probabilities

Question 64

Is there another distribution apart from the normal one that can be used for the maximum likelihood method?

Answer 64

In principle, you can do that, but you won’t get OLS as a result. 200+ years ago, Laplace (who kind of started this study) used Laplace distribution which later turned out to be not as good as normal.

Question 65

In some experimental data analysis, I got a negative R^2. Is it possible at all?

Answer 65

The baseline model for linear regression is the model where the prediction is the average value of the target value for any value of X. The Negative R2 implies that it’s worse than the baseline model.

Question 66

What is a good correlation to say that there is a strong correlation?

Answer 66

It depends on the problem at hand but generally, a value greater than 0.7 is considered a strong correlation.

Question 67

What is the range for the value of R squared?

Answer 67

R-Squared indicates the percentage of the variance in the dependent variable that the independent variables explain collectively. Hence, the value of R-squared lies between 0 and 1.

Question 68

Does more fitting have a bigger R square?

Answer 68

In overfitting, we would likely get a high R-Squared on the training data but a relatively small R-Squared on the test/validation data.

Question 69

Should we standardize the data before calculating regression coefficients?

Answer 69

It is useful to standardize data to make it computationally feasible but it’s not great for interpretation.

Question 70

Is the variation due to the randomness in the sample, or in the population itself?

Answer 70

Variation exists both in-sample and in population. We try to capture the true statistics of the population using the sample statistics.

Question 71

If we do not reject the null hypothesis, do we eliminate the corresponding variable and rerun the regression?

Answer 71

Yes, we need to re-run regression after eliminating insignificant variables.

Question 72

Wouldn’t it be more logical to make the null hypothesis theta > zero?

Answer 72

In that case, it would be difficult to conclude with rejection as there are many possible values we can test for.

Question 73

Is there a fundamental difference between a z-test and a wald test?

Answer 73

No, they are essentially the same tests.

Question 74

Is there a Bayesian equivalent to this method?

Answer 74

Yes - there is a bayesian variation for almost everything.

Question 75

Why don’t we use a t-test for coefficients?

Answer 75

We use a t-test, instead of a z-test, to check the significance of independent variables because the population variance is unknown.

Question 76

Do we perform dimensionality reduction before performing supervised learning?

Answer 76

Yes, it is generally a good idea when you have a lot of variables and the interpretation of variables is not required.

Question 77

For use in science say one is trying to determine a model with known inputs and outputs but unknown interactions. if we fit a model to data that has been collected over years of experimentation from multiple events/experiments. what is the best test to provide how well the model fits?

Answer 77

The most important part of research is to keep iterating experiments with little changes to seek the best performance that we can achieve. As the professor, showed how different equations fit the data differently and yield different accuracies. So no set method can be called ‘best’. It is about identifying which test would be the most relevant in what situation.

Question 78

If a model works better for some ranges of Xs, would it be valid to have multiple linear regressions for specific ranges of X?

Answer 78

Yes, but linear but assumption is broken then. So, one way is to use k spline linear regression or can try other non-linear algorithms.

Question 79

Why is it called “supervised” learning?

Answer 79

A supervised machine learning algorithm is trained on input data that has been labeled for a particular output. The name “supervised” learning originates from the idea that training this type of algorithm is like having a teacher supervise the whole process.

Question 80

What are some examples of contexts in which ML and traditional statistical models differ?

Answer 80

Machine Learning models and Statistical models are just two flavors of the same thing. ML models are generally algorithmic, and they help in prediction. We can fearlessly apply machine learning algorithms on a dataset to get predictions, while statistical models tend to be more mathematical and help to generate statistical insights from the data, rather than only trying to make predictions on it.

Question 81

What does it mean when we say that when it comes to Linear Regression, the two paths (the Machine Learning path and the Statistics/Probability path) are the same?

Answer 81

In Linear Regression, while using the statistical path we use ordinary least squares to get the final result, while in linear regression using machine learning, we try to optimize the error term using a gradient descent optimization algorithm to get the final result. But it doesn’t matter how you approach the problem - you will get similar results.

Question 82

Is estimator the same as likelihood?

Answer 82

No. An estimator is a statistic used to estimate an unknown parameter and likelihood is the probability of getting a result for a given value of the parameters.

Question 83

What is meant by noisy data?

Answer 83

Noisy data is meaningless data that cannot be understood and interpreted correctly. For example, negative or zero values for height variables can be considered as noisy data.

Question 84

What is the purpose of the estimator?

Answer 84

The purpose of the estimator is to find estimates of coefficients (beta0, beta1,….) that in some sense define the “best” fit line for the data.

Question 85

Can a distribution have multiple parameters (thetas)?

Answer 85

Yes, we allow theta to be a vector of multiple components that characterizes that distribution for eg: mean and standard deviation in Normal distribution.

Question 86

Does the estimator include outliers?

Answer 86

The estimator has nothing to do with data. If the data has outliers, it takes them into account. The main aim of estimators is to estimate parameters based on the data.

Question 87

Is there a minimum number of data records required to build an estimator?

Answer 87

No, not necessarily, but a good amount of data helps to improve the quality of the estimates. You can estimate even if you have just one measurement, but it might be a poor estimate.

Question 88

Let’s say we are interested more in the outliers and understanding the outliers more. Would that be less Machine Learning and more of a Statistical Testing/Modelling based approach?

Answer 88

Outliers are extreme values that deviate from other observations on the data. Identification of outliers is important for the following reasons. 1. An outlier may indicate bad data. For example, the data may have been coded incorrectly or an experiment may not have been run correctly 2. Outliers may occur due to random variation or may indicate something scientifically interesting. Identifying outliers are part of Exploratory Data Analysis(EDA) techniques rather than applying Machine Learning algorithms (or) Statistical testing.

Question 89

At what stage do you consider computational capacity before the model building or after?

Answer 89

The computational capacity mainly depends on the size of the data and the complexity of the algorithm. Generally, computations are not a big concern for simple algorithms like linear regression, but if we want to reduce the computational time, we can build the model on a sample of the original data instead of full data.

Question 90

What is overfitting?

Answer 90

Overfitting occurs when a statistical model fits exactly against its training data including the noise in the data. When this happens, the algorithm cannot perform accurately against unseen data, defeating its purpose. The generalization of a model to new data is ultimately what allows us to use machine learning algorithms to make predictions.

Question 91

Is eliminating outliers necessary in Linear Regression?

Answer 91

The slope of the regression line will change due to outliers in most of the cases. So, linear regression is sensitive to outliers, and it is good to eliminate outliers before building the linear regression model.

Question 92

What is the degree of freedom in the context of regression?

Answer 92

In machine learning, the degrees of freedom refer to the number of parameters in the model. In linear regression, it refers to the number of coefficients. If there are more degrees of freedom (model parameters) in machine learning, then the model is expected to overfit the training dataset.

Question 93

Why are we trying to minimize theta - the vector of coefficients?

Answer 93

We are finding the theta that minimizes the sum of squares of residuals because we need the predicted value to be as close as possible to the true value.

Question 94

Wouldn’t the number of coefficients be the same as the number of independent variables ?

Answer 94

Nearly, because we also have one additional coefficient called the intercept of regression. The intercept (often labeled as constant or theta_0) is the mean of the dependent variable when you set all of the independent variables in your model to zero. Having an intercept gives our model the freedom to capture ALL the linear patterns while a model with no intercept can capture only those patterns that pass through the origin.

Question 95

What is the difference between Simple Linear Regression and Multiple Linear Regression?

Answer 95

Simple linear regression has one independent variable and one dependent variable. Multiple regression is performed when we have more than one independent variable, but only one dependent variable.

Question 96

How do we decide which variables are important in Regression?

Answer 96

You can’t compare the regular regression coefficients because they use different scales. Fit the regression model using the standardized independent variables and compare the standardized coefficients. Because they all use the same scale, you can compare them directly. Standardized coefficients signify the mean change of the dependent variable given a one standard deviation shift in an independent variable.

Question 97

What if we have data points that are distributed randomly around a sin() wave, the line will be a wrong solution, right? How do we create more complicated prediction functions?

Answer 97

Linear regression does not work well in these cases as it simply provides the line of best fit, and cannot capture complex non-linear relationships in the data. So we would need to use more sophisticated algorithms capable of detecting non-linear patterns in the data, such as neural networks.

Question 98

What is Wi and what if it’s not normal?

Answer 98

Wi is the error of regression. It should be normally distributed. This is the basic assumption which should not be violated. Maximum likelihood estimation does not work if this assumption doesn’t hold true, in which case we should not use linear regression.

Question 99

Is there any test that determines the p-value for each coefficient (theta)?

Answer 99

The ‘t-tests’ are used to conduct hypothesis tests on the regression coefficients (thetas) obtained in linear regression. The p-value for each variable tests the null hypothesis that the coefficient is equal to zero (no effect). A low p-value (< 0.05) indicates that you can reject the null hypothesis. We can use these coefficient p-values to determine which variables are significant in the regression model.

Question 100

What are Ȳ (Y bar) and Ŷ (Y hat)?

Answer 100

Y bar is the mean of all the Yi, the bar symbol indicates the mean and Y hat is the predicted value by the model.

Question 101

What is the total sum of squares in regression?

Answer 101

The total sum of squares, denoted TSS, is the squared differences between the observed dependent variable and its mean. You can think of this as the dispersion of the observed variables around the mean (Y bar).

Question 102

Are RSS and Wi the same in regression?

Answer 102

RSS is the sum of the differences between the predicted value and the mean of the dependent variable. Think of it as a measure that describes how well our line fits the data. Wi is the difference between any data point and the regression line.

Question 103

What is R squared and what it means when the value is 1?

Answer 103

R2 is a statistic that will give some information about the goodness of fit of a model. In regression, the R2 coefficient of determination is a statistical measure of how well the regression predictions approximate the real data points. An R2 of 1 indicates that the regression predictions perfectly fit the data.

Question 104

What do the diagonal elements represent in the covariance matrix?

Answer 104

The covariance matrix is a symmetric matrix that shows covariances of each pair of variables. The diagonal element shows the covariance of a variable with itself which nothing but the variance of that variable. So, each diagonal element represents the variance of the respective variable.

Question 105

Is it possible or common to see the p-value high but the CI band not including the value zero? And vice versa?

Answer 105

No, the two things are equivalent. If the confidence interval includes zero, that means there is no statistically meaningful or statistically significant proof that the variable helps to predict the target variable. It is the same as saying the p-value is high (>0.05). Similarly, when CI doesn’t capture zero the p-value will be low.

Question 106

How do we decide when to use linear regression or other methods?

Answer 106

Linear regression is a basic model and it can be the first model to try for a regression problem. The linear regression model is widely used in many situations before attempting non-linear and more complicated models. It is the most accomplished theoretical model and helps in interpretability; some key concepts can also be explained well using the linear regression model.
In practice, we use multiple models of different kinds and the algorithm that gives the best results depends on the data and the problem on hand.