Model Evaluation Metrics

Question 1

What is the difference between noise and variance?

Answer 1

While collecting data, the data acquisition medium (either human or machine) may make errors. Such errors are called noise. Variance measures the variation of data points from the mean of the data.

Question 2

What is heterosekadascity? What is generally the shape of the heteroscedastic data?

Answer 2

Heterosekadascity is the phenomenon of having a different variance of data points along the regression line. Heteroscedastic data is generally having an irregular shape i.e. Cone.

Question 3

How does the weight help with making a prediction?

Answer 3

When weights are established in a model the relationship between the dependent and independent variables is established. This expression is called the model. To predict unknown data, the values of independent variables are passed in the model to get the dependent variable.

Question 4

Are the coefficients stand for weights in the model?

Answer 4

Yes, the coefficients established after the training process are the weights for each variable in the model. They play a vital role in making the prediction.

Question 5

Is it necessary to run PCA to resolve multicollinearity?

Answer 5

Yes, it can be but not in all cases. PCA is used to reduce the number of features when they are very high in count. This process is called dimensionality reduction. To get rid of multicollinearity one out of two correlated features can be removed for all the pairs of correlated variables.

Question 6

What is the maximum likelihood function?

Answer 6

The likelihood function is a function that is the combination of the likelihood of occurrence of all the events of a sample. When this function is maximized to get the parameters of the model then it is called the maximum likelihood function. It is a function of parameters not the variables of the data.

Question 7

In regression, is it needed to have a categorical variable?

Answer 7

A regression problem is not necessary to have only continuous or categorical variables. It is possible that it can have either only continuous variables, only categorical variables, or a mix of the two also.

Question 8

Would log(x2) and x2 have a very correlation that could cause multicollinearity issues?

Answer 8

Log(x2) is a transformed version of x2 and it is an increasing function. Creating such a column may cause multicollinearity because in the case of log transformation they will have some correlation.

Question 9

Why do we need a non-linear function added to the model?

Answer 9

It is not always possible to get a linear relationship between the dependent and independent variables. In real life, most of the time a non-linear relation captures the actual relation between the variables better. Due to this, we need non-linear functions added to the model.

Question 10

Would the product of two variables be an interaction between the features?

Answer 10

Yes, it is an interaction between the two variables. When two variables are multiplied, as an effect of interaction a new feature is generated.

Question 11

Can you take the log of Y and still have linear regression?

Answer 11

Yes, it is possible. A linear relationship between variables means that the index/power of the coefficients is not equal to 1. Linearity is seen with the index of the parameter, not with that of the variables. So Log(y) and the independent variables can have a linear relationship still.

Question 12

Are rescaling, normalizing, and standardizing are different?

Answer 12

Normalizing and standardizing are two methods of doing rescaling/scaling. In normalizing we measure how much standard deviation is the actual data away from the mean of the data while in standardization we bring back the data to a certain range of numbers.

Question 13

Can you provide an example of the weighted least square algorithm? How do we pick the weight?

Answer 13

Weighted least square algorithm is a method of finding the parameters of a model. It can be applied with any algorithm where this is used as the cost function for the algorithm for example Linear regression. The weights should, ideally, be equal to the reciprocal of the variance of the measurement. Corresponding to each record there is a different weight associated with it.

Question 14

Can you give some examples where scaling/ normalizing is necessary and where the regression will work just fine without transforming the data first?

Answer 14

Scaling and normalizing are used when different features of the dataset are at a different scale, for example, if a dataset contains weight in kg and height in meter then they have different scales and it is needed to do scaling of the data. Linear regression works relatively better if there is no big-scale difference between the features.

Question 15

Will the addition of more variables cause overfitting?

Answer 15

When new variables are added to the model then it makes the model more complex and hence it will try to capture the noise in the data while being trained. This will cause the overfitting of the model.

Question 16

If we take the log of Y, can we still account for outliers in our prediction model?

Answer 16

It is possible that even in the transformed data (log transformation) some points are far away from the main herd of the data points and they can be outliers. Transformation is not a fixed solution to outliers.

Question 17

Can a regression model include both continuous and categorical variables at the same time?

Answer 17

Yes, a regression model can include both the categorical and continuous variables at the same time. The only thing is the categories in the categorical data need to be converted into numbers so that the model can be established properly.

Question 18

Can outliers affect Linear regression?

Answer 18

In linear regression, outliers can adversely affect the prediction of the model. A variable with outliers dominates over the other variables in terms of contribution to the model. It causes to increase in the variance in the prediction and the original dataset also.

Question 19

What is a hyperparameter?

Answer 19

When we train a machine learning model some parameters are estimated during the training process. These are the model parameters. Along with them, there are some parameters that we need to pass to the model while training it. We have the freedom to pass different values to these parameters and check at what value the model is performing better. Such parameters are called hyperparameters of the model.

Question 20

What does sparsity mean?

Answer 20

Sparsity means out of a given number of values vanishing of some of the values. For example in the case of a 10x10 matrix, out of 100 entries, 60 are zero then it is a sparse matrix. In general, if the percentage of such values is high we refer to it as a sparse matrix.

Question 21

Is it a subset of the entire dataset? If the validation set and training set work well then why would we still get errors in our test set?

Answer 21

A validation set is a subset of the data that is chosen to validate the model (whether it is performing well or not). Validating a model is a process of making the model as accurate and generic as possible, but this does not mean that the accuracy or performance of the model is 100%. Irrespective of the accuracy of the model on the validation set, most of the models are bound to make an error on the test set data. This is because the model can not capture the existing pattern in the test set data properly.

Question 22

Could you randomly choose a different validation set rather than setting aside a test set?

Answer 22

One can choose the validation set randomly if desired, it is not necessary that it has to be the last few records of the dataset. While doing cross-validation in machine learning different validation sets are chosen to validate the model.

Question 23

In the k folds method, how do we combine different regression models?

Answer 23

Cross-validation is a method to find the best working model out of a set of models over a given set of data. If the average of the performance metrics of a model M1 is better than that for the model M2 then the model M1 will be preferred with the best set of parameters.

Question 24

Can simulation tell us anything about bias?

Answer 24

Machine learned models exhibit bias, often because the datasets used to train them are biased. This causes the resulting models to perform poorly on records that are minorities within the training set and ultimately present higher risks to them. Computer simulations are used to interrogate and diagnose biases within ML classifiers.

Question 25

Are "synthetic datasets" created from the population?

Answer 25

Yes, synthetic data is created from the population. If we are given a certain set of features X with n number of records then the synthetic data is created using these columns and records.

Question 26

So we are creating new sample data sets based on the permutation /combinations of the original data set in bootstrapping?

Answer 26

In bootstrapping, we create a certain number of datasets from a given original dataset. This is done with replacement and without replacement of the data. With replacement means when we take one record from the data then we keep it back before extracting the second record. This causes repetition of the records in the bootstrap sample. Without replacement means when we take one record from the data then we do not keep it back into the original dataset before taking the next record.

Question 27

Is bootstrapping different from K-fold cross-validation?

Answer 27

Bootstrapping is different from K-fold cross-validation. Bootstrapping is a method of creating different samples from the same dataset while K-fold cross-validation is a process of splitting the dataset into a number of equal-sized folds in order to train different models.

Question 28

For the simulation method, how do you draw the new data set? Do you draw from a pdf of the new data points using the current model parameters?

Answer 28

Given the original dataset and model parameters, the new datasets are created in simulation. It is used to create a dataset of the desired choice either to make the data more realistic or for some other real-world use purposes.

Question 29

Does bootstrapping require a minimum amount of samples (n)? Is Monte Carlo simulation some type of bootstrapping?

Answer 29

There is not a hard and fast rule for selecting the number of records while applying to bootstrap. What is the difference between Monte Carlo simulation and bootstrapping? Montecarlo simulation is different from the bootstrapping methods. Bootstrapping uses the original samples as the population from which it extracts samples to create a new dataset whereas Monte Carlo simulation is based on setting up a data generation process (with known values of the parameters).

Question 30

Is bootstrapping done on a labeled dataset?

Answer 30

Bootstrapping can be done on both labeled and unlabelled data both. It is all about creating a new set of data from a given set of records. It does not need labeled data to create a bootstrap model.

Question 31

When we bootstrap are we jointly estimating all parameters in multivariate regression or one at a time?

Answer 31

When we do bootstrap we have n number of datasets with a certain number of features. While applying multivariate regression we train the model on each of the datasets and make the final prediction as to the majority of predictions done by the models. So for every model parameters are estimated separately.

Question 32

Is bootstrapping is a way of simulating the sampling distribution?

Answer 32

Bootstrapping is different from simulation. In bootstrapping we create different samples from the same datasets while in simulation-based on certain parameters we create different samples.

Question 33

Does Bootstrapping require a size >= 30 to assure normal distribution of the sample?

Answer 33

In bootstrap, we create a minimum of 20-30 datasets but the size of each dataset is equal to the size of the original dataset.

Question 34

How taking duplicates makes it different data in case of bootstrapping

Answer 34

When we bootstrap a dataset, there are a few records that are duplicated during the process but not all the records. Even if we do bootstrap by sampling with replacement there is a 63% chance of unique records getting selected in each sample dataset. If the dataset is having duplicate records it is preferred to remove those duplicates before applying the bootstrap.

Question 35

There will be different estimates of model parameters for different folds? Which one to select?

Answer 35

Cross-validation is a process to find which model is working better among a given set of models. If you have two models M1 and M2 and we applied cross-validation over them and the average performance metrics of one model are found to be better than the other then that model is supposed to perform better on the unseen data. It can generalize better in an unseen scenario.

Question 36

What's the main difference between MSE and standard error?

Answer 36

The standard error is a statistical term that measures the accuracy with which a sample distribution represents a population by using standard deviation. While MSE is the mean squared error that is used to find the parameters of a model by optimizing mean squared error. Then seem to be two types of errors but they have a different purpose to fulfill.

Question 37

Is the benefit of neural nets vs machine learning, that in NN’s, you don’t have to do feature section…the algorithm does it for you?

Answer 37

There are a lot of benefits of using neural networks than the classical machine learning models. What is mentioned here is one of those benefits of using neural nets over the classical machine learning models.

Question 38

For supervised is there a restriction on the number of samples to be used for a model?

Answer 38

It depends on the computational complexity. Generally larger the data better is the training of the supervised model and better in predicting the result since it has more data to train.

Question 39

Are we already assuming Normal distribution to construct confidence interval?

Answer 39

The equation we derive will be correct if we assume normal distribution but even if we do not assume it is approximately normal distribution due to the Central Limit Theorem and if the data is more and say for more than 50 then the assumption of the normal distribution is pretty safe.

Question 40

Does the wald test always have to have the null hypothesis with theta equal to zero?

Answer 40

Yes, In the wald test, we try to find if the standard error of a particular component in the test is zero. Since this revolves around the standard error, we compute it with zero only.

Question 41

Difference between estimator and predictor?

Answer 41

None, they can be used interchangeably. Both are the same.

Question 42

What if the normality test on residuals fails?

Answer 42

The residuals should be normally distributed. This is the basic assumption which should not be violated. Maximum likelihood estimation does not work if this assumption doesn't hold true, in which case we should not use linear regression.

Question 43

Could you please explain "matrix full rank"?

Answer 43

A matrix is said to have full rank if its rank i.e. the number of independent columns equals the largest possible for a matrix of the same dimensions, which is the lesser of the number of rows and columns. A matrix is said to be rank-deficient if it does not have full rank.

Question 44

How do we detect here high correlation between two variables?

Answer 44

We can check by computing the correlation between two desired variables. The correlation coefficient, r, is a number between -1 and 1 and tells us how well a regression line fits the data. It gives the strength and direction of the relationship between two variables. The relationship between two variables is generally considered strong when their r value is larger than 0.7.

Question 45

Can multicollinearity be addressed by using R-Square?

Answer 45

R-Square cannot define multicollinearity. R-Square explains the proportion of variance for a target variable explained by the independent variables.

Question 46

Can we remove collinearity by removing variables? And which variable to remove?

Answer 46

There are many ways one of them is to drop the variable which has less correlation to the target variable or we can also check the Variance inflation factor (VIF) to remove the variable. If VIF gives higher values for those variables then they can be eliminated. It is one of the industrial approaches.

Question 47

Will the regression generate the best fit line?

Answer 47

Yes. The line of best fit has to be generated by regression which gives us the prediction with minimum errors.

Question 48

Can we use discrete(categorical) variables in regression?

Answer 48

Regression is fine if your independent variables are discrete. It does not affect the model in any way.

Question 49

Do we have independent assumptions of the variables?

Answer 49

We never know the variables are completely independent of each other. There may be a slight association with one another. If there is a very strong correlation that leads to collinearity then they should be taken care of.

Question 50

Why do we use logarithm?

Answer 50

It is a part of the data transformation technique to improve the model to best fit the data. Using the logarithm of one or more variables improves the fit of the model by transforming the distribution of the features to a more normally-shaped bell curve.

Question 51

If we add non-linear data, will the model stay linear?

Answer 51

Even though we add data points that are nonlinear the coefficients that are learned in the model will be linear. Hence linear regression can be applied.

Question 52

Is augmentation is like using more harmonics in the Fourier equation?

Answer 52

Yes. Instead of different frequencies we try to augment and add new variables using interaction, polynomial to best fit the data.

Question 53

Does augmentation affect the model explainability?

Answer 53

It slightly affects the explainability but it is always a tradeoff between explainability and accuracy of the model.

Question 54

Will the new feature replace the old one or will be added to the data set?

Answer 54

New features will be added along with the existing features and not replaced in the dataset. Features are only removed if they are not helpful for the model building process.

Question 55

What does “ combined effect of TV and Radio “ mean intuitively? TV has affected, Radio has affected, and they seem to be independent variables.

Answer 55

It can be considered as the combined effect of TV and Radio can bring a bigger effect when compared to them individually since in some cases individually they may not be effective but together will be more effective.

Question 56

If there is more noise in the data does it lead to more false positives and less true positives?

Answer 56

Yes, the noise would mean a presence of more false positives and true negatives since it affects the performance of the model.

Question 57

Can R square be used for measuring the prediction error?

Answer 57

R-Square explains the proportion of variance for a target variable explained by the independent variables and it is not used as an error metric. MSE, RMSE, etc are used as a metric for measuring the error.

Question 58

Is the problem with overfitting that it produces too many errors or the error is due to very high computation?

Answer 58

It produces misleading values as we made the model more complex and learn the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. It is not due to computation issues and error is high due to overfitting only.

Question 59

What is overfitting and what does high variance mean?

Answer 59

Overfitting happens when a model learns the detail and noise in the training data to the extent that it negatively impacts the performance of the model on new data. So it leads to high variance and overfitting as it learns more on the training data as we made the model more complex and when trying to predict on a new data it fails and leads to poor performance of the model.

Question 60

What does bias mean?

Answer 60

The bias is known as the difference between the prediction of the values by the ML model and the correct value. Being high in biasing gives a large error in training as well as testing data. It is recommended that an algorithm should always be low-biased to avoid the problem of underfitting. In general, bias is too simple and we should increase the complexity of the model by adding more variables.

Question 61

What is the difference between the validation set and the test set?

Answer 61

We use the validation set to test the performance of each method and the test set to test the performance of the final model/method that is chosen. The test set is untouched till the final testing of the model.

Question 62

With small sample sizes, this third test set is recommendable?

Answer 62

Yes, the test dataset will be smaller in size. The main dataset can be split for example like 80:10:10 for train, validation, and test dataset. Then the test is finally used for testing the selected model. This third dataset split i.e test dataset is kept hidden from the training and validation process. The other more useful approach for small datasets is to use to cross-validation.

Question 63

Would it be advisable to repeat training+validation several times with different splits of training/validation? Then use statistics of error

Answer 63

Yes, we can think about doing that. We can maybe form a grid of values for training/validation split, then whichever one gives us the best result we go with that

Question 64

How big should each of these data set sizes be? Seems like we're dividing our data up, again and again, to be more thorough, but the resulting subsets are smaller.

Answer 64

Generally, the training dataset will be larger in size since the model is trained on this training data and the validation dataset is used for evaluating the model. Then the test dataset which is kept hidden is used to test the final selected model to see how well the model performs. Generally, they are kept in the ratio of 70:20:10 or 80:10:10. The split depends on the size of the dataset and the train-validation-test split is an ideal method.

Question 65

Could LOOCV lead to overfitting?

Answer 65

When the data set is small and of the order of a few thousand, it can lead to overfitting.

Question 66

After doing the regression n times in LOOCV, do we take the average thetas?

Answer 66

Not really. Cross-validation is only used to check the performance of the model. After that regression is done on the training dataset to train the model.

Question 67

Does irreducible noise mean that the noise is constant at that point?

Answer 67

Irreducible noise cannot be constant even if we know everything about the variable, there will be still some noise or randomness in the behavior of the variable data.

Question 68

Do you vote to select the theta of the best performing model when we do cross-validation?

Answer 68

No - cross-validation is to test the performance and not to build the model. You have to run the regression again with a different set of variables, parameters to improve performance

Question 69

Are the folds in k-fold sequential or random?

Answer 69

It is chosen randomly for non-time series data. For Time-series data we need to use Sequentially

Question 70

Do all the folds need to be the same size?

Answer 70

Ideally, it should be the same. They can be unequal but there is no reason to not divide them equally.

Question 71

Does MSE accounts for both Bias + Variance?

Answer 71

Yes. MSE accounts for bias as well as variance. MSE tells us about the error in our prediction which accounts for both bias and variance.

Question 72

Can I use k-fold in clustering?

Answer 72

K-Fold is used for supervised learning. Once the model is trained on K-1 folds, it can be tested on the hold-out set. In unsupervised learning, we cannot test the model trained on K-1 folds because we don't have labels.

Question 73

Why not use random subsets of the dataset?

Answer 73

Because the sample size of a random subset is different than or original data set, then the amount of noise will vary, (noise will be more in less data, and less in more data). We want to avoid the difference in the estimator's performance.

Question 74

In the bootstrap method is there sampling with replacement?

Answer 74

Bootstrap Sampling is a method that involves drawing sample data repeatedly with replacement from a data source to estimate a population parameter. It means a data point in a drawn sample can reappear in future drawn samples as well.

Question 75

What is the mean of a sample?

Answer 75

Sum of all data points in the sample divided by the number of data points in that sample.

Question 76

What does pick a 'random sample' mean?

Answer 76

To randomly pick a sample means every data point is equally likely to be picked.

Question 77

Must the sample be equal size to the original data set in the bootstrap method?

Answer 77

No, the sample generated using bootstrap doesn't have to be of the same size as the original data but usually, it is kept of the same size, and the samples not selected while sampling can be used to validate the model.

Question 78

Bootstrapped parameters will lead to normally distributed parameters by CLT?

Answer 78

Yes, it should ideally if one is to believe CLT gives a true estimate of the population parameters.

Question 79

Isn't k-fold cross-validation for testing the generalizability of our model?

Answer 79

Yes, if the model performs equally overall folds or subsets we can go ahead with the model.

Question 80

How we can pick up the number of k in k-fold?

Answer 80

There is a bias-variance trade-off associated with the choice of k in k-fold cross-validation. Typically, given these considerations, one performs k-fold cross-validation using k = 5 or k = 10, as these values have been shown empirically to yield test error rate estimates that suffer neither from excessively high bias nor from very high variance. However, it is a heuristic to choose k.

Question 81

In the Heteroskedasticity slide, the linear regression image does not look wrong in the case of TV/Sales. Can you give an example for a prediction that does not work well with Linear Regression?

Answer 81

The regression line seems good in that image but it is difficult to interpret the results using the regression line alone. You must interpret all the results before making final decisions. For example, whether 1. All the variables are independent of each other, 2. All the variables are significant 3. Error terms are normally distributed etc. Consider a dataset with one continuous independent variable and a continuous dependent variable. Imagine, those values are distributed over 2-dimensional space which forms a circular shape. If linear regression is applied to this dataset, the prediction will not be efficient. The residuals will be very high for most of the points.

Question 82

What is meant by “full rank”?

Answer 82

Multicollinearity - The extent to which independent variables are correlated. A basic assumption of the linear regression model is that the rank of the matrix of observations on independent variables is the same as the number of explanatory variables. In other words, such a matrix is of full column rank. This implies that all the independent variables are independent of each other, and there is no linear relationship among them.

Question 83

Does multicollinearity happen when some of the independent variables are very correlated?

Answer 83

Yes. Multicollinearity happens when independent variables in the regression model are highly correlated to each other. The regression algorithm assumes that the independent variables are not correlated with each other, so this assumption must be met if we are to proceed with building a linear regression model.

Question 84

Will PCA explain the relative effect of TV vs Radio? If no, why not?

Answer 84

No. It will not explain the relative effect. PCA just tells if our data has redundancy. Then we can throw away those redundant variables. PCA only looks at the independent variables. But to explain the relative effect of these two variables, we have to consider the dependent variable as well.

Question 85

Is there some way to measure multicollinearity?

Answer 85

Yes. In the matrix (Xt * X)^(-1), look at the small eigenvalue of that matrix, if the smallest eigenvalue is close to zero, then that is the evidence of multicollinearity. Here, Xt represents the transpose of X matrix and (-1) represents the inverse of the matrix In practice, we can also use Variance Inflation Factor (VIF) to check the multicollinearity among variables.

Question 86

What if the 'X' is multicollinear only for a segment of the data?

Answer 86

That is not a problem. If the correlation is significant, it will still give enough evidence for multicollinearity that 'X' is correlated with some other variables.

Question 87

How to differentiate between multicollinearity vs correlation?

Answer 87

Correlation refers to the linear relationship between 2 variables while Multicollinearity is defined for regression model, where some features have a strong relationship with other variables or combination of other variables. It may affect the results and the interpretability of the linear regression model.

Question 88

Correlation does not mean causation - Is that true?

Answer 88

Yes, correlation does not imply causation. Causation applies to cases when action A causes outcome B. But correlation is simply a relationship, i.e., action A relates to action B.

Question 89

When to add/remove variables?

Answer 89

It is like a heuristic approach. In linear regression, add variables, see whether it improves our prediction in terms of R-squared. If it seems to give significant improvements we can put them in. Also, try to take out the variables, we can observe if there is any damage to the R-squared values. If not, probably those variables may not be relevant. We can take those variables away.

Question 90

What if a categorical variable has more than two values?

Answer 90

There can be categorical variables with more than one category. We can create dummy variables for each category, where each column represents different categories, and entries are 1 in the columns belonging to the specific category else 0.

Question 91

Do we want to check the adjusted R^2 for each new feature separately?

Answer 91

Yes, it can be helpful because the R-Squared value never decreases no matter the number of variables we add to our regression model. That is, even if we are adding redundant variables to the data, the value of R-Squared does not decrease. It either remains the same or increases with the addition of new independent variables. On the other hand, the Adjusted R-squared takes into account the number of independent variables used for predicting the target variable. In doing so, we can determine whether adding new variables to the model increases the model fit. It can decrease if that is not the case.

Question 92

How do we know when to stop adding variables?

Answer 92

We can observe the adjusted R-squared and see if it improves much by adding new variables. We can use the validation set to get a good estimate of whether new variables add value to the model or not. Note: There is also a trade-off between the computational time and the model performance. If we add, say, 5 variables to the data for the increase of 0.1 Adjusted R-Squared, then depending on the resources and application, we can decide whether it is a significant increase for the dimension and the computation time that we are increasing.

Question 93

Is there a threshold for sparsity?

Answer 93

It depends on how much data we have. If we have little data, we are not going to believe a model that has 100 coefficients. We may look for a sparse structure that has fewer thetas. But if we have billions of data, then even a structure with 1000 coefficients may be fine. It is relative to the size of the dataset.

Question 94

Is the validation set the same as the test set?

Answer 94

No, both data are different. The validation set is the data that is used to validate the results of our trained model and tuning model hyperparameters. The test set is the unseen data that is used to check whether the model is giving a generalized performance or not.

Question 95

What would be the minimum sample size recommended for this validation method?

Answer 95

It depends on the amount of data we have. Generally, we consider 60% data as a train set, 20% data as a validation set, and 20% as a test set.

Question 96

Could we change our validation set and run multiple iterations of our experiment until we get a good model?

Answer 96

Yes, this is what we do in K-fold cross-validation. Initially, we split the dataset into k groups. For each unique group, iterate the following procedure. -Take one group as a validation dataset -Take the remaining groups as a training dataset -Fit a model on the training set and evaluate it on the validation set We can summarize the performance of the model using the average scores on each group.

Question 97

Is there any difference between the validation process and cross-validation?

Answer 97

Validation - Divides the original training dataset into two different subsets, say training set and validation set. The training set is used for training and the validation set is used for assessing the performance of the model and tune hyperparameters. Here, the validation set is never getting trained by the model. Cross-validation- The original training data is divided into 'k' number of subsets. In one epoch, use 'k-1' subsets of data for training and use the remaining dataset for validating. Like this, for every epoch, the validation dataset will be different. This is also called K-fold cross-validation.

Question 98

Could we use different random states and have different training/validation/test sets?

Answer 98

The random_state parameter is used for initializing the internal random number generator. Setting random_state a fixed value will guarantee that the same sequence of random numbers is generated each time we run the code. We can get different training/validation/test sets if we change random states, but we keep it the same throughout our analysis to avoid any bias and use cross-validation to assess the model performance.

Question 99

How does RSS compare to MSE?

Answer 99

The residual sum of squares (RSS) measures the level of variance in the error term of a regression model. The smaller the residual sum of squares, the better the model fits the data; the greater the residual sum of squares, the poorer the model fits the data. The mean squared error (MSE) is used to test the performance of the fitted models. It is related to RSS by the following equation: MSE(Mean Squared Error) = (1/N) * RSS, where 'N' is the number of samples.

Question 100

How do we judge our model performance based on MSE?

Answer 100

Mean Square Error (MSE) is defined as the Mean (or Average) of the square of the difference between actual and estimated values. The smaller value of MSE indicates that the predicted values are closer to the actual values, i.e., a better model.

Question 101

In cross-validation, what would happen if the value of ‘k’ is the same as the total number of data points?

Answer 101

In cross-validation, if 'k' is equal to the number of data points, then for each fold we have only one data point in the validation set and the rest of the observations are used to train the model. It is called Leave-One-Out-Cross-Validation (LOOCV).

Question 102

What is meant by "With Replacement" in bootstrapping?

Answer 102

Bootstrapping is a resampling technique that generates a sampling distribution by repeatedly taking random samples from the known sample, with replacement. Here, 'with replacement' means a data point in a drawn sample can reappear in future drawn samples. This makes each drawn sample independent of the samples previously drawn. If we did not sample with a replacement, we would always get the same data (if the number of data points selected is equal to the number of data points in the original data).

Question 103

In the bootstrap method, what is the relation between 'n' & 'm'? Is m

Answer 103

Here, 'n' is the size of the original dataset. 'm' is the number of new bootstrap datasets. We can pick any value for 'm'. It can be smaller, equal, or larger than 'n'. In general, we keep it 'm' to be equal to 'n' but in certain scenarios, they can be different. For example, if we want to know details of the tails of the sampling distribution, we would need a bigger 'm'.

Question 104

Is bootstrapping the same as permutation?

Answer 104

No. A permutation would be to choose each one of the data points just once. If every data point is chosen once, then the dataset would be essentially the same as the original dataset. But in bootstrapping, we pick some duplicates (sampling with replacement) to create a new dataset.

Question 105

Does bootstrap allow us to approximate the sampling distribution?

Answer 105

Yes, this is where bootstrap helps us. With bootstrap, we get samples out of an approximation of the sampling distribution and when we make a histogram out of those samples, that histogram is an approximation of the true sampling distribution. Once we have a sampling distribution, we see that how wide the standard deviation is and determine the standard error.

Question 106

Does the bootstrap method create a new dataset with the same mean and standard deviation of the sample set, or does it choose actual records randomly from the sample set?

Answer 106

Bootstrap chooses samples randomly from the actual dataset with replacement. Bootstrapping is a general approach to statistical inference based on building a sampling distribution for a statistic by resampling from the data that we have.

Question 107

What do the diagonal elements represent in the covariance matrix?

Answer 107

The covariance matrix is a symmetric matrix that shows covariances of each pair of variables. The diagonal element shows the covariance of a variable with itself which nothing but the variance of that variable. So, each diagonal element represents the variance of the respective variable.

Question 108

Is it possible or common to see the p-value high but the CI band not including the value zero? And vice versa?

Answer 108

No, the two things are equivalent. If the confidence interval includes zero, that means there is no statistically meaningful or statistically significant proof that the variable helps to predict the target variable. It is the same as saying the p-value is high (>0.05). Similarly, when CI doesn't capture zero the p-value will be low.

Question 109

How do we decide when to use linear regression or other methods?

Answer 109

Linear regression is a basic model and it can be the first model to try for a regression problem. The linear regression model is widely used in many situations before attempting non-linear and more complicated models. It is the most accomplished theoretical model and helps in interpretability; some key concepts can also be explained well using the linear regression model. In practice, we use multiple models of different kinds and the algorithm that gives the best results depends on the data and the problem on hand.