Statistical inference

Question 1

What is the difference between drawing values from a distribution with or without replacement?

Answer 1

With replacement means that after we have drawn a value we put it back and this way the draws does not change the distribution of values. Replacement is good to use when we want to simulate a big or infinite population where one draw does not change the probabilities of the next draw so much. Without replacement means that we do not put the value back and the distribution changes with each draw. This should be used to simulate draws from a smaller finite population.

Question 2

What is a probability density function?

Answer 2

When we make stochastic observations with real values the distribution of those values are described by probability density functions. If we were to draw samples from an urn 1000 times and create a histogram of the drawn samples, the shape of the histogram would follow the probability density function for that variable where the highest number of drawn values will be the mean value and the variation the standard deviation (if the variable is normally distributed).

Question 3

How can we get the probability of drawing a value x from a mixture of multiple different distributions?

Answer 3

Having different distributions means that we have a mixture of pdfs and they have different means and variations. When we have different object types from different distributions, the distribution we end up drawing from is the pdfs multiplied by their weight summed together.p(x) = w1 * p1(x) + w2 * p2(x) + w3 * p3(x)1. Draw one sample from a discrete multiple class distributions with probabilities of the weights. 2. If the outcome is class i, then draw a sample from pi.

Question 4

What parameters do you need to define to draw from a univariate normal distribution vs a multivariate normal distribution?What do the results look like?

Answer 4

Univariate:We need to define the sd and the mean. Result is one value from the distribution.Multivariate:Vector of means for the different distributions and a covariance matrix that describes how the individual variables vary and how they vary together.Result is a vector with as many values as the number of distributions you have.

Question 5

Why is it important to incorporate noise when we simulate stochastic observations?

Answer 5

When we simulate stochastic observations we should take into account that real values usually have some noise that affects the values we draw. The noise is important to know about because it can make a linear relationship appear not to be linear if we have too few samples. The impact of the noise gets smaller with a higher number of samples.

Question 6

What is a Hidden Markov model?

Answer 6

Markov models are used to simulate stochastic processes. They are memoryless models (only looks at the present state) that are based on probabilities. The model will simulate the most probable process by using state and emission probabilities.

Question 7

What is a conventional confidence interval?

Answer 7

Conventional confidence interval: This interval will contain the true mean with 95% chance and tells you that if you were to sample many times and construct 95% confidence intervals each time, the true mean will fall within that interval 95% of the time. This because the conventional confidence interval is frequentistic There is still a 5% chance that the true mean falls on the “extreme values” not included in the confidence interval and 1/20 created confidence intervals might miss the true mean because of this.

Question 8

What is a bootstrap 95% percentile interval? What is the benefit and setbacks of using these intervals?

Answer 8

These are created for resampling. If you were to create many bootstrap datasets and calculate the mean of each and create a 95% percentile interval of the bootstrap means that interval will tell you nothing about if the true mean is inside that interval since it is not designed to have that coverage. It instead tells you how uncertain you are of your estimated mean in your samples. The 2.5 and 97.5 percentiles of the bootstrap means indicate the lower and upper bounds of the percentile interval. These percentile intervals capture the middle 95% of the distribution, leaving 2.5% in the lower tail and 2.5% in the upper tail. If that interval is very wide it means that we should be more uncertain of our estimated mean.The benefit is that with bootstrap intervals you do not need to worry about the distribution of the variable like you have to with conventional confidence intervals.

Question 9

How do we get the 95% bootstrap percentile interval?

Answer 9

We draw bootstrap datasets from the original dataset, calculate the metric we are interested in for each bootstrap set and save in a list (could be correlations, mean-values, fractions, standard deviations ect.) Then calculate the 2.5 and 97.5 percentiles of the bootstrap values and those values are the upper and lower bounds of the percentile interval [a,b].This interval tells you how certain you are of your estimated mean because it tells you how much it can vary just by random chance.The bigger the bootstrap set is the more data we have to look at and this will lead to a smaller interval that gives us more certainty because then the bootstrap represents the original dataset better.

Question 10

Why is it more efficient to create a bootstrap interval after you have collected your sample population?

Answer 10

We do bootstrap sets instead of doing the observations 1000 times and bootstrapping is also beneficial because we do not have to worry about the distribution of the variable we are looking at. A conventional confidence interval will crash if the variable is not normally distributed.

Question 11

What is hypothesis testing?

Answer 11

To do a hypothesis testing you are testing how likely it is to get an observed value under the null hypothesis.

Question 12

How is hypothesis testing done?

Answer 12

You are calculating your observed value. Then creating bootstrap sets from the observed distribution and calculating the metric for each bootstrap set. You are then calculating the p-value by summing how many times the bootstrap metrics were as extreme or more extreme as the observed value and dividing the sum with the number of bootstrap sets. If you only saw very few extreme values (under 5% usually) the null is not likely and we should reject it.

Question 13

How is hypothesis testing with percentile intervals done?

Answer 13

There we see if 0 (null) is inside of the 95% percentile interval, and if it is then the null is likely and we cannot reject it. This is usually only done when the null states that a difference between two groups is 0.

Question 14

What is the interquartile range? What is the benefit of using this over standard deviation?

Answer 14

IQR = Q3 - Q1. Removes the lowest 25% and the highest 25% values and only looks at the median of the remaining 50%. This metric is more robust as a metric of variety than standard deviation because the standard deviation is more sensitive to outliers because it looks at the entire dataset.

Question 15

What is regression?

Answer 15

Regression: The problem of using a set of training examples to build a prediction modelf() that produces predictions f(x) where the observed response values yn are real valued (continuous).

Question 16

What is mean absolute error?

Answer 16

Mean absolute error = Sum of all absolute errors between true response and predicted response divided by the number of samples. So the average of the absolute errors.

Question 17

What is mean squared error?

Answer 17

Mean squared error = Sum of each error squared / number of samples.

Question 18

What is root mean squared error?

Answer 18

Root mean squared error = Square root of MSE.

Question 19

What is the R^2 value?

Answer 19

How much of the variation in response is described by the features. calculated by taking 1- (sum of squared errors / sum of all differences between yi and sample mean of y)

Question 20

What is mean absolute percentage error?

Answer 20

Average of absolute value of ratio between absolute error and true value.

Question 21

What is k-fold cross validation?

Answer 21

K-fold cross validation is a kind of resampling model where a model is trained and tested on training data and the training and testing is done on different subsets of the data.

Question 22

Explain how k-fold cross validation is done?

Answer 22

The training dataset is divided into k number of subsets and the model is trained on k-1 subsets and tested on the one the subset that is excluded from training. This is done k number of times so that the model is trained and tested on all subsets of the training dataset and so that test is always done on data that the model was not trained on. We then take the average of all k number of performance values to get a good estimate of how the model will perform on unseen data of the same size as the training set.

Question 23

In k-fold cross validation, what are the pros and cons of using large vs small k?

Answer 23

If we were to use fewer folds, it would reduce the training time and computational cost but smaller K also leads to more overlap between training sets, because each training set will cover more of the dataset. This is potentially allowing the model to capture more consistent patterns in the data but with this we also get the risk of overfitting the model to the training data.Larger K will therefore reduce the risk of overfitting the data because the overlap between training sets will be smaller but the computational cost does however get bigger with bigger K.

Question 24

What is a hyper parameter? What is the hyperparameter of ridge regression and k-nearest neighbor regression?

Answer 24

A hyperparameter is a setting or configuration for a machine learning model that is set before the training process begins. Unlike the model parameters, which the algorithm learns from the training data, hyperparameters are external factors that influence how the learning process takes place.The hyper parameter of ridge regression is the alfa penalty value and for k nearest neighbor it is the number of nearest neighbors we choose to look at (k).

Question 25

What is ridge regression? What is the difference from OLS?

Answer 25

Ridge regression is similar to an ordinary least squares fitting in the sense that we are trying to find the coefficients of x so that the residuals are minimized. The difference is that with ridge regression we also have an alfa value that is described as a penalty for choosing large coefficients in the produced model. If alfa = 0 then it is the same thing as an ordinary least squares fitting because there is no penalty and the model is free to distribute the coefficients so that one feature is much more important than another.

Question 26

Why is ridge regression better if you have correlation between some of the features in your dataset?

Answer 26

If alfa is large then the model will be encouraged to distribute the coefficients equally across all features. This means that if we have multicollinearity, at least we reduce the risk of the correlated features being a lot more important for the change of the response value than the other features. The alfa also helps to reduce the risk for overfitting the model to the training data which is a risk with large coefficients.

Question 27

Explain k-nearest neighbor regression.

Answer 27

It is a regression model where we predict the values of the response of a new observation by looking at the response values of the previously observed observations. When we add a new observation, we assign to it the y-value of the closest observation. If k=1 we use the closest observation. If k=2 then we use the 2 closest observations and the average of their y-values.

Question 28

In k-nearest neighbor regression, what happens if we set k to be equal to the number of observations we have?

Answer 28

If K is set to the same value as the number of observations we will just take the same average each time for the predicted response of the new observed x.

Question 29

What is the workflow of PyCaret?

Answer 29

1. The dataset is divided into 70% training (Dtrain) and 30% testing (Dtest).2. Use the function compare_models() to fit all models in a family (here regression models) to Dtrain using k-fold cross validation with default K=10 and rank performance of the models where the performance value is the average of each fold. 3. Tune the model to find the optimal set of hyper parameters that optimize the performance. The function tests different hyperparameters for the model and trains and tests them again using k-fold cross validation. 4. The performance is only based on the training set. Therefore we use the function predict_model to let the model predict the responses on Dtest. When we look at the performance values from Dtest they should not be significantly lower than the mean of all cross validations since this would indicate that the model is overly fitted to Dtrain.5. Finalize_model() is a function that trains and evaluates the model again on the entire dataset. After this we let the model predict responses on completely unseen data.

Question 30

What is the setback of using the finalize_model function in PyCaret?

Answer 30

When training the final model on the entire dataset we have more data that could have meant different hyperparameters if that data was a part of the tuning. This means that the hyperparameters as well as the model family might be suboptimal now that we use the entire dataset and have more information.Another problem is that we use up all data we have for training and we have no data left to treat as “unseen” to test the models performance on new data.

Question 31

What is the setback of using the compare_models function in PyCaret?

Answer 31

The problem here is that the function does not consider other hyperparameters than the defaults and therefore it might favor one model over another. The knn with default k might be better than ridge with default parameters, but ridge might be the better choice if we looked at another hyperparameter value. This means that the model chosen could be suboptimal.

Question 32

In experimental design, what does it mean to run a full factorial design?

Answer 32

Running a full factorial design means that all possible combinations of factor levels are tested. If you have a 2 level design meaning that you have 2 values that the variable can take on (maximum and minimum, -1 and 1) and 5 variables, then the full factorial design is 2^5=32 . So the number of unique experiments to run is k^p and when you run the full factorial design, all unique experiments are tested.

Question 33

What is a fractional factorial design? Why do we use it?

Answer 33

The full factorial design usually leads to a large number of experiments to perform. We can reduce the number of experiments by doing a fractional factorial design by treating the higher order variables as the product of the other variables. If we for example treat x5 as the product of x1x2x3*x4, we do a full factorial (test all unique combinations) for x1-x4 and we will instead get 2^5-1=16 experiments.

Question 34

What is the setback of using a fractional factorial design?

Answer 34

We will not know if x5 or the product of x1-x4 is the important feature if x5 got a big coefficient.

Question 35

How do we build a fractional factorial design?

Answer 35

To build the fractional factorial we use the factorial.build_factorial(5, 2**(5-1)) function from the dexpy library. The first 5 indicates that we have 5 independent variables, 2 indicates the level and 5-1 is the fractional factorial. From this we will get a table with all 5 variables as columns and the 16 experiments as rows.

Question 36

What are counterpoints in experimental design and why do we need them?

Answer 36

We add center points to the design if the model we want to fit has quadratic elements. This is because a 2 level design cannot capture the models deviations from a straight line since we do not have enough levels - we need at least 3 for a quadratic model. Therefore we add center points to the features that have real values. The center points added are extra experimental runs with values for the features that are the midpoints of the two existing levels.This will lead to an extra number of experiments that corresponds to the number of unique combinations we can add to the experiment.

Question 37

What does a plain linear model look like?

Answer 37

y = w0 + w1x1 + w2x2 + w3x3….. + wnxn.

Question 38

What does a full quadratic model with 2 variables look like?

Answer 38

y = w0 + w1x1 + w2x2 + w12x1x2 + w1x1x1 + w2x2x2.

Question 39

What is yˆ = w^Tz?

Answer 39

Any model we choose to fit to our data can be described by yˆ = w^Tz where:w is a vector of the coefficients.z is a vector/matrix of the x-variables/the experimental designy^ is the predicted response from the model.

Question 40

How can we use yˆ = w^Tz to find the coefficients?

Answer 40

We can find the coefficients if we have the x values and the response values by using ordinary least squares that minimizes the sum of the squared residuals (minimizes (y - y^)^2).Searching for the minimum gives the closed form:wOLS = (Z^TZ) ^-1 = Z^Ty which is the same thing as Z^TZ = Z^Tw.This means that for any model we choose to fit we can use the closed form to find the coefficients that minimizes the residuals.

Question 41

Why do we encode the values of the features in the design matrix?

Answer 41

The values in the design will be encoded values to get them on the same scale so that they are directly comparable. When we later want to perform the actual experiment we would need to use the actual lows and highs in the design matrix.

Question 42

What is D-optimal design?

Answer 42

Instead of using the fractional factorial design to reduce the number of experiments we can use the D-optimal design.This means that we use the k number of experiments that maximizes the determinant of the design matrix, meaning that we optimize the design.

Question 43

In D-optimal design, what does it mean that we find the experiments that maximizes the determinant? What experiments are going to be included in the design?

Answer 43

A low determinant indicates that we have low values in the matrix and a determinant = 0 means that the matrix is singular and cannot be inverted. This means that the D optimal design is always going to include the “corners” of the squared box of all possible experiments because those values are the highest and will maximize the determinant.

Question 44

When we want to reduce the number of experiments to perform with D-optimal design, what is the smallest number of experiments we can choose?

Answer 44

The smallest number of experiments we can perform is the number of parameters of the model. If we have a full quadratic model with 2 variables then the smallest number of experiments will be 6 because we need at least w0 + w1x1 + w2x2 + w12x1x2 + w11x1x1 + w22x2x2.

Question 45

What is the function for creating the D-optimal design?

Answer 45

dexpy.optimal.build_optimal (number of factors, number of experiments, type of model).

Question 46

How can we test if a D-optimal design finds the same coefficients as the full factorial design?

Answer 46

We can test if a D-optimal design gives the same importance for the coefficients as the full factorial design by using the D-optimal design to predict the response and then use ordinary least squares to find the coefficients.