Statistical Learning Methods

Question 1

mean

Answer 1

arithmetic mean

-> greek letter “mu”

Question 2

median

Answer 2

The median of a set of data is the middlemost number in the set. The median is also the number that is halfway into the set. To find the median, the data should first be arranged in order from least to greatest. If there is an even number of items in the data set, then the median is found by taking the mean (average) of the two middlemost numbers.

Question 3

standard deviation

Answer 3

Deviation just means how far from the normal

• is just the square root of Variance
• while Variance gives you a rough idea of spread, the standard deviation is more concrete, giving you exact distances from the mean
• The standard deviation is an especially useful measure of variability when the distribution is normal or approximately normal because the proportion of the distribution within a given number of standard deviations from the mean can be calculated. For example, 68% of the distribution is within one standard deviation of the mean and approximately 95% of the distribution is within two standard deviations of the mean. herefore, if you had a normal distribution with a mean of 50 and a standard deviation of 10, then 68% of the distribution would be between 50 - 10 = 40 and 50 +10 =60. Similarly, about 95% of the distribution would be between 50 - 2 x 10 = 30 and 50 + 2 x 10 = 70.

=> a measure of the spread of the values around the mean

Question 4

Sample

Answer 4

a selection taken from a bigger Population

Question 5

normal distribution

Answer 5

It is a continuous, bell-­shaped distribution (single peak)
which is symmetric about its mean and can take on values from negative infinity to positive infinity

Each normal curve is characterized by two parameters (is completely described by):

• the mean
• the standard deviation (its symbol is the greek letter sigma)

Question 6

variance

Answer 6

measures how far a data set is spread out.
The technical definition is “The average of the squared differences from the mean,” but all it really does is to give you a very general idea of the spread of your data.
A value of zero means that there is no variability: All the numbers in the data set are the same.

Question 7

population

Answer 7

a sample is a part of a population. A population is a whole, it’s every member of a group.
A population is the opposite to a sample, which is a fraction or percentage of a group. Sometimes it’s possible to survey every member of a group. A classic example is the U.S. Census, where it’s the law that you have to respond. If you do manage to survey everyone, it actually is called a census: The U.S. Census is just one example of a census.
In most cases, it’s impractical to survey everyone. In addition, sometimes people either don’t want to respond or forget to respond, leading to incomplete censuses. Incomplete censuses become samples by definition.

Question 8

p-value

Answer 8

probability value: probability to get the sample result if the null hypothesis is assumed to be true -> if very small, then H_0 should most likely be rejected

Question 9

test statistic

Answer 9

s

Question 10

Gaussian distribution

Answer 10

= Normal distribution = bell curve

Question 11

simple linear regression

Answer 11

s

predict quantitative values

Question 12

least squares method

Answer 12

s

estimate coefficients β_0, β_1, …, β_p st RSS is minimized (i.e. has the smallest possible value)

Question 13

intercept

Answer 13

s

Question 14

Linear models

Answer 14

Linear models describe a continuous response variable as a function of one or more predictor variables. They can help you understand and predict the behavior of complex systems or analyze experimental, financial, and biological data. Linear regression is a statistical method used to create a linear model.

Question 15

residuals

Answer 15

s

Question 16

qualitative response

Answer 16

s

Question 17

quantitative response

Answer 17

s

Question 18

regression

Answer 18

s

Question 19

classification

Answer 19

s

Question 20

prediction

Answer 20

s

Question 21

inference

Answer 21

• when inference is the goal, there are clear advantages to using simple and relatively inflexible statistical learning methods
• In some settings, however, we are only interested in prediction, and the interpretability of the predictive model is simply not of interest
• if we seek to develop an algorithm to predict the price of a stock, our sole requirement for the algorithm is that it predict accurately—interpretability is not a concern

Question 22

prediction vs inference

Answer 22

s

Question 23

parametric

Answer 23

reduces the problem of estimating function f down to one of estimating a set of parameters

Question 24

overfitting (the data)

Answer 24

fitting a too flexible model can lead to overfitting the data, i.e. the model follows the errors, or noise, too closely
- As model flexibility increases, training MSE will decrease, but the test MSE may not. When a given method yields a small training MSE but a large test MSE, we are said to be overfitting the data.
- This happens because our statistical learning
procedure is working too hard to find patterns in the training data, and may be picking up some patterns that are just caused by random chance rather than by true properties of the unknown function f. When we overfit the training data, the test MSE will be very large because the supposed patterns that the method found in the training data simply don’t exist in the test data. Note that regardless of whether or not overfitting has
occurred, we almost always expect the training MSE to be smaller than the test MSE because most statistical learning methods either directly or indirectly seek to minimize the training MSE. Overfitting refers specifically
to the case in which a less flexible model would have yielded a smaller test MSE.

Question 25

parametric methods

Answer 25

s

Question 26

non-parametric methods

Answer 26

• do not make explicit assumptions about the functional
  form of f
• by avoiding the assumption of a particular functional form for f, they have the potential to accurately fit a wider range of possible shapes for f
• a major disadvantage: since they do not reduce the problem of estimating f to a small number of parameters, a very large number of observations (far more than is typically needed for a parametric approach) is required in order to obtain an accurate estimate for f
• We will often obtain more accurate predictions using
  a less flexible method. This phenomenon, which may seem counterintuitive at first glance, has to do with the potential for overfitting in highly flexible methods

Question 27

supervised learning

Answer 27

We wish to fit a model that relates the response to the
predictors, with the aim of accurately predicting the response for future observations (prediction) or better understanding the relationship between the response and the predictors (inference).

Question 28

unsupervised learning

Answer 28

for every observation i = 1, . . . , n, we observe
a vector of measurements xi but no associated response yi. It is not possible to fit a linear regression model, since there is no response variable to predict. In this setting, we are in some sense working blind; the situation is referred to as unsupervised because we lack a response variable that can supervise our analysis

• > two types of unsupervised learning:
  1) principal components analysis
  2) clustering

=> in unsupervised learning, there is no way to check our work because we don’t know the true answer—the problem is unsupervised

Question 29

variables

Answer 29

can be characterized as either quantitative or qualitative (also quantitative qualitative known as categorical)

Question 30

regression problems

Answer 30

problems with a quantitative response

Question 31

classification problems

Answer 31

problems with a qualitative response

Question 32

degrees of freedom

Answer 32

a quantity that summarizes the flexibility of a curve
- a more restricted and hence smoother curve has fewer degrees of freedom than a wiggly curve
-

Question 33

variance of a statistical learning method

Answer 33

Variance refers to the amount by which fˆ would change if we estimated it using a different training data set. Since the training data are used to fit the statistical learning method, different training data sets will result in a different fˆ.
But ideally the estimate for f should not vary too much between training sets. However, if a method has high variance then small changes in the training data can result in large changes in fˆ.
In general, more flexible statistical methods have higher variance

As a general rule, as we use more flexible methods, the variance will increase and the bias will decrease

Question 34

bias of a statistical learning method

Answer 34

bias refers to the error that is introduced by approximating a real-life problem, which may be extremely complicated, by a much simpler model.

• It is unlikely that any real-life problem truly has such a simple linear relationship, and so performing linear regression will undoubtedly result in some bias in the estimate of f
• Generally, more flexible methods result in less bias.

As a general rule, as we use more flexible methods, the variance will increase and the bias will decrease

Question 35

training error rate vs test error rate

Answer 35

s

Question 36

logistic function

Answer 36

s

Question 37

maximum likelihood

Answer 37

s

Question 38

student distribution vs normal distribution

Answer 38

s

Question 39

t-test

Answer 39

s

Question 40

RSS

Answer 40

• > residual sum of squares / sum of squared residuals
• > the least squares regression seeks coefficients β_0, β_1, …, β_p s.t. the sum of squared residuals is as small as possible

Question 41

R-square

Answer 41

s

Question 42

MSE

Answer 42

s

Question 43

t_0 statistic

Answer 43

s

Question 44

bias vs. variance

Answer 44

s

Question 45

stratification

Answer 45

s

Question 46

t-statistic vs t-test

Answer 46

s

Question 47

confidence interval

Answer 47

s

Question 48

covariance

Answer 48

s

Question 49

Support Vector machines (SVM)

Answer 49

• classifier

Question 50

PCA

Answer 50

s

Question 51

why normalize data?

Answer 51

s

Question 52

rejection of outliers

Answer 52

• Value incompatible with the variable domain
• Value too large or too small according to the domain
• Value larger or smaller the mean +/-­ 3 times the standard deviation
• Coding problem (a string instead of a real value

Question 53

10-CV

Answer 53

s

Question 54

bias-variance trade-off

Answer 54

p 36

as we use more flexible methods, the variance will
increase and the bias will decrease

Question 55

covariance

Answer 55

s

Question 56

normal distribution

Answer 56

A normal distribution is fully described with just two

parameters: its mean (μ) and standard deviation.

Question 57

outliers

Answer 57

e.g. when value larger or smaller the mean +/-­ 3 times the standard deviation

Question 58

population regression line

Answer 58

the best linear approximation to the true relationship between X and Y as in
Y = β_0 + β_1X + e.

the least squares plane is an estimate for the true population regression plane

Question 59

RSE

Answer 59

residual standard error

it is the estimate of standard deviation

Question 60

t-statistic

Answer 60

s

Question 61

normalize data

Answer 61

s

Question 62

variability

Answer 62

Schwankungen
Variability refers to how “spread out” a group of scores is
-> refers to how spread out a distribution is

There are four frequently used measures of variability: the range, interquartile range, variance, and standard deviation

Question 63

correlation matrix

Answer 63

s

Question 64

F-statistic

Answer 64

s

Question 65

quantitative vs qualitative predictors

Answer 65

s

Question 66

outlier

Answer 66

An outlier is a point for which y_i is far from the value predicted by the outlier model. Outliers can arise for a variety of reasons, such as incorrect recording
of an observation during data collection.

Question 67

cross-validation

Answer 67

s

Question 68

validation set approach

Answer 68

s

Question 69

maximal margin classifier

Answer 69

• In order to construct a classifier based upon a separating hyperplane, we must have a reasonable way to decide which of the infinite possible separating hyperplanes to use
  -> A natural choice is the maximal margin hyperplane which is the separating hyperplane that is farthest from the training observations. That is, we can compute the distance from each training observation to a given separating hyperplane; the smallest such distance is the minimal distance from the observations to the hyperplane, and is known as the margin. The maximal
  margin hyperplane is the separating hyperplane for which the margin is largest—that is, it is the hyperplane that has the farthest minimum distance to the training observations. We can then classify a test observation
  based on which side of the maximal margin hyperplane it lies -> maximal margin classifier
• > the maximal margin hyperplane depends directly on only a small subset of the observations!!
• > the generalization of the maximal margin classifier to the non-separable case is known as the support vector classifier

-> the distance of an observation from the
hyperplane can be seen as a measure of our confidence that the observation was correctly classified

Question 70

hyperplane

Answer 70

hyperplane is a flat affine subspace of dimension p-1 in a p-dimensional space

• > it divides a p-dimensional space into two halves
• > determine where a point lies by calculating the sign of the left hand side of (9.2)

Question 71

concept of separating hyperplane

Answer 71

the goal is to develop a classifier based on the training data that will correctly classify the test observation using its feature measurements

• In order to construct a classifier based upon a separating hyperplane, we must have a reasonable way to decide which of the infinite possible separating hyperplanes to use
  -> A natural choice is the maximal margin hyperplane which is the separating hyperplane that is farthest from the training observations. That is, we can compute the distance from each training observation to a given separating hyperplane; the smallest such distance is the minimal distance from the observations to the hyperplane, and is known as the margin. The maximal
  margin hyperplane is the separating hyperplane for which the margin is largest—that is, it is the hyperplane that has the farthest minimum distance to the training observations. We can then classify a test observation
  based on which side of the maximal margin hyperplane it lies -> maximal margin classifier

Question 72

support vectors

Answer 72

• > Observations that lie directly on the margin, or on the wrong side of the margin for their class. These observations do affect the support vector classifier.
• since support vectors are vectors in p-dimensional space and they “support” the maximal margin hyperplane in the sense that if these points were moved slightly then the maximal margin hyperplane would move as well
• > for SVM only the support vectors are relevant as for the other training observations alpha_i is zero

Question 73

soft margin

Answer 73

s

Question 74

support vector classifier

Answer 74

• > the generalization of the maximal margin classifier to the non-separable case is known as the support vector classifier
• > also called “soft margin classifier”
• > The hyperplane is chosen to correctly separate most of the training observations into the two classes, but may misclassify a few observations
• the support vector classifier’s decision rule is based only on a potentially small subset of the training observations (the support vectors) means that it is quite robust to the behavior of observations that are far away from the hyperplane
• > is a linear classifier
• > is a natural approach for classification in the two-class setting, if the boundary between two classes is linear
• > what to do with a non-linear class boundary?

-> note that the support vector classifier is equivalent to a SVM using a polynomial kernel of degree d = 1

Question 75

decision rule

Answer 75

s

Question 76

Type I error

Answer 76

a false positive conclusion
The rejection of the hypothesis H_0 which is true

und genauglich isch wenn me H_0 ablehnt (nei zu H_0 seit) aber in würklechkeit H_0 true isch, denn isches e false positive (e type I error) gemäss definition u nid e false negative

Practically, the type I error can be interpreted as the
probability of deciding that a significant effect is
present (reject H0) when it isn’t (H0 true).

Why?
The sample tends to demonstrate a significant effect
but it is due to random variability.
The sampling provides an extreme (but still possible)
sample.

Question 77

Type II error

Answer 77

a false negative conclusion
The acceptance of the hypothesis H0 which is false
(and so H1 true).

u zwar wenn du H_0 acceptisch (du seisch ja zu H_0) aber in würklechkeit H_0 falsch isch, isches äbe kei false positive sondern e false negative (e type II error) gemäss definition

Practically, the type II error can be interpreted as the
probability of not detecting a significant effect
(accept H0) when one exists (H0 false).

Why?
The true effect (H1) is too close to the H0 effect.
The effect is too small to be detected.
The sample size is small to detect the difference.

Question 78

LDA

Answer 78

linear discriminant analysis
-> approximates the Bayes classifier

• why another method aside LR?
• > if n is small and the distribution of the predictors X is approximately normal in each of the classes, the linear discriminant model is more stable than the logistic regression model
• > LDA is more popular for more than 2 response classes

-> the LDA classifier results from assuming that the observations within each class come from a normal distribution with a class-specific mean vector and a common variance σ^2, and plugging estimates for these parameters into the Bayes classifier

Question 79

LR

Answer 79

logistic regression

• > logistic regression models the probability that Y belongs to a particular category
• > uses logistic function
• > to fit that model use a method called maximum likelihood

Question 80

classifiers

Answer 80

• logistic regression
• linear discriminant analysis
• k-nearest neighbours
• support vector classifier

Question 81

supervised vs. unsupervised learning

Answer 81

supervised: regression and classification
- > goal: predict Y using X_1, X_2,…,X_p given p features and n observations

unsupervised: only a set of features X_1, X_2,…,X_p available measured on n observations.
- > goal: not interested in prediction (don’t have Y), goal is to discover interesting things about the measurements on X_1, X_2,…,X_p

two types of unsupervised learning:

1) principal components analysis
2) clustering

Question 82

parametric vs non-parametric methods

Answer 82

parametric method: boils the problem of estimating f down to estimating a set of parameters

Question 83

P(default = Yes|balance)

Answer 83

the probability of default given a value for balance

-> model it using a function that gives outputs between 0 and 1 for all values of X -> use the logistic function

Question 84

logistic function

Answer 84

s

Question 85

Bayes theorem

Answer 85

s

Question 86

Bayes classifer

Answer 86

cond. probability: Pr(Y=j | X = x_0)
- it assign each observation to the most likley class given its predictor values, i.e. it assigns the test observation with predictor vector x_0 to class j for which Pr(Y=j | X = x_0) is largest
- in a two-class problem, the Bayes classifier corresponds to predicting class 1 if Pr(Y=1 | X = x_0) > 0.5, and class 2 otherwise.

=> minimizes the probability of misclassification
-> the Bayes classifier is a useful benchmark in statistical classification

Question 87

Bayes error rate

Answer 87

the Bayes classifier produces the lowest possible test error rate, called the Bayes error rate.
Since the Bayes classifier will always choose the class j for which Pr(Y=j | X = x_0) is largest, the error rate at X = x_0 will be 
1 - max_j Pr(Y=j | X = x_0)

Question 88

linear vs non-linear classifier

Answer 88

• a linear classifier has a linear decision boundary

- a non-linear classifier has a non-linear decision boundary

Question 89

how to address non-linearity?

Answer 89

cf chap 7?

Question 90

support vector machine

Answer 90

• extension of support vector classifier
• > enlarge the feature space of support vector classifier to allow for non-linear boundaries between classes using KERNELS

=> When the support vector classifier is combined with a non-linear kernel such as (9.22) p 352, the resulting classifier is known as a support vector machine

• mainly for binary classification i.e. classification in the two-class setting
• > the concept of separating hyperplanes upon which SVMs are based does not lend itself naturally to more than two classes
• but two approaches for the K-class case exist:
  1) one-versus-one
  2) one-versus-all

Question 91

kernel

Answer 91

A kernel is function that quantifies the similarity of two observations.

• linear kernel
• polynomial kernel -> leads to a much more flexible decision boundary
• radial kernel

=> When the support vector classifier is combined with a non-linear kernel such as (9.22) p 352, the resulting classifier is known as a support vector machine

Question 92

ROC curve

Answer 92

s

p. 148

Question 93

deviance

Answer 93

s

Question 94

Principal component analysis (PCA)

Answer 94

PCA is a popular approach for deriving a low-dimensional set of features from a large set of variables

refers to the process by which principal
components are computed, and the subsequent use of these components in understanding the data. PCA is an unsupervised approach, since it involves only a set of features X1,X2, . . . , Xp, and no associated response
Y

• dimensional reduction
• pre-­process for the linear regression
• use to extract pattern or interpretation

Unsupervised approach
not target signal
more difficult to evaluate
but could be useful to understand the data!

Question 95

LDA vs QDA

Answer 95

Why does it matter whether or not we assume that the K classes share a common covariance matrix? In other words, why would one prefer LDA to QDA, or vice-versa? The answer lies in the bias-variance trade-off.

LDA tends to be a better bet than QDA if there are relatively few training observations and so reducing
variance is crucial. In contrast, QDA is recommended if the training set is very large, so that the variance of the classifier is not a major concern, or if the assumption of a common covariance matrix for the K classes is clearly
untenable.