Week 8 DSE

Question 1

What is machine learning?

Answer 1

field that develops algorithms designed to be applied to datasets, with the main focus being prediction, classification, clustering or grouping tasks

Question 2

What is yi in (yi,xi)?

Answer 2

dependent varaible ( OR RESPONSE VARIALE)

Question 3

What is Xi in (yi,xi)?

Answer 3

P-dimensional vector of independent variables or covariates (in ML speak: features).

Question 4

For a high dimensional dataset, how does p relate to N?

Answer 4

P»N

number of potential varibales that we can use in the model is way larger

Question 5

What does P and N stand for

Answer 5

P: Potential variable
N: number of observations

Question 6

What is supervised learning algorithm?

Answer 6

uses a training dataset (i.e., the estimation sample) (yi, xi), i = 1,2,…,N to determine the
conditional prediction (or forecast) rule Yˆ( X )

Question 7

When yi is continuous, it is called a ________problem; when it is categorical, it is called a _________ problem.

Answer 7

regression
classification

Question 8

What is unsupervised learning algorithm?

Answer 8

uses observations xi, i = 1,2,…,N of a random P-dimensional vector X with joint density p(X) to infer some properties of p(X).

trying to infer some strucutre within dataset

Question 9

What is reinforcement learning?

Answer 9

Algorithm gets told when the answer is wrong, but has no feedback on how to correct it

has to explore different possibilities until it works out how to get the answer right.

Question 10

What is the forecast distribution

Answer 10

cumulative distribution (s shaped)
F(y) = P(Y ≤ y).

Question 11

What is Yˆ

Answer 11

point forecast
best guess for the unknown value

Question 12

What is the formula for forecast erorr?

Answer 12

difference between the actual value and the forecast
See notes

Question 13

Why will there always be forecast error if Y is continuous random variable?

Answer 13

Is because when you are dealing with continuous variable, then the probability of getting a value is 0

P(x) =0
Integration of density at one point is 0

Question 14

What is used to calculate cost of forecast erorr?

Answer 14

loss function L(e)

can also be written as L(Y, Y HAT)

Question 15

What is a forecasst?

Answer 15

Predictive distribution .

Action that must be constructed given loss function and forecast distribution

Question 16

What are the conditions for an appropriate loss function

Answer 16

i)L(0) = 0 (minimum loss is 0) (when error is 0 and get exactly correct answer)

ii) L(e) ≥ 0 for all e

iii)Nonincreasing in e for e < 0, nondecreasing in e for e > 0: L(e1) ≤ L(e2) if e2 < e1 <0; L(e1) ≤ L(e2) if e2 > e1 >0. (Never be rewarded for making an error)

Question 17

What are common choices for loss funciton and what is their similarities? Are they symmetric and what does it mean?

Answer 17

quadratic ( L(e)=e^2 ) and absolute (L(e) =|e| )

Both are symmetric: penalize positive and negative errors of the same magnitude in the same way.

Question 18

What is the difference between the 2 common loss function

Answer 18

Quadratic loss penalizes large errors much more severely than small errors.

Question 19

What is the purpose of having asymmetric loss functions?

Answer 19

If you have something that is safer to overestimate than to undersestinate

Question 20

What is positive erorr and negative error?

Answer 20

If error is positive, you undershot
If error is negative, then you overshot

Question 21

What is risk

Answer 21

expected loss

Question 22

What is the optimal forecast under quadratic loss?

Answer 22

mean of F(y)

Question 23

We can improve our forecast further by ________________________

Answer 23

conditioning on predictors

Question 24

What happens when you allow model to tailor to specific noise in data?

Answer 24

overfitting

Question 25

What is the goal of ML from predictive modelling?

Answer 25

“learn” f(X) from data in a way that yields good out-of-sample forecasting performance or get a forecasting rule that minimizes expected loss

get a good estimate of E(Y|X)

Question 26

What is the optimal conditional point forecast?

Answer 26

optimal conditional point forecast Yˆ( X ) is the function of the conditional predictive distribution F(y|x) which minimizes the risk (expected loss).

Question 27

What is the optimal conditional point forecast for qudratic loss?

Answer 27

conditional mean

Question 28

What can traditional non parametrics be used to help fit ?

Answer 28

flexible nonlinear functions (kernels, series). (for estimating the structure of data)

Question 29

For modelling, we need to trade off _____ with __________

Answer 29

signal extraction (bias)
overfitting (variance).

Question 30

What is the problem an dits solution for using bayes classifier?

Answer 30

must know true F(y|x), so it is unattainable.

estimate F(y|x) and classify based on the highest
estimated conditional probability.

Question 31

What is the difference between parametric and nonparametric?

Answer 31

Parametric: “fixed” model, number of parameters fixed, faster computation, but stronger assumption on F(y|x)

Nonparametric: “flexible” model, number of parameters may grow with available data, computation becomes harder (or untractable) with larger datasets.

Question 32

Is knn parametric or nonparametric?

Answer 32

nonparametric (lazy /instance based learner)

Question 33

What are the steps for knn for k=3?

Answer 33

Find the 3 closest points of the datapoint to the pooint to classify

calculate euclidean distance and place them in the set for the k nearest neighbours

estimate conditional probabilities

classify test observation to class with highest prob

(FIND K MOST SIMILAR AND LET THEM VOTE)

Question 34

In order to make a predcition we need to ____________

Answer 34

look at the entire data

Question 35

Prediction step for knn is _______________ (speed)

Why is it such?

Answer 35

slow (need to ssort through data every time to determine nearest points)

Question 36

What must we watch out for for some non parametric methods?

Answer 36

SCALING (esp KNN)

Question 37

Why is scaling important for KNN? How to scale?

Answer 37

typically uses Euclidean distance, so we would get a very different answer depending on the scaling of X (e.g., income in SGD vs. in thousands of SGD). S

Standardiize or scale to [0,1]

Question 38

What must we watch up for euclidean distance calculation? (as in when does it not make sense to use it)

Answer 38

CATEGORICAL PREDICTORS

assign zero if classes coincide with the test observation, and 1 if not.

Question 39

What is 0-1 loss?

Answer 39

assign zero if classes coincide with the test observation, and 1 if not.

(1 can be replaced by a higher number for some mismatches based on domain knowledge.)

Question 40

Do we need to standardize variables for KNN in r?

Answer 40

NO

Question 41

What is good about knn for regression task and tuning value of k?

Answer 41

readily extends to predicting continuous variables.

Question 42

How should you choose the k value if you have many predictors?

Answer 42

OUT sample MSE

Question 43

How to discriminate between models? What are the steps ?

Answer 43

using data not used in estimation to compute out of sample risk estimate

1. Training sample: data used to estimated prediction rule(s).
2. Validation sample: data used to test estimated rule(s) on NEW data (a.k.a. hold-out sample).

Question 44

We should allow the algorithm minimize the loss over the training set. True/False

Why?

Answer 44

FALSE

any function that passes through all the datapoints would set it to 0

likely result in terrible out-of-sample performance – such model would be overfit to the training data and will generalize poorly, since we chase noise specific to the sample.

Question 45

What is the solution to overfitting?

Answer 45

introduce some penalty for complexity – this is referred to as regularization. In this example, regularization comes in through the choice of K.

Question 46

Generally, _________decreases with complexity, but ___________increases.

Answer 46

bias
variance

Question 47

What are the roles of conditioning?

Answer 47

By conditioning on available information we can make the forecasts more accurate.

Conditioning reduces the risk of the forecast.

Ignoring estimation error considerations, conditioning on more information is always better in the sense of reducing risk.

Question 48

What is the optimal point forecast under absolute loss ? What loss function do we normally use ? Why do we not use nonstandard losses ?

Answer 48

Under absolute loss, the optimal point forecast is the median.

Other loss functions lead to different solutions. Working out optimal forecasts under nonstandard losses could be tricky.

Sometimes we do not have an explicit loss function, so we take the simplest approach and use the quadratic loss.

In some real-world applications (e.g., policymaking) a very explicit loss function may arise. In that case, it would be best to use the estimators and forecasts tailored to the loss.

Question 49

What is the goal of most predictive modelling?

Answer 49

Get a good estimate of E(Y|X)

Question 50

What is the approach to modelling E(Y|X) esp in econometrics

Answer 50

traditional parametrics (fit flexible non linear function)
(cause most econometric model assume e(Y|X) approx linear at least in parmaeters

while trading off signal extraction(bias) with overfitting(variance)

Question 51

What are the approaches to classifciation

Answer 51

parametric and non parametric

Question 52

What is the problem of bayes classifer and the solution?

Answer 52

Problem: must known true F(y|x) so it is unattainable
Solution: estimate it and classify based on highest estimated CONDITIONAL PROBABILITY

Question 53

Why is knn called lazy learner?

Answer 53

doesn’t produce any model and hence any “understanding” of how X relates to Y: just lets the K most similar training data points “vote” on the class of the test observation.

Question 54

What is the issue of categorical predictors for knn? How to solve it?

Answer 54

Euclidean distance doesn’t make much sense when applied to categorical predictors. If assign number to different categories, the error might be differnet depending on which categoresi are assigned to which number, although all should be same error

Solution: 0-1 loss

Question 55

Can 1 be replaced by higher number in 0-1 loss?

Answer 55

Yes depending on domain knowledge

Question 56

What is regularization? How to do it in knn?

Answer 56

penalty for complexity,
To restrict choice available f(X).
Comes in through choice of k

Question 57

How does a line of higher k value look like as compared to lower k value

Answer 57

very stable from sample to sample: low var
misses many X values: high bias