7.1 - Types of Learning and Overfitting Problems

Question 1

_______ _________ uses labeled training data to guide the ML program toward superior forecasting accuracy.

Answer 1

Supervised learning

Question 2

Typical tasks for supervised learning include _________ and __________.

Answer 2

classification ; regression

Question 3

If the target variable is ___________, the model involved is a regression model.

Answer 3

continuous

Question 4

Multiple regression is an example of ________ learning.

Answer 4

supervised

Question 5

Classification models are used in cases where the target variable is ________ or _________ (e.g., ranking).

Answer 5

categorical ; ordinal

Question 6

Algorithms can be designed for __________ classification (e.g., classifying companies as likely to default vs. not likely to default) or _____________ classification (e.g., a ratings class for bonds).

Answer 6

binary ; multicategory

Question 7

In ____________ learning, the ML program is not given labeled training data; instead, inputs (i.e., _________) are provided without any _____________ about those inputs.

Answer 7

unsupervised ; features ; conclusions

Question 8

___________ is an example of an unsupervised ML program.

Answer 8

Clustering

Question 9

In __________ learning, in the absence of any ________ variable, the program seeks out structure or interrelationships in the data.

Answer 9

unsupervised ; target

Question 10

_______ _________ algorithms are used for complex tasks such as facial (image) recognition, natural language processing, etc.

Answer 10

Deep learning

Question 11

Programs that learn from their own prediction errors are called _________ ____________ algorithms.

Answer 11

reinforced learning

Question 12

Deep learning algorithms and reinforced learning algorithms are based on ________ ________ , a group of ML algorithms applied to problems with significant _____________ .

Answer 12

neural networks; nonlinearities

Question 13

__________ is an issue with supervised ML that results when a large number of ________ (i.e., independent variables) are included in the data sample.

Answer 13

Overfitting; features

Question 14

In overfitting, ___________ is misperceived to be a pattern, resulting in high __-_____ __-________.

Answer 14

randomness; in-sample R-squared

Question 15

Overfitting has occurred when the noise in the ________ ________ seems to improve the model fit.

Answer 15

target variables

Question 16

Overfit models do not _________ ________ to new data (i.e., ____-___-____ ___-______ will be low).

Answer 16

generalize well ; out-of-sample R-squared

Question 17

You “train” the model using __-_____ data.

Answer 17

in-sample

Question 18

When a model _______ ______, it means that the model retains its explanatory power when it is applied to new, (i.e., ___-__-_____) data.

Answer 18

generalizes well; out-of-sample

Question 19

To measure how well a model generalizes, data analysts create three non-overlapping data sets. List them and provide a brief explanation of each.

Answer 19

(1) training sample = used to develop the model. (2) validation sample = used for tuning the model. (3) test sample = used for evaluating the model using new data.

Question 20

In-sample prediction errors occur with the ________ sample, while prediction errors in the ______ and ______ samples are known as out-of-sample errors.

Answer 20

training validation and test

Question 21

Which supervised learning model is most appropriate (1) when the Y-variable is continuous and (2) when the Y-variable is categorical Continuous Y-variable Categorical Y-variable A) Decision trees Regression B) Classification Neural Networks C) Regression Classification

Answer 21

When the Y-variable is continuous, the appropriate approach is that of regression (used in a broad, ML context). When the Y-variable is categorical (i.e., belonging to a category or classification) or ordinal (i.e., ordered or ranked), a classification model is used. C

Question 22

In machine learning, out-of-sample error equals: A) forecast error plus expected error plus regression error. B) Standard error plus data error plus prediction error. C) bias error plus variance error plus base error.

Answer 22

Out-of-sample error equals bias error plus variance error plus base error. Bias error is the extent to which a model fits the training data. Variance error describes the degree to which a model’s results change in response to new data from validation and test samples. Base error comes from randomness in the data. C

Question 23

Overfitting is least likely to result in: A) higher forecasting accuracy in out-of-sample data. B) higher number of features included in the data set. C) inclusion of noise in the model.

Answer 23

Overfitting results when a large number of features (i.e., independent variables) are included in the data sample. The resulting model can use the “noise” in the dependent variables to improve the model fit. Overfitting the model in this way will actually decrease the accuracy of model forecasts on other (out-of-sample) data. A

Question 24

Which of the following about unsupervised learning is most accurate? A) There is no labeled data. B) Classification is an example of unsupervised learning algorithm. C) Unsupervised learning has lower forecasting accuracy as compared to supervised learning.

Answer 24

In unsupervised learning, the ML program is not given labeled training data. Instead, inputs are provided without any conclusions about those inputs. In the absence of any tagged data, the program seeks out structure or inter-relationships in the data. Clustering is one example of the output of unsupervised ML program while classification is suited for supervised learning. A

Question 25

Which of the following statements about supervised learning is most accurate? A) Typical data analytics tasks for supervised learning include classification and prediction. B) Supervised learning requires human intervention in machine learning process. C) Supervised learning does not differentiate between tag and features.

Answer 25

Supervised learning utilizes labeled training data to guide the ML program but does not need “human intervention.” Typical data analytics tasks for supervised learning include classification and prediction. A

Question 26

The technique in which a machine learns to model a set of output data from a given set of inputs is best described as: A) deep learning. B) supervised learning. C) unsupervised learning.

Answer 26

Supervised learning is a machine learning technique in which a machine is given labelled input and output data and models the output data based on the input data. In unsupervised learning, a machine is given input data in which to identify patterns and relationships, but no output data to model. Deep learning is a technique to identify patterns of increasing complexity and may use supervised or unsupervised learning. B

Question 27

A rudimentary way to think of machine learning algorithms is that they: A) “synthesize the pattern, review the pattern.” B) “find the pattern, apply the pattern.” C) “develop the pattern, interpret the pattern.”

Answer 27

One elementary way to think of ML algorithms is to “find the pattern, apply the pattern.”Machine learning attempts to extract knowledge from large amounts of data by learning from known examples in order to determine an underlying structure in the data. The focus is on generating structure or predictions without human intervention. B

Question 28

The degree to which a machine learning model retains its explanatory power when predicting out-of-sample is most commonly described as: A) predominance. B) generalization. C) hegemony.

Answer 28

Generalization describes the degree to which, when predicting out-of-sample, a machine learning model retains its explanatory power. B

Question 29

Which statement about target variables is most accurate? A) They can be continuous, ordinal, or categorical. B) They are not specified for supervised learning. C) They refer to independent variables.

Answer 29

Target variables (i.e., dependent variables) can be continuous, ordinal, or categorical. Target variables are not specified for unsupervised learning. A

Question 30

Which statement most accurately describes supervised learning? A) It uses labeled training data. B) It requires periodic human intervention. C) It is best suited for classification.

Answer 30

Supervised learning uses labeled training data, and it does not need human intervention. Classification algorithms can be used for both supervised and unsupervised learning. A

Question 31

A model that has poor in-sample explanatory power is most likely to have a high: A) bias error. B) variance error. C) base error.

Answer 31

Bias error is the in-sample error resulting from models with a poor fit. (LOS 7.b) A

Question 32

The problem of overfitting a model would least appropriately be addressed by: A) imposing a penalty on included features that do not add to explanatory power of the model. B) using cross validation. C) using a smaller sample.

Answer 32

To reduce the problem of overfitting, data scientists use complexity reduction and cross validation. In complexity reduction, a penalty is imposed to exclude features that are not meaningfully contributing to out-of-sample prediction accuracy. C

Question 33

Cross validation occurs when: A) training and validation samples change over the learning cycle. B) prediction is tested in another heterogeneous sample. C) the performance parameter is set by another algorithm.

Answer 33

In cross validation, the training and validation samples are randomly generated every learning cycle. (LOS 7.b) A

Question 34

Out-of-sample error originates from what three sources?

Answer 34

(1) Bias error (2) Variance error (3) Base error

Question 35

________ error is the in-sample error resulting from models with a poor fit. Describe with a diagram in terms of accuracy rate and sample size.

Answer 35

Bias

Question 36

_______ error is the out-of-sample error resulting from overfitted models that do not generalize well. Describe with a diagram in terms of accuracy rate and sample size. .

Answer 36

Variance

Question 37

Use a diagram to describe a robust model in terms of accuracy rate and sample size.

Answer 37

Robust model:

Question 38

______ error is the residual errors due to random noise (the randomness in the data).

Answer 38

Base

Question 39

Is there any effective way to determine the three individual error components within the overall error of a model? What are these three components?

Answer 39

No. (1) Bias error (2) Variance error (3) Base error

Question 40

________ error increases with complexity, while ______ error decreases with complexity.

Answer 40

Variance; bias

Question 41

To reduce the problem of overfitting, data scientists use ________ ________ and ______ ___________.

Answer 41

complexity reduction; cross validation

Question 42

In _______ _______, a penalty is imposed to exclude features that are not meaningfully contributing to out-of-sample prediction accuracy.

Answer 42

complexity reduction

Question 43

In ______ _______, the penalty value increases with the number of independent variables (features) used by the model.

Answer 43

complexity reduction

Question 44

The sampling technique known as _____ ________ estimates out-of-sample error rates directly from the validation sample.

Answer 44

cross validation

Question 45

In a __-___ _____ ______, the validation sample is randomly divided into “k” parts in order to estimate the out-of-sample error rates. The ________ sample comprises (k - 1) parts, with one part left for _________. ______ is then measured for the model in each of the parts. this process is repeated ___ times, and the average in-sample and out-of-sample error rates are compiled.

Answer 45

k-fold cross validation; training; validation; Error; “k”

Question 46

The objective of ML is to determine _______ or generate ________ without ______ __________.

Answer 46

structure; forecasts; human intervention