Test 2

Question 1

How can a paired t-test could be used to compare two models that have been developed for a classification problem?

Answer 1

A paired T test could be used to compare two classification models by applying both models to the test and comparing their error rates. The differences between these two will be calculated and the mean and standard dev of said differences are used in the t test to determine if there is a stat sig difference between the models

Question 2

How might a data set be manipulated to simulate weighting instances?

Answer 2

By duplicating the instances you want to see with a higher weight in the data set

Question 3

Briefly describe how the back-propagation algorithm works in MLPs.

Answer 3

It works by shifting the weights of the connections using gradient descent. This occurs iteratively until the error is acceptably low

Question 4

Why might we wish to weight some instances over others?

Answer 4

To fix underrepresented data or to bias a model to a certain outcome to avoid very bad results from improper classification

Question 5

What activation function is often used in an MLP?

Answer 5

Sigmoid function, ReLU, and tanh

Question 5

How is a signal propagated from one layer of a multilayer perceptron (MLP) to the next?

Answer 5

Through a system of weights and biases between nodes. The input is multiplied by the weight and biases are added.

Question 5

What is a bias node in an MLP? What purpose does it serve?

Answer 5

Additional node in each layer of a MLP. It has a constant output of 1 and is connected to all nodes in the next layer. This node helps the network learn more complex decisions by shifting the activation functions horizontally

Question 6

What role does the training set, validation set, and test set play in model development

Answer 6

Training set is the set that the model is trained on. While training the model is tested on the validation set and when the model is done it is compared against the test set

Question 7

In the context of training and evaluation of ML models, what is holdout? What is cross-validation?

Answer 7

A holdout is method where part of the data is set aside and used to test the data. Cross validation is a model technique that splits the data into multiple datasets, and averages the results

Question 8

If your training data is very large and representative of the population to which the final model will be applied, should you perform
cross-validation?

Answer 8

Not necessary as it would take to long and not provide a much better model

Question 8

In a regression tree, we do not use information gain as a splitting criterion. Assuming all attributes are numeric, how then are splits performed? How do we know when to terminate the splitting process?

Answer 8

We use a variance of the target attribute values and split to minimize that. You terminate based on a threshold set

Question 9

What are the distinctions between a regression tree and a model tree?

Answer 9

A regression tree predicts a single value while a model tree has a linear model

Question 10

How could a ML algorithm such as C4.5 (J48) be used for attribute selection? How about linear regression

Answer 10

By training a decision tree on a dataset and selecting the attributes that appear in the tree.
Linear regression can be used for attribute selection by fitting a linear model to the data and selecting attributes with the highest absolute coefficients or lowest p-values, indicating their significance in predicting the target variable.

Question 11

In C4.5, how are instances with missing attribute-values utilized during training? During classification?

Answer 11

By using a probabilistic approach distribution of known values. For classification the missing values are propagated down multiple branches and the final prediction is based on a weighted average of the outcomes

Question 12

What are forward selection and backward selection in the context of attribute selection? Which is likely to produce a set containing
more features?

Answer 12

Forward starts with no attributes and then adds them one-by-one until a suitable set is found. Backward selection starts with all
attributes and then eliminates individual attributes. The latter will generally produce larger sets.

Question 12

We sometimes want to discretize numeric attributes. Two methods to do that are equal-interval binning and equal-frequency
binning. Explain the basic ideas underlying each

Answer 12

Equal interval binning would have splits based upon the range of numeric values of the attributes (i.e., we divide the range of values into multiple intervals of the same size), while equal frequency binning would choose splits that result in sets of roughly
the same size.

Question 12

As discussed, C4.5 uses error on the training set rather than the test set to drive pruning. However, to avoid overfitting, an estimate is made. What is this estimate?

Answer 12

The upper bound of the training error

Question 13

What is meant by recursive feature elimination?

Answer 13

It’s an attribute selection method whereby one repeatedly applies an ML algorithm that provides coefficients for attributes commonly used with linear regression is used. The attribute with the lowest value is removed and the algorithm applied again. This process is repeated until no attributes remain. In general, the scheme provides a way of ranking attributes.

Question 14

What is an ensemble learner? Describe one or more (simple) techniques for combining results from multiple models

Answer 14

A collection of learners that have been combined to solve a problem.
Voting: each base model makes a prediction, final prediction is the majority vote (for classification) or average (for regression)

Question 15

What are some of the methods used to evaluate the quality of a feature set in attribute selection?

Answer 15

Choose attributes that are individually correlated with the target. Alternatively, we could examine sets of attributes, looking for sets containing attributes that are individually correlated to the target but with low inter-correlation

Question 16

What do we mean when we say that an attribute selection method is scheme independent?

Answer 16

Attributes are selected based upon characteristics of the data set and not based on performance of a machine learning scheme. For instance, we could use correlation with the target attribute to select attributes this does not utilize any ML algorithm.

Question 17

In the context of ensemble learning, what is a weak learner?

Answer 17

A model that performs only slight better than random guessing

Question 17

What computationally cheaper technique might be used to derive an attribute set rather than PCA?

Answer 17

Choosing the vectors randomly appears to work well in practice.

Question 17

What is stacking?

Answer 17

An ensemble learning technique that combines multiple base models by training a meta-model to learn how to best combine their predictions.

Question 17

In informal terms, how can principal component analysis be used to reduce the number of attributes used in a machine learning algorithm? What can be said about the attributes ultimately produced by PCA?

Answer 17

PCA will produce new attributes that are based on one or more of the original attributes. It works by choosing a vector in which
variance (of target values, when instances are projected onto it) is maximal. Subsequent vectors are chosen in the same way, but
orthogonal to those previously selected. As such, the same information encoded in multiple original attributes could be encoded more concisely

Question 17

What are bagging and boosting, and how are they different?

Answer 17

Techniques that combine multiple base models to improve performance Bagging creates multiple subsets of the training data by sampling with replacement, trains a base model and combines the predictions
Boosting iteratively trains weak learners on weighted versions of the dataset where misclassified instances receive higher weights.

Question 18

Does bagging utilize every instance in a training set? Explain.

Answer 18

Bagging uses sampling with replacement. As such, its possible for the re-sampled set to contain every instance of the underlying training set, but in realistic settings it is improbable.

Question 18

In post-pruning of decision trees using subtree replacement, when is a subtree replaced with a leaf node?

Answer 18

When doing so improves performance

Question 18

Why is bagging recommended in cases where the underlying machine learning scheme is sensitive to variations in the data?

Answer 18

Bagging is recommended when the underlying machine learning scheme is sensitive to variations in the data because it helps reduce variance and improve stability.

Question 18

What is a paired T-test?

Answer 18

A statistical method used to compare two related samples or measurements to determine if there is a significant difference between their means.

Question 18

In stacking, why is it wrong to use the data used in developing the level-0 models to combine them?

Answer 18

Because the resulting learner will naturally prefer level-0 models that over-fit the training data used to produce them. If we use a separate set to develop the level-1 model, then this will be better estimate of performance on unseen data.

Question 18

Suppose you have a data set with 1000 instances and 100 attributes. Discuss how random forests make use of training instances and their attributes to create a classifier.

Answer 18

Random forests uses bagging and attribute selection to generate a set of simple decision trees. Each tree is developed using only a subset of the full number of attributes. E.g., each tree could be developed using only 500 instances (possibly with duplicates) taken from the 1000 and only 20 randomly selected attributes from the original 100.

Question 19

In AdaBoost.M1, what happens if the current classifier has an accuracy level of less than 50

Answer 19

The instances are reweighted and the algorithm continues (another model is trained). It’s only
when the error is 0 or ≥ 0.5 that that the algorithm terminates.

Question 19

What is a confidence interval?

Answer 19

Confidence intervals provide a range of values that are likely to contain the true population parameter with a certain level of confidence. They give us a way to estimate an unknown population parameter based on a sample statistic, while accounting for the uncertainty in our estimate due to sampling variability.

Question 19

Do bagging and boosting typically combine multiple, distinct ML schemes?

Answer 19

Typically not. Bagging uses the same ML scheme but varies the data sets used to develop models, while boosting generates a sequence of models based on a common scheme and using a common data set but re-weighting instances for each subsequent
model.

Question 20

What is the Backpropagation Algorithm?

Answer 20

A supervised learning method used to train multi-layer perceptrons (MLPs) and other types of feedforward neural networks.

Question 20

Briefly describe how AdaBoost.M1 works. How are instances weighted?

Answer 20

It works by iteratively training weak learners (e.g., decision stumps) on weighted versions of the dataset. Initially, all instances have equal weights. In each iteration, a weak learner is trained on the weighted data, and its error rate is calculated. The weights of misclassified instances are increased, while the weights of correctly classified instances are decreased. This forces the next weak learner to focus more on the misclassified instances. The process is repeated for a specified number of iterations. The final classifier combines the weak learners’ predictions weighted by their accuracy.

Question 20

What is Cross-entropy?

Answer 20

A loss function commonly used in machine learning, particularly in classification problems. It measures the dissimilarity between two probability distributions: the true distribution of the data (usually represented by one-hot encoded class labels) and the predicted distribution (usually the output of a softmax function).

Question 20

What is Softmax?

Answer 20

An activation function commonly used in the output layer of neural networks for multi-class classification problems. It converts a vector of real numbers into a probability distribution over multiple classes, ensuring that the outputs sum up to 1 and can be interpreted as class probabilities.

Question 21

What is Cross Validation?

Answer 21

A method for partitioning data and assessing model performance. In k-fold cross-validation, the data is divided into k equally sized folds. The model is trained and evaluated k times, using each fold once as the testing set and the remaining folds as the training set. The final performance estimate is the average of the k evaluations.

Question 21

What is a training set?

Answer 21

The training set is the portion of the data used to train the machine learning model. The model learns from the examples in the training set by adjusting its parameters to minimize the loss function.

Question 22

What is stratified sampling?

Answer 22

When partitioning data, it’s important to ensure that the class distribution in each subset is representative of the original dataset. Stratified sampling techniques, such as stratified k-fold cross-validation, maintain the class proportions in each fold.

Question 23

What is a testing set?

Answer 23

The testing set is used to evaluate the performance of the trained model on unseen data. It provides an unbiased estimate of the model’s generalization ability, i.e., how well it performs on new, previously unseen examples.

Question 24

What is a validation set?

Answer 24

The validation set is used for model selection and hyperparameter tuning. It helps in choosing the best model architecture, regularization techniques, and other hyperparameters that maximize the model’s performance on unseen data. The validation set is used during the training process to guide the model selection and prevent overfitting.

Question 24

What is Post-prunning?

Answer 24

It involves recursively removing or collapsing subtrees that do not improve the model’s performance on a separate validation set or based on a complexity criterion. The subtrees are replaced with leaf nodes assigned the majority class or the average value of the target variable.

Question 25

What is pre-prunning?

Answer 25

Stopping the tree growth early, before it reaches its maximum depth or all the leaves are pure. This is done by setting a threshold on the minimum number of instances required to split a node, the maximum depth of the tree, or the minimum improvement in impurity reduction required for a split.

Question 25

What is Decision Tree pruning?

Answer 25

A technique used to simplify decision trees and prevent overfitting.

Question 26

What is Reduced error pruning?

Answer 26

In this post-pruning approach, each subtree is considered for pruning. If replacing a subtree with a leaf node reduces the error rate on a separate validation set, the subtree is pruned. This process is repeated until no further improvements can be made.

Question 26

What is cost-complexity pruning (weakest link prruning)

Answer 26

This post-pruning method introduces a complexity parameter (α) that controls the trade-off between the tree’s complexity and its goodness of fit. The algorithm finds the subtrees that minimize the cost complexity, defined as the sum of the misclassification rate and the product of α and the number of leaf nodes. The optimal value of α is determined using cross-validation.

Question 27

What is PCA (principal component analysis)?

Answer 27

A linear transformation technique that reduces the dimensionality of the data by projecting it onto a lower-dimensional space while retaining the maximum amount of variance. PCA identifies the principal components, which are the directions of maximum variance in the data, and transforms the features into a new set of uncorrelated variables.

Question 27

What is Random Projections?

Answer 27

A dimensionality reduction technique that projects the high-dimensional data onto a lower-dimensional subspace using a randomly generated matrix. It preserves the pairwise distances between data points and is computationally efficient.

Question 28

What is Recursive Feature Elimination?

Answer 28

A wrapper-based feature selection method that recursively removes the least important features from the dataset. It starts with all the features and iteratively trains a model, ranks the features based on their importance (e.g., using the model’s coefficients or feature importances), and eliminates the least important features. This process is repeated until the desired number of features is reached.

Question 29

What is Forward Stepwise Selection?

Answer 29

Forward Stepwise Selection is an iterative feature selection method that starts with an empty feature set and gradually adds the most relevant features one at a time. In each iteration, the feature that provides the greatest improvement in the model’s performance is added to the feature set. The process continues until a stopping criterion is met, such as a maximum number of features or no further improvement in performance.

Question 30

What is Backward Stepwise Selection?

Answer 30

Iteratively removes the least relevant features one at a time. In each iteration, the feature whose removal leads to the smallest decrease in the model’s performance is eliminated. The process continues until a stopping criterion is met.

Question 31

What is minimum description length pruning?

Answer 31

MDL pruning is based on the principle of finding the simplest model that encodes the data well. It considers the tree’s complexity and its ability to compress the data. The goal is to minimize the sum of the description length of the tree and the description length of the data given the tree.

Question 31

What is TF-IDF?

Answer 31

A numerical statistic used to reflect the importance of a word in a document within a collection of documents (corpus). It is commonly used as a feature transformation technique in text mining and information retrieval.

Question 31

What is Bagging?

Answer 31

Bagging is an ensemble method that creates multiple subsets of the training data by random sampling with replacement (bootstrap sampling). Each subset is used to train a separate base learner, typically a decision tree The results are then averaged out to get a final prediciton

Question 32

What is TF?

Answer 32

Measures the importance of a word across the entire corpus. The TF-IDF score is the product of TF and IDF, which helps in identifying the words that are highly frequent in a document but rare in the corpus, indicating their importance.

Question 32

What is Discretizing?

Answer 32

The process of converting continuous or numeric attributes into discrete or categorical attributes. It involves dividing the range of the continuous attribute into a set of intervals or bins and assigning each interval a discrete value or label.

Question 33

What is Unsupervised Discretization?

Answer 33

Dividing the attribute range into intervals without considering the target variable. Equal-width binning creates bins of equal size, while equal-frequency binning creates bins with an equal number of instances in each bin.

Question 34

What is One-hot encoding?

Answer 34

A technique used to convert categorical attributes into a binary vector representation. It creates a new binary attribute for each unique category in the original attribute, where a value of 1 indicates the presence of the category and 0 indicates its absence.

Question 34

What is ensemble learning?

Answer 34

A machine learning paradigm that combines multiple individual models, known as base learners or weak learners, to create a stronger and more accurate predictive model. The idea behind ensemble learning is that by combining the predictions of multiple models, we can reduce the errors and improve the overall performance compared to using a single model.

Question 35

What is Supervised Discretization?

Answer 35

Taking the target variable into account when creating the intervals. They aim to find the optimal split points that maximize the information gain or minimize the impurity with respect to the target variable.

Question 36

What is Boosting?

Answer 36

Combines weak learners to create a strong learner. The weak learners are trained sequentially, with each learner focusing on the instances that were misclassified by the previous learners.

Question 36

What is Adaboosting?

Answer 36

A popular boosting algorithm that assigns weights to the training instances based on their difficulty. Initially, all instances have equal weights.

Question 36

What is stacking?

Answer 36

An ensemble method that combines the predictions of multiple base learners using a meta-learner. The base learners are trained on the original training data, and their predictions are used as input features for the meta-learner.

Question 37

What is Random Forests?

Answer 37

An ensemble method that combines bagging with random feature selection. It creates multiple decision trees using bootstrap sampling of the training data. At each split in the decision trees, a random subset of features is considered for the split, introducing additional randomness and reducing the correlation between the trees.
The final prediction is obtained by majority voting (for classification) or averaging (for regression) of the predictions of all the decision trees.
Random Forest is known for its robustness, high accuracy, and ability to handle high-dimensional data.