General

Question 1

Stochastic Gradient Descent

Answer 1

Gradient descent algorithm that increments the parameters using only single observations at a time.

More efficient than batch gradient descent, especially with large datasets.

Question 2

Batch Gradient Descent

Answer 2

Gradient descent algorithm in which it is required to scan through the entire training set before taking a single step.

Question 3

Localized Linear Regression

Answer 3

A variant of traditional linear regression that uses only local data points around Xi to predict Yi

Question 4

Type I Error (False Positive)

Answer 4

Incorrectly rejecting the null hypothesis in favor of the alternative hypothesis when the null is true.

Same as alpha, set at the beginning of the experiment.

Question 5

Type II Error (False Negative)

Answer 5

Failing to reject the null hypothesis when it is false

Also known as beta. Note that power is (1 - beta)

Question 6

A\B Testing

Answer 6

Processing of testing two groups against a given desired measured to determine if there is a statistical difference

General Strategy:

1. Identify comparative statistic
2. Determine sample size
3. Analyze results

Question 7

SVMs : General description

Answer 7

Simple: Machine learning model that uses a hyperplane to differentiate and classify different groups of data

Detailed: SVM identified an appropriate hyperplane by attempting to maximize the margins between points between the closest points of each class to boundary.

If data that cannot be separated linearly, use transformations to map data into high dimensions.

Question 8

SVMs: Soft Margin Classification

Answer 8

A mechanism that serves to reduce the overfitting of maximum margin classification by penalizing misclassifications

Question 9

Bias / Variance Tradeoff

Answer 9

A tradeoff in machine learning models where you have the choice of reducing bias (how well a model fits a specific set of data) vs. reducing variance (how much performance of a mode varies across many datasets).

Question 10

Precision

Answer 10

TP / (TP + FP)

Measures the accuracy of positive predictions (but not necessarily identifying all of them).

Question 11

Recall

Answer 11

TP / (TP + FN)

Measures completeness of positive predictions

Question 12

F1 Score

Answer 12

2 / ((1/Precision) + 1/Recall))

Harmonic mean of precision and recall, ranging between 0 and 100%

Question 13

ROC Curve

Answer 13

Plots true positive rate (recall) against false positive rate. A good ROC curve goes toward the top left of the chart.

X = FPR
Y = TPR

Question 14

False Positive Rate
(1 - Specificity)

Answer 14

Proportion of negative instances that are incorrectly classified as positive (i.e. false positive)

FP / (FP + TN)

Question 15

Lasso Regression

Answer 15

Check this

Question 16

Elastic Net

Answer 16

Check this

Question 17

Early Stopping

Answer 17

A way of regularizing a model by stopping training once validation error reaches a minimum

Question 18

Soft max Regression

Answer 18

Also known as multinomial logistic regression.

Classification with multiple classes. For each instance x, assigns a score s(x) for each class k, then estimates probability by applying a soft max function.

Soft max function is as followed:

Exp(s(x)) / summation(exp(s(x)))

Question 19

Cross-entropy

Answer 19

Loss function used to measure difference between predicted and true probability distributions. Penalizes low probability on true labels significantly.

-1/m ∑ ∑y log (p(k))

Essentially the mean of the -log(estimated probabilities).

Question 20

Accuracy

Answer 20

Number of correctly classified instances / number of all classified instances

Question 21

True Positive Rate
(Sensitivity)

Answer 21

Proportion of positive instances that are correctly classified as positive (i.e. true positive)

TP / (TP + FN)

Question 22

Specificity

Answer 22

True Negative Rate

Proportion of negative instances that are correctly classified as negative (i.e. true negative)

TN / (TN + FP)

Question 23

Gradient Descent

Answer 23

An algorithm that minimizes a particular function (in ML the loss function) by taking small steps in the direction of the steepest descent for that function.

Step 1: Take the derivative of the loss function for each parameter (i.e. take the gradient of the loss function).

Step 2: Initialize parameters with random values

Step 3: Plug parameters into the partial derivatives (gradient)

Step 4: Calculate step sizes (Calculated slope from step 3 * learning rate)

Step 5: Calculate the new parameters (New = Old - Step Size)

Step 6: Repeat 4-5 until convergence

Question 24

Steps for K-fold Cross Validation

Answer 24

Step 1: Shuffle data into equally sized blocks (folds)

Step 2: For each fold k, train model on all data except fold i, and evaluate validation error using the remaining fold i.

Step 3: Average the validation errors from step 2 to get estimate of the true error.

Question 25

Bootstrapping

Answer 25

Drawing observations from a large data sample repeatedly (sampling with replacement) and then estimating some quantity of a population by averaging estimates from multiple smaller samples.

Useful for small data sets and helping to deal with class imbalance.

Question 26

Hyperparameter tuning:
Grid search

Answer 26

Forming a grid that is the Cartesian product of all parameters and then sequentially trying all such combinations and seeing which yields best results.

Question 27

Hyperparameter tuning:
Random Search

Answer 27

Randomly sample from the joint distribution of all parameters.

Question 28

ROC Curve
AUC?

Answer 28

Plots the true positive rate (y) against the false positive rate (x) for various thresholds.

Area under the curve (AUC) measures how well the classifier separates classes.

Question 29

Conditional Probability

P(A/B)

Answer 29

P(A ∩ B) / P(B)

Question 30

Bayes Theorem

Answer 30

P(H/E) = P(H)*P(E/H) / P(E)