Supervised Learning

Question 1

What is supervised learning?

Answer 1

A subcategory of M.L. defined by the use of labeled input/output sets.

Question 2

What is the difference between regression and classification?

Answer 2

Regression is used to predict continuous values such as price or income. The goal is to find a best-fit line. Classification is used to predict a discrete class label, goal: decision boundary.

Question 3

What kind of problems can you solve with classification and regression?

Answer 3

Regression: weather prediction, housing price prediction
Classification: spam detection, speech recognition, cancer cell identification.

Question 4

Why is training set error performance unreliable?

Answer 4

Doesn’t generalize to unseen data. Perfect training set performance equals overfitting.

Question 5

What is machine learning?

Answer 5

A field of artificial intelligence concerned with algorithms that can learn from data.

Question 6

Two main branches of Machine Learning?

Answer 6

Supervised learning
Unsupervised learning

Question 7

Two main branches of Machine Learning?

Answer 7

Supervised learning
Unsupervised learning

Question 8

3 requirements for machine learning?

Answer 8

1) A pattern exists
2) that cannot be pinned down mathematically
3) We have data on it

Question 9

Define data (for M.L)

Answer 9

Input - correct output pairs (feature, label)
input - real-valued or categorial
output - real-valued (regression) or categorical (classification)

Question 10

Goal of supervised learning?

Answer 10

To model dependency between features and labels.

Question 11

Goal of a supervised learning model?

Answer 11

To predict labels for new instances.

Question 12

What is a training set?

Answer 12

A set of input - output pairs.

Question 13

Classification output value types?

Answer 13

Categorical or binary (-1,1)

Question 14

Regression output value type?

Answer 14

Real numbers.

Question 15

Examples of supervised learning problems?

Answer 15

Junk mail:
features - word frequencies
class - junk/not junk

Access Control System:
features - images
class - ID of the person

Medical diagnosis:
features: BMI, age, symptoms, test results
class: diagnostic code

Question 16

Formal components of learning.

Answer 16

Input (x) - e.g. customer application
Output (y) - (approval/denial of application)
Target function: x -> y (ideal credit approval formula)
Data {(x1, y1), … (xn, yn)} (historical records)
Hypothesis: g; X -> Y
Hypothesis set (H): group of functions where we look for our solution
Supervised learning uses test data to learn this function from H that can be applied to new data.

Question 17

Building blocks for an M.L. algorithm?

Answer 17

Model class (hypothesis set) e.g.
-linear or quadratic function
-decision tree
-neural network, clustering
Error measure (Score function)
Algorithm - good model defined by the score function
Validation

Question 18

Dangers of overfitting

Answer 18

The model memorizes training data and does not generalize beyond it.
100% accuracy on training data, can’t do better than random guessing on new instances.

Question 19

Dangers of underfitting

Answer 19

Model not expressive enough, for ex. linear functions on non-linear problems.

Question 20

Approximation-Generalization tradeoff

Answer 20

Goal: to approximate target function as closely as possible.
More complex hypothesis set: better chance of approximating target function f
Less complex hypothesis set: better chance of generalizing f outside of the training set

Question 21

Ideal hypothesis set H

Answer 21

H = {f}, we already know the target function, no need for M.L.

Question 22

Occam’s Razor

Answer 22

the principle that favors the simplest hypothesis (set) that can well explain a given set of observations.

Question 23

Criteria for a good model

Answer 23

Interpretability
Computational complexity

Question 24

How to control Hypothesis set complexity?

Answer 24

With hyperparameters.
-max degree of polynomials
-no of nearest neighbors
-regularization parameter
-depth of decision tree

Question 25

What kind of methods should you start with?

Answer 25

Simple: linear regression, kNN, naive bayes
- easier to understand
-less tuning, less risk of overfitting
-often just as good as more advanced methods.

Question 26

K Nearest Neighbors

Answer 26

-Classic method (1951)
-Classification based on k most similar training instances
-parameter k tunes model complexity
-can learn complex non-linear functions

Question 27

kNN Classification

Answer 27

Choose the majority class among k nearest neighbors for prediction.

Question 28

kNN Regression

Answer 28

Take the mean value of k nearest neighbors for prediction

Question 29

Disadvantages of kNN predictor

Answer 29

All k nearest neighbors have the same influence on prediction. Maybe closer neighbors should have more influence?

Question 30

Distance measure in kNN

Answer 30

Standard: euclidean distance
Others: Manhattan, Mahalanobis, Chebyshev, Hamming

Question 31

Small vs large k in kNN

Answer 31

small: local complex model, depends on a handful of instances
large: global, simpler model, averaged over large set of instances

Question 32

kNN, k=1?

Answer 32

Overfitting! 0% training error, but won’t generalize

Question 33

Advantages of kNN

Answer 33

-simple
-non-linear modeling
-simple model complexity tuning (k)
-customizable (distance measure, feature/neighbor weighing)
-good results in many applications

Question 34

Disadvantages of KNN

Answer 34

-Large computational/memory complexity (O(nm) where m is the dimensionality of the data)
-sensitive to scaling
-Irrelevant features problematic
-black box-
-not state of the art

Question 35

Criteria to be balanced in learning.

Answer 35

Fit to data (low error) vs model complexity

Question 36

4 main ingredients of a kNN algorithm.

Answer 36

Distance metric
Number of neighbors (k)
Weighting function for neighbors
Prediction function

Question 37

Method to automatically determine the appropriate k value for kNN.

Answer 37

Cross-validation.