Lecture 2

Question 1

k-Nearest Neighbors (k-NN)

Answer 1

Given a set of labeled instances (training set), new instances (testing set) are classified according to the majority label of the k nearest neighbors

Question 2

Decision Boundary

Answer 2

A model of the separation between two classes. Can be a straight or wiggly line

Question 3

What is the complexity of the k-NN model proportional to?

Answer 3

The complexity is proportional to the wigglyness of the decision boundry. The more complex, the more wiggly

Question 4

What does a model do with data in a classification problem?

Answer 4

In classification, a model trained from data defines a decision boundry that separates the data

Question 5

What does a model do with data in a regression problem?

Answer 5

In regression, a model fits data to describe the relation between i) 2 features or ii) a feature and the label

Question 6

What happens when there is an equal number of classes of the nearest neighbors (a tie) or two neighbors are equidistant to the new data point?

Answer 6

Either a random selection of the class assigned to the new point is made or how large k is changes to break either the class or distance tie

Question 7

What is the label (class) of a point on the decision boundary?

Answer 7

It’s ambiguous

Question 8

Uniform Weighted k-NN

Answer 8

When the majority class of nearest neighbors determines the class of the new data point

Question 9

Distance Weighted k-NN

Answer 9

each neighbor has a weight which is based on its distance to the new data point

Question 10

Inverse Distance Weighted k-NN

Answer 10

each neighbor has a weight which is based on the inverse of its distance to the new data point so closer neighboring points have a higher vote

Question 11

What are two types of Kernel Functions

Answer 11

Gaussian kernel (bell curve) and tricube kernel

Question 12

Euclidean distance

Answer 12

A straight line

Question 13

Manhattan distance

Answer 13

Distance between two projections on the axis (you can’t walk through walls/buildings, you have to go around them)

Question 14

When does a k-NN model have a danger of overfitting?

Answer 14

When k is too low; the model has a high complexity

Question 15

When does a k-NN model have a danger of underfitting?

Answer 15

When k is too high; the model has a low complexity

Question 16

How do you determine the model complexity?

Answer 16

Depends on the complexity of the separation between classes. Start with the simplest model (large k) and increase complexity (smaller k)

Question 17

How do you choose k?

Answer 17

Typically an odd number k for an even number of classes. The data-miner’s rule of thumb is k = sqrt(n)

Question 18

Nearest centroid classification

Answer 18

Takes a new sample, and compares it to each of these class centroids. The class 
whose centroid it is closest to, in squared distance, is the predicted class for that new 
sample.

Question 19

Nearest shrunken centroid classification

Answer 19

“shrinks” each of the class centroids toward the overall centroid for all classes by an amount called the threshold . This shrinkage consists of moving the centroid towards zero by threshold, setting it equal to zero if it hits zero.

After shrinking the centroids, the new sample is classified by the usual nearest centroid rule, but using the shrunken class centroids.

Question 20

k-NN Advantages

Answer 20

• The cost of the learning process is zero
• No assumptions about the characteristics of the concepts to learn have
to be done
• Complex concepts can be learned by local approximation using simple
procedures

Question 21

k-NN Disadvantages

Answer 21

•The model can not be interpreted (there is no description of
the learned concepts)
•It is computationally expensive to find the k nearest neighbors
when the dataset is very large
•Performance depends on the number of dimensions that we have (curse of dimensionality)

Question 22

Curse of Dimensionality

Answer 22

The curse of dimensionality refers to various phenomena that arise when analyzing and organizing data in high-dimensional spaces that do not occur in low-dimensional settings such as the three-dimensional physical space of everyday experience.

For example: when the dimensionality increases, the volume of the space increases so fast that the available data become sparse.

Question 23

When does the k-NN algorithm require more computation?

Answer 23

The k-NN algorithm requires more computation for testing than for training.

Question 24

What does training the kNN algorithm consist of?

Answer 24

only storing the data

Question 25

What does testing the kNN algorithm consist of?

Answer 25

Testing involves comparing every test instance to all the instances in the training set and calculating which training instances are closest, before assigning a class label.

Question 26

Is the kNN algorithm used for classification or regression?

Answer 26

It can be used for both classification and regression.

Question 27

What is the relationship between k and the complexity of the model?

Answer 27

As you increase k, the model gets less complex (risk of overfitting decreases).

Question 28

If the model preforms well of the training data but poorly on the test data, what’s the issue?

Answer 28

The model is overfitting

Question 29

Can k in kNN be negative or a float?

Answer 29

No, k is always a positive integer