Final Review

Question 1

Who developed the VC dimension?

Answer 1

Dr. Vladimir Vapnik and Dr. Chervonenkis

Question 2

What does VC stand for?

Answer 2

Vapnik-Chervonekis

Question 3

Why was the VC dimension developed?

Answer 3

To help scientists develop better machine learning models that would be better at classifying data

Question 4

What is the VC dimension?

Answer 4

The Vapnik-Chervonenkis dimension of a hypothesis space is the maximum number of points that can be “shattered” by the hypothesis of that space

Question 5

In terms of VC dimension, what does “shattering” mean?

Answer 5

“Shattering” means that for every possible way of labeling these points (with binary labels), there is a hypothesis in the space that correctly classifies the points according to that labeling

Question 6

Is either of these true? Why or why not?

The VC dimension of the hypothesis class of circles is lower than that of squares.

The VC dimension of the hypothesis class of squares is lower than that of circles.

Answer 6

No, because both have a VC dimension of 3.

Question 7

Is this true? Why or why not?

The VC dimension of the hypothesis class of rings is higher that that of circles.

Answer 7

Yes, because rings have a VC dimension of 4 while circles have a VC dimension of 3.

Question 7

What are 2 important notes when discussing the VC dimension of a hypothesis space?

Answer 7

• Knowing if the hypothesis space is centered on the origin
• Knowing that the VC dimension is NOT always equivalent to 1 + (number of dimensions)

Question 8

Is either of these true? Why or why not?

The VC dimension of circles is higher than that of squares.

The VC dimension of squares is higher that that of circles.

Answer 8

No, because both have a VC dimension of 3.

Question 9

Who Invented Bayes’ Rule?

Answer 9

Thomas Bayes

Question 10

What was the goal of Bayes’ Rule?

Answer 10

To solve the problem of inverse probability - inferring the probability of causes (hypotheses) from observed effects (data)

Question 11

What is Bayes’ Rule?

Answer 11

A fundamental theorem in probability theory for updating the probability of a hypothesis based on new evidence

Question 12

What is the formula for Bayes’ Rule?

Answer 12

P(h|D) = [P(D|h) * P(h)] / P(D)

Question 13

In Bayes’ Rule, what is P(h|D) and what does it represent?

Answer 13

The posterior probability of the hypothesis given the data.

It represents our updated belief in the hypothesis after seeing the data.

Question 14

In Bayes’ Rule, what is P(D|h) and what does it represent?

Answer 14

P(D|h) is the likelihood, which is the probability of observing the data assuming the hypothesis is true.

It tells us how likely the observed data is under the assumption of the hypothesis.

Question 15

In Bayes’ Rule, what is P(h) and what does it represent?

Answer 15

P(h) is the prior probability of the hypothesis before observing any data.

It represents our belief in the hypothesis based on prior knowledge or assumptions.

Question 16

In Bayes’ Rule, what is P(D) and what does it represent?

Answer 16

P(D) is the marginal likelihood (evidence), which normalizes the posterior distribution by ensuring the total probability sums to 1.

It represents the total probability of observing the data across all possible hypotheses.

Question 17

For Bayes’ Rule, when would we ignore P(D)?

Answer 17

When we interested in finding the hypothesis that maximizes the posterior probability - P(h|D)

Question 18

What is MLE?

Answer 18

Maximum Likelihood Estimation is a way to estimate the parameters of a model by finding the values that make the observed data most likely

Question 19

What does MLE mean in the context of binary classification?

Answer 19

This means finding the parameters of the hypothesis h(x) that make the observed outcomes y more likely.

Question 20

What is MSE? How is it written?

Answer 20

Mean Squared Error
MSE = (1/n) * (Σ(y - h(x))^2

Question 21

Why is MSE unsuitable for binary classification?

Answer 21

• Binary classification treats targets as discrete (0 or 1), while MSE assumes targets are continuous values.
• MSE doesn’t penalize incorrect predictions made with high confidence as effectively as cross-entropy loss does
• MSE here can result in poor performance because it focuses on shortening distance between predicted and actual values, ignoring the probability estimates produced by models in classification tasks

Question 22

What function is typically used for a binary classification neural network’s output layer? Why?

Answer 22

A sigmoid activation function is typically used because it outputs a value between 0 and 1 which can be evaluated with cross-entropy loss to predict how well it matches the actual binary label.

Question 23

Describe Cross-Entropy Loss (CEL)

Answer 23

• Used in binary classification
• Measures the difference between the predicted probabilities and the true binary labels
• Directly handles probabilistic predictions
• Strongly penalizes wrong predictions made with high confidence

Question 24

Describe Mean Squared Error (MSE)

Answer 24

• Typically used in regression tasks where the target variable is continuous
• Measures the squared difference between predicted values and true values
• Treats output as continuous, making it LESS suitable for binary classification

Question 25

Describe Neural Networks with Sum of Squared Error (SSE)

Answer 25

• If trained for a binary classification task, it may treat the output as continuous rather than probabilistic
• For binary classification, use CEL (Cross-Entropy Loss) with a sigmoid output layer

Question 26

What is a limitation of K-Means?

Answer 26

• K-Means struggles with non-spherical clusters
• K-Means requires a pre-defined number of clusters
• K-Means may converge to local minima

Question 27

What are the steps of spectral clustering?

Answer 27

• Construct a similarity graph
• Compute the graph Laplacian
• Compute the eigenvalues and eigenvectors
• Perform clustering in the reduced eigenspace

Question 28

What is the kernel trick?

Answer 28

This trick maps data into a higher-dimensional space where linear separation is possible

Question 29

Why does the kernel trick matter for concentric circles?

Answer 29

This trick allows spectral clustering to transform into the original space, enabling the separation of non-linearly separable points, such as inner and outer circles

Question 30

What is the primary advantage of Isomap over traditional MDS?

Answer 30

Isomaps approximate geodesic distances, capturing global structure

Question 31

Advantages and Limitations of Isomap

Answer 31

Advantage:
- Captures global structure effectively, especially for data with a natural manifold shape

Limitations:
- Computationally intensive on large datasets
- Struggles with non-manifold high-dimensional data

Question 32

What is the core concept of Laplacian Eigenmaps?

Answer 32

Laplacian eigenmaps use spectral graph theory to focus on local neighborhood relationships, preserving local structure

Question 33

What is the process for Laplacian Eigenmaps?

Answer 33

• Build a weighted similarity graph of the data
• Compute the graph’s Laplacian matrix
• Perform eigenvalue decomposition to generate a low-dimensional embedding

Question 34

Laplacian Eigenmaps are best suited for preserving what type of structure?

Answer 34

Local neighborhood structure

Question 35

Name applications of Laplacian Eigenmaps

Answer 35

• Image Processing
• Clustering in High-Dimensional Data
• Sensor Data Analysis
• Gene Expression Data

Question 36

Advantages and Limitations of Laplacian Eigenmaps

Answer 36

Advantage: Efficient for preserving local structures, which is useful for clustering tasks

Limitation: May lose global relationships as it focuses solely at the local level