Clustering

Question 1

What is unsupervised learning?

Answer 1

Learning patterns in data without labeled outputs or a “teacher”.

Question 2

What is the goal of clustering?

Answer 2

To partition data into groups or clusters based on similarity.

Question 3

What does the K-means algorithm minimize?

Answer 3

The within-cluster point scatter/variance.

Question 4

What are the two main steps of the K-means algorithm?

Answer 4

1) Assign points to nearest cluster center, 2) Update cluster centers. Until convergence

Question 5

What is the K-means++ algorithm used for?

Answer 5

To initialize the cluster centers for K-means in a way that improves convergence (by spreading out the initial cluster centers)`

Question 6

How can the number of clusters K be determined in non-probabilistic models?

Answer 6

By computing the Mean Square Error (MSE) for different values of K and using Elbow Method or picking K value where SSE has a change of slope.

Question 7

What is a mixture model?

Answer 7

A probabilistic model that represents the presence of subpopulations within an overall population.

Question 8

What are the two steps in generating samples from a Gaussian mixture model?

Answer 8

1) Draw a categorical variable Z to select a component, 2) Draw an observation from the selected Gaussian component.

Question 9

What is the EM algorithm used for in mixture models?

Answer 9

To estimate the parameters of the mixture model by maximizing the likelihood.

Question 10

What are the two main steps of the EM algorithm?

Answer 10

The Expectation (E) step and the Maximization (M) step (Repeat until convergence as in stable assignments and parameters)

Question 11

In the context of Gaussian Mixture Models, what does the E-step compute?

Answer 11

the E-step computes the expected value of the latent variables, specifically the posterior probabilities (responsibilities) that each data point belongs to each Gaussian component.

Question 12

In the context of Gaussian Mixture Models, what does the M-step compute?

Answer 12

the M-step updates the parameters of the model (means, covariances, and mixing coefficients) to maximize the expected log-likelihood found in the E-step. This involves re-estimating the parameters to better fit the data based on the current responsibilities.

Question 13

How does K-means compare to Gaussian Mixture Models?

Answer 13

K-means is usually faster due to fewer iterations and less computation. K-Means assumes spherical clusters with equal variance while GMM can have clusters with different shapes and sizes. GMM also have soft-assignment (probability of belonging to each cluster) while K-Means have hard assignments.

Question 14

What is an advantage of mixture models over K-means?

Answer 14

Mixture models allow for distributional assumptions and can assess the fit of the data by computing likelihood.

Question 15

What is an issue with mixture models in terms of identifiability?

Answer 15

The likelihood is invariant to permutation of class memberships, making the estimators valid only up to permutation.

Question 16

What is hierarchical agglomerative clustering?

Answer 16

A clustering method that starts with singleton clusters and merges the most similar clusters iteratively.

Question 17

What is the purpose of the Bayes classifier in mixture models?

Answer 17

To obtain class posterior probabilities when parameters are known: P(y|x, Θ) ∝ fΘ(x|y)πy

Question 18

What is the main challenge in maximizing the likelihood for mixture models

Answer 18

The lack of a closed-form solution, requiring an iterative procedure like EM.

Question 19

What is the relationship between K-means and Gaussian Mixture Models?

Answer 19

K-means is equivalent to a special case of GMM where all clusters have the same diagonal covariance matrix as σ approaches 0.

Question 20

What is a disadvantage of mixture models compared to K-means?

Answer 20

Mixture models require explicit distributional assumptions.

Question 21

How does the EM algorithm’s performance depend on initialization?

Answer 21

EM’s performance can vary significantly based on parameter initialization due to multiple local maxima.

Question 22

How can missing values be handled in mixture models?

Answer 22

Mixture models can naturally infer missing values as part of the model fitting process.