Lecture 11 - Clustering: EM Algorithm for GMM

Question 1

What is a Gaussian Mixture Model (GMM)?

Answer 1

A probabilistic model representing data as a mixture of Gaussian distributions.

Question 2

Priors - 𝜋𝑖

Answer 2

𝜋𝑖 : Probabilities of each cluster.

Question 3

Means - 𝜇𝑖

Answer 3

𝜇𝑖: Centers of Gaussian clusters.

Question 4

Covariances - Σ𝑖

Answer 4

Σ𝑖: Shapes and spreads of clusters.

Question 5

Why use GMM instead of K-Means?
GMM vs. K-Means

Answer 5

Soft Clustering: Assigns probabilities to clusters, not hard assignments.

Handles Arbitrary Shapes: Covariances allow ellipsoid-shaped clusters.

Probabilistic Reasoning: More robust for complex data distributions.

Mnemonic: “GMM Gives More Meaningful shapes.”

Question 6

What is the Expectation-Maximization (EM) Algorithm?

Answer 6

An iterative algorithm to optimize parameters (π,μ,Σ) in GMM

Question 7

What happens in the E-Step?

Answer 7

Compute the posterior probability hi,t that a point belongs to a cluster:

Question 8

What happens in the M-Step?

Answer 8

Update parameters using hi,j.

Question 9

How does EM converge?

Answer 9

Converges when the log-likelihood stabilizes.

Alternative: Stop when the change in log-likelihood is below a threshold.

Tip: Imagine EM climbing a mountain of likelihood to the peak (maximum).

Question 10

Challenges with GMM and EM

Answer 10

Initialization Sensitivity: Poor starts may lead to local optima.

Choosing 𝑘: Methods include Elbow, BIC, and AIC.

Computational Cost: Iterative nature makes it slow for large datasets.

Question 11

Applications of GMM. Where is GMM used?

Answer 11

Image Segmentation: Dividing an image into meaningful regions.

Speech Recognition: Modeling phonemes or accents.

Anomaly Detection: Identifying outliers with low probabilities.