Gaussian Mixture Model Flashcards

1
Q

What is the goal of Gaussian mixture models (GMM)?

A

to model a probability distribution for data by assuming that the data is generated from a mixture of several Gaussian distributions. Each Gaussian component in the mixture is characterized by its own mean and covariance, and the GMM assigns a probability (weight) to each Gaussian.

GMM aims for flexibility in modeling distributions.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

What is the common strategy for obtaining a flexible distribution in GMM?

A

To make p(x) a combination of simpler, more tractable elements.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

What is the definition of the multivariate normal distribution for an M-dimensional vector?

A

It is defined as the density of a multivariate normal distribution.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

What does K represent in the context of GMM?

A

The number of components (or clusters) in the model.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

What is a one-hot vector in GMM?

A

A binary vector where exactly one entry is 1 and all other entries are 0.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

What does the binary variable z indicate?

A

It indicates the selected component in the GMM.
z = [0,1,0,0]

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

What are the mixing probabilities in GMM?

A

They are the weights of the k-th Gaussian component in the mixture.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

What does the log-likelihood function represent in the EM algorithm?

A

How well the model explains the observed data X.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

What are the two main steps of the EM algorithm?

A
  • Expectation step (E-step)
  • Maximization step (M-step)
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

What is the purpose of the E-step in the EM algorithm?

A

To calculate the probability (responsibility) that each data point belongs to component k.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

How does the M-step update model parameters in the EM algorithm?

A

Using the responsibilities to better fit the data in each iteration.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

What is the main advantage of GMM in density estimation?

A

It provides a flexible way to represent continuous densities.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

What is a kernel density estimator (KDE)?

A

A non-parametric tool for estimating the data’s probability density, providing a flexible alternative to the Gaussian mixture model without assuming a specific distribution.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

What does the kernel width λ control in KDE?

A

The size of the kernel’s ‘spread’ around each data point.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

What does a small λ in KDE lead to?

A

A very detailed density estimate, possibly leading to overfitting.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

What is the purpose of leave-one-out cross-validation in KDE?

A

To avoid overestimating the density by excluding the i-th training sample.

17
Q

What is Average Relative Density (ARD)?

A

A method for identifying outliers by comparing density at a point x with average density of its K-nearest neighbors.

18
Q

How is density around a point x computed in ARD?

A

As the inverse of the average distance to the K-nearest neighbors.

19
Q

What is a key limitation of GMM in outlier detection?

A

It can be affected by initialization and may have difficulties with regions of different density.

20
Q

What is the initialization process for GMM?

A

Start with K Gaussian components defined by mean, covariance, and weight.

21
Q

What are the benefits of using GMM?

A
  • Flexible modeling of complex distributions
  • Probabilistic framework
  • Unsupervised learning capability
22
Q

What are the limitations of GMM?

A
  • Assumption of Gaussianity
  • Sensitivity to initialization
  • Overfitting potential
23
Q

What does the ARD ratio indicate?

A

A low ARD value suggests that x is in a lower-density region compared to its neighbors.

24
Q

What is GMM?

A

GMM is a probabilistic model which assumes the data is generated from a mixture of several gaussian components (clusters)

25
what is p(x|z=1) = N(x|μ_1, Σ_1 )?
The probability of the data point x given that it was generated from the first gaussian component
26
What does π_k mean?
The weight of the mixture proportions, where all weights for each gaussian components has to be summed to 1
27
what is N(x|μ_k, Σ_k)?
Normal distribution for the k-th gaussian component of the mixture, determining what the chance is that this given x (datapoint) is part of the k-th gaussian
28
How is KDE more flexible?
KDE more flexible when you have no prior knowledge of the data distribution and when you want to estimate the density without making any strong assumptions (i.e., not assuming the data is Gaussian).
29
what is a sign of a bad GMM?
Visual grouping in gaussian component clusters look bad or the log-likelihood never stabilizes, indicates a bad model and KDE might be useful
30
When is GMM often used?
The model is often used for tasks like clustering, density estimation, and anomaly detectio