Gaussian Mixture Model Flashcards
What is the goal of Gaussian mixture models (GMM)?
to model a probability distribution for data by assuming that the data is generated from a mixture of several Gaussian distributions. Each Gaussian component in the mixture is characterized by its own mean and covariance, and the GMM assigns a probability (weight) to each Gaussian.
GMM aims for flexibility in modeling distributions.
What is the common strategy for obtaining a flexible distribution in GMM?
To make p(x) a combination of simpler, more tractable elements.
What is the definition of the multivariate normal distribution for an M-dimensional vector?
It is defined as the density of a multivariate normal distribution.
What does K represent in the context of GMM?
The number of components (or clusters) in the model.
What is a one-hot vector in GMM?
A binary vector where exactly one entry is 1 and all other entries are 0.
What does the binary variable z indicate?
It indicates the selected component in the GMM.
z = [0,1,0,0]
What are the mixing probabilities in GMM?
They are the weights of the k-th Gaussian component in the mixture.
What does the log-likelihood function represent in the EM algorithm?
How well the model explains the observed data X.
What are the two main steps of the EM algorithm?
- Expectation step (E-step)
- Maximization step (M-step)
What is the purpose of the E-step in the EM algorithm?
To calculate the probability (responsibility) that each data point belongs to component k.
How does the M-step update model parameters in the EM algorithm?
Using the responsibilities to better fit the data in each iteration.
What is the main advantage of GMM in density estimation?
It provides a flexible way to represent continuous densities.
What is a kernel density estimator (KDE)?
A non-parametric tool for estimating the data’s probability density, providing a flexible alternative to the Gaussian mixture model without assuming a specific distribution.
What does the kernel width λ control in KDE?
The size of the kernel’s ‘spread’ around each data point.
What does a small λ in KDE lead to?
A very detailed density estimate, possibly leading to overfitting.
What is the purpose of leave-one-out cross-validation in KDE?
To avoid overestimating the density by excluding the i-th training sample.
What is Average Relative Density (ARD)?
A method for identifying outliers by comparing density at a point x with average density of its K-nearest neighbors.
How is density around a point x computed in ARD?
As the inverse of the average distance to the K-nearest neighbors.
What is a key limitation of GMM in outlier detection?
It can be affected by initialization and may have difficulties with regions of different density.
What is the initialization process for GMM?
Start with K Gaussian components defined by mean, covariance, and weight.
What are the benefits of using GMM?
- Flexible modeling of complex distributions
- Probabilistic framework
- Unsupervised learning capability
What are the limitations of GMM?
- Assumption of Gaussianity
- Sensitivity to initialization
- Overfitting potential
What does the ARD ratio indicate?
A low ARD value suggests that x is in a lower-density region compared to its neighbors.
What is GMM?
GMM is a probabilistic model which assumes the data is generated from a mixture of several gaussian components (clusters)
