lecture 1 - normal distribution

Question 1

What is precision (β) in the context of normal distributions?

Answer 1

• inverse of the variance
• 1/(σ^2)
• represents how concentrated/sharp the distribution is

Question 2

How does precision change as variance increases or decreases?

Answer 2

• When variance (σ^2) increases, precision (β) decreases.
• As variance approaches infinity, precision approaches zero.
• When variance decreases, precision increases, resulting in a sharper peak.

Question 3

Why is precision (β) used instead of variance (σ^2) in some cases?

Answer 3

• For brevity and convenience
• In Bayesian statistics and machine learning, expressing inverse variance directly simplifies equations.

Question 4

How are precision and variance related?

Answer 4

• Precision is the reciprocal of variance
• σ^2 = 1/β
• β = 1/(σ^2)

Question 5

What does the Probability Density Function (PDF) of a univariate normal distribution tell us?

Answer 5

• how likely a given value of x is, centered at the mean μ and spreading out based on the variance σ^2.
• The height of the distribution curve at a given value of x

Question 6

What does (x−μ)^2 represent in the normal distribution?

Answer 6

• The squared distance from the mean.
• Values further from the mean have lower probabilities, creating a tapering effect on both sides of the curve.

Question 7

What is the expected value of a univariate normal distribution?

Answer 7

• The expected value of x (first moment)
• E[x] = μ

Question 8

What is the expected value of x^2 in a univariate normal distribution?

Answer 8

• The expected value of x^2 (second moment)
• E[x^2] = μ^2 + σ^2
• So it’s slightly higher than μ^2 as it accounts for the variance as well

Question 9

How is variance derived from the moments of the normal distribution?

Answer 9

• Variance is the second moment minus the square of the first moment
• var(x) = E[x^2] - (E[x])^2 = σ^2

Question 10

What are the properties of the normal distribution PDF?

Answer 10

1. The PDF is always positive (N(x|μ,σ^2) > 0)
2. the total area under the curve is 1

Question 11

What is a multivariate normal distribution?

Answer 11

• A multivariate normal distribution is a generalization of the univariate normal distribution to higher dimensions.
• It describes the distribution of a vector of variables.

Question 12

What are the key parameters of a multivariate normal distribution?

Answer 12

1. μ: A D-dimensional mean vector.
2. Σ: A D×D covariance matrix (symmetric and positive definite).
3. D: The dimensionality of the data.
4. ∣Σ∣: The determinant of the covariance matrix.
5. x: a vector of variables

Question 13

What does (x−μ)^T Σ^−1 (x−μ) represent in the multivariate normal distribution?

Answer 13

• It represents the “distance” of the vector x from the mean μ, scaled by the spread and orientation of the distribution.
• This term measures how likely it is to observe x given the distribution.

Question 14

What happens when the covariance matrix of a multivariate distribution is an identity matrix?

Answer 14

• When the covariance matrix is an identity matrix, the variables are independent and not correlated.
• The distribution is spherical, with equal spread in all directions.

Question 15

What changes about the multivariate distribution when covariances are introduced?

Answer 15

• When covariances are introduced, relationships between variables emerge.
• The distribution becomes elliptical, indicating correlations between variables.

Question 16

What are the properties of marginal and conditional distributions for jointly Gaussian variables?

Answer 16

• If two sets of variables are jointly Gaussian, the conditional distribution of one set given the other is also Gaussian.
• Similarly, the marginal distribution of either set is Gaussian.

Question 17

What is the precision matrix (Λ)?

Answer 17

• The precision matrix (Λ) is the inverse of the covariance matrix (Σ): Λ=Σ^−1
• It represents the concentration of the multivariate normal distribution.

Question 18

What are the properties of the precision matrix (Λ)?

Answer 18

1. Symmetric: The precision matrix is always symmetric.
2. Inverse of the covariance matrix: It provides information about how tightly the variables are distributed around the mean.

Question 19

How are the covariance matrix and precision matrix related?

Answer 19

The covariance matrix describes the spread of the data, while the precision matrix (its inverse) describes how concentrated the distribution is around the mean.

Question 20

What is the Mahalanobis distance (Δ) in the Gaussian distribution?

Answer 20

• The Mahalanobis distance (Δ) measures the distance from a point x to the mean
  μ, taking into account the spread and orientation of the data described by the covariance matrix Σ: Δ^2
  = (x−μ) ^T Σ^−1 (x−μ)
• It reduces to the Euclidean distance when Σ is the identity matrix.

Question 21

What is “completing the square” in the context of Gaussian distributions?

Answer 21

Completing the square is a method used to rewrite a quadratic form in the exponent of a Gaussian distribution to identify the corresponding mean (μ) and covariance (Σ).

Question 22

What is the general form of the exponent in a multivariate Gaussian distribution?

Answer 22

(-1/2) (x−μ)^T Σ−1 (x−μ)

Question 23

How can the mean and covariance be derived by completing the square?

Answer 23

• The second-order term in x corresponds to Σ^−1, allowing the covariance matrix to be identified as
  Σ (inverting the matrix)
• The coefficient of the linear term in x corresponds to
  Σ^−1μ, from which the mean μ can be obtained.

Question 24

What does the term “const” refer to in the completed square expression?

Answer 24

Terms that are independent of x, which do not affect the form of the Gaussian distribution but may influence normalization factors.

Question 25

Why is completing the square useful in machine learning?

Answer 25

Completing the square is useful for deriving the parameters of Gaussian distributions in probabilistic models, such as in Bayesian inference and maximum likelihood estimation.