Chapter 3/4/5 Flashcards
Moment Generating Functions
- A moment generating function (MGF) can be used to generate moments (about the origin) of the distribution of a random variable (discrete or continuous), i.e., E(X), E(X²), E(X³), etc.
- The moment generating function, M_X(t), is given by:
M_X(t) = E[e^(tX)]
for all values of t for which the expectation exists. - M_X(0)=1
- To calculate moments (E(X^r))
-Differentiate the MGF with respect to t.
-Set t = 0.
-The rᵗʰ derivative gives the rᵗʰ moment about the origin.
Series expansion of MGF
Expanding the exponential function in the definition of the MGF and taking expected values throughout gives:
1. M_X(t) = E[e^(tX)]
= E[1 + tX + (t²X²)/2! + (t³X³)/3! + …]
1. This simplifies to:
M_X(t) = 1 + tE[X] + (t²E[X²])/2! + (t³E[X³])/3! + …
1. From this expansion, the rᵗʰ moment of the distribution about the origin, E[Xʳ], is obtainable as the coefficient of tʳ/r! in the power series expansion of the MGF.
Cumulant generating function
- The cumulant generating function (CGF) is the logarithm of the moment generating function (MGF):
K_X(t) = ln(M_X(t)) - The cumulants of a distribution are obtained by differentiating the CGF:
- The rᵗʰ cumulant is given by the rᵗʰ derivative of K_X(t), evaluated at t = 0:
κ_r = K_X⁽ʳ⁾(0) - The coefficient of tʳ / r! in the Maclaurin series of K_X(t) is called the rᵗʰ cumulant and is denoted by κ_r:
The first few cumulants correspond to well-known statistical measures:
κ₁ = Mean (E[X])
κ₂ = Variance (Var(X))
κ₃ = Skew
κ₄ = Kurtosis-related measure
Linear functions of the MGF
- Suppose X has MGF M_X(t), and we are interested in the distribution of Y = a + bX.
- The MGF of Y is:
M_Y(t) = E[e^(tY)] = E[e^(t(a + bX))] - This simplifies to:
M_Y(t) = e^(ta) * M_X(bt)
Linear functions of the CGF
If Y = a + bX.
Since the cumulant generating function (CGF) is the logarithm of the MGF, we have:
C_Y(t) = ln(M_Y(t))
Using the expression for M_Y(t), we have:
M_Y(t) = e^(at) * M_X(bt)
Now, applying the logarithm to both sides:
C_Y(t) = ln(e^(at) * M_X(bt))
This simplifies to:
C_Y(t) = at + ln(M_X(bt))=at +C_X(bt)
Independence of RVs
The random variables X and Y are independent if and only if their joint probability function/PDF is the product of their marginal probability functions/PDFs for all (x, y) in the support of the variables.
For continuous random variables:
f_{X,Y}(x, y) = f_X(x) * f_Y(y) for all (x, y)
For discrete random variables:
P(X = x, Y = y) = P(X = x) * P(Y = y)
This means that probability statements about (X, Y) can be broken down into statements about X and Y separately.
If X and Y are continuous, the double integral required to evaluate a joint probability splits into the product of two separate integrals, one for X and one for Y:
P(x₁ < X < x₂, y₁ < Y < y₂) = P(x₁ < X < x₂) * P(y₁ < Y < y₂)
This means that in the continuous case, if the two random variables are independent, we can factorise the joint PDF into two separate expressions, one of which will be a function of x only, and the other will be a function of y only.
Expectation of a product
For independent random variables X and Y:
E[g(X) * h(Y)] = E[g(X)] * E[h(Y)]
This holds because the joint density function factorises into the two marginal density functions.
Covariance
The covariance of two random variables X and Y is defined as:
cov(X, Y) = E[(X - E[X])(Y - E[Y])]
This simplifies to:
cov(X, Y) = E[XY] - E[X]E[Y]
- cov[X,X] =var[X]
- cov[aX+b,cY+d]=ac cov[X,Y]
- cov[X,Y+Z]=cov[X,Y]+cov[X,Z]
- If X,Y independent cov[X,Y]=0
The covariance between X and Y is a measure of the strength of the linear association or linear relationship between the variables. However it suffers from the fact that its value is dependent on the units of measurement of the variables.
Pearsons correlation coefficient
- A related quantity to the covariance is Pearson’s correlation coefficient, which is a **dimensionless measure **(i.e., it has no units).
- Pearson’s correlation coefficient, denoted as corr(X, Y) or ρ(X, Y), is defined by:
corr(X, Y) = cov(X, Y) / (√var(X) * √var(Y)) - This measures the strength and direction of the linear relationship between X and Y. It takes values between -1 and 1, where:
ρ = 1 indicates a perfect positive linear relationship, (one variable is a linear function of the other)
ρ = -1 indicates a perfect negative linear relationship,
ρ = 0 indicates no linear relationship.
In simple terms, ‘independent’ means that ‘probabilities factorise’, and ‘uncorrelated’ means that ‘expectations factorise’.
Variance of a sum
For any random variables X and Y, the variance of their sum is given by:
var(X + Y) = var(X) + var(Y) + 2cov(X, Y)
Can be extended:
For a set of random variables X₁, X₂, …, Xₙ, and constants c₁, c₂, …, cₙ, the variance of a linear combination of the random variables is given by:
Var(c₁X₁ + c₂X₂ + … + cₙXₙ) = ∑ cᵢ²Var(Xᵢ) + 2∑∑ cᵢcⱼCov(Xᵢ, Xⱼ)
This result is an extension of the variance formula for the sum of two random variables:
Var(X₁ + X₂) = Var(X₁) + Var(X₂) + 2Cov(X₁, X₂)
For n random variables, the formula can be generalized as follows:
Var(∑ cᵢXᵢ) = ∑ cᵢ²Var(Xᵢ) + 2∑∑ cᵢcⱼCov(Xᵢ, Xⱼ)
This expression accounts for both the variances of the individual random variables and the covariances between pairs of them.
If X₁, X₂, …, Xₙ are random variables and are pairwise uncorrelated, the variance of their linear combination can be simplified. In this case, the covariance terms between distinct random variables drop out (i.e., Cov(Xᵢ, Xⱼ) = 0 for i ≠ j). Thus, for n random variables, the variance formula becomes:
Var(c₁X₁ + c₂X₂ + … + cₙXₙ) = ∑ cᵢ²Var(Xᵢ)
Convolution
Discrete Case:
* Consider the sum of two discrete random variables, Z = X + Y, where (X, Y) has a joint probability function p(x, y).
* The probability function of Z is found by summing p(x, y) over all values of (x, y) such that x + y = z:
P(Z = z) = Σ p(x, y) for all (x, y) where x + y = z
* If X and Y are independent, the joint probability function factorizes into the product of the two marginal probability functions:
*P(Z = z) = Σ p_X(x) * p_Y(z - x)
This process of obtaining the probability function of Z = X + Y from the probability functions of X and Y is called convolution.
Continuous Case:
Similarly
f_Z(z) = ∫ f_X(x) * f_Y(z - x) dx
This integral expresses f_Z(z) as the convolution of f_X(x) and f_Y(y), written as:
f_Z = f_X * f_Y
So, in both discrete and continuous cases, the probability function (or density) of Z = X + Y is obtained by convolving the individual probability functions (or densities) of X and Y.
Sum of expectations
For any random variables X₁, X₂, …, Xₙ (not necessarily independent) and constants c₁, c₂, …, cₙ, the expectation follows the linearity of expectation rule:
E[c₁X₁ + c₂X₂ + … + cₙXₙ] = c₁E[X₁] + c₂E[X₂] + … + cₙE[Xₙ]
This property holds regardless of whether the random variables are dependent or independent.
E[∑ cᵢXᵢ] = ∑ cᵢE[Xᵢ]
Using MGFs to derive distributions of linear combinations of independent random variables
Suppose X₁ and X₂ are independent random variables with MGFs M₁(t) and M₂(t), respectively, and let S = c₁X₁ + c₂X₂. Then:
M_S(t) = E[e^(tS)] = E[e^(t(c₁X₁ + c₂X₂))] = E[e^(c₁Xt)] * E[e^(c₂Xt)]
M_S(t) = M₁(c₁t) * M₂(c₂t)
In the case of a simple sum, Z = X + Y, the MGF of the sum is the product of the individual MGFs:
M_Z(t) = M_X(t) * M_Y(t)
This result extends to the sum of more than two independent random variables. If Y = X₁ + X₂ + … + Xn, where the Xᵢ are independent and each with MGF Mᵢ(t), then:
M_Y(t) = M₁(t) * M₂(t) * … * Mₙ(t)
If the Xᵢ’s are identically distributed, each with MGF M(t), then:
M_Y(t) = M(t)ⁿ
Conditional Expectation
The conditional expectation of Y given X = x is the mean of the conditional distribution of Y given X = x. This mean is denoted as E[Y | X = x], or simply E[Y | x].
For discrete random variables X and Y, this is:
E[Y | X = x] = Σ (y * P(Y = y | X = x))
In the case when X and Y are continuous random variables, the equivalent expression is:
E[Y | X = x] = ∫ y * f_Y|X(y | x) dy=∫ y * f(x,y)/f(x) integrating over y
Where f_Y|X(y | x) is the conditional probability density function of Y given X = x.
Regression of Y on X
The conditional expectation E[Y | X = x] represents the expected value of Y given the value of X = x. This is denoted as E[Y | X], and is sometimes referred to as the regression of Y on X. In some cases, the conditional expectation is a linear function of X, for example:
E[Y | X = x] = 34x + 3(1)
This can be considered as the observed value of the random variable Y, given a particular value of X. More generally, we can express the conditional expectation as a function of X, denoted g(X), where g(X) is a function that depends on the random variable X.
The regression of Y on X, denoted as E[Y | X], is an important concept in statistics, and it can be represented as a function of X. This function describes how Y is expected to change as X changes.
