Week 5: Statistical Modelling

Question 1

Probability Distributions

Answer 1

This approach models uncertainty and quantifies our degree of belief that something will happen.

Question 2

Probability Distribution Function (PDF)

Answer 2

The area under the curve between two points of a PDF is the probability of the outcome being within the two points.

Question 3

Cumulative Distribution Function (PDF)

Answer 3

The height of the curve at a point is the chance that the outcome is less than or equal to the point.

Question 4

Joint Distribution

Answer 4

It’s the probability distribution of all the random variables in the set.

Question 5

Independence

Answer 5

A and B and independent if

P(A|B) = P(A),
P(B|A) = P(B),
P(A,B) = P(A)*P(B)

Question 6

Conditional Independence

Answer 6

A and B are conditionally independent given C

iff P(A,B|C) = P(A|C) * P(B|C), or
iff P(A|B,C) = P(A|C)

Conditional independence doesn’t imply unconditional independence or the other way around.

Question 7

Representative Sample

Answer 7

A sample from a population that accurately reflects the characteristics of the population.

Question 8

Prior

Answer 8

The initial probability that hypothesis h holds without having observed the data.

P(h)

Question 9

Likelihood

Answer 9

The probability of observing data D, given some world where the hypothesis h is true.

P(D|h)

Question 10

Posterior

Answer 10

The probability that hypothesis h is true, given that we have observed dataset D.

P(h|D)

Question 11

Likelihoods

Answer 11

When modelling a random process, we don’t know the hypothesis h. We estimate the parameters of a model h by maximising the probability P(D|h) (or L(h|D)) of observing D. Hypotheses aren’t always mutually exclusive and there can be an infinite number of them.

Question 12

Maximum Likelihood Estimate (MLE)

Answer 12

Calculate the parameters of so to maximise the likelihood L(h|D).

Goal is \arg \max_h \left{L(h \mid D) \right}

L(h \mid D) = P(D \mid h) = \prod_{i=1}^m P(\boldsymbol{x}_i \mid h)

Question 13

Bayesian Estimation

Answer 13

Compute a model h of maximum posterior probability Pr(h \mid D)

Goal is \arg \max_h \left{ P(h \mid D) \right}

Using Bayes Rule,

P(h \mid D) = \frac{P(D \mid h) \cdot P(h)}{P(D)}

This assumes conditional independence.

Question 14

Conditional Independence Assumption

Answer 14

Can multiply all probabilities given assumption and the probability of the assumption together in the numerator divided by the probability of the data.

Question 15

Probability Density Function for Normal Distribution

Answer 15

f(x) = \frac{1}{\sigma \sqrt{2 \pi}}e^{-\frac{1}{2} \left( \frac{(x - \mu)^2}{\sigma^2} \right)}

Question 16

Laplace Estimation

Answer 16

When computing likelihoods for each possible attribute value, add 1 to the numerator and \ell to the denominator. This allows for non-zero denominators

Question 17

Density Estimation

Answer 17

Given a dataset, compute an estimate of an underlying probability density function.

Question 18

Parametric Models

Answer 18

The number of parameters is fixed and independent of the training set size. These are approximations of reality and incorporate stronger assumptions than non-parametric models. They’re generally more explainable and enable deeper investigations.

Examples include:
- Multivariate Linear Regression
- Neural Networks
- k-Means
- Gaussian

Question 19

Non-parametric Models

Answer 19

The number of parameters grows as the sample size increases. They have modelling power for getting stronger representations.

Examples include:
- Decision Tree
- DBSCAN

Question 20

Multivariate Gaussian/Normal Distribution

Answer 20

f(\boldsymbol{x}) = \frac{1}{(2\pi)^{\frac{n}{2}} \left\lvert \boldsymbol{\Sigma} \right\rvert ^{\frac{1}{2}}} e^{-\frac{1}{2}(\boldsymbol{x} - \boldsymbol{\mu})^T \boldsymbol{\Sigma}^_{-1} (\boldsymbol{x} - \boldsymbol{\mu})}

Question 21

Iso-density Contours

Answer 21

In these contours, all the points x have equal density. f(x) = c.

This is similar to the elevation maps used in topography.

Question 22

Poisson Distribution

Answer 22

f(x) = \frac{\delta^x e^{-\delta}}{x !}

\delta = rate at which the events occur
x = random variable corresponding to the number of events

Question 23

Mixture Model

Answer 23

Consists of multiple component models each one specified by its own parameters.

f(\boldsymbol{x}) = \sum_{k=1}^K \pi_k f_k (\boldsymbol{x}; \boldsymbol{w}_k)

Question 24

Log-likelihood of a Mixture Model

Answer 24

L(\boldsymbol{\pi}, \boldsymbol{w}_1,…,\boldsymbol{w}K) = \log \left[ \sum{k=1}^K \pi_k f_k (\boldsymbol{x}; \boldsymbol{w}_k) \right]

Question 25

Gaussian Mixture Model (GMM)

Answer 25

P(\boldsymbol{x}i) = \sum{k=1}^K P(C_k) P(\boldsymbol{x}_i \mid C_k)

Question 26

Expectation-Maximisation (EM) Algorithm

Answer 26

Well-known algorithm for computing GMM’s.

E-Step: compute \pi_{i,k} = P(C_k \mid \boldsymbol{x}i). Using Bayes rule, compute P(\boldsymbol{x}i \mid C_k) P(C_k). m_k = \sum{i=1}^m \pi{i,k}

M-step: compute the new means, covariances, and component weights
\boldsymbol{\mu}k \leftarrow \sum{i=1}^m \left( \frac{\pi_{i,k}}{m_k} \right) \boldsymbol{x}i
\boldsymbol{\Sigma}k \leftarrow \sum{i=1}^m \left( \frac{\pi{i,k}}{m_k}\right) (\boldsymbol{x}_i - \boldsymbol{\mu}_j) (\boldsymbol{x}_i - \boldsymbol{\mu}_j)^T
\pi_k \leftarrow \frac{m_k}{m}