Part 1 : Probabilistic Data Models

Question 1

Deterministic Models Vs Probabilistic Models

Answer 1

• Deterministic models do not explicitly model uncertainties or
  ‘randomness’ in data.
• Variability of inferences derived from the data is not included.
• In many tasks, we benefit from modelling uncertainty and randomness.
• This is explicit in Probabilistic Models.

Question 2

Maximum-Likelihood Estimation

Answer 2

is a method of estimating the parameters of a probabilistic model.

• Assume θ is a vector of all parameters of the probabilistic model
• MLE is an extremum estimator obtained by maximising an objective
  function of θ

Question 3

Difference between Deterministic and Probabilistic Models

Answer 3

• A deterministic model would give one value, the most likely.
• A probabilistic model quantifies the chance/probability of the selected point being one of the possible classes.

Question 4

MLE - Mathematics

Answer 4

Assume f(θ) is an objective function to be optimised (e.g. maximised), the
arg max corresponds to the value of θ that attains the maximum value of
the objective function f.

θˆ = arg maxθ f(θ)

Note: this is different than maximising the function (i.e. finding the
maximum value [max f(θ)])

Question 5

Probabilistic MLE Approach

Answer 5

• Derive expression for conditional probability of observing data D given
  parameter a.

p(D|a)

- Using observed data, find parameter value which maximises the
conditional probability (i.e. the likelihood).

aML = arg maxap(D|a)

• Assume that observations are independent - a common assumption often
  referred to as i.i.d. independent and identically distributed.

p(D|a) = Π p(yi | xi, a)

– Note :: Π

Question 6

Probabilistic MLE Large Sample

Answer 6

• The average of yi value will be a x

- The ‘spread’ will be the same as for (epsilon), defined by σ^2

Question 7

Probabilistic Variance

Answer 7

Var(aML) = σ^2/Σx^2

Variance is dependant on input variables.

Question 8

Binomial Distribution

Answer 8

gives the probability distribution for a discrete
variable to obtain exactly D successes out of N trials, where the probability
of the success is α and the probability of failure is (1 − α) and 0 ≤ α ≤ 1,

Question 9

Maximum a Posteriori (MAP) Estimation

Answer 9

When you build prior knowledge into MLE.

θML = arg maxθ p(D|θ) p(θ)

• Likelihood
• Prior
• Posterior :: (Combine Likelihood and Prior)

Question 10

Conclusion

Answer 10

• Probabilistic models encode randomness in the data
• They enable predicting confidence (as a probability)
• Parameters of the model are tuned
• Maximum Likelihood Estimation (MLE) is a recipe used for training model parameters
• MLE does not encode our prior knowledge of possible parameters
• Maximum a Posteriori (MAP) maximises likelihood along with prior