lecture 3 - bayesian inference

Question 1

What is a common pitfall when using frequentist statistics?

Answer 1

Frequentist statistics often minimize error based on averages without accounting for the uncertainty around predictions.

Question 2

How is data modeled in the frequentist framework?

Answer 2

• The full dataset is modeled as y(x,w)
• A datapoint is modeled as a distribution where the mean is given by y(x_0, w) for an input x_0

Question 3

What does p(t∣x) represent in the frequentist framework?

Answer 3

• For each fixed x, p(t∣x) represents the probability of observing t, which is assumed to be normally distributed.

Question 4

What is the definition of the expected loss function E[L] in machine learning?

Answer 4

the average loss across possible values of t, weighted by their probability, expressed as a double integral over x and t.

Question 5

What does minimizing the expected loss function involve?

Answer 5

Taking the derivative with respect to the model’s prediction y(x) and setting it to zero.

Question 6

What result do you get when taking the derivative of the expected loss function with respect to y(x)?

Answer 6

• 2 ∫ {y(x)-t} p(t|x) dt
• indicates that adjustments should be made to reduce the difference between the prediction y(x) and the target t.

Question 7

How do you find the optimal prediction y(x) after taking the derivative of the expected loss function?

Answer 7

By setting the derivative to zero and solving, you find that the optimal prediction y(x) is the expected value of t given x, denoted as E[t∣x].

Question 8

If you do not use a model, how can you estimate the best value of t for a given x?

Answer 8

By taking the expected value of t at that fixed value of x.

Question 9

What is the purpose of adding zero in the form of −E[t∣x]+E[t∣x]=0 when decomposing the expected loss function?

Answer 9

The purpose is to separate the prediction y(x) from the true expected value E[t∣x], which serves as the best predictor of t given x.

Question 10

What happens to the cross term 2(y(x)−E[t∣x])(E[t∣x]−t) when taking the expectation of the expanded loss function?

Answer 10

The cross term disappears because E[t∣x]−t has zero mean.

Question 11

What are the two main components left after taking the expected value of the expanded loss function?

Answer 11

1. (y(x)−E[t∣x]): Error of the model in approximating the true expected value.
2. Var(t∣x): Inherent variance in the data, independent of the model (noise).

Question 12

Why is variance related to
t always present in the model?

Answer 12

• Variance related to t is intrinsic noise in the random variable t and cannot be changed, meaning there will always be some noise-related error.
• Var(t∣x)

Question 13

What are the components of the bias-variance decomposition of the expected loss?

Answer 13

• (Bias)^2: The squared difference between the expected model prediction and the true value.
• Variance: The variability of the model prediction.
• Noise: The inherent variability in the data that cannot be controlled by changing the model.

Question 14

How can bias and variance be controlled in a model?

Answer 14

• The model bias and variance are related to eachother in that if you increase one, the other decreases (and vice versa)
• Bias can be reduced by increasing model complexity, but this may increase variance.
• Variance can be reduced by simplifying the model, but this may increase bias.
• In deep learning, it’s possible to reduce both bias and variance through techniques like regularization and large datasets.

Question 15

What is the final form of the expected loss after decomposition?

Answer 15

Expectedloss = (Bias)^2 + Variance + Noise

Question 16

How does regularization affect the bias-variance tradeoff in a model?

Answer 16

Regularization controls the tradeoff between bias and variance, influencing the model’s performance by adjusting flexibility.

Question 17

What happens when the regularization parameter λ is high?

Answer 17

1. Little model variance, meaning minimal difference in the estimates produced by different models (low variance).
2. High bias, as the model is too rigid and far from the true function.

Question 18

What are the effects of low regularization (λ is low)?

Answer 18

1. High variance because the model has more flexibility to fit the data.
2. Low bias, as the model better approximates the true function.

Question 19

What happens when the model is over-regularized (extremely low λ)?

Answer 19

The model exhibits even more variability (high variance), allowing it to be highly flexible, but with very little bias.

Question 20

What happens to bias as regularization decreases?

Answer 20

Bias decreases as regularization decreases because the model becomes more flexible and can better fit the data.

Question 21

What happens to variance as regularization decreases?

Answer 21

Variance increases as regularization decreases because the model becomes more complex and can overfit the data.

Question 22

What is the relationship between test error, bias, and variance?

Answer 22

The test error behaves similarly to the combined bias squared and variance curve, but at a slightly higher error value.

Question 23

What is the goal when tuning the regularization parameter?

Answer 23

The goal is to find a balance where the sum of bias and variance is minimized, leading to the lowest test error.

Question 24

How does model complexity affect bias and variance?

Answer 24

Model complexity increases variance but decreases bias, as more complex models can fit the data more closely.

Question 25

How does regularization affect bias and variance?

Answer 25

Regularization decreases variance but increases bias, as it simplifies the model and reduces its flexibility.

Question 26

What happens when you increase the number of coefficients or basis functions in a model (e.g., linear regression or neural networks)?

Answer 26

Increasing the number of coefficients or basis functions adds complexity, which increases variance but decreases bias.

Question 27

In the context of bias-variance tradeoff, what does the red line in the plot represent?

Answer 27

The red line represents a simple model (e.g., a flat line) that underfits the data, resulting in high bias.

Question 28

In the context of bias-variance tradeoff, what does the blue line in the plot represent?

Answer 28

The blue line represents a very complex model that overfits the data, resulting in high variance.

Question 29

What is the “sweet spot” of model complexity?

Answer 29

The sweet spot is a level of complexity where the bias and variance are balanced, minimizing the overall error, which can be achieved through proper regularization.

Question 30

Why is the bias-variance tradeoff valuable in machine learning?

Answer 30

It provides useful intuition for model selection and regularization by explaining the tradeoff between underfitting and overfitting.

Question 31

Why is the bias-variance tradeoff of limited practical value?

Answer 31

• In theory, the bias-variance tradeoff relies on analyzing an ensemble of datasets to compute the exact bias and variance.
• In real-world scenarios, we typically work with only a single dataset, not an ensemble.
• Because bias and variance cannot be precisely calculated from a single dataset in practice, making it difficult to directly measure or apply the tradeoff.

Question 32

What is Bayesian linear regression?

Answer 32

Bayesian linear regression is a type of linear regression that incorporates Bayesian principles, allowing you to quantify uncertainty in predictions by providing a distribution over possible parameters (weights).

Question 33

How does Bayesian regression differ from traditional linear regression?

Answer 33

Traditional linear regression provides a single “best” estimate for the parameters, whereas Bayesian regression provides a probability distribution over possible parameters, reflecting uncertainty.

Question 34

In Bayesian regression, how are weights w treated?

Answer 34

Weights w are treated as random variables with a distribution, and the goal is to find the posterior distribution of the weights.

Question 35

What is the formula for the posterior distribution in Bayesian regression?

Answer 35

• The posterior distribution is proportional to the product of the likelihood and the prior
• p(w∣t)∝p(t∣w)p(w)

Question 36

What is the role of the likelihood p(t∣w) in Bayesian regression?

Answer 36

The likelihood describes how well the data is explained given the current set of weights.

Question 37

What is the role of the prior p(w) in Bayesian regression?

Answer 37

The prior encodes our initial belief about the parameters before observing the data.

Question 38

What does the posterior p(w∣t) represent?

Answer 38

The posterior represents the probability of a set of weights given the observed data, providing an updated estimate of the weights based on the prior and likelihood.

Question 39

What is the key idea behind Bayesian regression?

Answer 39

The key idea is to use a probability distribution over the weights and update this distribution based on the observed data, resulting in both the data and weights following probability distributions.

Question 40

What is the purpose of using a conjugate prior distribution in Bayesian linear regression?

Answer 40

Using a conjugate prior simplifies the computation by ensuring that the posterior distribution is in the same family as the prior, allowing easy tracking of the mean and covariance.

Question 41

What happens if the prior and likelihood are conjugate in Bayesian inference?

Answer 41

The posterior distribution will be a normal distribution with a mean m_N and variance S_N.

Question 42

How is the prior distribution p(w) typically represented in Bayesian linear regression?

Answer 42

• The prior is represented as a normal distribution p(w)=N(w|m_0, S_0)
  ​
• m_0 is the prior mean and
• S_0 is the prior covariance matrix.

Question 43

How is the posterior distribution represented in Bayesian linear regression?

Answer 43

• The prior is represented as a normal distribution p(w|t)=N(w|m_N, S_N)
  ​
• m_N is the posterior mean and
• S_N is the posterior covariance matrix.

Question 44

How is the posterior mean m_N calculated?

Answer 44

m_N = S_N(S_0^−1, m_0 + βΦ^T t)

Question 45

How is the posterior covariance S_N calculated?

Answer 45

S_N^−1 = S_0^−1 + βΦ^TΦ

Question 46

Why do we want the posterior mean m_N to change its location and decrease its variance?

Answer 46

To ensure that the model converges to an optimal set of weights with reduced uncertainty, improving confidence in predictions.

Question 47

what do the posterior mean m_N and posterior covariance S_N depend on

Answer 47

the prior parameters m_0 and S_0, and the observed data

Question 48

What is a common choice for the prior in Bayesian linear regression?

Answer 48

• A common prior is p(w)=N(w∣0,α^−1 I)
• mean of the weights is zero and the variance is 𝛼^−1 multiplied by the identity matrix

Question 49

What happens before observing any data (initial prior)?

Answer 49

1. Likelihood: Irrelevant, as no data points have been observed.
2. Prior/Posterior: Broad circular distribution centered around zero, indicating high uncertainty.
3. Data Space: The possible lines span a wide range of slopes and intercepts with no specific fit.

Question 50

How does the posterior update after observing one data point?

Answer 50

1. Likelihood: Influences the posterior, forming a diagonal ridge based on the first data point.
2. Prior/Posterior: Becomes more concentrated, reducing uncertainty.
3. Data Space: Possible lines start converging toward the correct slope and intercept.

Question 51

What changes after observing two data points?

Answer 51

1. Likelihood: Becomes narrower and more informative, further restricting possible values.
2. Prior/Posterior: More concentrated, increasing confidence in weight estimates.
3. Data Space: Lines converge closer toward the actual relationship, though some uncertainty remains.

Question 52

What happens after observing 20 data points?

Answer 52

1. Likelihood: Highly concentrated, showing certainty about the correct slope and intercept.
2. Prior/Posterior: Tightly concentrated around the true values, with low uncertainty.
3. Data Space: Lines cluster closely around the true relationship, indicating a near-perfect fit.

Question 53

What are the key takeaways from Bayesian linear regression’s updating process?

Answer 53

1. Prior Influence: Initially dominates, with high uncertainty.
2. Data Updates: As more data points are observed, the posterior becomes more concentrated.
3. Confidence Grows with Data: Confidence in the weight estimates increases with more data.
4. Role of Likelihood: Each data point provides new information, refining the prior to create the posterior.

Question 54

How does Bayesian linear regression adjust predictions with more data?

Answer 54

• The process reduces uncertainty about weights by updating the posterior as more data is observed.
• Initially, there’s high uncertainty, but the posterior narrows with each data point, leading to more confident and accurate predictions.

Question 55

How is the log of the posterior distribution expressed in Bayesian linear regression?

Answer 55

lnp(w∣t)= −β/2 SUM(t_n - w^Tϕ(x_n))^2 − α/2 w^Tw+const

Question 56

What does the log likelihood term

Answer 56

• −β/2 SUM(t_n - w^Tϕ(x_n))^2
• penalizes the difference between the predicted value (w^T ϕ(x_n) and the actual observed value t_n.
• the further the prediction deviates from the data, the lower the posterior probability for that particular set of weights.

Question 57

What is the role of the prior term in the log of the posterior

Answer 57

• − α/2 w^Tw
• penalizes large weights, ensuring the model does not stray too far from the belief that weights should be small.
• This introduces (free) regularization, which helps prevent overfitting.

Question 58

How does the strength of the prior affect the model?

Answer 58

A stronger prior (larger α) discourages large weights more, influencing the regularization level and the implication of different priors on the posterior distribution.

Question 59

Why can the constant term in the log of the posterior be ignored when maximizing the posterior?

Answer 59

It is unrelated to w, so it does not affect the optimization process and can be ignored when working with relative probabilities

Question 60

What are the key takeaways from the log of the posterior formula?

Answer 60

1. The log of the posterior is a combination of the likelihood (how well the model fits the data) and the prior (how much we trust our initial belief about the weights).
2. The posterior distribution becomes sharper (more peaked) around weight values that fit the data well and respect the prior.
3. The likelihood term pulls the weights toward values that fit the observed data, while the prior term pulls the weights toward zero or plausible values based on the prior.

Question 61

What happens in Bayesian linear regression if there is no prior information (α→0)

Answer 61

• High uncertainty: Precision goes to 0.
• The influence of the prior vanishes, so posterior relies entirely on the data, making it equivalent to standard frequentist linear regression (OLS) (reverts back to MLE.
• There is no regularization, and the posterior is only influenced by observed data.

Question 62

What is the posterior mean in the absence of prior information? (α→0)

Answer 62

• simplifies to the ordinary least squares (OLS) solution
• m_N = (Φ^T Φ)^−1 Φ^T t
• This is the same as the maximum likelihood estimate (MLE) or least squares solution in traditional linear regression.

Question 63

What happens if the prior information is very precise (α→∞)?

Answer 63

• Precision goes to infinity, and the prior dominates entirely.
• The posterior ignores the data, and predictions are determined solely by the prior.
• The posterior mean simplifies to m_N=0, meaning the model is fully biased by the prior and does not learn from data.

Question 64

What happens if there is infinite data (N→∞) in Bayesian regression?

Answer 64

• With infinite data, the posterior converges to the maximum likelihood estimate (MLE), and the impact of the prior diminishes.
• m_N = (Φ^T Φ)^−1 Φ^T t
• The data dominates, and the influence of the prior becomes negligible.

Question 65

How is the predictive distribution for a new input
x computed in Bayesian linear regression?

Answer 65

p(t∣x,α,β)= N(t|m_N^T ϕ(x),σ^2_N(x))

• uses a normal distribution with the posterior mean and the variance prediction to model the distribution of t

Question 66

How is the variance prediction σ^2_N(x)

Answer 66

• ϕ(x),σ^2_N(x)= 1/β + ϕ(x)^T S_N ϕ(x)
• this captures the inherent noise (1/β) and model uncertainty (ϕ(x)^T S_N ϕ(x))

Question 67

What does the predictive distribution p(t∣x) provide?

Answer 67

1. a mean prediction (the most likely value)
2. a variance (uncertainty) for the new input x.

Question 68

What does Bayesian linear regression capture in terms of uncertainty?

Answer 68

1. Data Dependence: The uncertainty is directly tied to the quantity and distribution of observed data points.
2. Expressive Uncertainty: It reflects high uncertainty in regions with sparse data and low uncertainty in densely populated regions.
3. Gradual Learning: As more data points become available, the posterior distribution sharpens, leading to more precise predictions and reduced uncertainty.

Question 69

What are the conclusions about Bayesian linear regression versus frequentist approaches?

Answer 69

1. Maximum likelihood or least squares can lead to overfitting if complex models are trained on limited data.
2. Bayesian methods avoid overfitting by quantifying uncertainty in model parameters.

Question 70

How can you explain the Bayesian approach for exam success?

Answer 70

1. Start with the Prior:
   “We begin with a prior belief about model parameters, representing our knowledge before observing any data.”
2. Incorporate the Data:
   “As data points are observed, the likelihood tells us how well each line fits the data. Combining it with the prior gives a new belief about the parameters.”
3. Predict with Uncertainty:
   “For a new input, instead of predicting a single value, we provide a distribution that includes both the prediction and the uncertainty due to limited data.”

Question 71

Why are machine learning problems considered optimization problems?

Answer 71

because they involve minimizing the error related to a set of weights.

Question 72

What does the convexity of the parameter space imply in linear models?

Answer 72

Convexity ensures that there is a single global optimal solution, meaning the optimization process is guaranteed to find the best solution.

Question 73

How does the gradient guide the optimization process?

Answer 73

1. The gradient indicates the direction of the steepest ascent.
2. Taking the negative gradient gives the direction of steepest descent.
3. Steps are taken in this direction, and the process is repeated until convergence.

Question 74

How does the step size affect the optimization process?

Answer 74

1. The influence of the step size depends on the curvature of the parameter space.
2. In steep regions, a large step size may overshoot the minimum, while in flat regions, the same step size may result in slow convergence and oscillations.

Question 75

How does feature scaling help in optimization?

Answer 75

Feature scaling normalizes the data so that step sizes in all directions are approximately the same, improving the efficiency and stability of the optimization process.

Question 76

What is the general update rule for gradient descent?

Answer 76

• w_j := w_j −α (∂E(w_0,w_1) / ∂w_j)

Question 77

How do you correctly update weights in gradient descent?

Answer 77

1. Calculate the updated values separately without updating them yet and store them in temporary values
2. Update all weights at the same time using the temporary values

Question 78

Why is it important to store the updated values separately during gradient descent?

Answer 78

Storing updated values separately ensures that each weight update is based on the original values, preventing incorrect updates caused by using already updated weights.

Question 79

Why is step size important in gradient descent?

Answer 79

1. Too large: The values oscillate and may diverge.
2. Too small: Convergence becomes very slow.

Question 80

Why is convexity important in optimization problems?

Answer 80

• Convexity is crucial because it ensures global optimality.
• In a convex parameter space, gradient descent is guaranteed to find the global minimum.

Question 81

What challenge arises when optimizing a non-convex parameter space?

Answer 81

the trajectory followed by gradient descent depends on the initialization point, which may lead to finding local minima instead of the global minimum

Question 82

In which models does gradient descent guarantee global optimality?

Answer 82

only in models with convex parameter spaces, such as linear models and logistic regression