7 | Parameter estimation 1: nonlinear least squares

Question 1

Name a method discussed in the lecture to estimate the parameters of a systems biology model from experimental data. When is a set of parameters considered to be optimal in this method?

(2023_1, 2021_1 - question on NFkB Hoffman paper)

Answer 1

Nonlinear least squares.
Optimal when parameters minimize sum of sq. R between observed data, model predictions.

OR

Likelihood based methods.
Optimal when parameters maximize likelihood function

Question 2

Which of the following regression functions are nonlinear?

f(z;θ) = θ1 + zθ2 + z²θ3

f(z;θ) = θ1 + (z²θ2) / (1 + z²)

f(z;θ)= θ1e^−zθ2

The regression function corresponding to the reaction system
∗ ⟶(θ1) S ⟶(θ2) ∗
with observation function h(x)=x

(ungraded quiz)
.

Answer 2

f(z;θ)= θ1e^−zθ2

The regression function corresponding to the reaction system
∗ ⟶(θ1) S ⟶(θ2) ∗
with observation function h(x)=x

(nonlinear regression function = nonlinear in the parameters, not x)

Question 3

What is the difference between the errors and the residuals?

(ungraded quiz)

Answer 3

Errors describe randomness in the data generating process, while residuals describe misfit of a model assuming a certain parameter (e.g., least squares estimate).

Question 4

True or false

Residuals describe randomness in the data generating process, while errors describe misfit of a model assuming a certain parameter (e.g., least squares estimate).

(ungraded quiz)

Answer 4

F

Question 5

True or false

Errors describe randomness in the data generating process, while residuals describe misfit of a model assuming a certain parameter (e.g., least squares estimate).

(ungraded quiz)

Answer 5

T

Question 6

True or false

Errors and residuals can be used synonymously

(ungraded quiz)

Answer 6

F

Question 7

Consider the nonlinear regression model:

y_i = f(z_i;θ) + ε_i.

What can be done if the scatter in the residuals r_i increases with values of the dependent variable y_i?

☐ Use weighted least squares estimation
☐ This is no problem for the nonlinear regression model.
☐ Use the transform-both-sides-approach
☐ Use a more complex regression function

(ungraded quiz)

Answer 7

Use the transform-both-sides-approach:
involves transforming both the dependent variable and the independent variables to stabilize the variance. For example, taking the logarithm of both sides can help if the variance increases with the mean.

Use weighted least squares estimation
method that assigns different weights to different observations based on the variance of their errors. When the scatter of residuals increases with the dependent variable, WLS can be used to give less weight to observations with higher variance, thereby stabilizing the variance across all observations.

Question 8

True or false

The least squares estimate can always be found analytically.

(ungraded quiz)

Answer 8

F

Question 9

True or false

For ODE-based models, the regression function cannot always be stated analytically.

(ungraded quiz)

Answer 9

T

Question 10

True or false

Residual plots allow to detect misspecifications of the error model, but not the regression function.

(ungraded quiz)

Answer 10

F

Question 11

True or false

The least squares estimate solves the equation R(θ)=0, where R is the sum of squared residuals.

(ungraded quiz)

Answer 11

F

Question 12

True or false

For nonlinear regression functions, the least squares estimate cannot be found analytically.

(ungraded quiz)

Answer 12

T

Question 13

True or false

Residual plots allow to detect misspecifications of the regression function or the error model

(ungraded quiz)

Answer 13

T

Question 14

Assume we want to find a polynomial of a suitable degree that fits a dataset consisting of x-y pairs.

For a fixed degree K of the polynomial, specify the regression function; then explain how least squares can be used to estimate the coefficients of the polynomial.
(2022_2)

Answer 14

Model:
f(x_i; θ) = θ₁ + θ₂x² + θ₃x³ + θ_Kx^K
Observed values vs model:
y_i = f(x_i;θ) + ε_i.

How to do nonlinear least squares:
find the coefficients θ that minimize the sum of the squared differences between the observed values and the values predicted by the polynomial model

Question 15

Assume we want to find a polynomial of a suitable degree that fits a dataset consisting of x-y pairs.

What is the problem with using the sum of squared residuals R to find the best polynomial degree? What could be done instead?

(2022_2)

Answer 15

main problem is overfitting:
- As degree of polynomial increases, R always decrease / stay same, even if higher degree polynomial not actually better model
- (higher degree polynomial has more flexibility to fit the specific data points, including noise and random fluctuations, rather than capturing the true underlying relationship)

As a result, using R alone will always favor the highest degree polynomial possible, which may not generalize well to new data.

Instead of solely relying on the sum of squared residuals, model selection criteria that penalize model complexity can be used, for example
- likelihood ratio test
- Akaike Information Criterion (AIC):