lecture 4 - model fitting

Question 1

types of models

Answer 1

1. model animal
2. algorithmic model
3. artificial neural network
4. data driven models

Question 2

model animal

Answer 2

• model animals allow researchers to draw conclusions that may generalize across species
• e.g. mice are often used as models to study biological processes and behaviors relevant to humans

Question 3

algorithmic model

Answer 3

• never touches data
• relies on algorithms or theoretical constructs
• they are abstract and typically focus on understanding or simulating processes in a hypothetical or idealized way

Question 4

artificial neural network

Answer 4

• more of a tool than a scientific model
• typically applied in engineering contexts to process data, rather than to explain underlying biological mechanisms
• do mimic some properties of biological neural networks

Question 5

data-driven model

Answer 5

• used by scientists to explain data
• explicitly created to analyze and interpret real data, helping scientists draw insights directly from empirical observations

Question 6

George Box: ‘All models are wrong. But some models are useful’

Answer 6

• emphasizes that no model can perfectly represent reality, because models are simplifications of complex systems.
• they leave out details and assumptions, and therefore cannot fully capture the intricacies of real-world phenomena.
• however, despite their limitations, models can still be valuable tools.

Question 7

descriptive models

Answer 7

• mathematical description of the data
• ‘fitting’ is important
• fitted parameters can be assessed, but are properties of the data more than of the model

Question 8

process models

Answer 8

• mathematical description of the process that gave rise to the data
• ‘fitting’ is important
• fitted parameters have meaning because they tell us something about the generative process - i.e., how the data was produced

Question 9

utility of descriptive models

Answer 9

1. gaussian distribution (central limit theorem): models noise
2. n-degree polynomial: describe the shape of data

Question 10

utility of process models

Answer 10

• parameters have cognitive/neural meaning

E.G.,

• parameters quantify the process by which the brain reaches a decision
• modeler commits to latent variables (e.g., action value in RL)

Question 11

why are process models harder to formulate

Answer 11

• because you have to think about the underlying process, not just the data
• i.e., they require a deep understanding of the cognitive or neural mechanisms, not just the ability to fit a dataset

Question 12

process models or descriptive models

Answer 12

• many models are a bit of both descriptive and process categories
• e.g., SDT

Question 13

occam’s razor/rule

Answer 13

• helps decide what the better model is
• lex parsimoniae: suggests that “entities are not to be multiplied without necessity.”
• if two models give an accurate description of the data, the simpler model is to be preferred

Question 14

why choose the simpler model

Answer 14

1. generalization: a good model is not dependent on the experiment: it generalizes.
2. there is always a model with more parameters, giving a better fit

Question 15

key questions for model selection

Answer 15

1. does the data require more complexity: if a simpler model fits the data adequately, adding complexity might not be necessary.
2. are you fitting the process or the noise: the goal is to model the actual process that generated the data, not the random fluctuations or noise within it.

Question 16

overfitting

Answer 16

happens when there are too many parameters for the data, and are fitting the noise in a specific dataset, rather than the true underlying pattern

Question 17

cross-validation

Answer 17

• main method against overfitting
• splits the data into fit (train) and test datasets
• if you’re fitting noise that is unique to the training set, your model should fail to predict the test data (CVr^2 = N(0,σ)

Question 18

comparing model fits

Answer 18

1. cross validation: splits data into training and test sets to check if the model generalizes well to unseen data
2. information criteria: downweight the quality of fit of a model by penalizing the number of parameters.

Question 19

types of information criteria

Answer 19

1. bayesian information criterion (BIC): -2ln(L) + k * ln(n)
2. akaike information criterion (AIC): -2ln(L) + 2k

k = nr parameters, n = sample size, ln(L) = loglikelihood

Question 20

how do information criteria work

Answer 20

• If two models fit the data equally well, the one with fewer parameters will be preferred because it incurs a smaller penalty.
• the criteria favor simpler models unless the more complex model demonstrates a significantly better fit to the data

Question 21

BIC & AIC: quantity of parameters

Answer 21

• there is flexibility in the number of parameters in a model
• it is possible to formulate a model that decreases the number parameters without losing predictive power

Question 22

BIC and AIC: what does it mean to say that they are conservative metrics

Answer 22

• BIC and AIC are conservative: they favor simpler models that avoid overfitting by penalizing added complexity
• BIC is a more conservative criterion as it includes a stronger penalty for the number of parameters than AIC, especially when the sample size is large
• therefore, first do cross validation. if you can’t then you look at information criteria

Question 23

what does it mean to ‘fit a model to data’

Answer 23

• it means we found the parameters for our model that, when used to create a prediction (simulate), best explain our data

Question 24

model fitting methods

Answer 24

1. quantify explanation differently
2. search the parameters differently

Question 25

we want to find parameters that

Answer 25

1. minimise the (euclidean) distance between the model and the data (SSE)
2. maximise the likelihood of the data, given the model parameters (MLE)

Question 26

likelihood

Answer 26

• p(y∣θ)
• tells us how likely the data is given a specific set of parameters

Question 27

maximum likelihood

Answer 27

directly uses the likelihood function to find the set of “best” parameters that maximizes the likelihood of the observed data

Question 28

How does a model account for variability or noise in data?

Answer 28

By assigning probabilities to data points instead of deterministic values, the model assumes the data is generated as y=f(θ)+N(0,σ), where noise follows a normal distribution.

Question 29

what is f(θ)

Answer 29

• function of θ
• can be a vector, matrix, distribution, etc. of parameters
• each setting of θ defines a specific model instance

Question 30

What does it mean for a model to be a probability density function (pdf)?

Answer 30

• each set of parameters defines a model instance and a probability distribution over the outcomes of y, rather than a deterministic value
• MLE identifies the parameter set (θ-hat) that maximizes p(y∣θ).

Question 31

how are p(y∣θ) and p(θ|y) related to each other

Answer 31

• p(y∣θ) - likelihood: by varying θ, we can quantify the probability of the data given θ
• p(θ|y) - posterior: from this, you can make a distribution that shows the value of θ that maximizes the likelihood of the data p(θ|y)

Question 32

MLE vs bayes

Answer 32

MLE
1. only considers the likelihood term
2. ignores prior probability term as MLE does not use strong prior expectations
3. ignores marginal probability term since it uses fixed data

bayes
1. combines evidence, expectations, and hypotheses

Question 33

MLE problems and solutions

Answer 33

• problem 1: works with probabilities of observations, which can become really small (p = 0.00001) for the entire data set really fast if something is unlikely
• solution 1: use logarithm of the likelihood (loglikelihood)
• problem 2: many methods are used to work with distances (e.g., SSE), which are minimised, not maximized
• solution 2: minimize negative log-likelihood, as this is mathematically equivalent to maximizing log(L), but it aligns with standard minimization-based optimization frameworks

Question 34

smoothness of the likelihood landscape

Answer 34

• depends on the quality and amount of data
• with more data, the landscape becomes smoother, making optimization easier and more reliable.
• sparse or noisy data can lead to a rugged likelihood surface with many local minima.

Question 35

relation of the parameters in the likelihood landscape

Answer 35

they are dependent on each other

Question 36

grid search

Answer 36

• simplest optimiser
• exhaustively goes through all parameter combinations in a grid to find the best model

Question 37

grid search: downside

Answer 37

• higher number of parameters leads to exponential growth of the number of evaluations, causing computational explosion
• therefore only feasible for models with very few parameters and small search spaces

Question 38

gradient descent

Answer 38

• more complicated optimiser
• systematically searches for the minimum of the likelihood landscape by following the gradient
• more efficient than grid search — can be done in fewer iterations
• efficient for larger parameter spaces

Question 39

gradient descent: downside

Answer 39

• relies on a smooth likelihood landscape.
• if the landscape has multiple local minima, e.g., for complicated fits (non convex parameter space), gradient descent might converge to a suboptimal solution

Question 40

gradient descent: how it works

Answer 40

1. set up a cost function J(θ)
2. set up an objective function that minimizes J(θ) based on distance (SSE) or likelihood (MLE)
3. on each iteration, determine the direction to step in based on the gradient at the present step
4. update based on the gradient multiplied by the learning rate
5. repeat until convergence

Question 41

learning rate value: too high

Answer 41

potentially overshooting the minimum or leading to divergence, oversensitive to noise

Question 42

learning rate value: too low

Answer 42

slow convergence and potentially getting stuck in local minima if the parameter space is not smooth enough

Question 43

gradient descent and reinforcement learning

Answer 43

gradient descent is mathematically the same to RL speed-accuracy trade off: a good choice of the learning rate balances speed and stability

Question 44

gradient descent: problem & solution

Answer 44

• problem: complicated fits (non-convex) run the risk of getting stuck in a local minimum/maximum
• solution: difficult to find if you really found the global solution, so choose starting values wisely and check results to see if multiple intializations converge to the same solution

Question 45

bayesian fitting: hierarchical bayesian modeling

Answer 45

• allows us to specify the same model for all participants, and fit per-subject and group-level parameter estimates.

Question 46

parameter recovery

Answer 46

• fitting comes with different problems, like sometimes we have different sets of parameters that produce the same outputs.
• we need to know if our problem is fittable
• recovery ensures your modeling process produces results that are interpretable and replicable

Question 47

parameter recovery recipe

Answer 47

1. simulate using fitted parameters
2. add noise
3. refit
4. check if parameters can be recovered

Question 48

parameter recovery outcome

Answer 48

• if parameters come from this distribution, they should be able to simulate data that is close to the original dataset
• if your model cannot recover known parameters, it may be overfitting, underfitting, or poorly specified.

Question 49

George Box

Answer 49

‘All models are wrong. But some models are useful’