Hypotheses, Prediction, Scientific Method

Question 1

what are hypothesis testing alternatives?

Answer 1

Bayesian methods and likelihood techniques

Question 2

what are the main characteristics of “new” hypothesis testing alternatives?

Answer 2

prior information
several hypotheses
inferences from multiple models
parameters which refer to biological processes/states

Question 3

how can the scientist use his knowledge in these “new” hypothesis testing alternatives?

Answer 3

the knowledge of the scientist is used to formulate hypotheses (testable ideas) and select models that represent them

Question 4

how detailed should a model be?

Answer 4

this can be objectively assessed depending on the data available to estimate parameters for model

Question 5

what is the probability of observing this data if the null hypothesis was true? (which statistical method is used to answer this question?)

Answer 5

we answer that with p-value, which shows the probability of observing current or more extreme values considering that the null hypothesis is true

Question 6

what is missed by a p-value based analysis?

Answer 6

does not evaluate if hypothesis/model is strong; as it only evaluates chance of extreme events

Question 7

what is the likelihood of competing hypotheses considering the data?

Answer 7

proportional to likelihood of data given the hypotheses

Question 8

give 3 characteristics of likelihood methods for testing hypotheses

Answer 8

competing models can have different parameters data is fixed and hypothesis is variable
relative support of each model given the data

Question 9

what is goal and main assumption of maximum likelihood estimations?

Answer 9

• goal: find parameter estimates which give a distribution which will in turn maximize chance of observing data
• assumption: considering that each observation is independent, then the total probability of observing all of data is the product of observing each data point individually

Question 10

how to evaluate likelihood models?

Answer 10

by AIC (Akaike’s Information Criterion): the best model loses less information when attempting to approximate reality (lowest AIC value); assessment of a given model depends on performance of other models which are being compared together

Question 11

what are Akaike weights for?

Answer 11

to account for uncertainty (if we selected these models many times, what is the probability of model x being the best one?)

Question 12

compare a traditional hypothesis testing with a likelihood method in terms of how groups are compared

Answer 12

instead of comparing means, we compare competing hypotheses in terms of how much support there is in the data for effects of each treatment; can be multivariate

Question 13

what is the main difference between bayesian and likelihood methods?

Answer 13

pre-existing info are probabilities instead of likelihoods

Question 14

describe a Bayesian analysis

Answer 14

previous knowledge for each parameter is a requirement; prior = pre-existing datasets; new knowledge from new set of data = previous knowledge modified by what was learned from the new data

Question 15

what is a unique feature of bayesian methods?

Answer 15

output of Bayesian analyses: probabilities of hypotheses (no other method does it)

Question 16

describe output for frequentist, likelihood, and bayesian methods

Answer 16

frequentist stats say that if experiment is repeated many times, parameter estimate would fall in the 95% confidence interval 95% of the times
bayesian: probabilities of hypotheses (no other method does it)
likelihoods are proportionate to probabilities

Question 17

what is a confidence interval?

Answer 17

range of values we are fairly sure our true value lies in

Question 18

confidence, in statistics, is another way to describe probability. exemplify this.

Answer 18

if you construct a confidence interval with a 95% confidence level, you are confident that 95 out of 100 times the estimate will fall between the upper and lower values specified by the confidence interval

Question 19

what happens with the confidence interval if I am comparing a highly variable group with a uniform group?

Answer 19

the confidence interval will be narrower for the uniform group (more certainty)

Question 20

which method answers these questions: what is the odd of alternative results? What are the expected values of alternative decisions? Or the frequency distribution of results?

Answer 20

these are questions which can only be answered by Bayesian methods

Question 21

compare likelihood and bayesian methods in terms of parameters

Answer 21

likelihood: maximization of possible values across parameters; Bayesian: integration across parameters

Question 22

problem with Bayesian methods: when we don’t know much about parameters; what is the solution?

Answer 22

meta-analysis: summarizing pre-existing knowledge

Question 23

what are meta-analysis types in terms of parameter estimation?

Answer 23

stimate the value of a parameter based on several studies; several parameters whose values are different in each population (can be visualized in a histogram)

Question 24

what is a problem with meta-analyses in terms of how samples are taken?

Answer 24

we can’t guarantee that the samples were randomly taken (eg., more productive samples usually are chosen)

Question 25

why are ecological data and conclusions different than other fields?

Answer 25

natural variability and complex interactions interferes with results, is of interest to us, and makes data analysis a complex task
instead of having a single correct hypothesis, we usually end up with several partially correct hypotheses. Thus, we obtain a solid understanding of a phenomenon by looking at several studies over time instead of looking at a single one (eg., how trophic interactions shape communities)

Question 26

why are new statistical methods more adequate for ecological studies?

Answer 26

ecological processes are characterized by several forces acting together; each portion of this complex system will have hypotheses that are more or less true. Thus, new statistics (likelihoods and Bayesian probabilities) are more aligned with ecology because they allow us to evaluate several hypotheses in terms of their relative importance to explain a given process

Question 27

what can be the contribution of new statistics to the precautionary principle and for adaptive management of natural resources?

Answer 27

• Precautionary principle: practice is forbidden unless you can prove inexistence of effects; this is questionable: how large should an effect be to justify regulation? New statistics can help understand the probability of competing hypotheses and the size of the effects.
• Adaptive management of natural resources: improved data (future, after harvest for example) is estimated with Bayesian methods

Question 28

what was the original idea behind p-values?

Answer 28

could my results be produced by random chance?

Question 29

what is a null hypothesis?

Answer 29

what we want to disprove

Question 30

what is a p-value?

Answer 30

what are the chances of getting results at least as extreme as the ones I actually got? This is the p-value (the smallest the value, the greater the chance of the null hypothesis being false)

Question 31

p-value of 0.01 means that there is a…

Answer 31

99% chance of value falling in the 99% confidence interval; it summarizes the data given one specific null hypothesis

Question 32

does a p-value of 0.01 mean that there is only 1% of accepted hypothesis being wrong?

Answer 32

no, as it does not calculate the odds of an effect being true in the first place

Question 33

describe a problem with p-value

Answer 33

deviates attention from effect size analysis

Question 34

what is p-hacking?

Answer 34

dropping conditions or doing other manipulations so that significant p-value is achieved

Question 35

what are the recommendations for analyzing data moving forward from traditional p-value based analyses?

Answer 35

• Confidence intervals (relative importance) and effect sizes (magnitude of effect)
• move to Bayesian methods (probabilities of outcome instead of potential frequency of outcome)
• try several methods and compare results
• disclosure written on paper (how was sample size determined? Was data excluded? How were manipulations and measures made?)
• use an open science framework database to publish results of a first round of experiments, then replicate studies and publish

Question 36

complete the sentence for traditional hypotheses testing: what is the chance of observing current effects (or more extreme)…

Answer 36

if the true effect is zero?

Question 37

when we want to falsify a hypothesis, what analysis is appropriate?

Answer 37

p-values

Question 38

what is a difficulty of likelihood techniques?

Answer 38

how to select candidate set of models to represent hypotheses?

Question 39

what is the problem when analyzing with small populations in a frequentist approach?

Answer 39

a small decrease in pop growth, for example, will be likely be non-significant (small statistical power); this would be better analyzed as an exercise of parameter estimation only

Question 40

when are frequentist approaches useful?

Answer 40

frequentist approaches are useful for carefully designed experiments which aim to falsify a given hypothesis

Question 41

what are three distinct goals for statistical modeling in ecology?

Answer 41

data exploration, inference, and prediction

Question 42

Populations, communities, and ecosystem-level
properties fluctuate through time in response to internal
and external drivers. Describe them.

Answer 42

Internal drivers include intraspecific density dependence, demographic stochasticity, interspecific interactions, and food web dynamics. External drivers are typically related to environmental conditions,
and weather is perhaps the most variable.

Question 43

Why is quantifying the relative impacts of internal forcing versus weather on ecological dynamics a core goal of ecology with
new relevance?

Answer 43

As we attempt to predict ecological responses to climate change.

Question 44

what is the goal of exploratory studies?

Answer 44

describe patterns in the data and generate hypotheses about nature

Question 45

what is a type 1 error? how can it happen during exploratory analysis?

Answer 45

making false discoveries. it can happen when trying to indiscriminately combinate many potential covariates (a few of them will be highly correlated by chance)

Question 46

what are the 3 good practices for exploratory analyses?

Answer 46

only consider plausible relationships
correct p-values for multiple testing to reduce chance of false discoveries
communicate exploratory nature of analysis (proposed hypotheses will only be valid if confirmed by subsequent independent studies)

Question 47

what is inference analysis?

Answer 47

evaluate the strength of evidence in a data set for some statement about nature

Question 48

what is the main difference between exploratory and inferential analysis?

Answer 48

exploratory: hypotheses are generated
inference: tests hypotheses based on a priori knowledge of ecological systems; eg. null hypothesis significance testing
we need independent datasets for both of these techniques

Question 49

are conclusions obtained via statistical inference automatically accepted as scientific fact?

Answer 49

no, as the reliability of statements is tied to a particular set of data obtained by particular methods under particular conditions; thus, it requires replication and validation across a range of conditions before they are accepted as scientific facts

Question 50

what happens when we are comparing too many models in inferential analyses?

Answer 50

the true goal of the analyses may be exploratory or predictive. inferential analyses have a small risk of false discoveries only when a small set of groups (2 or 3) are compared.

Question 51

what is distinguishing feature of modeling for prediction?

Answer 51

the need to test predictive models “out of sample”: using independent data that were not used to fit the model. Exploratory modeling can help identify important predictors even if the causal mechanisms are currently not understood, but without validation on independent data, the predictive skill of the model is unknown.

Question 52

what is the difference between predictive and inferential analyses when looking at a linear model?

Answer 52

predictive analyses focus on the response variable (which model will best predict y values for new observations of covariates in x?
inferential analyses focus on estimates of the regression coefficients (which ones are positive, negative, non-zero? which covariates are more important?)

Question 53

what are the 2 main components of the AIC (Akaike’s Information Criterion) formula?

Answer 53

number of parameters
maximum likelihood estimates for model parameters

Question 54

what are the pitfalls of AIC analyses?

Answer 54

Proponents of AIC argue that this approach avoids arbitrary P-value cut-offs (Burnham and Anderson 2002), but in practice researchers have relied on equally arbitrary and less interpretable cut-offs for the
difference in AIC values required to conclude that one model is more supported by the data than another.

Question 55

what is regularization (eg., 2p in AIC equations)? what would happen if it was not there?

Answer 55

2p = regulates model complexity by penalizing models with too many parameters
without this penalty, models with the highest number of parameters would always be the chosen as the best ones (more complex models)

Question 56

define regularization.

Answer 56

regularization involves selecting a model or estimating parameter values based on a weighted average of a goodness-of-fit measure and a complexity penalty.

Question 57

what is model validation? why is it important?

Answer 57

comparing model predictions with observations that were not used to
“train,” or fit, the model. Out-of-sample validation is important because it is easy for a model to reproduce patterns in the training data, but much harder to accurately predict the outcome in another situation, which is what we expect predictive models to do.

Question 58

what is the process of out-of-sample validation like?

Answer 58

• randomly splitting the available data into training and validation sets
• we then fit candidate models using the training data, use
  each model to make predictions for the validation data,
  and quantify model errors using a summary measure
  such as root mean squared error.

Question 59

what can we do if our dataset is too small to do an out-of-sample validation technique?

Answer 59

when data sets are too small for out-of-sample validation,
we can use cross-validation

Question 60

how can we screen for potentially important covariates when doing an exploratory analysis?

Answer 60

we can calculate the correlation coefficient between population growth rate and each covariate and establish an arbitrary cut-off value (eg., 0.3)

Question 61

describe the steps for exploratory analysis

Answer 61

• define the research question (eg., which covariates are associated with population growth rates)
• screen many covariates for potential associations (eg., correlation coefficients with arbitrary threshold)
• assess statistical evidence more rigorously (linear mixed-models including covariates with high coefficient; followed by likelihood methods comparing full model with reduced versions to exclude covariates which do not change full model results)
• fit final model and check for inconsistencies (eg., estimates are positive in Nov but negative in Dec?)
• correct for multiple comparisons (control for expected proportion of of false significant results)
• consider alternative approaches (sliding climate windows to average weather data from a group of periods instead of analyzing periods individually)

Question 62

describe the steps for inference.

Answer 62

formulate competing hypotheses: priori hypotheses based on the
literature or deduced from theory
translate hypotheses into alternative models: eg. null-hypothesis significance testing

Question 63

describe the steps for prediction.

Answer 63

• define the predictive goal: eg., objective is to predict butterfly population size (on log scale) at time t given data on log population size in year t − 1 and information about weather during the period in between the observations
• choose model selection approach (eg., regularization or AIC?)
• choose model validation approach (eg., select sub dataset as training data and validation data from dataset)

Question 64

does an understanding of the process translate into improved predictive skill?

Answer 64

• not necessarily, as using covariates from an inference study can lead to loss of predictive accuracy, for example

Question 65

give an example to explain why a given predictive model testing effects of climate on population growth may be weak

Answer 65

a given population may be responding to many weak climate signals instead of some stronger ones

Question 66

describe classical errors for each exploratory, inference, and predictive studies.

Answer 66

exploratory: overfitting, coming up with fake discoveries
inference: misrepresenting exploratory analyses as inference studies
predictive: not testing predictive power of models in new, independent data

Question 67

while in traditional statistics we are aiming at establishing significant differences among means, what kind of comparisons can we make using new methods (maximum likelihood and bayesian)?

Answer 67

because p-values in traditional analysis do not estimate evidence, new techniques are more appropriate for finding in the data the relative support for various competing hypotheses regarding the effects of treatment.