Week 6: MCC-ROI analysis

Question 1

Definition of the p-value

Answer 1

• P(T >= t | H0); the probability of finding a result as extreme or more extreme than the observed statistic
• Expresses the surprise of observing the data if the null hypothesis was actually true
• Can only be used to refute H0 and doesn’t provide evidence for the truth of the H0

Question 2

Power

Answer 2

• The chance that a testing procedure correctly rejects H0 when there is a true effect; percent of TP that we will detect; one minus the Type II error rate
• Varies as a function of: 1) size of the true effect; 2) the efficiency of the statistical procedure; 3) and the sample size

Question 3

Type I error

Answer 3

the probability of rejecting the null hypothesis given that it is true. The test is designed to keep the type I error rate below a prespecified bound called the significance level, usually denoted by the Greek letter α

Question 4

Type II error

Answer 4

the probability of failing to reject the null hypothesis when it is actually false.

Question 5

Expected number of false positives given the H0

Answer 5

e.g., if our alpha level is set to 0.05, then we are implying that it is acceptable to have a 5% probability of incorrectly rejecting the true null hypothesis
(think of the normal distribution of thr H0 and of the little tail on the right, that means that we are taking that amount of risk of committing a type I error > same reasoning for type II error, but think of Ha distribution)

Question 6

High sensitivity leads to…

Answer 6

…few false negatives

Question 7

Low specificity leads to…

Answer 7

…many false positives

Question 8

Levels of inference

Answer 8

• voxel level
• cluster level
• peak level
• set level

Question 9

Voxel level

Answer 9

testing each individual voxel for significance and retaining the ones above a certain threshold (think of y-axis threshold); gives best spatial specificity IF the threshold is picked correctly; we can say something about a specific voxel.

Question 10

Cluster level

Answer 10

• takes into account the spatial information available in the images, by finding connected clusters of activated voxels and testing the significance of each
• has two stages: 1) defining clusters given a width and 2) retaining those clusters according to another threshold

Question 11

Why would we generally expect the fMRI signal to be spatially extended?

Answer 11

Because 1) the brain regions that are activated in fMRI are often much larger than the size of a single voxel and 2) fMRI data are often spatially smoothed, which results in a spreading of the signal across many voxels in the image

Question 12

Using the cluster level inference gives better…, but also worse spatial … . Picking a low cluster retaining threshold will cause a …, while picking a very high threshold will cause …

Answer 12

sensitivity;specificity;very large cluster that encompasses most of the brain;way less and smaller clusters

Question 13

When a 1,000 voxel cluster is marked as statistically significant, all we can conclude is that…

Answer 13

…one or more voxels within that cluster have evidence against the null. This is not a problem when cluster sizes are small, but it is if on the contrary we have a big cluster size; the only solution is to increase the retaining threshold, but this is not scientifically sound

Question 14

Peak level

Answer 14

similar to cluster level, so first we define the clusters (e.g., given their width) but then here we retain those clusters that go above a certain peak

Question 15

Set level

Answer 15

asks “is there any significant activation anywhere”?; it tests whether anyhwhere across the whole brain there is any activation, but cannot localise the activation; it is an omnibus test; has no localizing power whatsoever

Question 16

Types of error rates (+ error rate definition)

Answer 16

Error rate = a measure of the degree of prediction error of a model made with respect to the true model.
* per comparison error rate (PCER)
* family-wise error rate (FWER)
* false discovery rate (FDR)

Question 17

Multiple comparison (MC) problem

Answer 17

if a statistic image has 100,000 voxels, and we declare all voxels with P < 0.05 to be “significant,” then on average 5% of the 100,000 voxels – 5,000 voxels – will be false positives. This problem is referred to as the multiple testing problem and is a critical issue for fMRI analysis.

Standard hypothesis tests are designed only to control the ‘per comparison rate’ and are not meant to be used repetitively for a set of related tests.

To account for this multiplicity, we have to measure false positive risk over the number of tests we will conduct; error rate * the number of independent tests (voxels) we are going to conduct

Question 18

PCER

Answer 18

• Individual voxel’s error rate; for each voxel independently, the % of null pixels that are FP; the probability of a Type I error in the absence of any multiple hypothesis testing correction.
• example: if 10 significance tests were each conducted at 0.05 significance level, then the per-comparison error rate would be 0.05
• by ensuring that PCER <= alpha, the probability of making one or more errors for a single voxel is controlled by alpha

Question 19

FWER

Answer 19

• the probability of making at least one Type I error among a family of comparisons.
• increases as the number of comparisons increases, because the chance of getting a false positive becomes higher.
• in this case, the alpha level represents the amount of risk we take in accepting false positives throughout the whole map
• by ensuring that FWER <= alpha, the probability of making one or more errors in the family is controlled by alpha

Question 20

Ways to deal with the MC problem

Answer 20

• Bonferroni correction
• RFT correction
• Parametric correction
• Non-parametric correction

Question 21

Difference between FWER and FDR

They refer to two different things!

Answer 21

FWER tells us the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests; FDR tells you the expected proportion of significant tests (not of all tests) that will be Type I errors.

Put another way, where a level 0.05 FWE procedure is correct 95% of the time – no more than 5% of experiments examined can have any false positives - a level 0.05 FDR procedure produces results
that are 95% correct - in the long run the average FDR will be no more than 5%.

FDR’s greater sensitivity comes at the cost of greater false positive risk

Question 22

Important questions

Answer 22

What statistics we are working with & what error rate we are controlling?

Question 23

Spatial smoothing can cause…

Answer 23

…correlated data

Question 24

Classic way to deal with MC

Answer 24

Bonferroni correction; adjusts the alpha value by dividing it by the number of independent tests

Question 25

BUT, due to spatial smoothing, it is hard to assess the amount of independent things / voxels we deal with…this makes Bonferroni…

Answer 25

fail.

Question 26

Why does more smoothing of the fMRI signal increase the chance of false positives?

Answer 26

Smoothing is intended to improve the signal-to-noise ratio (SNR). If there is less noise, there will be fewer false positives. However, if the smoothing is excessive, the size of clusters can be exaggerated leading to more false positives. So, we need to strike a balance with enough smoothing to improve SNR, but not so much that we are spreading activation. The expected size of the clusters is typically a good guide to determine the right amount of smoothing. The range of smoothing using in fMRI analysis is usually quite restricted as well. It’s rare to find spatial smoothing kernels outside of the 4-8 mm range.

Question 27

RFT

Answer 27

• method that accounts for the spatial dependence between voxels when correcting for MC (and hence adjsuting our p-value) > post-hoc test!
• accounts for the degree of smoothness in the data

Question 28

In RFT, smoothness is defined by…

Answer 28

…[FWMH_x, FWMH_y, FWMH_z]; this is NOT the size of the Gaussian kernel applied, but the the combination of the intrinsic and applied smoothing.

Question 29

RESEL

Answer 29

• a virtual voxel of size FWHM_x × FWHM_y × FWHM_z
• the number of resels is given by: V / size of the smoothness parameter

Question 30

In RFT

Through the smoothness and resels parameters, we can adjust the p-value to account for the smoothness in the data: how?

Answer 30

there is a formula in the book; this formula shows that, for a given statistic value t and search volume V:
1. as the product of FWHM’s increase (hence more smoothing), the RESEL count R decreases and so does the corrected P-value, producing increased significance; the intuition is that greater smoothness means there is a less severe multiple testing problem, and a less stringent correction is necessary.
2. as the search volume in RESELs grows, so does the corrected P-value, producing decreased significance for the same statistical value. This should also make sense, as the larger the search volume, the more severe the multiple testing problem.

Question 31

RFT limitations

Answer 31

• Requires estimating parameters
• Must be sufficiently smooth: if you do not spatially smooth the data enough, RFT will not work

Question 32

Parametric tests

Answer 32

making parametric assumptions about the data to approximate P-values

Question 33

Non-parametric tests

Answer 33

use the data themselves to obtain empirical null distributions of the test statistic of interest and estimate the FWE-corrected p-values

Question 34

Permutations tests

Answer 34

• Using just a single voxel, suppose you have two groups of ten subjects, high performers (H) and low performers (L), each of whose BOLD response data you wish to compare. Under the null hypothesis of no group difference, the group labels are arbitrary, and one could randomly select ten subjects to be the H group, reanalyze the data, and expect similar results.
• principle of the permutation test: repeatedly shuffling the assignment of experimental labels to the data, and analyzing the data for each shuffle to create a distribution of statistic values that would be expected under the null hypothesis

Question 35

How to account for the MC with permutation tests and obtain the FWE-corrected P-values?

Answer 35

• A FWE-corrected P-value is found by comparing a particular statistic value to the distribution of the
  maximal statistic across the whole image > why?
• With repeated permutation a distribution of the maximum statistic (for voxels, the highest intensity, for clusters the biggest size) is constructed, and the FWE corrected P-value is the proportion of maxima in the permutation distribution that is as large or larger than the observed statistic value

Question 36

ROI analysis

Answer 36

• reduces the stringency of the
  correction for multiple testing.
• it is crucial that the ROI is defined
  independently of the statistical analysis of interest.

Question 37

Circularity

it refers to ROI analysis methods

Answer 37

• the issue lies in trying to derive the magnitude of an activation effect using a ROI derived from the same data.
• arises when one selects a subset of noisy variables
  (e.g., voxels) from an initial analysis for further characterization
• When a voxel exceeds the threshold (and is thus selected for further analysis), this can be due
  either to signal or to noise
• In the case in which there is no signal, the only voxels
  that will exceed the threshold will be the ones that have a very strong positive noise value.
• If we then estimate the mean intensity of only those voxels that exceed the threshold, they will necessarily have a large positive value; there is no way that it could be otherwise, since they were already selected on the basis of exceeding the threshold
• In the case where there is both true signal and noise, the mean of the voxels that exceed threshold will be inflated by the positive noise values, since those voxels with strong negative noise contributions will not reach threshold
• Thus, the mean effect size for voxels reaching threshold will over-estimate the true effect size

Question 38

Forward inference

Answer 38

what brain activity/region is associated with a
given experimental condition

Question 39

Reverse reference

Answer 39

what cognitive process/behavior etc. is
occurring given the brain activity.

Question 40

Double dipping (another way to refer to circular analysis)

think of training vs test set

Answer 40

the use of the same dataset for selection and selective analysis, will give distorted descriptive statistics and invalid statistical inference whenever the results statistics are not inherently independent of the selection criteria under the null hypothesis

Question 41

Validity

Answer 41

Internal validity: Degree to which the observed effect can be related to a cause (or could there be an alternative cause).
Construct validity: Are we measuring the supposed
underlying construct.
External validity: Do my results generalize?

Question 42

Reliability

Answer 42

Degree to which it is possible to replicate reported results