Wk3 - Cautionary Tales and Dead Fish Flashcards
What is a problem with many fMRI experiments showing correlations between brain activation and personality tests?
Researchers perform non-independent correlation analyses
It is not always possible to see the research methods and data analysis methods that studies used
What in the results section might indicate that non-independent analysis has been used?
Very high correlations between brain activation during a task and scores on personality tests
What in the method section might indicate that non-independent analysis has been used?
Brief method section
Doesn’t explain how the analysis was run
Doesn’t explain how the researcher’s chose which parts of the brain to look at
Doesn’t explain how the researcher’s corrected for multiple comparisons etc.
Who noticed puzzlingly high correlations in many papers?
Vul et al. (2009)
What 2 things do correlations measure?
Relationship between the variables
Reliability of the two measures used
What is considered a high correlation?
A correlation over .8 or .9
What does a correlation of .9 mean?
90% of the variance in a subject’s scores on a questionnaire can be explained by the amount of brain activity in a region. This is extremely high and rare.
In terms of measure reliability, in what circumstances might we find a high correlation?
If both measures had PERFECT reliability. Rare.
Can a correlation be higher than the reliability of the scales used?
No
What is typically the highest reliability most personality scales have?
.8
What is typically the highest reliability most BOLD studies have found?
.7
If the BOLD measurement reliability is .7 and the personality measurement reliability is .8, then what is the maximum correlation a study should find?
r = .74
How is BOLD reliability calculated?
Test-retest of a simple task and seeing whether the same activations occur.
What makes up an fMRI image?
Lots of measurements of the BOLD signal in regions called voxels
What is the range of number of voxels in ONE fMRI image?
40,000 - 500,000 voxels
How many mm of brain tissue does each voxel cover?
1mm3 - 125mm3
How often are fMRI images taken?
Every 2 to 3 seconds during the scan to give a time series of these images (multiple snapshots)
How would researchers get a contrast between task conditions?
They would take away the activation in one condition from the activation in another condition
Compare the difference in activation
What does a contrast between tasks yield?
Matrices for hundreds of thousands of numbers indicating activation levels in different voxels
What can researchers base their choice of selecting a subset of voxels on?
Can select a subset of voxels based on anatomical constraints, functional constraints, or a mixture of both anatomical and functional constaints
Give an example of selecting voxels based on anatomical constraints
Selecting a subset of all voxels in the amygdala
Give an example of selecting voxels based on functional constraints
Selecting a subset of all voxels that are more active to one condition than another
Give an example of selecting voxels based on a mixture of anatomical and functional constraints
Selecting a subset of all voxels that are more active to one condition in the amygdala
Why do researcher’s have to select a subset of voxels to run further statistical analyses on?
To reduce the likelihood of finding significant results by chance.
Because if they run multiple tests, you can increase the likelihood on findings purely based on chance (meaningless)
If the researcher’s run a correlation analysis on hundreds of thousands of numbers, by chance they are bound to find a highly significant correlation
The more tests you run on a sample, the more likely you are to find false positives / high correlations by chance
What is a type 1 error?
Finding a false positive
Finding a significant result just by chance
How can type 1 error be reduced?
Choosing a subset of voxels to run the analysis on
Run less multiple tests
Outline what Vul et al. (2009) did
Conducted a literature review of papers reporting correlations between BOLD signal and trait measure
Found out the exact methodology and statistical analyses the authors used (non-independent analyses)
What did Vul et al. (2009) find?
High correlations in one paper (Sander et al., 2005) were the result of voxels selected on the basis of regression across subjects
What is the process of selecting voxels based on regression across subjects?
Separate correlations are run between each voxel activity and the trait measure for the individual
Select the voxels which have statistically significant correlations
Report the voxel with the highest correlation or report the average correlation across the subset of voxels
What is the name for the voxel which has the highest correlation?
Peak voxel
What is cluster analysis?
The average correlation across the subset of voxels is reported
What is the problem with selecting voxels based on regression across subjects?
The researchers are choosing the voxels on the basis of their correlation and then they are just restating this correlation
The researchers are reporting a correlation that they found across up to 500,000 separate analyses
What is the non-independence error?
The non-independence error refers to the fact that the analysis that is used to select the voxels is not independent from the analysis that is then presented as the result
How did Vul et al. (2009) illustrate the non-independence error?
Ran a simulation with 10 subjects, 10,000 voxels, and 1 individual difference measure
Made-up data / random numbers
Ran correlations between the made-up data for each of the voxels and the individual measure
Found high correlations in pure noise data
What does non-independent analysis mean?
Voxels are selected based on a functional analysis
Researchers then report the results of that same analysis and functional data from just the selected voxels
The analysis used to select the voxels is not independent from the final analysis that is reported as the result
Why are high correlations naturally found by chance in non-independent analyses?
Analysis uses the same data they used to pick the voxels in the first place
Increases the chance of high correlations (?)
Why is there a high probability of finding a false positive correlation in fMRI experiments?
A large number of voxel activations are yielded by fMRI experiments
How might the correlations from a non-independent analysis differ from an independent analysis?
Correlations will be stronger in a non-independent analysis
Correlations will be weaker or might not exist at all in independent analysis
Are results from non-independent analyses meaningful?
No, they are just found from chance. There is just as much chance that there isn’t a real effect going on at all. The real correlation might not actually exist if independent analysis was used.
What did Vul et al. (2009) conclude about emotion, personality, and social cognition fMRI research?
fMRI research uses defective research methods (53% from their literature review used non-independent analyses)
Produces numbers/results that should not be believed - personality research wants to publish exciting findings, even if they’re not true
Why is non-independent analyses likely to be worse in fMRI studies?
fMRI studies contain the most numbers to look at.
There are enormous matrices of thousands and thousands of numbers corresponding to activations in different voxels.
What might suggest use of non-independent analyses?
High/strong correlation - can say that the correlation might be spurious
What can you say about strong correlations if the details of choosing voxel subset are not given in the paper’s method section?
Might be spurious
Worried about the strength of the correlation between a personality measure and activation because it is possible that they came by it through non-independent means
Briefly outline the dead fish experiment
1 dead Salmon placed in an fMRI scanner
Shown a series of photographs of people and asked to determine what emotion the individual in the photo must have been experiencing
What was found from the dead salmon experiment regarding the BOLD signal?
There was a significant increase in the BOLD signal during the photo task compared to resting state
What was found from the dead salmon experiment regarding voxel activity?
Several active voxels were discovered in a cluster located within the salmon’s brain cavity
How many voxels out of 8064 were significantly active?
16
Was the dead salmon engaging in the perspective-taking task?
No
What can we conclude from the dead salmon experiment about multiple comparisons and chance findings?
Random noise in the EPI timeseries may yield spurious results if multiple comparisons are not controlled for
Absolutely by chance, we have found activation in the dead salmon’s brain while taking part in the task compared to not taking part in the task
What can we conclude from the dead salmon experiment regarding standard statistical thresholds and minimum cluster sizes?
Relying on statistical thresholds (e.g., p < .001) and low minimum cluster sizes (k > 8) is an ineffective control for multiple comparisons.
What should fMRI studies utilise as standard practice in the computation of their statistics?
Should utilise multiple comparisons correction
Why can’t we rely on less than 8 voxels in a cluster?
It is dangerous to call a region of activation a cluster if you have less than 8 voxels in the cluster
POINT THIS OUT IF fMRI papers report clusters where there are fewer than 8 voxels.
What is the problem with statistics from neuroimaging studies such as fMRI studies?
Neuroimaging studies sometimes use dodgy/dubious statistical procedures
Brief method sections which do not describe the methods or analysis used
What should we look at in fMRI studies?
Methodology
Number of voxels in a cluster
How did they choose their subset of voxels to look at?
Analysis
High correlations
What is the amygdala important for?
Fear recognition
Fear conditioning
What does the reward circuitry include?
NAcc
Ventral striatum
Insula
Prefrontal cortex
Are there different circuits for positive and negative affect?
No, there is overlap between circuits underlying negative and positive affect
