Multisensory Processing

Question 1

1. Superaddivity principle and where in the brain it is usually found

Answer 1

Superadditivity
• Response enhancement, that may exceed the sum of the unisensory processes
• Mostly found when the different component inputs are weak
• To ensure that weak stimuli are not missed  ideal to guide orienting

Found in the superior colliculus, but also rarely in the cortex

Question 2

2. Principle of inverse efficiency

Answer 2

 Superadditivity is increasingly likely as the salience of the components decreases

• Makes sense, because superadditivity is not needed if a stimulus already elicits a robust response in each modality

Question 3

3. Response depression

Answer 3

 Simultaneous stimuli with different spatial locations – if one falls within a unit’s receptive field and the other one adjacent to it, this will elicit a lower response than either component alone

Question 4

4. Traditional unimodal view, new multimodal view and main evidence to support the latter

Answer 4

Traditional view
• Each modality initially processes information independently
• Binding happens at later stage, in association areas
• Made early research feasible

Multimodal view
• Multisensory processing already happens much earlier, even in primary sensory cortices
o
• There are neuronal connections and areas that support this view
o Direct connections between early auditory and visual cortices
o Integration subcortically in the superior colliculus of visual, auditory and somatosensory information
• Still, there are of course ‘later’ cortical regions, like superior temporal sulcus and PFC that provide multisensory convergence

Multisensory processing examples in primary sensory areas
• V1 reacts to auditory stimulation or sensory (Braille reading)
• A1 processes somatosensory inputs and visual input (during lip reading)
• TMS disruption of V1 can impair performance on a tactile task

Sensory deprivation:
• V1 in blind people is activated by auditory or tactile tasks, A1 in deaf people by visual tasks
o This activation is functionally relevant, as disruption of the area leads to impairment
• Already effects for short-term deprivation (including reversal)
•  more likely to be associated with use of pre-existing inter-cortical connections than with rewiring of the brain

Question 5

5. Modality appropriateness hypothesis

Answer 5

= Vision dominates in spatial tasks, audition in temporal tasks (visual or auditory ‘capture’)

• However, capture is not absolute, but this is more a rule of thumb
o E.g. visual dominance can be reversed when visual signals are degraded (‘reverse ventriloquism’)
• Spatial factors: ventriloquist effect
o Spatial sensitivity is higher in the visual domain than in the auditory
• Temporal factors: auditory driving, e.g. of perceived frequency
o Particularly strong for high frequency, as visual resolution declines
o Expertise plays a role in asynchrony detection
• Occurrence of temporal synchrony plays an important role in guiding behaviour
o However, this is at least partly due to reduction of uncertainty

Question 6

6. MLE and its advantages

Answer 6

Maximum-likelihood estimation
• Based on Bayesian probability theory
• Sums up signals that are weighted by their reliability  result is the most reliable estimation
• Reliable means less variable in this case
• Combined estimate will always have a lower variance than either of the unimodal estimates

Advantages
• Model seems to be close to reality (sensory modalities are often integrated in a similar fashion closely resembled by the MLE model)
• Quantitative alternative to modality appropriateness hypothesis
• Flexible combination rule, that can also explain findings like reverse ventriloquism
• Explains findings that one domain never fully dominates, but there is always a residual contribution by the non-dominant modality

Question 7

7. Are attentional resources supramodal or modality specific?

Answer 7

Differing findings for both views
• Supramodal
o Vision appears to be dominant, with e.g. auditory stimuli getting lost in a flow of visual stimuli
o Neuroimaging support for supramodal parietal attentional system
• Modality specific
o Dual task experiments: little or no cost when doing two tasks cross-modally, but large costs when doing two tasks within one modality
o TMS disruption of right parietal cortex impairs visual but not somatosensory orienting

Hybrid models
• Evidence of independent processing is rarely absolute, there are always attentional links
• Task speed: more evidence for common resources in speeded tasks
• Task and stimulus level: More evidence for independent processing with simple stimuli, while higher level tasks hint more towards common supramodal resources

 No consensus yet