Multisensory Integration

Question 1

Why is it important to integrate across the senses accurately?

Answer 1

Integrating visual and auditory signals is important to understand speech (particularly in noisy environments, hearing impaired people, unfamiliar languages)

Question 2

What is the McGurk Effect

Answer 2

We integrate mismatched information of lip movements and sound, which changes perception in either sense.
The effect is stronger when the stimuli appear at approximately the same time, but there is some temporal tolerance.
The effect is stronger when noise is added to auditory components - this is because the effect of the multi-sensory integration is greater when the unimodal stimulus is weaker.
Effect is not reduced by gender differences - suggesting early stage of phonetic processing

Question 3

Why is it important to integrate across the senses correctly?

Answer 3

Interactions between the senses direct our attention
- sound can direct visual attention to a specific location
- a movement can help localise the source of sound
- a tactile sensation can direct visual attention

Question 4

Multisensory effects on detection: crossmodal cueing

Answer 4

An irrelevant auditory ‘cue’ can be valid or invalid:
If auditory cues can affect visual detection, the irrelevant cue being on the same side as the target will create a faster response and less errors.

Question 5

Ventriloquist effect:

Answer 5

vision affects our auditory localisation –> assumption that the sound is coming from the moving lips

Temporal synchrony is crucial:
the light flash occuring shortly after the sound results in audio-visual spatial integration

this does not integrate if it occurs 1 second later

Question 6

Mechanisms of multisensory processing:

Answer 6

initial processing in sensory specific (unimodal) areas - and then subsequent processing by multisensory ‘convergence zones’ such as the SUPERIOR COLLICULUS

Question 7

Provide an example of the multisensory convergence in the superior colliculus

Answer 7

Single cell recording:
- a neuron responds to light in a certain part of the visual field, as well as when the monkey was touched on a certain part of the face
- but it fires MORE during SIMULTANEOUS stimulation (superadditive response)

Question 8

What is the principle of inverse effectiveness?

Answer 8

Effectiveness of unisensory stimuli (inverse effectiveness): multisensory interactions are more likely or stronger when the relevant unisensory stimuli evoke relatively weak responses by themselves

example is how adding white noise to auditory signals boosts the mcgurk effect

therefore: as stimulus effectiveness increases, the benefit from spatiotemporal synchronous V+A decreases

Question 9

What’s the ‘but’ in the standard view of multisensory processing

Answer 9

Standard view: inital processing in unimodal sensory cortex, subsequent processing in multisensory convergence zones (i.e. feedforward process)

BUT
- evidence for multimodal responses in primary sensory areas (feedback and lateral connections)

Seen in Anatomical tracing studies: its a highly interactive network that integrates information from the senses for conscious perception

Question 10

What is temporal synchrony?

Answer 10

Multisensory interactions are stronger when the constituent stimuli arise at the same time

Degree of temporal tolerance: difference in propagation and processing time for different sense

The temporal binding window is the epoch of time within which multisensory interactions are more likely

Question 11

Spatial correspondence

Answer 11

Spatial correspondence: multisensory interactions are more likely/stronger when the unisensory stimuli are from the same location
There is spatial tolerance: we can integrate information even when it isn’t coming form the same location –> ventriloquist effect and rubber hand illusion