Multi-sensory integration

Question 1

What does a sensory system consist of?

Answer 1

• Sensory receptors
• Neural pathways
• Brain (sensory) areas

Question 2

What is sensory processing?

Answer 2

We have an input which is a physical energy that is then transferred to the brain through encoding procedures and then we have an output – a representation of the environment/our body or perhaps a motor action to interact with the external environment)

Sensory Input –> Processing –> Output/Response

Question 3

Where does sensory processing occur?

Answer 3

Different brain areas selectively respond to sensory stimuli.
Primary and Secondary (Associative) Sensory Processing occurs in the cortex.

Question 4

Where are the different cortices located?

Answer 4

In the occipital lobe we have the primary visual cortex (V1)
In the parietal lobe there is the primary somatosensory cortex (involved in touch and pain)
In the temporal lobe is the auditory cortex

Question 5

What sort of stimuli do neurons respond to?

Answer 5

In order to have the cortex respond to the stimulus, the stimulus needs to be optimal, neurons only respond to stimuli that are appropriate)

Question 6

What is sensory bombarding and how does our brain provide a unified representation?

Answer 6

Our brain is constantly “bombarded” by sensory information from different senses

The brain combines relevant information to provide a unified representation of the object, the external environment, and our own body.

Question 7

What is multisensory integration?

Answer 7

Combining information from different senses

Most events in life require our brain to combine different sensory information.

Question 8

What does multisensory integration require?

Answer 8

Multisensory integration requires a temporal and spatial congruency – events need to happen pretty much at the same time, and they need to be close in space. So, spatial and temporal factors are key for multisensory integration.

Question 9

Why is the temporal factor key for multisensory integration?

Answer 9

Multisensory integration is more likely to occur the closer that the stimuli in different modalities are presented in time.

Question 10

Why is the spatial factor key for multisensory integration?

Answer 10

Spatial coincidence facilitates multisensory integration. (e.g., if vision and sound stimuli come from nearby, then it’s much easier for them to be integrated – if they were far apart there is no point in them being integrated because they probably didn’t belong to the same event.)

Question 11

How does multisensory integration work?

Answer 11

Multisensory integration is assessed by considering the effectiveness of a cross-modal stimulus combination, in relation to that of its component stimuli, for evoking some type of response from the organism.
(the response to an event that has both visual and auditory components is compared with that for the visual and the auditory stimuli alone)

Question 12

Explain, in detail, how does multisensory integration work?

Answer 12

• Normally we have a unimodal neuron responding to a sensory signal e.g., visual unimodal neurons respond to visual information by producing AP’s and increasing the number of spikes which shows excitation.
• In multisensory integration there is a different type of neuron that responds to more than one signal. – We look at the activity of the neuron responding to one modality, then the other modality and finally when it responds to both modalities at the same time. The activity is much higher when the neuron responds to the two modalities at the same time.
• The cross-modal combination (two modalities at the same time) of stimuli produces a response that is much bigger than the response evoked by each of the sensory modalities separately.

Question 13

How is multisensory integration operationally defined?

Answer 13

Multisensory integration is defined operationally as: a statistically significant difference between the number of impulses evoked by a cross-modal combination of stimuli and the number evoked by the most effective of these stimuli individually.
(There is a difference in excitability from the cross-modal condition compared to the single modal condition.)

Question 14

What is the difference between unimodal and bimodal neurons?

Answer 14

Unimodal neurons respond to one modality; therefore, they won’t process anything related to a possible second sensory modality.
The bimodal neuron responds to both modalities if they are presented independently but they also respond to both modalities happening at the same time by integrating the signal and shaping the response.

Question 15

What does the superior colliculus play a role in?

Answer 15

SC plays a role in the motor control of orientation behaviours of the eyes, ears and head.
It has a high proportion of multisensory neurons
The SC has a high proportion of bimodal neurons responding to touch and vision, and sound and vision.

Question 16

Where is the superior colliculus located?

Answer 16

The superior colliculus (SC) is located in the midbrain, superior to the brainstem and inferior to the thalamus

Question 17

Explain the structure of the superior colliculus?

Answer 17

The superior colliculus contains seven layers of alternating white and grey matter:

• superficial layers: topographic maps of the visual field (have a full representation of the visual field, which is well organised in function of where the stimuli are on the visual field)
• deeper layers contain spatial maps of visual, auditory and somatosensory modalities

We can make a distinction between the layers and see that the superficial ones encode information about visual modality and in particular where the stimuli are in the external environment. The deeper layers also have representation of touch and sound.

Question 18

What are the consequences of SC ablation?

A brain lesion or stroke in the SC can be an ablation

Answer 18

• disturbances in visual attention (visual neglect)
• disturbances in orientation behaviours (« where » system)
• loss of the ability to respond appropriately to contralateral (to the lesion) touch and auditory stimuli
• bilaterally symmetrical lesions are far less disruptive to visual auditory and somatosensory behaviour than unilateral ones
• As well as the visual domain, there are impairments in tactile localisation and auditory processing. In patients that have had a SC ablation or a stroke in that area, there is more than one sensory modality involved.

Question 19

What are the key behavioural functions of the superior colliculus?

Answer 19

1) Integrating sensory modalities:
• associates the sensory inputs to redirect the organs the input originates from
• in order to localize (capture-avoid) the source of the stimulus
(receiving information and integrating them to create perfect representation)

2) Sensorimotor transduction:
• Transforms incoming sensory inputs into motor commands
• By virtue of the convergence and intermixing of sensory inputs and motor circuits (different sensory modalities access to the same output motor circuits)
• Many neurons have sensory and motor properties and are involved in a variety of circuits and functions
(gets the signal ready for motor command)

3) Realization:
• Alignment of the different sensory and motor maps (receptive fields and movement fields in register)
• Sometimes with use of multisensory neurons which can have also a premotor role
(create a multisensory representation of the external environment)

Question 20

What are the 4 principles of multisensory integration (in superior colliculus neurons)?

Answer 20

Multisensory enhancement
Spatial congruency
Temporal congruency
Inverse effectiveness

Question 21

What is multisensory enhancement?

Answer 21

• Multisensory enhancement is an increased likelihood of detecting and/or initiating a response to the source of the signal
• The largest enhancements are due to super additive combinations of cross-modal influences

Question 22

Explain what is meant by a superadditive combination?

Answer 22

If we have a bimodal neuron only receiving vision which induces a weak response. The same neurone receiving the auditory modality also gives a weak response.
However, if we have the sound and visual event happening at the same time, there is a huge response. The neuron is responding in a super additive way (the combination is much more than the sum of the single events).

Question 23

Why is Spatio/Temporal congruency a principle of multisensory integration?

Answer 23

If they are to be integrated, different sensory stimuli must also be linked in space and time

Question 24

Explain the steps of a visual-auditory bimodal neuron responding to an event

Answer 24

• The neuron is going to respond when the visual event happens in the visual receptive field.
• It will also respond when there’s a sound in the auditory receptive field.
• If the event is where the two fields are overlapping there will be a huge increase in the response – the neuron will integrate the two signals and recognise it comes from the same event.
• Similarly, if stimuli occur at the same time in the same space the neuron will respond. Integration takes place in a certain time window – around 100ms. If the stimuli are farther apart in time, the brain won’t integrate them, and the bimodal neuron won’t respond.

Question 25

Explain the factor of inverse effectiveness on multisensory integration

Answer 25

• Multisensory enhancement is inversely related to the effectiveness of the individual cues that are being combined
• Individual cues that are highly salient will be easily detected and localized as no need to be integrated!
• Weak cues evoke few neural impulses, and their responses are subject to substantial enhancement when stimuli are combined.

Weak stimuli are more likely to be integrated, (think about the example with the lady and her cat responding to the dog) – when the dog was far away, we saw the maximum multisensory enhancement because the unimodal stimuli were weak.

Why? Because if the stimulus is already loud, and visible there is no need for it to be integrated as it’s already a strong powerful stimulus that is processed. – Already able to evoke a response.

Question 26

What is the intensity of a multisensory response in comparison to the individual sensory responses?

Answer 26

Multisensory response can exceed the arithmetic sum of the individual sensory responses

Question 27

What are the key areas for multisensory integration in humans?

Answer 27

Key areas for multisensory integration in humans are the posterior parietal cortex, posterior superior temporal sulcus, inferior pre-frontal cortex and the premotor cortex (a little bit).

They integrate different types of signals – e.g., the posterior parietal cortex does a lot of different sensory integration.

Question 28

What is sensory illusion and give an example?

Answer 28

When sound and vision are combined, because they happen at the same time.
These illusions are based on the assumption that, detecting something by different senses at the same time, must have been caused by the same event (we implicitly combine sound and vision without even being aware of it).

E.g., The ventriloquist effect

Question 29

How does the ventriloquist effect happen?

Answer 29

• Auditory location perception is shifted toward a visual cue.
• When the visual cue is unambiguous, visual capture reliably occurs. (if the stimuli is weak and you can’t see the man speaking multisensory integration can happen, but if the stimuli are very clear this won’t happen – inverse effectiveness)
• In this case you are relying on the visual input to localise the sound.
• Time is important! the lips of the puppet have to be moving at the same time as the sound.
• Higher level cognitive effects will also play a role!

Question 30

What is Mandatory fusion?

Answer 30

Mandatory Fusion (or optimal fusion) is when in some cases, we combine percept’s regardless of information required.

E.g., the McGurk effect
(When presented with an audio track playing “pa” and a separate video track of a person silently articulating “ka” viewers will most often claim to hear a unique fusion sound “ta”)