lecture 4 - multisensory integration Flashcards
definitions
-multisensory
-cross-modal
-integration
Multisensory: More than one modality is used in perception
Cross-modal: Interactions between different
modalities → one sense affects perceptions provided by a different sense
Integration: Merging information from different modalities into a unified percept
how is perception multisensory
Perception is multisensory in natural interactions with the environment
- Information from different senses can either be complementary or redundant/ overlapping
multisensory information ________ reliability of the percept and provides a more complete representation of the world
Multisensory information increases
reliability of the percept and provides a more complete representation of the world
(+ increases resistance to interference)
different modalities can provide convergent information about…
-what does this mean for the CNS
Different modalities can provide convergent information about the same external event/properties
- CNS has to disentangle cases where stimulation of different senses is unrelated and where it is related
simple heuristics for integration
(3) (how do we know information from 2 senses belong together)
1) Temporal correlation:
Stimulation of different modalities occurs at, (roughly) the same time.
2) Spatial congruency:
Stimuli in the different senses come from approximately the same location
3) Inverse effectiveness:
Reduced benefit of multisensory integration the stronger the unimodal signal of a cross-modal
cue is → Multisensory response is stronger when one stimulus by itself is quite weak
what is inverse effectiveness
Degree to which a multisensory response exceeds the response of the most effective modality specific
stimulus component declines as the effectiveness of the modality-specific stimulus component increases
inverse effectiveness on a neural level
-what is the point of this
example of one neuron and its reaction to a weak stimulus
Multi-modal neurons in Superior Colliculus (SC) (relevant for rapid orienting of attention):
-when neuron reacts to JUST visual input there is a weak respons, same as when it reacts to auditory input and has a weak response
- Spikes produced by combination of visual and auditory event (5) is larger than the individual
neural spikes in response to visual (1) and auditory stimuli (2) - Superadditivity of spike counts:
Multisensory response is greater than the sum of uni-sensory responses - Sum usually only larger for weak inputs (near threshold) → aids detection of weak stimuli
→ speeds up behavioural responses
inverse effectiveness
-photos of the dog , 1,2,3
1) dog far away from window, weak auditory stimuli, and periphery
2)
Superadditivity: Both cues are weak – response exceeds the sum of the separate inputs
- Additivity: As cues become stronger unisensory responses become stronger → integrated response is not different from the sum of the responses to each component
- Subadditivity: Combined input is smaller than the sum of the two uni-sensory inputs (but still exceeds the largest single input response)
neural mechanisms
subcortical areas : the superior colliculus
-where is it located
-how many multinsensory neurons
-do neurons overlap
Located in the mid-brain –
important for orienting
behaviour and fast motor
reactions
- High(est) proportion of
multisensory neurons
(extensively studied) - Neurons show overlapping
spatial maps for visual, auditory
and somatosensory modalities
neural mechanisms
-cortical areas
-how many multisensory neurons and where
- what did the studies of this area focus on
- Multisensory neurons are
found in most areas – often in combination with unimodal neurons
-Even in areas previously considered modality specific (e.g., neurons in visual cortex respond to tactile cues, and neurons in primary auditory cortex are activated by
visual lip movements
Studies in primates primarily focussed on posterior parietal cortex (converging information from visual, vestibular, tactile
and auditory system
cross modal integration
-how is input from two senses combined perceptually?
Different modalities are combined to yield the best estimate of the external properties
The modality that provides more reliable information is given more weight (greater reduction in uncertainty)
cross modal integration examples
→ e.g., vision strongly influences auditory localisation
(ventriloquist effect) – vision is spatially more accurate
→ audition can dominate vision in temporal properties,
e.g., auditory flutter drives perception of visual flicker (the two flash illusion
→ Modality appropriateness hypothesis
what is semantic congruency
Semantic congruency (consistent
meaning of two stimuli) Semantic congruency refers to the degree of similarity or compatibility between two pieces of information, such as words or stimuli, based on their meaning or semantic content.
semantic congruency _______ multinsensory stimulus integration and…
Semantic congruency (consistent
meaning of two stimuli) strengthens
multisensory stimulus integration and corresponding behavioural
performance
semantic congruency and its affect on speed of participants responses
Semantic congruency of visual and
auditory stimuli affects the speed of
participants responses
→ faster target detections when
visual stimulus is accompanied by a
semantically congruent sound
two visuals of- teleophone and frog
two sounds of teleophone and frog
-did a combination- press a button whenever you see or hear the telephone
-if you only see the teleophone (visual stimulus) you have a reaction time of jus under 5 ms, and if you only hear it there is similar reaction time
-incogruent : see the teleophone and hear the frogor vice versa (slow reaction time)
-
crossmodal illusions
-what is the parchment-skin Illusion
-complementary information improves the reliability of our perception
-sound modifies tactile sensations
-enhanced high frequency feedback makes the skin feel drier
-temporal coincidence required
incongruent information can result in unexpected precepts due to _______ _______
sensory interactions eg audio-visual illusion of the MCGurk effect)
-you listen to auditory stimulus ‘ba’, have a visual stimulys/actor pronouncing ‘ga’
-usually the perception is a fusion response (da / tha)
explain the parchment skin illusion in depth
exp
-relationship between auditory information and skin perception
-sound modifies tactile sensations
deprivation of one modality can modify the development and integration of remaining modalities
-enhanced high frequency feedback makes the skin feel drier
-temporal coincidence required
parchment skin illusion in blind people
Less susceptible to the illusion – ability to ignore irrelevant auditory input in the tactile task
parchment skin illusion ,
vision, audition and touch
study on blind people
-what does this show
-study
-control
-10 early onset blind participants
-had to rate their tactile perception on scale -5 to +5
-if high frequencies accentuated (makes it feel rougher), works well in sighted participants
Robust illusion in sighted humans
* 7 of the early blind participants were not/ minimally susceptible to the illusion
* Only 3 of the early blind participants showed small effects in the expected direction
- Multisensory perception might not be innate but is – at least to some extent - based on experiences during early
development → following visual deprivation extensive cross-modal changes occur (re-organisation of perceptual system)
Cross modal plasticity in the Cerebral cortex
study by hamilton et al 2000
-what does this case suggest
Case of blind woman (born blind) who lost ability to read Braille (code) following bilateral occipital lesions (usually processing vision) following stroke in primary visual cortex (processes vision area) but didn’t loose any other tactile sensitivity , just specific to reading braille
- Occipital cortex involved in decoding spatial and tactile information for Braille reading -
- Suggests that there may be a critical period of susceptibility for the recruitment of the occipital cortex for haptic information processing (in congenitally blind)
Cross-modal Plasticity in the Cerebral Cortex
TMS – Study
Cohen et al., (1997),
TMS briefly disrupts the electric activation patterns of the neurons in the cortical area it is applied to. short bursts of magnetic stimulation applied to certain areas of the brain
- Blind Braille readers and sighted participants (blindfolded) who had to identify embossed Roman letters
- Occipital stimulation (visual area) disrupted Braille reading in blind participants but not tactile discrimination in sighted participants (but disrupted their visual performance)
- Visual cortex recruited for somatosensory processing in early blind
Cross-modal Plasticity after Sensory Deprivation mechanism
-1 Reorganisation in areas associated with the deprived modality
if you don’t have vision, the occipital cortex seems to reorganise process
Primary sensory areas are able to process information from remaining modalities
- Sensory inputs shape the functional architecture of the brain
- Reorganisation likely to be limited to early-onset (sensitive period- when brain is still elastic)
- Caution:
Difficulty to clearly distinct between
primary brain areas and neighbouring areas (usually multi-modal) – small spatial resolution of TMS, PET and MRI it might not be clear where the reorganisation in primary areas occurs
Cross-modal Plasticity after Sensory Deprivation mechanism
2. Reorganisation of multi-modal
areas in the cortex:
Behavioural compensation for missing modality is mediated by enhanced
recruitment of multi-modal areas
- Reorganisation in multi-modal areas also for late-onset deprivations
- Example: Enhanced recruitment of
posterior STS (area of multi-modal
integration) in deaf individuals when attending to moving visual displays
perceptual distinction between one’s own body and the environment is critical for….
what does body ownership include
Perceptual distinction between one’s own body and the
environment is critical for our survival and part of human
conscious experience
- Body Ownership includes feeling the skin stretching around
joints and digits, feeling the coolness/ warmness on the skin,
feeling the tension from muscles and tendons etc. (all combined
is body ownership a single sense
Sense of Body Ownership is multisensory in nature (cannot be
reduced to a single modality) – not really a “sense” but a complex/multisensory perception
what is the rubber hand illusion
- 3 way interaction between vision, touch & proprioception
-participant has one hand in front and one hand behind the occlude. put a rubber hand in place of the real hand so it looks as if participants hand is there
-eventually after touching the rubber hand with the same objects etc as the real hand behind the occluder, participant starts to thinkit’s their own hand
how is the rubber hand illusion objectively quantified
- Questionnaires and Rating scales
- Proprioceptive Drift (reported hand
location changes) - Physiological responses (skin
conductance) in response to
perceived threat to the rubber hand
perceptual rules of body ownership
-temporal synchrony
-spatial rules
- Temporal synchrony: if visual and tactile stimulation are mismatched illusion disappears (~ 300-500 ms)
- Spatial rules: e.g., distance between real hand and rubber hand (peri-personal space as constraint), identical direction of
visual and tactile strokes, matching
orientation & postures of hands
multinsensory integration is the key mechanism in perceiving _____ ______
body ownership
Perceptual Rules are similar to principles of multisensory integration
- Sense of body ownership is governed by the same principles as multisensory perception
→ multisensory integration is the key mechanism in perceiving body ownership
what is peripersonal space (spatial rules rubber hand illusion)
-when does the illusion strength decreases
Space immediately surrounding our bodies in which objects can be grasped and manipulated
Illusion strength
decreases at distances > ~30 cm (peripersonal space)
Perceptual Rules of Body Ownership (RHI) cont
-Tactile Congruence Rule:
- Humanoid Shape Rule
Tactile Congruence Rule: Tools that touch real and rubber hand must be similar in texture and geometric features (subtle incongruences possible)
- Humanoid Shape Rule: Rubber hand must resemble a human hand in shape and structure (colour and material less critical) → e.g., ownership observed for realistic prosthetic hands or even images
of human hands
Congruent pattern of ______ _______ ________drive the perceptual phenomenon
Congruent pattern of multi-sensory signals drive the perceptual phenomenon!
Multisensory integration of body signals- where does this happen
there are Cortex areas specifically dedicated to multisensory integration of body related
signals in peripersonal space
meta analysis : areas in ventral premotor cortex and intraparietal sulcus that respond with greater activation for combined/congruent visuotactile stimulation than for unimodal visual or haptic stimulation or
incongruent stimulation
Certain areas in your brain get super excited when you see and feel something at the same time, like a toy and a high-five together. They respond more strongly to this combined experience than to just seeing or feeling something alone, or when the two don’t match. Your brain loves it when things happen together
Recent studies on limb ownership:
* Consistent activation in area EBA
of the ventral stream (processes body postures etc)
generally body ownership is associated with activation in ________________ areas in ______ and ______ lobes
Generally, body ownership is associated with activation in multisensory areas in frontal and parietal lobes
multisnesory integration : neuroimaging
graphs of intraparietal and premotor areas
-results
graphs show response of these areas to stimulation of the rubber hand illusion
-people in scanner, real hand is occluded, rubber hand in front of them
Manipulation of temporal congruency (synchrony) and spatial congruency (hand orientation)
Largest activation when stimulation was both spatially and temporally congruent
* Activation in intraparietal cortex is additive
* Response in the pre-motor cortex is super additive
entire body is perceived as ___ ____
Entire body is perceived as one’s own (not just the
stimulated parts) → requires multisensory perceptual binding
does the RH illusion work on the rest of the bdoy
illusion adheres to the same
perceptual rules as the RH-illusion
- Illusion seems to relate to similar
activation of brain patters as the
RH-Illusion (increased activity in
ventral premotor cortex, intraparietal cortex and LOC) - Activation in ventral premotor
cortex correlates with the strength
of the illusion
activation in _______ ______ cortex correlates with the strength of the illusion
Activation in ventral premotor
cortex correlates with the strength
of the illusion
-individual differences in illusions
Degree of illusory experience varies
(+about 30% of population are immune to the induction of the RH-Illusion)
- Factors influencing individual differences are largely unknown
- Multi-sensory account predicts that immunity should relate to how visual, tactile and proprioceptive information is weighted by the brain
- People that rely more on proprioceptive information (e.g., dancers, gymnasts etc.) maybe more resistant to the illusion
potential clinical applications
Projection of ownership to advanced hand prostheses: Indication that synchronised
brushing of participant’s stump and the fingers of a prosthetic hand produces RH-Illusion in some (~ 30%) amputees
- Projection of ownership to simulated bodies in VR: People can maintain ownership of a virtual
hand as long as its movements are temporally and spatially congruent with movements of real hand (allow paralysed people to “own a virtual
limb” in virtual and mixed reality applications)
