Week 7 Flashcards
Speech in Noise
Vision helps to resolve ambiguity Few “lip readers” but visual speech greatly improves perception in noise Automatic Greater than sum of parts
Warning Signals
Faster response to multimodal stimuli Flashing indicator Beeping indicator Flash/Beep more reliable faster
Flavour
“Flavour is in the brain” Taste Smell Somatosensory Modulators Sight Sounds Smells Expectation
A Unified Perceptual Experience
“… it is surely one of the great remaining scientific puzzles just
how it is that signals from such completely separated and
wholly dissimilar sensory epithelia as the haircells of the
cochlea, the photoreceptors of the retina and the corpuscles
of the skin can be integrated centrally to form such a seamless
unitary perceptual world”.
Molholm and Foxe, 2010 (p 1709)
Overview - Perception
A primary role of the brain • Determine what is “out there” in the world • Decide on best/most appropriate action/behaviour • Food to approach? • Danger to avoid? • Path to navigate? • Potential mate to interact with? Survival depends on speed and accuracy with which an organism can evaluate external events and properly react to them To determine what is “out there” need to • Gather information • Interpret that information • Represent the information
Multisensory Integration
Overview • Perception • Multisensory Perception • Example - Redundant Targets Effect Factors influencing integration • Example - Stream-Bounce Effect and Top-Down Factors Neuroscience of multisensory integration • Superior Colliculus • Superior Temporal Sulcus • Example – McGurk Effect • Posterior Parietal Cortex • Example – Flash/Beep Effect
Multisensory Perception
Objects and events in the world generate information in multiple ways light sound mechanical chemical
Multisensory Perception
… evaluate external events and properly react
to them
Many info sources so many senses good
• Different modalities can substitute when individually
compromised (eg vision lost in the dark)
• Different fields of operation – touch/smell/taste for up close and
vision/hearing for distance
• Resolve ambiguities – 2 things may sound the same but look
different; boost signal to noise
Multisensory Integration
Multisensory integration enhances our ability to perceive
and understand our environment, enabling us to better
interact with our surroundings
• Combine info from multiple cues to improve stimulus
detection and discrimination – increase speed and
accuracy (e.g. Audio-visual warning)
• Resolve perceptual ambiguities (e.g. Stream-Bounce)
• Create novel representations (e.g. Flavour)
Redundant Targets Effect
• Miller (1982) • Speeded response to either audio (A), visual (V), or audio-visual (AV) target Redundant since additional stimulus doesn’t provide any additional information 2 Models 1. Statistical facilitation - independent (parallel) processing 2. Neural coactivation - integrated signals
Redundant Targets Effect
Statistical Facilitation
• Both elements of AV stim processed along
independent channels
• One that reaches output stage first triggers
response
• On average, the time of the winner will be less than
the time for either racer
Redundant Targets Effect
Neural Coactivation
• Both components of a redundant signal influence
response on a single trial
• Activation from different channels combine in
satisfying a single criterion for response initiation -
activation builds over time until some criterion is
reached
• Activation builds faster when provided by two
sources rather than one
Redundant Targets Effect process
Visual signal processed + Auditory signal processed -> decision-> response
Visual + Auditory -> audio-visual signal processed -> decision-> response
Redundant Targets Effect cont.
• responses to redundant signals are too fast to be
explained as the faster of two responses to
individual signals
• The easiest way to explain the speed of responses
to redundant signals is to assume that signals
jointly contribute to the process of producing the
response
• Neural Coactivation - MSI
MSI – Key Issue
• Sensory environment is complex – there are
multiple sources in each modality – eg lots of visual
objects and lots of sounds
• Key function of MSI dissociate between stimuli
from different sources and single source
• Integration
• what to bind
• what to segregate
Factors Influencing MSI
- Temporal Coincidence
- Spatial Coincidence
- Temporal patterning
- Crossmodal correspondence
- Stored knowledge
- Context
- Recent experience
- Expectation
- Attention
Factors Influencing MSI- bottom-up processing: objective
- Temporal Coincidence
- Spatial Coincidence
- Temporal patterning
Factors Influencing MSI- top-down: subjective
- Stored knowledge
- Context
- Recent experience
- Expectation
- Attention
Bottom-up Factors
• Temporal Coincidence – things that happen at the same
time
• Spatial Coincidence – things that happen at the same
place
• Temporal patterning – things that are correlated over
time
Strong cues that stimuli were caused by the same event
and so belong together
Top-down Factors
Information that is already present in our brain
influences how crossmodal signals are combined
• Stored knowledge
• Recent experience
• Context
• Expectation
• Attention
Top-down: Context
Stream-bounce; 3 blocks of 200 trials each Zeljko and Grove Block 1 – all no sound Block 2 – mixed sound/no sound Block 3 – all no sound
Top-down: Expectation
When are cognitive influences (eg previous response, overall presence of the tone) impacting Does expectation lead to an early decision or bias? Typical S/B – observe entire motion sequence then provide response – subjective Want to see how percept evolves Record cursor position at each frame -1 target indicated to follow -No Sound or Sound at coincidence -Target tracked until motion stop Subjective- No Sound or Sound Objective- Stream or Bounce
Neuroscience of MSI
• Minimum 32 visual, 15 auditory, 8 somatosensory areas
identified in primate cortex
• Body control - proprioception, vestibular, vision, motor
control
• How are individual senses integrated?
• How is a unified perceptual experience created?
Multisensory neurons are
found at nearly every level
in the CNS
Multimodal Brain Areas
intraparietal sulcus
superior temporal sulcus
superior colliculus
inferior colliculus