PSY280 - 11. Multisensory Processing Flashcards
the unisensory approach
traditional approach to perception was to study the senses in isolation assigned areas as either/or areas emerging evidence that combining senses has a profound effect on perceptual experience not as dedicated as we thought
multisensory approach
studying how processing results in unique experiences unpredicted by sensory input
kicked off by single cell recordings in cats
Meredith & Stein (1983)
accidental discovery
recordings in superior colliculus of cat - important for recognizing unexpected visual events
reflexively driving attention to unexpected events
representation of space to guide attention
Meredith & Stein (1983)
Audiovisual signals presented at around the same time were super-additive
little activity with auditory
just visual, slightly bigger response
presented together, greater response
greatest activity in audiovisual - more than sum of its parts
more than addition of audio and visual - more often super-additive
Meredith & Stein (1983)
Audiovisual signals presented at different times resulted in response depression.
out of sync stimulus
strong response to V, small to A
offset of A just in front onset of V = depression
non-coincidence silences neuron
Meredith & Stein (1983)
superficial layers - vision only
deep layers - more
still fair to say it’s mostly a visual structure
created definition for multisensory integration - includes enhancement and depression
any change in responsivity that’s more than the sum of its parts
temporal order judgments
light or sound first
psychophysical curve
The average PSS for an auditory & visual signal is about +67ms.
temporal order judgments
betw temporal order and experience
negative values mean sound comes first, positive values mean light comes first
calculate proportions after hundreds of trials
find the 50/50 point - SOA where experience events as simultaneous
temporal order objects - point of subjective simultaneity
to feel like they occured at same time, visual stimulus has to come one +67 ms before the sound comes on
due to speed of processing of vision and audition
apparent motion
Vision updates [ between 8 - 32x/sec (Hz) illusory motion tracking changes over time vision doesn’t check that often for changes
apparent motion
easily fooled about timing of events - every 31 1/2 ms
each gray area is an epoch of time - 31 ms
highly dependent on interstimulus interval
1. no aparrent motion - just simultaneous presentation
2. partial movement - overlapping onsets - still look distinct
3. illusory motion - in next epoch
4. flicker - skipping epoch: temporal interval perceived
between 55-65 ms - put in diff epochs
apparent motion
visual has good spatial resolution, but poor temporal resolution: t
• vernier acuity: 0.3333% of 1° VA • film = 48 frames per second
31 ms is super slow, but has good spatial resolution
where is accurate, but when is not good
VA: ability to resolve objects in space is .33333 of 1 VA
to fool visual system, only need 32 fps, 48 is perfect of motion pictures
audition
audition has poor spatial resolution, but good temporal resolution:
• sample rates of stereos: 44 100 samples/ second
• localization in space: 1° VA
to produce illusion of continuous sound - sample rate needs 44 thousand samples
knows gist of where you are
audiovisual
if it comes first, know when it starts + when it ends
when V presented first, blurs when V happens
window of possibility when V comes first
system slow and lazy assumes it’s presented sometime
visual system locks in V to A
audition
at around same time + location, probably same event
vision has latitude in timing, audition doesn’t = comes first
to see it as simultaneour, V event needs to open up gate for integration earlier
V bang on in space, not A. Locks in to V within it’s area
Modality appropriateness
modality most appropriate for the task at hand dominates perception
perceptual response bias depends on time and space
ventriloquism effect
When you present a visual event and an auditory event at different locations simultaneously, the auditory event is perceived as emanating from the visual event.
in perception and psychophysics, we use low level V and A stimuli
visual cue, auditory target
task: report the location of the auditory target
at various time intervals - irrelevant
cue + target at same elevation - cued - faster response - 250 ms
cue + target at diff elevation - uncued - slower response - 260 ms
cuing helpful if positive effect size
visual cue, auditory target
When sound was hard to localize in space, people used the visual cue to determine the location.
presented pure tones and white noise bursts
task is about judging elevation, pure tones have no spectral shape
visual cue, auditory target
spectral shape cues change in composition based on how it bounces off pinna
when VS preceeded AS by 100ms - huge cuing effect for pure
visual cue, auditory target
no effect at longer interval
only works within time window of integration + when it is hard to localize - influenced more by VS
faster responses in these conditions
easier to localize in white noise - ignore redundant visual cues - based it on AS
visual cue, auditory target
task: visual TOJ with redundant sounds at various SOAs
temporal ventriloquism
temporal order judgements - order 2 visual events
redundant auditory cues just before 1st VS and just after 2nd VS
visual system blurs, but for the sound, we know exactly when we heard it
judgements on timing - defer to auditory stimulus - timelock VS to sound
sounds on either side acted to pull them apart in time
just noticeable difference
JND - how far apart in time they have to be to notice diff
A smaller JND means
the 2 visual events could
be closer in time & still
reliably be judged for
order.
smaller JND = 2 visual events closer in time + can reliably tell which came first
just noticeable difference
When sound was hard to localize in space, people used the visual cue to determine the location When vision was hard to localize in time, people used the sound cue to determine the timing. varied lag time 75 ms lag - JND decreases when visual hard to localize in time, ppl use sound cues to determine temporal order
just noticeable difference
pull blurry windows apart to make it more obvious which one happened first
the rules
• the spatial rule
• the temporal rule
• suppressive interactions • modality appropriateness • inverse effectiveness
max when individual inputs are weakest
McGurk Effect
senses combined at early stage of processing
top down understanding can’t overpower this bottom up perception
can’t override it - bottom up
Language perception
is incredibly multisensory
second language over the phone is hard, no visual cues
Silent lip-reading activates the primary auditory cortex.
despite lack of auditory stimulus
hypothesis
Multisensory integration will be greatest when vision and audition are equally weighted for speech recognition.
when noise absent, auditory is used
when noise present, visual cue use is high
speech perception relies on both modalities at mid range
multisensory integration greatest at x point
Is speech recognition subject to inverse effectiveness?
benefit for inverse effectiveness
increases ability to detect stimuli - all about preparing to act on unexpected
higher level semantic
condition: auditory vs only condition: audiovisual
auditory only - static mouth
audiovisual - lip movement
both at 50 dB, varying noise in environment
addition of coarticulation in visual modality significantly improves performance
condition: auditory vs only condition: audiovisual
% diff performance
greatest benefit to performance is at midlevel of noise
enough of auditory present and helpful for filling in the gaps
with respect to speech perception
Maximal enhancement of auditory signals by a visual signal is at intermediate levels.
Based on this finding, what is one prediction about inverse effectiveness for bottom-up vs. top-down processing?
just basic detection bottom up - inverse effectiveness is at work
engaging in recognition - inverse effectiveness might not be the rule
it’s the mid range that gains most benefit
McGurk effect is about meaningless stimuli
should still see inverse effectiveness for McGurk effect - when signals are more degraded
top-down contributions
Top-down processing occurs any time a past experience or prior knowledge influences your interpretation of lower level sensory data
what you know does dictate what you experience and also past experience
movement & sound
strong beats - louder, longer metrical structure - what we move to, derived from strong beats what we hear influences how we move ambiguous structure - no strong beats Metrical structure is derived from strong beats of a rhythm pattern.
movement & sound
heard unaccented music and experience of bouncing from mom
gave infants head turning paradigm a week later - prefer duple or truple
they picked what they experienced - listened more of the time to the beat that matched their proprioceptive experience a week earlier
bounce to duple, preferred duple/bounce to truple, preferred truple
color & the chemical senses
Humans possess an excellent ability to
detect & discriminate odors & flavors…
but they are hard to verbalize.
train themselves to link smell and taste to descriptors
Humans possess an excellent ability to detect & discriminate odors & flavors… but they are hard to verbalize
color & the chemical senses
used white wine descriptors when they thought they tasted white wine
used red wine descriptors when they thought they tasted red wine
change experience by adding another modality
could be response bias
response bias: How do people respond
when they’re unsure?
colour is just altering report, but not experience
unsure what they taste, defer to vision
not really about crossmodal
odor detection task
either with or without visual cues baseline: no vision, no odour visual: vision, no odour odour: just odour OVc: smell matches visual OVi: smell doesn’t match visual most likely to say yes in vision when unsure
odor detection task
vision more accurate than odour only
accuracy for vision is same as baseline so no bias
better odour if OVc, worse if OVi
odor detection
conscious experience of semantically congruent visual stimuli can facilitate odor detection top down because congruency matters based on semantic knowledge
color & the chemical senses
There are common associations that people
have between odors & colors.
IAT minimize response bias - force them to respond faster
implicit association test asks participants to categorize two kinds of stimuli using the same buttons.
more accurate and faster if compatible
color & the chemical senses
strawberry vs mint in odour condition
pink or green in colour condition
when buttons matched - compatible - faster and more accurate
ppl are rapidly accessing colour-odour associations, not about response bias, actually robust
automated impact on our perceptual experience, not just our response
vision (distance) & somatosensation
Distance is judged further when you’re carrying something heavy
expectation set up by heavy backpack impacts visual perception
audition & gustation (hedonics)
Eating potato chips with an altered timbre changes the perception of freshness. gustatory hedonics chips tasted more stale when it didn’t sound crunchy
vision & flavor (discrimination)
Popcorn tastes saltier in
a blue bowl compared
to a white bowl.
blue produces expectation of saltiness
somatosensory & flavor
Yogurt served in a heavy bowl was rated as more intensely flavored, denser, more expensive and was liked more.
multisensory brain areas
Researchers have
found anatomical
pathways between
V1 & A1.
integrated everywhere
superior colliculus first subcortical area
most are the primary sensory cortices - influenced by opposite input
inferotemporal cortex
contains object representations that are independent of modality increases in infero activation
intraparietal sulcus
hosts a convergence of multiple modality inputs.
representing space in multiple senses due to spatial rule
making sure that spatial rule is true
anterior cingulate cortex
involved in multisensory pain perception.
looking at pictures, distract
looking at wound + listening to crying - more painful
multisensory world
multisensory integration set to blur things
film tracks have to be out by a lot for you to notice
multisensory integration set to blur things
film tracks have to be out by a lot for you to notice
scary movies are scarier with the sound on
multisensory world
Eating is a multisensory experience on so many levels
first you eat with your eyes
eating is multisensory in terms of plates
based on expectation
if they can see it affects aroma, intensity
