PSY280 - 11. Multisensory Processing

Question 1

the unisensory approach

Answer 1

traditional approach
to perception was to
study the senses in
isolation
assigned areas as either/or areas
emerging evidence that combining senses has a profound effect on perceptual experience
not as dedicated as we thought

Question 2

multisensory approach

Answer 2

studying how processing results in unique experiences unpredicted by sensory input
kicked off by single cell recordings in cats

Question 3

Meredith & Stein (1983)

Answer 3

accidental discovery
recordings in superior colliculus of cat - important for recognizing unexpected visual events
reflexively driving attention to unexpected events
representation of space to guide attention

Question 4

Meredith & Stein (1983)

Answer 4

Audiovisual signals presented at around the same time were super-additive
little activity with auditory
just visual, slightly bigger response
presented together, greater response
greatest activity in audiovisual - more than sum of its parts
more than addition of audio and visual - more often super-additive

Question 5

Meredith & Stein (1983)

Answer 5

Audiovisual signals presented at different times resulted in response depression.
out of sync stimulus
strong response to V, small to A
offset of A just in front onset of V = depression
non-coincidence silences neuron

Question 6

Meredith & Stein (1983)

Answer 6

superficial layers - vision only
deep layers - more
still fair to say it’s mostly a visual structure
created definition for multisensory integration - includes enhancement and depression
any change in responsivity that’s more than the sum of its parts

Question 7

temporal order judgments

Answer 7

light or sound first
psychophysical curve
The average PSS for an auditory & visual signal is about +67ms.

Question 8

temporal order judgments

Answer 8

betw temporal order and experience
negative values mean sound comes first, positive values mean light comes first
calculate proportions after hundreds of trials
find the 50/50 point - SOA where experience events as simultaneous
temporal order objects - point of subjective simultaneity
to feel like they occured at same time, visual stimulus has to come one +67 ms before the sound comes on
due to speed of processing of vision and audition

Question 9

apparent motion

Answer 9

Vision updates [ between 8 - 32x/sec
(Hz)
illusory motion
tracking changes over time
vision doesn’t check that often for changes

Question 10

apparent motion

Answer 10

easily fooled about timing of events - every 31 1/2 ms
each gray area is an epoch of time - 31 ms
highly dependent on interstimulus interval
1. no aparrent motion - just simultaneous presentation
2. partial movement - overlapping onsets - still look distinct
3. illusory motion - in next epoch
4. flicker - skipping epoch: temporal interval perceived
between 55-65 ms - put in diff epochs

Question 11

apparent motion

Answer 11

visual has good spatial resolution, but poor temporal resolution: t
• vernier acuity: 0.3333% of 1° VA • film = 48 frames per second
31 ms is super slow, but has good spatial resolution
where is accurate, but when is not good
VA: ability to resolve objects in space is .33333 of 1 VA
to fool visual system, only need 32 fps, 48 is perfect of motion pictures

Question 12

audition

Answer 12

audition has poor spatial resolution, but good temporal resolution:
• sample rates of stereos: 44 100 samples/ second
• localization in space: 1° VA
to produce illusion of continuous sound - sample rate needs 44 thousand samples
knows gist of where you are

Question 13

audiovisual

Answer 13

if it comes first, know when it starts + when it ends
when V presented first, blurs when V happens
window of possibility when V comes first
system slow and lazy assumes it’s presented sometime
visual system locks in V to A

Question 14

audition

Answer 14

at around same time + location, probably same event
vision has latitude in timing, audition doesn’t = comes first
to see it as simultaneour, V event needs to open up gate for integration earlier
V bang on in space, not A. Locks in to V within it’s area

Question 15

Modality appropriateness

Answer 15

modality most appropriate for the task at hand dominates perception
perceptual response bias depends on time and space

Question 16

ventriloquism effect

Answer 16

When you present a visual event and an auditory event at different locations simultaneously, the auditory event is perceived as emanating from the visual event.
in perception and psychophysics, we use low level V and A stimuli

Question 17

visual cue, auditory target

Answer 17

task: report the location of the auditory target
at various time intervals - irrelevant
cue + target at same elevation - cued - faster response - 250 ms
cue + target at diff elevation - uncued - slower response - 260 ms
cuing helpful if positive effect size

Question 18

visual cue, auditory target

Answer 18

When sound was hard to localize in space, people used the visual cue to determine the location.
presented pure tones and white noise bursts
task is about judging elevation, pure tones have no spectral shape

Question 19

visual cue, auditory target

Answer 19

spectral shape cues change in composition based on how it bounces off pinna
when VS preceeded AS by 100ms - huge cuing effect for pure

Question 20

visual cue, auditory target

Answer 20

no effect at longer interval
only works within time window of integration + when it is hard to localize - influenced more by VS
faster responses in these conditions
easier to localize in white noise - ignore redundant visual cues - based it on AS

Question 21

visual cue, auditory target

Answer 21

task: visual TOJ with redundant sounds at various SOAs
temporal ventriloquism
temporal order judgements - order 2 visual events
redundant auditory cues just before 1st VS and just after 2nd VS
visual system blurs, but for the sound, we know exactly when we heard it
judgements on timing - defer to auditory stimulus - timelock VS to sound
sounds on either side acted to pull them apart in time

Question 22

just noticeable difference

Answer 22

JND - how far apart in time they have to be to notice diff
A smaller JND means
the 2 visual events could
be closer in time & still
reliably be judged for
order.
smaller JND = 2 visual events closer in time + can reliably tell which came first

Question 23

just noticeable difference

Answer 23

When sound
was hard to
localize in
space, people
used the visual
cue to determine
the location
When vision was hard to localize in time, people used the sound cue to determine the timing.
varied lag time
75 ms lag - JND decreases
when visual hard to localize in time, ppl use sound cues to determine temporal order

Question 24

just noticeable difference

Answer 24

pull blurry windows apart to make it more obvious which one happened first

Question 25

the rules

Answer 25

• the spatial rule
• the temporal rule
• suppressive interactions • modality appropriateness • inverse effectiveness
max when individual inputs are weakest

Question 26

McGurk Effect

Answer 26

senses combined at early stage of processing
top down understanding can’t overpower this bottom up perception
can’t override it - bottom up

Question 27

Language perception

Answer 27

is incredibly multisensory
second language over the phone is hard, no visual cues
Silent lip-reading activates the primary auditory cortex.
despite lack of auditory stimulus

Question 28

hypothesis

Answer 28

Multisensory integration will be greatest when vision and audition are equally weighted for speech recognition.
when noise absent, auditory is used
when noise present, visual cue use is high
speech perception relies on both modalities at mid range
multisensory integration greatest at x point

Question 29

Is speech recognition subject to inverse effectiveness?

Answer 29

benefit for inverse effectiveness
increases ability to detect stimuli - all about preparing to act on unexpected
higher level semantic

Question 30

condition: auditory vs only condition: audiovisual

Answer 30

auditory only - static mouth
audiovisual - lip movement
both at 50 dB, varying noise in environment
addition of coarticulation in visual modality significantly improves performance

Question 31

condition: auditory vs only condition: audiovisual

Answer 31

% diff performance
greatest benefit to performance is at midlevel of noise
enough of auditory present and helpful for filling in the gaps

Question 32

with respect to speech perception

Answer 32

Maximal
enhancement
of auditory signals by
a visual signal is at
intermediate
levels.

Question 33

Based on this finding, what is one prediction about inverse effectiveness for bottom-up vs. top-down processing?

Answer 33

just basic detection bottom up - inverse effectiveness is at work
engaging in recognition - inverse effectiveness might not be the rule
it’s the mid range that gains most benefit
McGurk effect is about meaningless stimuli
should still see inverse effectiveness for McGurk effect - when signals are more degraded

Question 34

top-down contributions

Answer 34

Top-down processing occurs any time a past experience or prior knowledge influences your interpretation of lower level sensory data
what you know does dictate what you experience and also past experience

Question 35

movement & sound

Answer 35

strong beats - louder, longer
metrical structure - what we move to, derived from strong beats
what we hear influences how we move
ambiguous structure - no strong beats
Metrical structure is derived from
strong beats of a rhythm pattern.

Question 36

movement & sound

Answer 36

heard unaccented music and experience of bouncing from mom
gave infants head turning paradigm a week later - prefer duple or truple
they picked what they experienced - listened more of the time to the beat that matched their proprioceptive experience a week earlier
bounce to duple, preferred duple/bounce to truple, preferred truple

Question 37

color & the chemical senses

Answer 37

Humans possess an excellent ability to
detect & discriminate odors & flavors…
but they are hard to verbalize.
train themselves to link smell and taste to descriptors
Humans possess an excellent ability to detect & discriminate odors & flavors… but they are hard to verbalize

Question 38

color & the chemical senses

Answer 38

used white wine descriptors when they thought they tasted white wine
used red wine descriptors when they thought they tasted red wine
change experience by adding another modality
could be response bias

Question 39

response bias: How do people respond

when they’re unsure?

Answer 39

colour is just altering report, but not experience
unsure what they taste, defer to vision
not really about crossmodal

Question 40

odor detection task

Answer 40

either with or without visual cues
baseline: no vision, no odour
visual: vision, no odour
odour: just odour
OVc: smell matches visual
OVi: smell doesn’t match visual
most likely to say yes in vision when unsure

Question 41

odor detection task

Answer 41

vision more accurate than odour only
accuracy for vision is same as baseline so no bias
better odour if OVc, worse if OVi

Question 42

odor detection

Answer 42

conscious experience of
semantically congruent
visual stimuli can facilitate odor
detection
top down because congruency matters based on semantic knowledge

Question 43

color & the chemical senses

Answer 43

There are common associations that people
have between odors & colors.
IAT minimize response bias - force them to respond faster
implicit association test asks participants to categorize two kinds of stimuli using the same buttons.
more accurate and faster if compatible

Question 44

color & the chemical senses

Answer 44

strawberry vs mint in odour condition
pink or green in colour condition
when buttons matched - compatible - faster and more accurate
ppl are rapidly accessing colour-odour associations, not about response bias, actually robust
automated impact on our perceptual experience, not just our response

Question 45

vision (distance) & somatosensation

Answer 45

Distance is judged further when you’re carrying something heavy
expectation set up by heavy backpack impacts visual perception

Question 46

audition & gustation (hedonics)

Answer 46

Eating potato chips
with an altered
timbre changes
the perception of
freshness.
gustatory hedonics
chips tasted more stale when it didn’t sound crunchy

Question 47

vision & flavor (discrimination)

Answer 47

Popcorn tastes saltier in
a blue bowl compared
to a white bowl.
blue produces expectation of saltiness

Question 48

somatosensory & flavor

Answer 48

Yogurt served in a heavy
bowl was rated as more
intensely flavored,
denser, more expensive and
was liked more.

Question 49

multisensory brain areas

Answer 49

Researchers have
found anatomical
pathways between
V1 & A1.
integrated everywhere
superior colliculus first subcortical area
most are the primary sensory cortices - influenced by opposite input

Question 50

inferotemporal cortex

Answer 50

contains object
representations that are
independent of
modality
increases in infero activation

Question 51

intraparietal sulcus

Answer 51

hosts a convergence of multiple modality inputs.
representing space in multiple senses due to spatial rule
making sure that spatial rule is true

Question 52

anterior cingulate cortex

Answer 52

involved in multisensory pain perception.
looking at pictures, distract
looking at wound + listening to crying - more painful

Question 53

multisensory world

Answer 53

multisensory integration set to blur things
film tracks have to be out by a lot for you to notice
multisensory integration set to blur things
film tracks have to be out by a lot for you to notice
scary movies are scarier with the sound on

Question 54

multisensory world

Answer 54

Eating is a multisensory experience on so many levels
first you eat with your eyes
eating is multisensory in terms of plates
based on expectation
if they can see it affects aroma, intensity