PSY280 - 11. Multisensory Processing Flashcards

1
Q

the unisensory approach

A
traditional approach
to perception was to
study the senses in
isolation
assigned areas as either/or areas
emerging evidence that combining senses has a profound effect on perceptual experience
not as dedicated as we thought
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

multisensory approach

A

studying how processing results in unique experiences unpredicted by sensory input
kicked off by single cell recordings in cats

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

Meredith & Stein (1983)

A

accidental discovery
recordings in superior colliculus of cat - important for recognizing unexpected visual events
reflexively driving attention to unexpected events
representation of space to guide attention

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

Meredith & Stein (1983)

A

Audiovisual signals presented at around the same time were super-additive
little activity with auditory
just visual, slightly bigger response
presented together, greater response
greatest activity in audiovisual - more than sum of its parts
more than addition of audio and visual - more often super-additive

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

Meredith & Stein (1983)

A

Audiovisual signals presented at different times resulted in response depression.
out of sync stimulus
strong response to V, small to A
offset of A just in front onset of V = depression
non-coincidence silences neuron

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

Meredith & Stein (1983)

A

superficial layers - vision only
deep layers - more
still fair to say it’s mostly a visual structure
created definition for multisensory integration - includes enhancement and depression
any change in responsivity that’s more than the sum of its parts

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

temporal order judgments

A

light or sound first
psychophysical curve
The average PSS for an auditory & visual signal is about +67ms.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

temporal order judgments

A

betw temporal order and experience
negative values mean sound comes first, positive values mean light comes first
calculate proportions after hundreds of trials
find the 50/50 point - SOA where experience events as simultaneous
temporal order objects - point of subjective simultaneity
to feel like they occured at same time, visual stimulus has to come one +67 ms before the sound comes on
due to speed of processing of vision and audition

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

apparent motion

A
Vision updates [ between 8 - 32x/sec
(Hz)
illusory motion
tracking changes over time
vision doesn’t check that often for changes
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

apparent motion

A

easily fooled about timing of events - every 31 1/2 ms
each gray area is an epoch of time - 31 ms
highly dependent on interstimulus interval
1. no aparrent motion - just simultaneous presentation
2. partial movement - overlapping onsets - still look distinct
3. illusory motion - in next epoch
4. flicker - skipping epoch: temporal interval perceived
between 55-65 ms - put in diff epochs

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

apparent motion

A

visual has good spatial resolution, but poor temporal resolution: t
• vernier acuity: 0.3333% of 1° VA • film = 48 frames per second
31 ms is super slow, but has good spatial resolution
where is accurate, but when is not good
VA: ability to resolve objects in space is .33333 of 1 VA
to fool visual system, only need 32 fps, 48 is perfect of motion pictures

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

audition

A

audition has poor spatial resolution, but good temporal resolution:
• sample rates of stereos: 44 100 samples/ second
• localization in space: 1° VA
to produce illusion of continuous sound - sample rate needs 44 thousand samples
knows gist of where you are

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

audiovisual

A

if it comes first, know when it starts + when it ends
when V presented first, blurs when V happens
window of possibility when V comes first
system slow and lazy assumes it’s presented sometime
visual system locks in V to A

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q

audition

A

at around same time + location, probably same event
vision has latitude in timing, audition doesn’t = comes first
to see it as simultaneour, V event needs to open up gate for integration earlier
V bang on in space, not A. Locks in to V within it’s area

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

Modality appropriateness

A

modality most appropriate for the task at hand dominates perception
perceptual response bias depends on time and space

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

ventriloquism effect

A

When you present a visual event and an auditory event at different locations simultaneously, the auditory event is perceived as emanating from the visual event.
in perception and psychophysics, we use low level V and A stimuli

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

visual cue, auditory target

A

task: report the location of the auditory target
at various time intervals - irrelevant
cue + target at same elevation - cued - faster response - 250 ms
cue + target at diff elevation - uncued - slower response - 260 ms
cuing helpful if positive effect size

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

visual cue, auditory target

A

When sound was hard to localize in space, people used the visual cue to determine the location.
presented pure tones and white noise bursts
task is about judging elevation, pure tones have no spectral shape

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

visual cue, auditory target

A

spectral shape cues change in composition based on how it bounces off pinna
when VS preceeded AS by 100ms - huge cuing effect for pure

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q

visual cue, auditory target

A

no effect at longer interval
only works within time window of integration + when it is hard to localize - influenced more by VS
faster responses in these conditions
easier to localize in white noise - ignore redundant visual cues - based it on AS

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

visual cue, auditory target

A

task: visual TOJ with redundant sounds at various SOAs
temporal ventriloquism
temporal order judgements - order 2 visual events
redundant auditory cues just before 1st VS and just after 2nd VS
visual system blurs, but for the sound, we know exactly when we heard it
judgements on timing - defer to auditory stimulus - timelock VS to sound
sounds on either side acted to pull them apart in time

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

just noticeable difference

A

JND - how far apart in time they have to be to notice diff
A smaller JND means
the 2 visual events could
be closer in time & still
reliably be judged for
order.
smaller JND = 2 visual events closer in time + can reliably tell which came first

23
Q

just noticeable difference

A
When sound
was hard to
localize in
space, people
used the visual
cue to determine
the location
When vision was hard to localize in time, people used the sound cue to determine the timing.
varied lag time
75 ms lag - JND decreases
when visual hard to localize in time, ppl use sound cues to determine temporal order
24
Q

just noticeable difference

A

pull blurry windows apart to make it more obvious which one happened first

25
the rules
• the spatial rule • the temporal rule • suppressive interactions • modality appropriateness • inverse effectiveness max when individual inputs are weakest
26
McGurk Effect
senses combined at early stage of processing top down understanding can’t overpower this bottom up perception can’t override it - bottom up
27
Language perception
is incredibly multisensory second language over the phone is hard, no visual cues Silent lip-reading activates the primary auditory cortex. despite lack of auditory stimulus
28
hypothesis
Multisensory integration will be greatest when vision and audition are equally weighted for speech recognition. when noise absent, auditory is used when noise present, visual cue use is high speech perception relies on both modalities at mid range multisensory integration greatest at x point
29
Is speech recognition subject to inverse effectiveness?
benefit for inverse effectiveness increases ability to detect stimuli - all about preparing to act on unexpected higher level semantic
30
condition: auditory vs only condition: audiovisual
auditory only - static mouth audiovisual - lip movement both at 50 dB, varying noise in environment addition of coarticulation in visual modality significantly improves performance
31
condition: auditory vs only condition: audiovisual
% diff performance greatest benefit to performance is at midlevel of noise enough of auditory present and helpful for filling in the gaps
32
with respect to speech perception
``` Maximal enhancement of auditory signals by a visual signal is at intermediate levels. ```
33
Based on this finding, what is one prediction about inverse effectiveness for bottom-up vs. top-down processing?
just basic detection bottom up - inverse effectiveness is at work engaging in recognition - inverse effectiveness might not be the rule it’s the mid range that gains most benefit McGurk effect is about meaningless stimuli should still see inverse effectiveness for McGurk effect - when signals are more degraded
34
top-down contributions
Top-down processing occurs any time a past experience or prior knowledge influences your interpretation of lower level sensory data what you know does dictate what you experience and also past experience
35
movement & sound
``` strong beats - louder, longer metrical structure - what we move to, derived from strong beats what we hear influences how we move ambiguous structure - no strong beats Metrical structure is derived from strong beats of a rhythm pattern. ```
36
movement & sound
heard unaccented music and experience of bouncing from mom gave infants head turning paradigm a week later - prefer duple or truple they picked what they experienced - listened more of the time to the beat that matched their proprioceptive experience a week earlier bounce to duple, preferred duple/bounce to truple, preferred truple
37
color & the chemical senses
Humans possess an excellent ability to detect & discriminate odors & flavors… but they are hard to verbalize. train themselves to link smell and taste to descriptors Humans possess an excellent ability to detect & discriminate odors & flavors... but they are hard to verbalize
38
color & the chemical senses
used white wine descriptors when they thought they tasted white wine used red wine descriptors when they thought they tasted red wine change experience by adding another modality could be response bias
39
response bias: How do people respond | when they’re unsure?
colour is just altering report, but not experience unsure what they taste, defer to vision not really about crossmodal
40
odor detection task
``` either with or without visual cues baseline: no vision, no odour visual: vision, no odour odour: just odour OVc: smell matches visual OVi: smell doesn’t match visual most likely to say yes in vision when unsure ```
41
odor detection task
vision more accurate than odour only accuracy for vision is same as baseline so no bias better odour if OVc, worse if OVi
42
odor detection
``` conscious experience of semantically congruent visual stimuli can facilitate odor detection top down because congruency matters based on semantic knowledge ```
43
color & the chemical senses
There are common associations that people have between odors & colors. IAT minimize response bias - force them to respond faster implicit association test asks participants to categorize two kinds of stimuli using the same buttons. more accurate and faster if compatible
44
color & the chemical senses
strawberry vs mint in odour condition pink or green in colour condition when buttons matched - compatible - faster and more accurate ppl are rapidly accessing colour-odour associations, not about response bias, actually robust automated impact on our perceptual experience, not just our response
45
vision (distance) & somatosensation
Distance is judged further when you’re carrying something heavy expectation set up by heavy backpack impacts visual perception
46
audition & gustation (hedonics)
``` Eating potato chips with an altered timbre changes the perception of freshness. gustatory hedonics chips tasted more stale when it didn’t sound crunchy ```
47
vision & flavor (discrimination)
Popcorn tastes saltier in a blue bowl compared to a white bowl. blue produces expectation of saltiness
48
somatosensory & flavor
``` Yogurt served in a heavy bowl was rated as more intensely flavored, denser, more expensive and was liked more. ```
49
multisensory brain areas
Researchers have found anatomical pathways between V1 & A1. integrated everywhere superior colliculus first subcortical area most are the primary sensory cortices - influenced by opposite input
50
inferotemporal cortex
``` contains object representations that are independent of modality increases in infero activation ```
51
intraparietal sulcus
hosts a convergence of multiple modality inputs. representing space in multiple senses due to spatial rule making sure that spatial rule is true
52
anterior cingulate cortex
involved in multisensory pain perception. looking at pictures, distract looking at wound + listening to crying - more painful
53
multisensory world
multisensory integration set to blur things film tracks have to be out by a lot for you to notice multisensory integration set to blur things film tracks have to be out by a lot for you to notice scary movies are scarier with the sound on
54
multisensory world
Eating is a multisensory experience on so many levels first you eat with your eyes eating is multisensory in terms of plates based on expectation if they can see it affects aroma, intensity