Cross-modal processing Flashcards
1
Q
what is modular processing?
A
- Fodor (1983) proposed that cognition is a collection of independent information-processing modules.
- Although this has been a very influential theory and there are many virtues, e.g. different areas in the brain are specialised for different tasks to optimise speed and efficiency, there is evidence that modules (1) can change in specificity, (2) may be correlated in some tasks and (3) can communicate with each other.
2
Q
what modules can change?
A
- Ferrets in which retinal projections are redirected neonatally to the auditory thalamus have visually responsive cells in auditory thalamus and cortex, form a retinotopic map in auditory cortex and have visual receptive field properties in auditory cortex that are typical of cells in visual cortex.
- von Melchner et al. (2000) reported that this cross-modal projection in the auditory cortex can mediate visual behaviour: When light stimuli are presented in the portion of the visual field that is ‘seen’ only by this projection, ‘rewired’ ferrets respond as though they perceive the stimuli to be visual rather than auditory.
3
Q
can modules be correlated?
A
- One example of this is in speech processing where visual and audio information are processed together.
- In a noisy environment, understanding someone’s speech can be facilitated by reading their lip movement. In fact, Spence (2002) demonstrated that visual cues can improve speech perception by up to 15-20dB.
- This facilitation is possible because visual and auditory cues are normally correlated. So a particular sound will be accompanied by a specific mouth shape. If the sound is unclear, you can rely on the visual, and vice versa.
4
Q
how do modules communicate with each other?
A
- Emergence highlights the brain’s ability to create new precepts by deeply integrating sensory inputs, fundamental for complex tasks like navigating environments or understanding social cues.
5
Q
what is the McGurk effect?
A
- When audio stimulus is /b/, but the visual stimulus is /g/, the actual perception becomes one of /d/. When the participants shut their eyes, they can hear the /b/, but when they see the incongruent /g/ lip movement, they hear /d/.
- This demonstrates that neither visual or audio takes precedence and both streams of information work together to allow a new coherent percept.
6
Q
what is cross modal cueing?
A
- Two pathways or information channels is better than one.
- When participants are required to detect a faint visual target on the left or right of fixation, if they are also given an additional auditory or tactile cue on the side of the stimuli, reaction times are faster (Butter et el., 1989; Spence et al., 2009).
7
Q
what did McDonald et al. 2000 study?
A
- McDonald et al. (2000) asked participants to detect a weak visual stimulus that was presented to the left or right of a fixation point.
- The visual target was preceded by an audio cue, either on the same side of the visual target (valid) or on the other side (invalid).
- McDonald et al. (2000) measured two variables:
- Low-level: Perceptual detectability measured as d’
- High-level: Decision criterion, called bias or β - The found that d’ was significantly higher when audio was presented on the same side, but there were inconsistent effects on bias β.
- The results indicated that the cross-modal cueing does involve some modulation of relatively low-level sensory processing, but there was no evidence of high-level involvement in this study.
8
Q
what is dominance?
A
- This relates to which sense is prioritised when there is conflicting information. Accordingly, which type of sensory dominance (visual or auditory) emerges may be dependent on the conditions with which the stimuli are presented.
9
Q
what is the colavita effect?
A
- Colavita (1974) presented suprathreshold auditory and visual targets to participants and asked them to press one button for auditory targets and another button for visual targets.
- On some trials, both the light and sound were presented at the same time. These trials were interleaved into the sequence of unimodal target trials.
- The results found that most participants perceived the visual on the light-sound trials. Some reported being unaware that the sound component. This demonstrates visual dominance over auditory stimuli.
10
Q
what is the modality appropriateness hypothesis?
A
- Certain properties of stimuli are better processed by a specific sense, leading to a particular modality dominating.
- Visual modality is superior at processing spatial information (e.g. Posner et al., 1976).
- Auditory modality is superior for temporal information (e.g. Shams, Kamitani, & Shimojo,2002).
11
Q
what is the biased/integrated competition hypothesis?
A
- Spence, Parise, and Chen (2012) suggest that brain systems dedicated to the different sensory systems may compete. They suggest that given that a large proportion of the brain is dedicated to visual processing (Sereno et al.,1995), visual stimuli and the visual system should be more likely to dominate and inhibit processing in other sensory systems.
12
Q
what is visual dominance?
A
- This is when the apparent location of a sensory stimulus can be influenced by visual information.
- During a ventriloquist’s act the mouth of the puppet does not correspond to the location of the sound source, yet observers perceive the puppet’s speech come from its mouth.
- Rubber hand illusion (Botvinick & Cohen, 1998): When participants placed an arm underneath a table and a false arm was stroked at the same time as a real arm and participants perceived strokes from the brush on an artificial arm and even felt that was their own.
13
Q
what is the sound-induced flash illusion?
A
- Shams et al., (2000) presented a visual illusion that is induced by sound. When a single flash is accompanied by two bursts of noise, you may perceive two flashes instead of one.
- Shams et al., (2000) showed how a visual stimulus can be altered by another modality even when the visual stimulus is not ambiguous. This demonstrates that although vision is usually very dominant, like other modalities, it is also malleable by other senses.
14
Q
how does one sense enhance another?
A
- Smell and colour: The strawberry smell of a liquid appears stronger when the liquid is coloured red (Sellner & Kautz, 1990).
- Sound and light: Auditory noise presented with light tends to be perceived as louder than noise presented alone (Odgaard et al, 2004).
- Light and touch: Tactile discrimination thresholds are lower during visual observation (Kennett et al., 2001).
- Sound and touch: Perturbing the sound made as hands are rubbed together can alter the perception of the skin’s texture (Jousmaki & Hari, 1998; Guest et al., 2002).
15
Q
what is the neural basis to multimodal processing?
A
- Convergence zones in the brain have been suggested for multimodal processing (see Driver & Noesselt, 2008). These are higher level areas, such as the superior temporal sulcus, the parietal cortex, the prefrontal cortex, and the subcortical nuclei.
- Some neurons in the subcortical nuclei show superadditive responses to multimodal stimulation, where stimulation from multiple senses is greater than the sum of response from the difference senses in isolation (Alvarado et al, 2007).
- Some researchers have also found multimodal responses in previously considered modality-specific areas, such as the visual cortex and auditory cortex.
- There is increased evidence that signals can travel laterally from one unimodal area to another, but there is also some downward travel from the higher order multimodal/polysensory cortex (STP).
16
Q
what is the Bouba-Kiki effect?
A
- Köhler(1929) showed two forms and asked readers which shape was called “takete” and which was called “maluma”. More recently, Ramachandranand Hubbard (2001) repeated the experiment with the words “kiki” and “bouba”, and asked American college undergraduates andTamilspeakers in India, “Which of these shapes is bouba and which is kiki?” In both groups, 95% to 98% selected the curvy shape as “bouba” and the jagged one as “kiki”, suggesting that the human brain somehow attaches abstract meanings to the shapes and sounds consistently.
- Ćwiek et al. (2022) found the bouba/kiki effect is across-cultural phenomenon where 917 participants speaking 25 different languages maintain a consistency in bouba/kiki identification, regardless of their native language.
17
Q
what is synaesthesia?
A
- This is the multisensory phenomenon in which stimulation of one sense causes sensory experience in another. People who experience this are calledsynaesthetes.
- One of the most common synaesthesia is the colour-grapheme association where words/letters/numbers have particular colours.
18
Q
what did Simner 2019 find?
A
- Simner (2019) reported 128 different types of synaesthesias and the number of people with synaesthesia amounted to just 4.4 per cent of the population.
- Synaesthesia is a multi-variant condition:
- Lexical-gustatory synaesthesia: Words have different tastes
- Sound-colour synaesthesia: Sounds have different colours
- Flavour-colour synaesthesia: Tastes have different colours
- Touch-colour synaesthesia: Textures have different colours
- Visual-auditory synaesthesia: Silent moving objects produce sounds
19
Q
what are the four important features of synaesthesia?
A
- Four important features:
- The experiences are elicited by particular stimuli that would not evoke such experiences in most members of the population. The triggering stimuli could be perceptual or conceptual.- The experiences are automatic and are extremely difficult to suppress.
- The nature of the synaesthetic experience itself is a conscious perceptual event.
- The synaesthetic experience is consistent over time, although this is still debated.
20
Q
what are the bottom up factors?
A
- Witthoft and Winawer (2006) showed that for some grapheme-colour synesthetes, altering the fonts and cases of letters can affect the saturation of synaesthetic colours. In their study, the colour category of the synaesthetic colours was not strongly affected by these manipulations, only the saturation.
21
Q
what did Dixon et al. 2006 study?
A
- presented a case study with a synaesthete called J.
- presented J with ambiguous graphemes (e.g., a grapheme that could either be interpreted as ‘S’ or ‘5’) in biasing contexts (e.g., 34_67 or MU_IC), and followed this by a Stroop-like task.
- This was followed this by a Stroop-like task where J had to name the coloured of the target. The colour of the target could match the synaesthetic colour for number 5 (pink) or for the letter S (green).
- found that the colour of the ambiguous grapheme was determined by the context.
- J was faster at naming pink when they saw
- J was faster at naming green when they saw
- The researchers suggested that it is the identity of a grapheme (top-down feature), and not just its physical appearance (bottom-up feature), that determines synaesthetic colour.
22
Q
what is the correlation between culture and synaesthesia?
A
- Colour-grapheme synaesthesia is common in aural-oral society and visual-literate one.
- Howes (2006) suggested that culture organises the senses which descend via the psychological to the physiological level of the brain. They suggested that drawing up an inventory of the range of cultural practices and technologies can help us understand different sensory combinations and types of synaesthesias across different cultures and historical periods.
23
Q
what is said about people hearing smells?
A
- Stevenson and Boakes (2004)’s claim that “Odor display taste properties but do not elicit auditory or visual sensations” may be true for western societies but there is evidence of audio-olfactory synasthesia in other cultures.
- For example, people in Papua-New Guinea and Trobriand Islands talk about hearing smells.
- This audio-olfactory synasthesia may have arisen because in these locations, most communication takes place face-to-face (i.e. within olfactory range of the other) and odoriferous substances (e.g. anointing the body with oil, chewing ginger) are regularly used to augment the power of a person’s presence and words (Howes, 2003).
24
Q
what is said about the Dogon culture and hearing smells?
A
- Hearing-smell associations are also common in various African languages, such as Dogon. To those who speak Dogon, speech has material properties that despite its invisible nature are more than just sounds. It has an odour because sound and odour both arise from vibrations (Calame-Griaule, 1986).
- Dogon also classify words by smell: good speech smells sweet, and bad or impetuous speech smells rotten. A Dogon expression states that “the mouth is too ready to speak is likened to the rectum”.
- Consistent with the intrinsic connection between sound and odour: The nose rings are believed to have the super-additive function of modulating both her hearing and her speech, by (1) promoting the reception and utterances of “good-smelling words” and by (2) deflecting or repressing of the bad-smelling ones.
25
Q
what did Wassily Kandinsky study?
A
- Kandinsky believed his art could evoke the sound of music and used the terms “improvisations” and “compositions” to describe his paintings.Composition VII—with its swirling vortex of colour and symbols—was often described as “operatic.”
- Kandinsky’s use of circles, grids, semicircles, triangles, and other mathematical forms reflect his belief that shape and colour alone could communicate emotion and sounds.
- According to Kandinsky, black signified silence while orange indicated the alto range. Yellow, forming a halo around the blue and red circles, represents explosions of sounds, such as loud trumpets and fanfares.
