Week 8 RF-The Role of Hand Gestures During Audiovisual Speech Perception

Question 1

True or false: Auditory speech is often enough to convey the message

Answer 1

True! Although visual information e.g., head/eye movements, gestures and facial movements helps to understand better.

Question 2

Is Speech multisensory?

Answer 2

Yes! Speech is often multisensory during social interactions (i.e., perceive speech through multisensory channels)

• Verbal information: voice
• Non-verbal information:
• Face: lip movements, eyebrows
• Head and shoulders nods
• Hand gestures

Question 3

Can listeners use rapid hand gestures to detect important parts of the speech content?

Answer 3

• Speech and gestures temporally align very well
• Common language system (McNeill, 2000; Iverson & Goldin-Meadow, 1998) ilustrated by the use that blind people use gestures even if never learnt too
• Continuum of movements
• Different categories depending on speech dependence e.g., emblems (related to culture and saying related to hand shape e.g., thumbs up), iconic (describing an action), metaphoric (needs contex provided with speech) and beats (most simplest gesture usually seen in political addresses).

Question 4

What are BEAT GESTURES AND PROSODY?

Answer 4

Beats are the visual versant of the speaker’s prosody through kinematics: direction, velocity, etc,

Extension – Apex – Retraction:
* Rapid flick of the hands; No semantic content in their shape
* Synchronize with pitch accentuations

Question 5

In which ways such as rapid and meaningless gestures, can influence speech perception?

Answer 5

• Beat gestures align with the stressed syllable of target words
• Listeners are experts in matching gestures with verbal speech
• (Yasinnik, Renwick & Shattuck-Hufnagel, 2004): apex is the most important part of the gestures

1 – Beat gestures increase the salience of words perceived by the listeners (Treffner, Peter & Kleidon; 2008): asked which word is emphasised which shifts dependent on the position of the gesture.

So, Sim & Low (2012):
2 – Beat gestures facilitate accompanied words encoding and memory.
* 3-s videoclips presenting isolated verbs with a related Iconic gesture, a Beat or no Gesture.
* Later, participants were asked to recall as many words as possible.
* Compared memory performance between adults and children (i.e., whether this is dependant on age).

Question 6

What were the findings of So, Sim & Low’s (2012) study?

Answer 6

• People recalled more words encoded
  with an iconic gesture than alone (expected as redundant info explaining semantic content was provided).
• Beat gestures only improved memory
  in adults but not children (children only benefitted from iconic gestures).
• Beats convey no semantic content
  therefore, attributing value requires
  concepts that are not yet developed
  in children < 4-5 years (i.e., relates to language development).

Question 7

What was Biau et al’s (in prep) study

Answer 7

• We tested the effect of beat gestures on continuous and spontaneous speech perception.
• 20-s segments of Q&A sessions at the House of Commons.
• 2 conditions: Audiovisual speech containing beat gestures vs. Auditory
  speech only.
• Subsequent memory test on words recognition (Old/New) to compare performances between words encoded with a beat gesture or alone.
• No effect found in the auditory condition BUT more words recalled and better memory when words were presented with gestures.

Question 8

Is the influence of speakers’ beat gestures on speech perception reflected in the brain activity during speech perception? Biau et al. (2013); Biau et al. (2015); Biau et al. (2017)

Answer 8

• The integration of co-speech gesture elicited a positive waveform in the ERP time-locked to word onsets.
• Modulation of early word processing stage corresponding to multisensory integration mechanisms (N1/P2 component).
• The effect peaked in the left centro-frontal areas related to speech perception.

Question 9

What is the Audiovisual speech synchrony detection Task - EEG?

Answer 9

• 10-s segments of an actor engaged in an interview, spontaneously gesturing and talking to a journalist.
• 2 Conditions: Intact natural synchrony between body gestures and speech sounds (synchronous), or subtle delay between the gestures and their corresponding word onsets (360ms; asynchronous).

*Semantic content was removed by filtering the auditory track i.e., muffled.

• Audiovisual synchrony detection task to test how listeners match gestures and sounds: Correct responses and reaction times (recorded using EEGs).

Question 10

What were the results of the Audiovisual speech synchrony detection Task - EEG? (Biau, Schultz, Gunter & Kotz, 2022)

Answer 10

• Participants correctly detected both synchronous and asynchronous stimuli > comparable correct response rates and reaction times.
• In contrast, the brain activation was increased in response to asynchronous audiovisual speech compared to synchronous speech.
• Suggesting a compensatory mechanism to a greater difficulty to integrate auditory and visual information together, created greater brain activation to perceive speech.

Question 11

Do beat gestures carry communicative values that are interpreted differently from simple body movements by the brain?

Answer 11

• Audiovisual speech presentation - fMRI
• Segments taken from a public addressee of a former Spanish prime minister.
• 2 Conditions: Natural speech or the hand trajectories were replaced by moving dots.
• 2 levels of audiovisual synchrony: Intact natural synchrony between gestures/dots and sounds (synchronous). Or subtle delay between the gestures/dots and the corresponding word onsets (360ms; asynchronous)

Question 12

What were the findings of the Interaction Movement (Beat vs. Dot) x Synchrony (Sync vs. Async) study? (Biau et al., 2016)

Answer 12

• The brain responded differently to audiovisual speech asynchrony, depending
  on whether movements were carried by speaker’s hands (beats) or dots.
• Greater activations in areas important for multisensory integration, speech and
  prosody as well as visual perception e.g., left medio temporal gyrus/left inferior-frontal gyrus.
• Suggesting that the brain might be sensitive to social values and interpret beat
  as markers of speakers’ communicative intentions, respect to trivial visual cues.

Question 13

What are the overall conclusions?

Answer 13

• Listeners are expert in mapping beat gestures with their target speech segments during perception.
• Although rapid flicks without any semantic content, beat gestures influence multiple aspects of speech processing, which are reflected both by behavioural and neural
  measures.
• The social and communicative value of beat gestures seems to require higher-order capacities acquired through language development.