Language 2

Question 1

processing spoken words overview

Answer 1

• Properties of spoken language
• Major elements of speech recognition:
  
  • Segmentation- speech is continuous so need to extract units of information e.g. words from this
  • Lexical selection- once we have isolated units we have to map the meaning/representation from mental dictionary
  • Access to meaning- want to understand the meaning of units
  • Context effects- all of this mechanism shows you how contexts can facilitate different meanings
    Speech processing in the brain

Question 2

speech processing

Answer 2

• A process of progressively extracting invariant, discrete representations from a variable, continuous input…
• Speech signal challenges:
  
  • Continuous. Distributed in time. Fast-fading Words are not neatly segmented (e.g., by pauses). Consecutive speech sounds blend into each other due to mechanical constraints on articulators.
    Variable Speaker differences; pitch affected by age and sex; different accents, talking speeds, often heard in noise.

Question 3

word segmentation (Cutler & Norris, 1988)

Answer 3

• The rhythmic structure of English is stress- timed (some syllables are emphasised)
• Lettuce Trousers Cigar (underlined = stressed syllable)
  Metrical Segmentation Strategy (MSS): In English, stressed (strong) syllables are likely onsets of words. Continuous speech is segmented at stressed syllables.

Question 4

syllable stress and word types

Answer 4

• Syllables
  
  • Stressed syllables- Full vowels (lettuce)
  • Unstressed syllables- Reduced vowels (behind)
• Words
  
  • Content words- Nouns, verbs, adjectives
  • Grammatical words- Articles, pronouns, prepositions, conjunctions
    Evidence in favour of MSS: the 74% of stressed syllables in English corresponds to sole or initial syllable of a content word. This is not the case for unstressed syllables - only 5% corresponds to content words (Cutler & Carter, 1987).

Question 5

metrical segmentation strategy

Answer 5

• It is not infallible – it is a strategy ( * examples of incorrectly segmented words with complex syllabic structure). Listeners need other source of information to segment successfully.
  
  • Alert *lert
  • Assassinate *sassinate
• MSS is language specific. Other languages may use different strategies.
  It solves the child’s paradox (how could a child segment a word, if the child does not know the word?)

Question 6

lexical selection

Answer 6

• Segmented stream is the input for lexical selection.
  
  • searching process that determines the best fit between the input and the abstract lexical representations.
  • fast: it starts as soon as there is some information about the word, and can finish before the word has been fully pronounced.
    Words in context can be recognised within 175-200 ms of their onset, or when only a part of their acoustic content has been presented.

Question 7

evidence from shadowing (Marslen-Wilson, 1975)

Answer 7

• Task: Participants hear a sentence and they repeat aloud what they heard.
• Results: Participants corrected the words when repeating them.
  
  • The corrections occurred before the incorrect word was presented in full.
    We are fast in recognising words - not mere repetition of sounds but they access known lexical representations

Question 8

evidence from gating (Tyler & Wessels, 1983)

Answer 8

• Task: Participants are given a word to listen. The word is chopped in different fragments/gates of different durations.
• The gates start from the beginning of the word, and become increasingly larger (e.g., +25ms every time).
  The task is to say what is the word.

Question 9

cohort model (marlin-Wilson & Welsh, 1978)

Answer 9

• 3 stages: access, selection, and integration
• Access: activation of initial set of candidates based on word-initial cohort
• Selection: words that mismatch the incoming signal removed from the cohort
• Integration: their syntactic and semantic properties integrated with context
• A word is recognised at the point where it is the only word still consistent with the input (Recognition Point).
  Optimally efficient system: maximally effective use of incoming signal, a word will be recognized as soon as the info is available to differentiate it from competitors, even before end of the word.

Question 10

salience of word onsets and additional late activation (Allopenna et al.m 1998)

Answer 10

• The Cohort model suggests that word onsets, i.e., information that we have at the very beginning helps to set up the search and the cohort;
  This is consistent with evidence showing that word onsets are particularly salient (Cole and Jakimik, 1980) but information coming at later points can also activate lexical entries

Question 11

access to meaning (swinney, 1979)

Answer 11

• Ambiguous context: Rumour had it that, for years, the government building had been plagued with problems. The man was not surprised when he found bugs in the corner of his room.
• Disambiguating context :Rumour had it that, for years, the government building had been plagued with problems. The man was not surprised when he found several spiders, cockroaches and other bugs in the corner of his room.
• Different meanings are initially activated: contextual info is not used to determine which words are considered for recognition;
  Contextual information is critical for the selection of the appropriate meaning amongst the activated alternatives.

Question 12

context effects (in monitoring) Marslen-Wilson, Brown & Tyler, 1988)

Answer 12

• Task: Listening to sentences & monitoring for specific words
• Results:
  
  • 1. Word in isolation: guitar ~300ms
  • 2. Normal: The boy held the guitar. ~240ms
  • 3. Pragmatic Anomalous: The boy buried the guitar. ~268ms
  • 4. Semantic Anomalous: The boy drank the guitar. ~291ms
  • 5. Categorical Anomalous: The boy slept the guitar. ~320ms
       Multiple types of contextual information are integrated during spoken word recognition

Question 13

speech processing in the brain (Gil-da-costa et al., 2006)

Answer 13

• Bilateral activity in Heschl’s gyrus (low level processing), STG and MTG for simple mapping of sound to meaning
• Left-lateralised activation in the dorsal stream, and especially left IFG, for more complex spoken inputs (e.g., sentences)
• Evolutionary context
  
  • Non-human primates also communicate by exchanging meaningful calls
  • This triggers comparable bilateral activity in the brain of a macaque
  • Suggests evolutionary continuity of the bilateral system that supports mapping from sound to meaning (ventral stream)
    Strong dorsal connections between temporal and frontal areas in the left hemisphere are unique to humans (important for syntax).

Question 14

sentence processing

Answer 14

• Sentences are not stored in the mental lexicon
  We need rapid mechanisms for parsing and extraction of the information they convey (‘who is doing what to whom’)

Question 15

syntactic rules

Answer 15

• Syntactic rules: the rules that govern how words can be combined
• They allow all permissible sentences to be generated; and rule out all illegal sentences
• Powerful little fruits produce open-source initiatives
• No meaning, but syntactically correct
• Principles for combining words into sentences that do not depend on meaning
  Based on knowledge about grammatical categories such as nouns, verbs, adjectives, prepositions etc

Question 16

syntactic parsing

Answer 16

• The girl hits the woman with a bag
• The girl hits the woman with a bag
• The girl hits the woman with a bag
  Comprehension depends (in part) on the grammatical relations that listeners can establish among the words in a sentence

Question 17

main theories of sentence processing

Answer 17

• Garden-path model (Frazier & Rayner, 1982) Initial parsing is purely syntactic, meaning is not involved in the selection of the initial syntactic structure
• Constraint-based theories (MacDonald et al, 1994) The initial interpretation depends on all available sources of information (syntactic, semantic, general world knowledge)
• Unrestricted race-model (Van Gompel et al, 2000)
  All sources of information used to identify a syntactic structure
  ‘Good-enough’ representations (Ferreira et al, 2002) Processing depth / type of information used depend on the task

Question 18

evidence- sentence parsing cues

Answer 18

• Types of information used to understand sentences:
  
  • syntactic principles
  • statistical regularities
  • grammatical categories
  • prosodic cues
  • semantic information
  • world knowledge
    (…and more)

Question 19

syntactic principles (Frazier, 1987)

Answer 19

• Late closure principle: new items are attached to the phrase or the clause most recently processed, if grammatically possible.
• Minimal attachment: Minimal attachment links each incoming word to the existing structure using the simplest syntactic structure that is consistent with the input.
  A strategy of parsimony: The parser builds the simplest structure that obeys the rules of language.

Question 20

statistical regularities (Slobin, 1966)

Answer 20

Word order: Evidence: Effects of word order have been demonstrated in sentence to picture matching tasks, where active sentence types are processed (canonical word order, more frequently used) faster than passive sentence

Question 21

grammatical categories

Answer 21

• Individual words provide reliable cues for sentence interpretation (esp. grammatical words like prepositions, articles, pronouns)
• A grammatical word and signals a new phrase node
  A complementiser (that, which, etc.) signals embedded sentence

Question 22

neurobiology of syntax

Answer 22

• Meta-analysis of 128 neuroimaging studies of syntactic processing
• Extended fronto-temporal processing network, more prominent on the left
• Requires intact left fronto-temporal white matter connections (dorsal and ventral)

Question 23

example of neurocognitive syntax

Answer 23

• Strict left-lateralization of grammatical function; primacy of Broca’s area
• Emphasis on distinguishing between simple and complex syntactic computations
• Two neural computational systems:
  
  • Ventral pathway (BA45 – aSTG) for simple syntax The cat sat on the mat
    Dorsal pathway (BA 44 – pSTG) for complex syntax The juice that the child spilled stained the rug

Question 24

communication

Answer 24

• Exchange of information between individuals using a common system of symbols, signs, or behaviour
• Broader than language as it also refers to:
  
  • non-verbal communication
    use of iconic and indexical signs - animal communication, etc.

Question 25

types of gestures

Answer 25

• Beats Simple, brief, repetitive movements. Coordinated with the speech prosody but bear no obvious relation to the meaning of the accompanying speech
• Pointing Simple conventionalised movements that refer to spatial locations, or bind objects to spatial locations
• Symbolic gestures Movements with specific, conventionalised meaning such as OK, thumbs up, fingers crossed, etc.
  Lexical gestures Hand movements of different shape and length, non repetitive and changing in form, that appear related to the meaning of the accompanying speech

Question 26

why gestures? (Iverson & Goldin-Meadow, 1998)

Answer 26

• Not necessarily because others can see us, or we understand that gestures are conveying useful information to the listener
• Congenitally blind people gesture as much as sighted people, even when they talk to other blind people
• Clearly some communicative and informative function
• However, gestures are often rather vague and meaningless without words, and performed in the absence of listeners
  Hp: Gestures may facilitate word production

Question 27

onset of gestures (Morrel- Samuels & Krauss, 1992)

Answer 27

• Most of the gestures start between 500ms and 1s before the words
• Familiar words:
  
  • closer gesture asynchrony
    briefer gesture

Question 28

multisensory communication

Answer 28

• Auditory speech often enough to convey the message
• Speech is often multisensory during social interactions
• Verbal information: voice
• Non-verbal information:
  
  • Face: lip movements, eyebrows
  • Head and shoulders nods
    Hand gestures

Question 29

hand gestures and speech production

Answer 29

• Speech and gestures temporally align very well
• Common language system (McNeill, 2000; Iverson & Goldin-Meadow, 1998)
• Continuum of movements
  Different categories depending on speech dependence

Question 30

beat gestures and prosody

Answer 30

• Beats are the visual versant of the speaker’s prosody through kinematics: direction, velocity, etc.
• Extension – Apex – Retraction
• Rapid flick of the hands; No semantic content in their shape
  Synchronize with pitch accentuations

Question 31

beat gestures and speech production

Answer 31

• Do we gesture to facilitate speech production?
• Beats do not convey any meaning
• We gesture on the telephone even if the other person cannot see us.
  Congenitally blind people gesture as much as sighted people, even when they talk to other blind people.

Question 32

beat gestures and speech production- Rauscher et al., 1996

Answer 32

• Describe cartoon clips of Road Runner with or without gesturing during description
• Preventing gestures has a negative effect on speech fluency
  Beat indeed facilitate word production if content spatial information > Causal role on production

Question 33

beat gestures and speech perception

Answer 33

• The primary goal of communication is to maximise chances for information to reach the listener and to be understood by them
• This begs the question: in which ways such rapid and meaningless gestures can influence speech perception on the listener’s side?
• Beat gestures align with the stressed syllable of target words
  Listeners are experts in mapping gestures and speech

Question 34

beat gestures and speech perception- Treffner, Peter & Kleidon, 2008

Answer 34

• Beat gestures increase words’ salience perceived by the listeners.
  “Which word is being emphasized in the sentence?”

Question 35

beat gestures and speech perception (So, Sim & Low, 2012)

Answer 35

• Isolated words in 3 different conditions: No gesture, Iconic or Beat gesture
• Later asked to recall as many words as possible
• Compared memory between adults and children
• Participants recalled more words encoded with a gesture than alone.
• Adults benefit from both types of gestures, while only iconic gestures improved memory in children
• Attributing a value to beat gestures required higher-level concepts that are not yet developed in children < 4-5 years.
  E.g., Signal emphasis or salience, turn-taking, speaker stance, social cues.

Question 36

beat gestures and speech perception (Biau et al., in prep

Answer 36

• Beat gestures and word encoding in spontaneous speech?
• Q&A sessions at the House of Commons (~20s)
• Encoding: Audiovisual vs. Auditory only clips
• Memory test: Old/New words recognition (Roediger et al., 1995)
• Compare memory accuracy between words encoded with or without a beat gesture
• Participants recalled more words encoded with a gesture than without gesture.
• Listeners integrated beat gestures and coinciding verbal information during speech perception
• It suggests that adults infer a communicative value of simple beats such as “emphasizing important information”
  Attention attraction towards crucial speech bits and improve speech processing?

Question 37

beat gestures and speech perception (Biau et al., 2013, Biau et al., 2015, Biau et al., 2017)

Answer 37

• Beat-speech integration modulates brain neurophysiology
• The presence of a gesture elicited a positive waveform in the ERPs time-locked to the corresponding word onsets.
• Early modulation of word processing stage corresponding to multisensory integration stages (N100-P200).
• The effect peaked across the left Centro-Frontal electrodes, covering speech-related regions.

Question 38

beat gestures and speech perception- audiovisual synchrony detection task- EEG

Answer 38

• 20-s Audiovisual clips of a female engaged in an interview and spontaneously gesturing while talking to a journalist in 2 conditions
• (1) Synchronous: Intact natural coincidence between gestures and sounds.
• (2) Asynchronous: 360ms offset between gesture and corresponding word onsets
• Audiovisual synchrony detection tests how listeners integrate gestures and sounds (correct responses and reaction times)
• Participants correctly detected both synchronous and asynchronous stimuli > Similar accuracy and reaction times
• Brain activation increased in response to asynchronous speech in the Left IFG and Motor cortex
  Compensatory mechanism to a greater difficulty to integrate visual and auditory streams to perceive multisensory speech (Biau, Schultz & Kotz, 2022)

Question 39

do beat gestures carry specific communicative values that are reflected in the brain activity? Biau et al., 2016

Answer 39

• 20-s audiovisual clips taken from a public addressee given by a former Spanish prime minister
• 2 condition of visual movement information: Beats or equivalent moving Dots
• 2 conditions of AV synchrony: Synchronous (Sync) or asynchronous with a 360ms offset between gesture and corresponding word onsets (Async)
• Asynchrony impacted brain activity differently depending on the type of movement (Beat vs. Dots) during speech perception
• Interaction Synchrony x Movement found in language areas important for multisensory integration, prosody and visual perception.
  The brain interprets beats as social values and markers of communicative intentions, beyond trivial visual cues.