Language 2 Flashcards
1
Q
processing spoken words overview
A
- Properties of spoken language
- Major elements of speech recognition:
- Segmentation- speech is continuous so need to extract units of information e.g. words from this
- Lexical selection- once we have isolated units we have to map the meaning/representation from mental dictionary
- Access to meaning- want to understand the meaning of units
- Context effects- all of this mechanism shows you how contexts can facilitate different meanings
Speech processing in the brain
2
Q
speech processing
A
- A process of progressively extracting invariant, discrete representations from a variable, continuous input…
- Speech signal challenges:
- Continuous. Distributed in time. Fast-fading Words are not neatly segmented (e.g., by pauses). Consecutive speech sounds blend into each other due to mechanical constraints on articulators.
Variable Speaker differences; pitch affected by age and sex; different accents, talking speeds, often heard in noise.
- Continuous. Distributed in time. Fast-fading Words are not neatly segmented (e.g., by pauses). Consecutive speech sounds blend into each other due to mechanical constraints on articulators.
3
Q
word segmentation (Cutler & Norris, 1988)
A
- The rhythmic structure of English is stress- timed (some syllables are emphasised)
- Lettuce Trousers Cigar (underlined = stressed syllable)
Metrical Segmentation Strategy (MSS): In English, stressed (strong) syllables are likely onsets of words. Continuous speech is segmented at stressed syllables.
4
Q
syllable stress and word types
A
- Syllables
- Stressed syllables- Full vowels (lettuce)
- Unstressed syllables- Reduced vowels (behind)
- Words
- Content words- Nouns, verbs, adjectives
- Grammatical words- Articles, pronouns, prepositions, conjunctions
Evidence in favour of MSS: the 74% of stressed syllables in English corresponds to sole or initial syllable of a content word. This is not the case for unstressed syllables - only 5% corresponds to content words (Cutler & Carter, 1987).
5
Q
metrical segmentation strategy
A
- It is not infallible – it is a strategy ( * examples of incorrectly segmented words with complex syllabic structure). Listeners need other source of information to segment successfully.
- Alert *lert
- Assassinate *sassinate
- MSS is language specific. Other languages may use different strategies.
It solves the child’s paradox (how could a child segment a word, if the child does not know the word?)
6
Q
lexical selection
A
- Segmented stream is the input for lexical selection.
- searching process that determines the best fit between the input and the abstract lexical representations.
- fast: it starts as soon as there is some information about the word, and can finish before the word has been fully pronounced.
Words in context can be recognised within 175-200 ms of their onset, or when only a part of their acoustic content has been presented.
7
Q
evidence from shadowing (Marslen-Wilson, 1975)
A
- Task: Participants hear a sentence and they repeat aloud what they heard.
- Results: Participants corrected the words when repeating them.
- The corrections occurred before the incorrect word was presented in full.
We are fast in recognising words - not mere repetition of sounds but they access known lexical representations
- The corrections occurred before the incorrect word was presented in full.
8
Q
evidence from gating (Tyler & Wessels, 1983)
A
- Task: Participants are given a word to listen. The word is chopped in different fragments/gates of different durations.
- The gates start from the beginning of the word, and become increasingly larger (e.g., +25ms every time).
The task is to say what is the word.
9
Q
cohort model (marlin-Wilson & Welsh, 1978)
A
- 3 stages: access, selection, and integration
- Access: activation of initial set of candidates based on word-initial cohort
- Selection: words that mismatch the incoming signal removed from the cohort
- Integration: their syntactic and semantic properties integrated with context
- A word is recognised at the point where it is the only word still consistent with the input (Recognition Point).
Optimally efficient system: maximally effective use of incoming signal, a word will be recognized as soon as the info is available to differentiate it from competitors, even before end of the word.
10
Q
salience of word onsets and additional late activation (Allopenna et al.m 1998)
A
- The Cohort model suggests that word onsets, i.e., information that we have at the very beginning helps to set up the search and the cohort;
This is consistent with evidence showing that word onsets are particularly salient (Cole and Jakimik, 1980) but information coming at later points can also activate lexical entries
11
Q
access to meaning (swinney, 1979)
A
- Ambiguous context: Rumour had it that, for years, the government building had been plagued with problems. The man was not surprised when he found bugs in the corner of his room.
- Disambiguating context :Rumour had it that, for years, the government building had been plagued with problems. The man was not surprised when he found several spiders, cockroaches and other bugs in the corner of his room.
- Different meanings are initially activated: contextual info is not used to determine which words are considered for recognition;
Contextual information is critical for the selection of the appropriate meaning amongst the activated alternatives.
12
Q
context effects (in monitoring) Marslen-Wilson, Brown & Tyler, 1988)
A
- Task: Listening to sentences & monitoring for specific words
- Results:
- Word in isolation: guitar ~300ms
- Normal: The boy held the guitar. ~240ms
- Pragmatic Anomalous: The boy buried the guitar. ~268ms
- Semantic Anomalous: The boy drank the guitar. ~291ms
- Categorical Anomalous: The boy slept the guitar. ~320ms
Multiple types of contextual information are integrated during spoken word recognition
- Categorical Anomalous: The boy slept the guitar. ~320ms
13
Q
speech processing in the brain (Gil-da-costa et al., 2006)
A
- Bilateral activity in Heschl’s gyrus (low level processing), STG and MTG for simple mapping of sound to meaning
- Left-lateralised activation in the dorsal stream, and especially left IFG, for more complex spoken inputs (e.g., sentences)
- Evolutionary context
- Non-human primates also communicate by exchanging meaningful calls
- This triggers comparable bilateral activity in the brain of a macaque
- Suggests evolutionary continuity of the bilateral system that supports mapping from sound to meaning (ventral stream)
Strong dorsal connections between temporal and frontal areas in the left hemisphere are unique to humans (important for syntax).
14
Q
sentence processing
A
- Sentences are not stored in the mental lexicon
We need rapid mechanisms for parsing and extraction of the information they convey (‘who is doing what to whom’)
15
Q
syntactic rules
A
- Syntactic rules: the rules that govern how words can be combined
- They allow all permissible sentences to be generated; and rule out all illegal sentences
- Powerful little fruits produce open-source initiatives
- No meaning, but syntactically correct
- Principles for combining words into sentences that do not depend on meaning
Based on knowledge about grammatical categories such as nouns, verbs, adjectives, prepositions etc
16
Q
syntactic parsing
A
- The girl hits the woman with a bag
- The girl hits the woman with a bag
- The girl hits the woman with a bag
Comprehension depends (in part) on the grammatical relations that listeners can establish among the words in a sentence
17
Q
main theories of sentence processing
A
- Garden-path model (Frazier & Rayner, 1982) Initial parsing is purely syntactic, meaning is not involved in the selection of the initial syntactic structure
- Constraint-based theories (MacDonald et al, 1994) The initial interpretation depends on all available sources of information (syntactic, semantic, general world knowledge)
- Unrestricted race-model (Van Gompel et al, 2000)
All sources of information used to identify a syntactic structure
‘Good-enough’ representations (Ferreira et al, 2002) Processing depth / type of information used depend on the task
18
Q
evidence- sentence parsing cues
A
- Types of information used to understand sentences:
- syntactic principles
- statistical regularities
- grammatical categories
- prosodic cues
- semantic information
- world knowledge
(…and more)
19
Q
syntactic principles (Frazier, 1987)
A
- Late closure principle: new items are attached to the phrase or the clause most recently processed, if grammatically possible.
- Minimal attachment: Minimal attachment links each incoming word to the existing structure using the simplest syntactic structure that is consistent with the input.
A strategy of parsimony: The parser builds the simplest structure that obeys the rules of language.
20
Q
statistical regularities (Slobin, 1966)
A
Word order: Evidence: Effects of word order have been demonstrated in sentence to picture matching tasks, where active sentence types are processed (canonical word order, more frequently used) faster than passive sentence
21
Q
grammatical categories
A
- Individual words provide reliable cues for sentence interpretation (esp. grammatical words like prepositions, articles, pronouns)
- A grammatical word and signals a new phrase node
A complementiser (that, which, etc.) signals embedded sentence
22
Q
neurobiology of syntax
A
- Meta-analysis of 128 neuroimaging studies of syntactic processing
- Extended fronto-temporal processing network, more prominent on the left
- Requires intact left fronto-temporal white matter connections (dorsal and ventral)
23
Q
example of neurocognitive syntax
A
- Strict left-lateralization of grammatical function; primacy of Broca’s area
- Emphasis on distinguishing between simple and complex syntactic computations
- Two neural computational systems:
- Ventral pathway (BA45 – aSTG) for simple syntax The cat sat on the mat
Dorsal pathway (BA 44 – pSTG) for complex syntax The juice that the child spilled stained the rug
- Ventral pathway (BA45 – aSTG) for simple syntax The cat sat on the mat
24
Q
communication
A
- Exchange of information between individuals using a common system of symbols, signs, or behaviour
- Broader than language as it also refers to:
- non-verbal communication
use of iconic and indexical signs - animal communication, etc.
- non-verbal communication
25
types of gestures
- Beats Simple, brief, repetitive movements. Coordinated with the speech prosody but bear no obvious relation to the meaning of the accompanying speech
- Pointing Simple conventionalised movements that refer to spatial locations, or bind objects to spatial locations
- Symbolic gestures Movements with specific, conventionalised meaning such as OK, thumbs up, fingers crossed, etc.
Lexical gestures Hand movements of different shape and length, non repetitive and changing in form, that appear related to the meaning of the accompanying speech
26
why gestures? (Iverson & Goldin-Meadow, 1998)
- Not necessarily because others can see us, or we understand that gestures are conveying useful information to the listener
- Congenitally blind people gesture as much as sighted people, even when they talk to other blind people
- Clearly some communicative and informative function
- However, gestures are often rather vague and meaningless without words, and performed in the absence of listeners
Hp: Gestures may facilitate word production
27
onset of gestures (Morrel- Samuels & Krauss, 1992)
- Most of the gestures start between 500ms and 1s before the words
- Familiar words:
* closer gesture asynchrony
briefer gesture
28
multisensory communication
- Auditory speech often enough to convey the message
- Speech is often multisensory during social interactions
- Verbal information: voice
- Non-verbal information:
* Face: lip movements, eyebrows
* Head and shoulders nods
Hand gestures
29
hand gestures and speech production
- Speech and gestures temporally align very well
- Common language system (McNeill, 2000; Iverson & Goldin-Meadow, 1998)
- Continuum of movements
Different categories depending on speech dependence
30
beat gestures and prosody
- Beats are the visual versant of the speaker’s prosody through kinematics: direction, velocity, etc.
- Extension – Apex – Retraction
- Rapid flick of the hands; No semantic content in their shape
Synchronize with pitch accentuations
31
beat gestures and speech production
- Do we gesture to facilitate speech production?
- Beats do not convey any meaning
- We gesture on the telephone even if the other person cannot see us.
Congenitally blind people gesture as much as sighted people, even when they talk to other blind people.
32
beat gestures and speech production- Rauscher et al., 1996
- Describe cartoon clips of Road Runner with or without gesturing during description
- Preventing gestures has a negative effect on speech fluency
Beat indeed facilitate word production if content spatial information > Causal role on production
33
beat gestures and speech perception
- The primary goal of communication is to maximise chances for information to reach the listener and to be understood by them
- This begs the question: in which ways such rapid and meaningless gestures can influence speech perception on the listener’s side?
- Beat gestures align with the stressed syllable of target words
Listeners are experts in mapping gestures and speech
34
beat gestures and speech perception- Treffner, Peter & Kleidon, 2008
- Beat gestures increase words’ salience perceived by the listeners.
“Which word is being emphasized in the sentence?”
35
beat gestures and speech perception (So, Sim & Low, 2012)
- Isolated words in 3 different conditions: No gesture, Iconic or Beat gesture
- Later asked to recall as many words as possible
- Compared memory between adults and children
-
- Participants recalled more words encoded with a gesture than alone.
- Adults benefit from both types of gestures, while only iconic gestures improved memory in children
- Attributing a value to beat gestures required higher-level concepts that are not yet developed in children < 4-5 years.
E.g., Signal emphasis or salience, turn-taking, speaker stance, social cues.
36
beat gestures and speech perception (Biau et al., in prep
- Beat gestures and word encoding in spontaneous speech?
- Q&A sessions at the House of Commons (~20s)
- Encoding: Audiovisual vs. Auditory only clips
- Memory test: Old/New words recognition (Roediger et al., 1995)
- Compare memory accuracy between words encoded with or without a beat gesture
-
- Participants recalled more words encoded with a gesture than without gesture.
- Listeners integrated beat gestures and coinciding verbal information during speech perception
- It suggests that adults infer a communicative value of simple beats such as “emphasizing important information”
Attention attraction towards crucial speech bits and improve speech processing?
37
beat gestures and speech perception (Biau et al., 2013, Biau et al., 2015, Biau et al., 2017)
- Beat-speech integration modulates brain neurophysiology
- The presence of a gesture elicited a positive waveform in the ERPs time-locked to the corresponding word onsets.
- Early modulation of word processing stage corresponding to multisensory integration stages (N100-P200).
- The effect peaked across the left Centro-Frontal electrodes, covering speech-related regions.
38
beat gestures and speech perception- audiovisual synchrony detection task- EEG
- 20-s Audiovisual clips of a female engaged in an interview and spontaneously gesturing while talking to a journalist in 2 conditions
- (1) Synchronous: Intact natural coincidence between gestures and sounds.
- (2) Asynchronous: 360ms offset between gesture and corresponding word onsets
- Audiovisual synchrony detection tests how listeners integrate gestures and sounds (correct responses and reaction times)
-
- Participants correctly detected both synchronous and asynchronous stimuli > Similar accuracy and reaction times
- Brain activation increased in response to asynchronous speech in the Left IFG and Motor cortex
Compensatory mechanism to a greater difficulty to integrate visual and auditory streams to perceive multisensory speech (Biau, Schultz & Kotz, 2022)
39
do beat gestures carry specific communicative values that are reflected in the brain activity? Biau et al., 2016
- 20-s audiovisual clips taken from a public addressee given by a former Spanish prime minister
- 2 condition of visual movement information: Beats or equivalent moving Dots
- 2 conditions of AV synchrony: Synchronous (Sync) or asynchronous with a 360ms offset between gesture and corresponding word onsets (Async)
- Asynchrony impacted brain activity differently depending on the type of movement (Beat vs. Dots) during speech perception
- Interaction Synchrony x Movement found in language areas important for multisensory integration, prosody and visual perception.
The brain interprets beats as social values and markers of communicative intentions, beyond trivial visual cues.
