Week 8

Question 1

State the audio features that have been used for emotion recognition

Answer 1

1) pitch, intensity & duration of speech
2) spectral energy distribution
3) Mel Frequency Cepstral Coefficients (MFCCs) & ∆MFCCs
4) average Zero Crossings Density (ZCD)
5) filter-bank energies (FBE) parameters

Question 2

Given a speech signal s, how can its pitch and intensity (or loudness) be measured?

Answer 2

1) Pitch via DFT (equ. 17.1)

2) intensity, syllable peak (equ. 17.3)

Question 3

What is meant by harmonics-to-noise ratio?

Answer 3

• represents rough voice

- ≈20 dB (i.e., 99% of energy of the signal is periodic, & the rest is noise) for young speakers

Question 4

What is the cepstrum of a speech signal?

Answer 4

1) equ. 17.10
2) cepstrum of sound source:
-voiced, pulses around n = 0, T, 2T, . . . where T is pitch
period, fundamental frequency (or pitch)
- unvoiced: no distinct features
3) Cepstrum of vocal tract IR:
- non-negligible for small number of samples
- number of samples < number of samples in a pitch period
- a low-time cepstrum window will extract it
4) Cepstrum Coefficients (CC) for separating signal from vocal tract IR e.g., use low coefficients to analyse
vocal tract

Question 5

Discuss the usefulness of audio features for emotion recognition from speech.

Answer 5

1) Not best feature set or most efficient recognition model ⇒ data-dependent
2) Select set of relevant features:
- prosodic features, e.g., pitch, energy, speaking rate
- lexical, disfluency, & discourse cues
- acoustic features, e.g., zero-crossing, spectrum, formants, MFCCs
3) Data-dependent ⇒
- use as many features as possible
- optimise choice of features with the recognition algorithm

Question 6

What are the characteristics of intonation groups of a pitch contour that are selected for feature extraction for emotion recognition?

Answer 6

1) complete IGs with largest pitch range or duration
2) monotonically decreasing or increasing IGs with largest pitch range or duration
3) monotonically decreasing or increasing IGs at start or end of a sentence

Question 7

What are the three types of intonation groups that are labelled in Figure 2? Which are selected for feature extraction?

Answer 7

1) Type 1: A complete pitch segment that starts from the point of a pitch rise
to the point of the next pitch rise
2) Type 2: A monotonically decreasing pitch segment
3) Type 3: A monotonically increasing pitch segment
4) Type 2 and 3

Question 8

What is meant by jitter and shimmer in a speech signal?

Answer 8

1) Jitter:
- cycle-to-cycle variations of pitch period
- affected by lack of control of vibration of the vocal cords
- patients with pathologies, higher value of jitter
- 2 measures: absolute and relative
2) Shimmer:
- cycle-to-cycle variations of the amplitude of pitch
period
- changes with reduction of glottal resistance & mass lesions on the vocal cords
- patients with pathologies - higher values of
shimmer
- 2 measures: absolute and relative

Question 9

Motivation for IG feature extraction:

Answer 9

1) emotional state can be characterised by many speech features
2) trembling speech, unvoiced speech, speech duration & hesitation - useful characteristics for emotion detection

Question 10

What articulatory mechanisms are involved in emotionally expressive speech?

Answer 10

1) emotional speech is associated more with peripheral articulatory motions than that of Neutral speech
2) tongue tip (TT), jaw & lip positioning are more advanced (extreme) in
emotional speech than in Neutral speech
3) classification recalls of using articulatory features are higher than those of
acoustic features
4) ⇒ articulatory features carry valuable emotion-dependent information

Question 11

Methods for collecting articulatory data

Answer 11

ultrasound, x-ray microbeam, electromagnetic

articulography (EMA), & magnetic resonance imaging (MRI)

Question 12

How foes EMA work?

Answer 12

1) measures position of parts of the mouth
2) uses sensor coils placed on tongue & other parts of mouth
3) induction coils around the head produce an electromagnetic field that induces a current in the sensors in the mouth
4) induced current is inversely proportional to the cube of the distance ⇒ sensor coil’s location

Question 13

What are the current challenges with emotion recognition via speech?

Answer 13

1) Variability in speakers:
- Inter-speaker variability, heterogeneous display of emotion & differences in individual’s vocal tract structures
- Intra-speaker variability: a speaker’s ability to express an emotion in a number of ways & can be influenced by the context
2) Variant nature of controls of speech production
components

Question 14

State the three types of acoustic low-level descriptors for emotion recognition,
and their characteristics.

Answer 14

1) Prosody - related to rhythm, stress & intonation of speech, and include
fundamental frequency (fo) or pitch, short-term energy, speech rate

2) Spectral characteristics - related to harmonic/resonant
structures, & include
Mel-frequency cepstral coefficients (MFCCs), Mel-filter bank energy coefficients (MFBs)

3) Voice quality-related measures - related to characteristics of vocal cord vibrations, & include
jitter, shimmer
harmonic-to-noise ratio

Question 15

Processing steps for acoustic LLDs

Answer 15

1) Compute various statistical functionals (i.e., mean, standard deviation) on these LLDs at different time scale granularities (e.g., at 0.1, 0.5, 1, & 10 sec, etc.)

2) Compute the dynamics at multilevel time scales using
statistical functional operators

3) Select features to reduce dimension using: stand-alone method or mutual information

Question 16

Affect recognition and modelling using speech requires

Answer 16

1) annotation scheme of emotion labels
2) feature normalisation technique
3) context-aware machine learning frameworks

Question 17

Emotion labelling, self-reports vs perceived rating

Answer 17

1) self-reports - ask subjects to recall how they have felt during a particular interaction

2) perceived rating - ask trained observers to assign emotion labels as they
watch a given audio video recording

Question 18

What are the three approaches to global normalisation of acoustic features? Why are they not always effective?

Answer 18

1) z-normalisation -transforms features by subtracting their mean & dividing by their standard deviation ⇒ each feature with zero mean & unit variance across all data
2) min-max - scales feature to predefined range
3) nonlinear normalisation -convert features’ distributions into normal distributions

4) not always effective as a single normalisation scheme across the entire corpus can adversely
affect the emotional discrimination of the feature

Question 19

With the aid of a diagram, outline the operation of the Iterative Feature Normalisation.

Answer 19

1) Normalise features by estimating the parameters of an affine transformation
(e. g., z-normalisation) using only neutral (non-emotional) samples

2) Iterative because neutral samples may not be available for every individual
3) Front-end scheme estimates the neutral subset of the data iteratively
4) ⇒ to estimate the normalisation parameters
5) Robust against different recording conditions

Question 20

Motivation for static emotion recognition for single utterance via hierarchical tree

Answer 20

1) to map an utterance to a predefined set of categorical emotion classes given acoustic features
2) appraisal theory of emotion (i.e., emotion is theorised to be in stages of a stimulus)
3) ⇒ Based on first processing the clear perceptual differences of emotion information in the acoustic features at root of tree, & highly ambiguous emotions are recognised
at leaves of tree

Question 21

Hierarchical tree for static emotion recognition for single utterance

Answer 21

1) Levels in tree solve the easiest classification tasks first, mitigates error propagation
2) Each node of tree a binary classifier i
3) Leaves of tree identify most ambiguous emotion class, which often is the neutral class

Question 22

What is recall?

Answer 22

Recall indicates what proportion of actual positives (i.e.,

correct emotion class) is identified correctly. Calculated: TP/(TP+FN)

Question 23

How could the physiological theory of emotion be exploited in affective computing?

Answer 23

1) Physical response - primary to feeling of an emotion
2) Stimulus ⇒ activity in autonomic nervous system (ANS) ⇒ emotional response in brain
3) Specific patterns of response correspond to specific emotions:
- anger, increased blood flow to hands, increased heart rate, snarling, increased involuntary nervous system arousal
- fear ⇒ high arousal state, decreased voluntary muscle activity, greater involuntary muscular contractions, decreased circulation in the peripheral blood vessels

Question 24

What bodily changes have been exploited for emotion recognition?

Answer 24

All bodily changes that can be sensed from surface of the skin & reflect ANS activity:
1) cardiac activity (heart rate, heart-rate variability, & blood
volume pulse)
2) galvanic skin response (skin conductivity)
3) surface electromyography (EMG)
4) respiration through expansion of chest cavity

Question 25

Explain ECG

Answer 25

1) ventricles depolarise, pumping blood into the rest of body, while the atria polarise by expanding their volume, receiving blood from the body ⇒ ventricles polarise, & the process is repeated
2) electrical changes due to polarisation of chambers detectable (ECG)
3) Inflection points P,Q,R,S,T:
- P, atrial depolarisation
- QRS, ventricular depolarisation
- T, ventricular repolarisation

Question 26

Drawbacks of ECG

Answer 26

Requires:
1) contact of electrode
adhesive patch with
subject’s skin
2) excess hair removed
3) skin cleaned at adhesion sites

Question 27

Outline the operation of a wrist pulse oximeter. How can its output be exploited
for emotion recognition?

Answer 27

1) device emits light
2) amount of blood in vessel measured by amount of light reflected by the vessel over time
3) more blood ⇒ higher reflectance reading
4) Vasoconstriction (can be
measured if subject is
stationary):
- defensive reaction in
which peripheral blood
vessels constrict
- ⇒ increases in
response to pain,
hunger, fear & rage
- ⇒ decreases in
response to quiet
relaxation

Question 28

What is a photoplethysmograph (PPG) sensor?

Answer 28

1) Measures blood volume pulse:
- every heartbeat pumps blood to blood vessels
- most pronounced in peripheral vessels e.g. fingers, earlobe

Question 29

PPG sensor placement and reading

Answer 29

1) Place sensor where the capillaries are close to the surface of the skin, e.g., finger
2) No gels or adhesives
3) Reading is very sensitive to variations in placement & to motion artefacts

Question 30

How do SNS (sympathetic) and PNS (parasympathetic) affect heart rate

Answer 30

1) SNS accelerates heart rate - related to stress or activation
2) PNS decelerates heart rate - responsible for relaxation, or rest & healing

Question 31

HRV metrics

Answer 31

1) standard deviation of time between successive heart-beats within a certain window (i.e., the recording epoch), best for short time windows
2) difference between maximum & minimum normal R-R interval lengths within the window
3) percent differences between successive normal R-R intervals that exceed 50 mSec (pNN50)
4) root mean square successive difference (RMSSD)
5) sympathovagal balance: ratio of (influence by PNS) to
(influence by SNS) on HRV, PNS modulates heart rate for frequencies between 0.04 &
0.5 Hz, SNS modulates heart rate with significant gain only below 0.1 Hz

Question 32

Non-emotion factors affecting HR and HRV

Answer 32

1) Age, age ↑ ⇒ HRV ↓, e.g., infants have a high level of sympathetic activity but
decreases between ages 5 to 10
2) Level of physical conditioning, congestive heart failure: HRV ⇒ zero; heart beats like a metronome, subject with pace maker or taking medication
3) Physical activity, talking & posture (sitting vs standing vs lying down)
4) Breathing frequency
5) Circadian cycle (24 hour cycle in the physiological
processes)

Question 33

What is Electroencephalogram (EEG)? How can it be used for emotion recognition?

Answer 33

1) Measures electrical activity of brain by placement of electrodes on the
surface of the head
2) Full EEG distinguishes between positive & negative
emotional valence, and different arousal levels
3) Determines orienting response by detecting alpha blocking, alpha waves (8 to 13 Hz) become extinguished & beta waves (14 to 26 Hz) become dominant when the person experiences a startling event

Question 34

In affective computing state how does Electromyogram (EMG) aid with emotional recognition

Answer 34

1) Measures activity by detecting surface
voltages via 3 electrodes that occur when a
muscle is contracted
2) on facial muscles to study facial expression
3) on body to study affective gestures
4) on both - emotional valence & emotional
arousal
5) need adhesives and gels
6) Signal is low-pass filtered ⇒ aggregate
muscle activity & is sampled at 10 to 20 Hz

Question 35

In affective computing state how does blood pressure (sphygmomanometry) aid with emotional recognition

Answer 35

1) Correlates with increases in emotional stress & with repression
of emotional responses
2) Difficult to measure continuously
3) Requires constricting a blood vessel ⇒ discomfort

Question 36

In affective computing state how does respiration aid with emotional recognition

Answer 36

1) Most accurately recorded by measuring gas exchange of the lungs (but cumbersome)
2) Measured by a strap sensor that incorporates a
strain gauge, a Hall effect sensor, or a capacitance sensor to measure chest activity expansion
3) Physical activity & emotional arousal
⇒ faster & deeper respiration
4) Peaceful rest & relaxation
⇒ slower & shallower respiration
5) Sudden, intense, or startling stimuli
⇒ momentary cessation of respiration
6) Negative emotions
⇒ irregularity in respiration patterns

Question 37

Skin conductance or galvanic skin response or electrodermal activity (EDA)

Answer 37

1) Indirectly measures the amount of sweat in
a person’s sweat gland, skin is an insulator & its conductivity changes in response to ionic sweat
filling the sweat glands
2) Sweat-gland activity
an indicator of sympathetic activation, GSR is a robust non-invasive way to
measure this activation
4) GSR is a component in lie detector, measures the emotional stress
associated with lying
5) Differentiates between states, e.g., anger &
fear; conflict & no conflict
6) Measures stress on anticipatory anxiety &
during task performance

Question 38

Where are the most emotionally reactive sweat glands?

Answer 38

1) palms of hands & soles of feet

2) place electrodes on lower segment of middle & index fingers of dominant hand

Question 39

Features of skin conductance/GS/EDA

Answer 39

1) Mean conductivity level, variance, slope, max. & min. levels
2) Orienting response, amplitude, latency, rise time, half-recovery time
habituation
3) Skin conductance response is neither linear nor time invariant, baseline drift & conductance changes due to increased or decreased contact between electrodes & skin, ⇒ introduce confounding factors into interpreting GSR & into pooling features from different time periods (e.g., morning vs
evening)