Week 10

Question 1

What are the prerequisites for collecting a multimodal dataset and what are the
challenges involved?

Answer 1

1) quantity of data
2) high diversity w.r.t. subjects’ age, gender & culture, and situational context
3) balanced distribution of instances among classes, or along the range (for continuous models)
4) quality of data (i.e., adequate, realistic & naturalistic)
5) adhering to ideal capture conditions

Question 2

Current situations about multimodal datasets

Answer 2

1) smaller in size than unimodal datasets
2) more often recorded in lab most are bimodal - audiovisual
3) physiological measures or speech alongside depth images becoming available

Question 3

What ethical issues must be addressed before creating a multimodal dataset?

Answer 3

1) affect can be very private ⇒ subjects might not always
agree with making genuine & spontaneous affect data
available for study, especially with video & audio
2) moral principles guiding research:
- how ethical issues influence selection & conduct
- subjects are informed & provided consent ⇒ might reduce spontaneity and naturalness of the data
3) different levels of release for different contained modalities
4) whether the research will be beneficial to subjects
5) subjects should not be harmed

Question 4

Challenges of multimodal data collection

Answer 4

1) To obtain naturalistic display of affects
2) Complex setups for multimodal recordings require careful control of lab conditions, observers’ paradox - presence of experimenter &
awareness of being recorded may influence the subject
3) Synchronisation of multimodal capture streams, different devices/timescales/sampling
4) Sufficient number of independent labellers (or self-labelling):
- not all modalities’ recorded data are sufficiently informative
for human labellers to make affect judgments
- may require self-assessment ⇒ disruptive w.r.t. an
awareness of being in an experiment

Question 5

10 steps to consider for multimodal datasets:

Answer 5

1) Step 1 - To consider ethics to guide the data collection
2) Step 2 - To consider type of new data and possible reusing of existing material
3) Step 3 - To consider collection of meta information including
demographic data
4) Step 4: The challenges in collecting data from multiple
devices.
5) Step 5: The choice of model or models, and temporal unit
of analysis.
6) Step 6: The labelling method for separate modality or in combination.
7) Step 7: Standardising to foster compatibility of the meta data & the annotation.
8) Step 8: Partitioning data for modelling, optimising & testing.
9) Step 9: Verifying perception and baseline results conducted
individually per modality or for modality combinations.
10) Step 10: The release of the data with highest spread and usage.

Question 6

What considerations should be made when choosing the appropriate model/s for
a multimodal dataset?

Answer 6

1) emotion model
continuous or categorical (influenced by modalities)
2) temporal unit of analysis:
- physiological measures & video - annotated on a per-frame basis
- acoustic parameters - extracted over larger chunks, e.g., words or turns (a turn is a time during which a subject speaks)
3) compromise - annotation in continuous dimensions (e.g., arousal & valence), but also in time (e.g., every 100 ms):
- for diverse mappings, e.g., averaging over a certain chunk
4) use multiple models:
- enriches flexibility of database
- requires considerable extra effort
- could be applied for modality-specific annotation

Question 7

What considerations should be made when recording/re-using for a multimodal dataset?

Answer 7

1) recording new data
2) reusing existing material
usually only sparsely available (especially for multimodal)
3) data cover acted, induced & naturalistic emotions
4) increasing use of mobile & wearable devices for naturalistic data

Question 8

What considerations should be made when synchronising streams for a multimodal dataset?

Answer 8

1) audio & video - a challenge if using several microphones & cameras
2) worn physiological devices not routed via same computer
3) use aligned time stamps or markers for later
synchronisation, may need to be repeated during a take (or trial) to compensate for temporal deviations

Question 9

What considerations should be made when labelling for a multimodal dataset?

Answer 9

1) not all modalities can be easily annotated by a human rater, e.g., physiological signals
2) self-assessment is not always an option:
- ⇒ several external labellers serve as “expertise of the mass”
- e.g., by majority voting or by taking mean & median (for continuous emotion models)
- number of labellers proportional to level of subjectivity or ambiguity of the labelling, & the complexity of the model
3) multimodal can be annotated modality-wise or in combination:
- acoustic & physiological data - better in conveying arousal
- video or textual data - well suited to convey valence
- not all modalities are necessarily present at all time, e.g., speech

Question 10

What considerations should be made when partitioning for a multimodal dataset?

Answer 10

1) divides data into partitions for modelling, optimising & testing
2) provides default or suggested form of partitioning, facilitates comparison of results & findings
3) development partitions in addition to training & testing partitions
4) use cross-validation to enable use of as much data as possible for all partitions
5) independence of subjects, context, etc. e.g., by leaving out a subject or subject group at a time
6) keep good balance of all factors throughout the partitions
7) transparent & easy to reproduce, noting that random partitioning is
suboptimal

Question 11

What considerations should be made when verifying perception and baseline for a multimodal dataset?

Answer 11

1) independent perception test with individuals other than the annotators
2) conducted individually per modality or for modality
combinations
3) via crowd sourcing
4) include machine-based baseline recognition results

Question 12

What is the role of the evaluator-weighted estimator in the creation of multimodal dataset?

Answer 12

1) to reach rater-weighted gold standard
2) average of individual evaluators’ responses takes into account that each evaluator is subject to an individual amount of disturbance during evaluation
3) weights measure the correlation between the individual annotator’s estimations & the average ratings of all evaluators
4) if the weights are constant among raters, the gold standard is the mean of the raters’ continuous labels

Question 13

Quality assessment for multimodal affect databases

Answer 13

1) Gold standard is practically never reliable:
- training & testing labels are ambiguous to a certain degree, as subject’s emotion is usually difficult to assess
- an emotion may not be mapped unambiguously to a single category or a point in space

2) Groundtruth - actual truth as measured

3) In interpreting results:
ideally ground ⇒ trained models that process affect data are error-prone, classification error might not be so wrong in ambiguous
cases

4) ⇒ use several annotators to achieve a reliable gold
standard close to the groundtruth

Question 14

What method for measuring reliability if affect is modelled continuously?

Answer 14

1) (mean) correlation coefficient (CC) or (average) mean linear/absolute error (MLE, MAE)
2) mean square error (MSE)
3) standard deviation
4) use correlation if using only one measure

Question 15

What method for measuring reliability if affect is modelled categorically?

Answer 15

Fleiss’ Kappa K (most frequently used):

• all raters to rate all data
• if labellers agree throughout ⇒ K equals 1
• if they agree only on the same level as chance would ⇒ K=0
• negative values ⇒ systematic disagreement
• values of 0.4 to 0.6 ⇒ moderate agreement
• values > 0.6 ⇒ good to excellent agreement

Question 16

Why are ethical issues important in affective computing?

Answer 16

1) AC has profound moral significance because:
- it raises prospect of creating things that mimic human free will or impinge on it
- the public may feel that it is ethically unacceptable
2) Concerns to be countered involve moral principles:
- certain kinds of unnaturalness are bad
- a computer which seems to have emotions is unnatural

Question 17

2. In what scenarios where it is not necessary to obtain prior consent when recording subjects’ voices and images?

Answer 17

1) research consists solely of naturalistic observations in
public places, & the recording will not be used in a manner that could cause personal identification or harm
2) research design includes deception, & consent for use of recording is obtained during debriefing

Question 18

Deception in research:

Answer 18

1) should not be conducted unless it is justified by the study’s significant prospective scientific, educational or applied
value, & there are no non-deceptive alternative procedures

2) do not deceive prospective participants about research that
is expected to cause physical pain or emotional distress

3) explain any deception that is an integral feature of an experiment to participants as early as is feasible, but no later than at the conclusion of the data collection, and permit participants to withdraw their data

Question 19

. In what scenarios is informed consent unnecessary?

Answer 19

1) Where research would not reasonably be assumed to
create distress or harm, & involves:
- study of normal educational practices, curricula or classroom
management methods conducted in educational settings
- only anonymous questionnaires, naturalistic observations, or archival research for which disclosure of responses would not place participants at risk of criminal or civil liability or damage their financial standing, employability or reputation,
& confidentiality is protected
- study of factors related to job or organisation effectiveness conducted in organisational settings for which there is no risk
to participants’ employability, & confidentiality is protected

Question 20

What does Article 8 of the EU Charter of Fundamental Rights say about personal
data?

Answer 20

1) “Everyone has the right to the protection of personal data concerning him or her”

2) Data may not be processed at all unless the subject of the data
has unambiguously given his/her consent

3) Very severe restrictions on the use of data revealing racial or ethnic origin

Question 21

What are the three ethical themes for affective computing?

Answer 21

Beneficence, deception, respect for autonomy

Question 22

Discuss the beneficence ethical theme

Answer 22

1) researchers should have the welfare of the research
participant as a goal of any research study
2) morally positive goals - to make technology better able to furnish people with positive experiences and/or less likely to impose negative ones
3) objections to AC:
- unintended damage that might outweigh intended gains in happiness
positive affect has no moral value
4) ⇒ implications:
- remedial, to spare people distress that would otherwise be caused by interactions with affectively incompetent systems
- countering misguided fears that might prevent the increase of the happiness of humanity

Question 23

Discuss the deception ethical theme

Answer 23

1) general charge that AC is deceptive & cannot avoid being deceptive ⇒ AC systems ‘feel’ emotions

2) systems should not be deliberately engineered to make people believe something that is actually false, natural to fear that flawless logic, endless patience and no conscience are supplemented with ability to manipulate
emotion ⇒ an irresistible persuader

3) between the above two extremes:
- object to signs of emotion that mislead users about the way a system is likely to behave
- object to a system showing some behaviours associated
with an emotion.

Question 24

Discuss the respect for autonomy ethical theme

Answer 24

1) for people to exercise autonomy, they must have procedural independence, i.e., freedom from factors that compromise or subvert their ability to achieve self-reflection & decide rationally

2) deception violates:
- A duty of honesty
- A duty not to infringe autonomy - if information about a person’s emotional state becomes available, it can restrict their opinions in ways that they would not choose

3) users need assurances that emotional-oriented systems should not undertake any actions that users do not or
cannot endorse

Question 25

What are the ethical difficulties in certifying competence of an affective computing system?

Answer 25

1) evaluation of AC system is problematic:
- its system function is intrinsically bound up with human systems that are not well understood ⇒ impossible to guarantee analyses of risk

2) reluctance to proclaim the limitations of a product that represents an enormous investment of effort & intelligence

Question 26

Ethical issues of potraying humans by AC

Answer 26

1) descriptions of emotion are rarely morally neutral
to use them is to pass a kind of moral judgment, not obvious when a machine has the right to pass that kind of judgment

2) presenting phenomena that are morally entitled to a certain kind of human response, typically empathic, that disguises
that kind of significance
e.g., systems that detect pain & distress, better to portray by a line on a graph showing levels of pain
or distress

Question 27

How has compliance to ethical codes been enforced?

Answer 27

1) FP7 program
2) Checklist followed by text explaining how the research will handle issues arising from the list, & any others
3) Last of five sections in the evaluators’ report, does proposal raises “ethical issues that need further
attention”, if so ⇒ specialist ethical review
4) Specialist ethical reviews, & to a lesser extent panels, are informed by European Group on Ethics in Science & New Technologies

Question 28

How can negative media presentations affect AC research?

Answer 28

1) Not sanctions in themselves, but risk in translating into
sanctions

2) Shape public opinion, & public opinion sways funding bodies

Question 29

What are the essentials of an affective computing system and what are the challenges still encountered in recognising affects?

Answer 29

1) Essentials of AC System:
- Ability to recognise affect
- ⇒ To be aware of user’s cognitive & mental states
- Learning a classification decision boundary given a set of observations

2) Challenges in recognising affects:
- Little labelled training data
- Labelling is time consuming & error prone
- To be able to identify important bits of information

Question 30

Pool-based active learning

Answer 30

1) consists of an interactive loop first, a prediction system is trained on existing labelled data
2) ⇒ reasons about the best
unlabelled data to present to the user for tagging
3) loop repeats until either all data are labelled or labelling budget is
exhausted
4) Most active learning assumes pool-based
setting:
- sets of labelled & unlabelled data are provided
- algorithm selects from the pool to query

Question 31

Stream-based learning

Answer 31

1) learns & adapts throughout its life cycle
2) learner sees a series of unlabelled data points
3) continues to make decisions about whether to query for missing labels

4) unlike pool-based scenarios, learners in stream-based settings
may not have complete information about the underlying data
distribution

5) In dynamic environments, data observed in the past can become outdated ⇒ adapt by:
- continuing usage of current
predictive model to decide if & when to probe users for feedback
- considering long-term value
associated with such feedback

Question 32

How has cross-validation been used to improve feature extraction for affect recognition?

Answer 32

1) Use cross-validation to find appropriate set of features, some validation data from the training set is left out, & models are trained using the remaining data:
- models differ in the sets of features being used to train a classifier
- the model that achieves highest accuracy informs about the important features

Question 33

Hand-designed features for feature extraction

Answer 33

1) e.g., EDA or GSR correlates very well with a subject’s arousal level ⇒ appropriately designed filters
2) provide discriminatory information
3) not easy to scale because:
a multimodal affect recognition system might have several signals ⇒ impossible to handcraft individual features
- statistical relationships across 2 or more sensors, beneficial for discrimination but difficult to be handcrafted

Question 34

How can automatic relevance detection aid feature extraction for affect recognition?

Answer 34

1) allows feature selection under a Bayesian paradigm,

2) assumes:
- a priori, all features are equally likely
- as more & more data are observed, only a few of those features would be relevant

3) sparse set of features is determined using hierarchical Bayesian modelling

4) Results of an ARD procedure on
multimodal data collected for building a frustration predictor:
- most discriminative
features: fidgets, velocity of
the head, & ratio of the
postures
- numerous outliers for
fidgets ⇒ unreliable due to
sensor failure & individual
differences

5) All parameters span a range of values ⇒ despite some statistical trends the discriminative power of a feature might depend on each individual learner