M06 - Multimodel Learning, Interaction and Communication

Question 1

What is machine learning?

Answer 1

“The field of machine learning is concerned with the question of how to construct computer programs that automatically improve with experience.”

Question 2

What is the definition of machine learning with E,T and P?

Answer 2

“A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E.”

Question 3

Define E, T and P for a robot learning project.

Answer 3

Task: object recognition with color and depth data
Experience: the iCub multisensor dataset
Performance measure: accuracy, precision, recall, F1 score

Question 4

What is modality?

Answer 4

Sensory data that are associated with different aspects of the observed phenomena

Question 5

Why do we need multimodel learning/integration?

Answer 5

1. To form a robust sensory representation
2. To leverage complementary characteristics of modalities

Question 6

How do you count modalities in a robot?

Answer 6

The number of types of data = The number of modalities

Question 7

What are the 5 challenges in Multimodel Machine Learning?

Answer 7

1. Representation: how to represent multimodal data [pixels, signals, symbols, etc.]
2. Translation: how to map data from one modality to another
3. Alignment:how to identify direct relations between modalities
4. Fusion:how to join information from two or more modalities [data level, decision level, intermediate]
5. Co - learning: how to transfer knowledge between modalities

Question 8

What are the steps in a machine learning pipeline?

Answer 8

1. Preprocessing (dimensionality reduction, features extraction, selection, scaling, sampling, denoising)
2. Learning (Initializing, Optimizing, Cross-Validation)
3. Evaluation (New model)

Question 9

What is the problem between color and depth data?

Answer 9

There is a huge semantic gap between color depth data and raw data matrices

Question 10

How can we extract representations?

Answer 10

• hand-crafted features
• automatic feature learning

Question 11

Which feature selection is quicker?

Answer 11

Automatic feature learning usually finds better solutions than hand-designed ones

Question 12

What are desired specifications of the representations?

Answer 12

1. Similar representation should indicate similar concepts
   [if you visualize the representation space, the distances for different carrots should be close to each other but far from the cars]
2. Representations should be robust
   [The extracted representations should be robust to deal with the noise]
3. We should know how to handle missing data in one modality

Question 13

What is unimodal learning?

Answer 13

Discrete probability distribution of objects

Question 14

What are the problems with unimodal learning?

Answer 14

External reasons:
- noise in the environment
- miscalibrated sensors

Model-related reasons
- wrong model selection
- non-regularized weights
- using raw data as input

Question 15

What are the goals of multimodal learning/integration?

Answer 15

1. To form a robust sensory representations
2. To leverage complementary characteristics of modalities

Question 16

What are the main characteristics of deep multimodel learning?

Answer 16

• both modality-wise representations (features) and shared (fused) representations are learned from data
• requires little or not preprocessing of input data (end-to-end training)
• deeper, complex networks typically require large amounts of training data (if trained from scratch)

Question 17

What are the main characteristics of conventional multimodel learning?

Answer 17

• features are manually designed and require prior knowledge about the underlying problem and data
• some techniques, like early fusions, may be sensitive to data preprocessing
• may not require as much training data

Question 18

What the levels of multimodel fusion techniques?

Answer 18

• data level
• decision level
• intermediate fusion

Question 19

What happens on the data level of multimodal techniques?

Answer 19

Fuse the inputs before performing machine learning

Question 20

What happens on the decision level of multimodal techniques?

Answer 20

Fuse the decisions of each model i.e. outputs of the machines learning algorithm

Question 21

What happens on the intermediate fusion level of multimodal techniques?

Answer 21

Fuse the representations at different levels of the model (you need to understand which part of network encodes what parts of your data) i.e. intermediate layers of the convolutional neural networks

Question 22

What are the assumptions of data level fusion?

Answer 22

conditional independence among modalities, depth and color

Question 23

What does data level fusion do?

Answer 23

• concentrate raw inputs
• reduce dimensions of input
• hand-crafted features
• observing same phenomena but each sensor observes different types

Question 24

What is the output of data level fusion?

Answer 24

Decide on output by looking at the majority or mean or use an algorithm

Question 25

What does decision level fusion do?

Answer 25

• employs different or the same model for different modalities
• collect decisions form separate models trained on different modalities
  fuse them by averaging, sum-them-maximg, or meta learning

Question 26

When do you use decision level fusion?

Answer 26

• the modalities are uncorrelated
• the modalities have different dimensions
• exploit different machine learning model for different modalities
  (CNNs for image, SVM for depth, MLP for touch, etc.)

Question 27

What are fusion methods?

Answer 27

• multimodal deep learnig for robust object recognition
• deep learning-based image segmentation on multimodal medical imaging
• multimodal representation models for prediction and control from information

Question 28

What is intermediate fusion?

Answer 28

• non-hand-crafted features
• fuse similar modalities together
• multi-modal architecture

Question 29

When is it best to use CNN?

Answer 29

CNN is best when you are planning to fuse the output of features

Question 30

What are the implications of multimodal learning results?

Answer 30

1. We form a robust sensory representation
2. We leverage complementary characteristics of modalities