Unit 5 - Representation Learning

Question 1

What is transfer learning?

Answer 1

Talk in human context

Traditional ML and DL models trained in isolation. We have to train a model in scratch whenever the feature space distribution changes

Idea of overcoming the isolated paradigm

Definition - transfer learning enables to adapt or utilise the knowledge acquired from one set of tasks and/or domains to another related set of tasks and/or domain

Question 2

What to transfer?

Answer 2

To answer this question, it is necessary to know which part of the knowledge can be transferred from the source to the target so that the performance of the target is improved

Question 3

Approaches to “What to transfer”:

Answer 3

Instance transfer: cannot use the source domain data directly. Use certain instances of the source domain.

Feature-representation: to minimise domain divergence and reduce error rates by identifying good feature representations that can be utilised from the source to the target

Parameter: related tasks share some parameters or prior distribution of hyper parameters. Apply additional weightage to the loss function of the target do to improve the performance.

Relational knowledge: handles non-IID data. Each data point has some relationship with other data points, for example social network data

Question 4

When to transfer?

Answer 4

Using transfer knowledge, for the sake of it may make matters worse

So we do not want to degrade the task, performance and aim to utilising class into improve target task, performance or results

Question 5

How to transfer?

Answer 5

When winner answer, think about identifying ways to transfer learning across domains or task

This may mean changes in the existing algorithms are using different techniques

Question 6

Types of transfer learning:

Answer 6

Inductive transfer: source and target domains are same, but the tasks are different from each other. Use of inductive biases to improve target task

Unsupervised transfer: similar to inductive transfer, but focus on unsupervised tasks on the target. No labelled data

Transductive transfer: similarities between source and target tasks, but corresponding domains are different. Source has a lot of label data target has none. Sub categories are based on settings that are either feature. Space is different or marginal probabilities.

Question 7

What is distributed representation?

Answer 7

Important concept in the field of deep learning

Simply a vector that represent some piece of data

Example of examining different types of shapes.

Ability to capture, meaningful semantic similarity between data

Question 8

Variants of CNN: DenseNet

Answer 8

Consist of:
Dense block
DenseNet architecture
Advantages of DenseNet

Question 9

What is dense block?

Answer 9

Uses element by element addition

an algorithm that is handed a state from one ResNet model to another

Each layer receives extra inputs from all the layers that came before it and also transmits it on feature map to all the letters that came after it

Uses concatenation

Generates a more compact and thin network with your channel

Greater memory and processing efficiency

Question 10

Advantages of DenseNet

Answer 10

1. Strong gradient flow
2. Parameter and computational efficiency
3. More diversified features
4. Maintain low complexity features