AutoEncoders

Question 1

AutoEncoder

Components and loss function

Answer 1

An autoencoder is a model designed to encode objects in a low dimensiona space - the latent space.
1. Encoder: reduces the dimensionality of the input while retaining its most valuable features.
2. Bottleneck: the latent vector.
3. Decoder: expands the dimension back using the latent vector so to approximate the original input with the best fidelty.

The loss function is the reconstruction loss.

Question 2

Reconstruction Loss

Answer 2

The l2 norm between the decoded output and the original input.

Insensitive to blurring.

Question 3

How can AutoEncoders be used as generative models?

Answer 3

Decode the interpolation of two encoded latent vectors

Question 4

Why are VAEs required? Why can’t we use regular AEs to generate?

Answer 4

The latent space isn’t necessairly continuous

Question 5

VAE

Variational AutoEncoder

Answer 5

An autoencoder model designed to encode each input as a distribution (paramaterized by mean and variance) which the decoder will later sample to generate the decoded output.
We can also sample from the distribution to generate new samples.

Question 6

What are the loss functions of VAE

Answer 6

1. Reconstruction Loss
2. KL Divergence between the encoded distribution and the N(0, 1) distribution.

Question 7

Downside of VAEs

Answer 7

Generations are blurry, low fidelity. The average of real samples often is out-of-distribution.