VQ-VAE

Question 1

VQ-VAE - what are the 3 main contributions of VQ-VAE in comparison to VAE

Answer 1

1 Quantisation of the latent space
2 Restrict the latent space to a linear combination of a set of vectors (codebook).
3 The prior of the codebook is learned rather than static

Question 2

VAE - what are the 4 main principles that VAE is based on

Answer 2

1 AE framework with continuous latents
2 A Gaussian sampling procedure
3 The prior of the latent vector follows a Gaussian distribution with u=0 and sigma=1
4 a reconstruction loss and a KL divergence loss

Question 3

VQ-VAE - how does it changes the AE framework

Answer 3

There’s a quantisation step in the middle which convert the continuous feature extraction output to a quantised entity

Question 4

VQ-VAE - How does it changes the sampling procedure?

Answer 4

It assumes that the prior is a uniform distribution over the codebook and the posterior of the decoder input is a delta function that gives back the nearest codeword

Question 5

VQ-VAE - what happens to the KL divergence

Answer 5

Becomes constant and is removed from the learning

Question 6

VQ-VAE - What happens to the prior

Answer 6

It is learned in the training process

Question 7

VQ-VAE - what is the loss?

Answer 7

The reconstruction loss, the codebook loss and the commitment loss

Question 8

VQ-VAE - What is the codebook loss?

Answer 8

MSE from the features of the input to the codeword (putting stop-gradient on the input features)

Question 9

VQ-VAE - What is the commitment loss?

Answer 9

The squared distance between the encoder to the stop-gradient of its closest codeword.