MSFT specific

Question 1

What improvement was made in the transducer model? What is a problem with our transducer model?

Answer 1

They reduced training time and improved efficiency by solving the memory issue in the speech transducer.

Indeed the transducer model consume a lot of memory and so it is hard to use big training batches

Question 2

Why transducer model consume all this memory?

Answer 2

They contain a 4-D joint output
One popular type of transducer model is the RNN-T (Recurrent Neural Network Transducer) model. The 4-D joint output in such models refers to the probability distribution over the combined set of input and output labels at each time step.

The 4-D joint output can be represented as P(c, l | x) where:
c: Represents the output label (characters or subword units)
l: Represents the length of the output sequence
x: Represents the input audio features
P(c, l | x): Represents the probability of a certain output label and sequence length given the input audio features.

The joint output is generated using a combination of the encoder and predictor networks. The encoder processes the input audio features, while the predictor generates a probability distribution over the possible output labels. The joint output is then computed as the outer product of these two probability distributions, producing a matrix that represents the probabilities of all possible combinations of input and output labels at each time step.

The 4-D joint output is used to compute the loss function during training and to search for the most likely output sequence during inference. This is typically done using techniques like beam search, greedy search, or dynamic programming algorithms such as the forward-backward algorithm.

Question 3

Is it always the case that RNN-T consumer more memory?

Answer 3

Not necessarily, it depends on the cases and architecture under consideration

• 4-D joint output is specific to RNN-T (Recurrent Neural Network Transducer) models in end-to-end (E2E) speech recognition.
• It represents the joint probability distribution over input and output labels at each time step.
• Memory footprint may be larger compared to some other E2E architectures due to the joint probability matrix.
• Listen-Attend-Spell (LAS) models have a different memory footprint, affected by the size of context vectors and attention mechanism’s internal states.

The 4-D joint output specifically refers to the output of the RNN-T (Recurrent Neural Network Transducer) model, which is an end-to-end (E2E) speech recognition model. It is different from other E2E speech models because it represents the joint probability distribution over both the input and output labels at each time step, whereas other models may have different output representations.

The memory footprint of an RNN-T model with a 4-D joint output may be larger compared to some other E2E architectures, mainly because of the matrix that stores the joint probabilities for every combination of input and output labels at each time step. However, the increase in memory footprint depends on the specific architecture and implementation details.

For example, in Listen-Attend-Spell (LAS) models, which are also E2E speech recognition models, the attention mechanism computes a context vector at each time step that is used to generate the output sequence. The memory footprint of LAS models is affected by the size of the context vectors and the attention mechanism’s internal states, which may be smaller or larger than the 4-D joint output of RNN-T models depending on the specific configuration.

Overall, whether the 4-D joint output increases the memory footprint of an RNN-T model compared to other architectures depends on several factors, such as the size of the input and output label sets, the number of time steps, and the complexity of the encoder and predictor networks. However, it is worth noting that modern hardware and optimized software implementations can help mitigate the memory footprint and computational cost of handling 4-D joint outputs in RNN-T models.

Question 4

What is one architecture option, deployed for the windows team that needs models <200MB for their ASR model?

Answer 4

They are deploying residual Transformer models. Of course there are other ways to make the model smaller without compromising the performance.