Latency

Question 1

Latency

Answer 1

Refers to the time delay between the input being fed into the NN and the output being produced (inference).

Question 2

7 factors that affect latency

Answer 2

• Number of layers in NN
• Number of parameters
  (These both increase computation time)
• The type of hardware used.
• The memory bandwidth
• Size of batch (the larger the batch, higher the latency)
• Data transfer speed between storage, memory, and processing units.
• Specifically in distributed systems, the transfer speed between different nodes on a network

Question 3

______ & _______ can _______ latency by using their __________ processing capabilities

Answer 3

TPUs, GPUs, reduce, parallel

Question 4

Batch size

Answer 4

The number of data points from the training dataset processed together in a single iteration before updating the model’s parameters.

Question 5

4 factors in how the training dataset can affect latency

Answer 5

• If the training dataset is very large, it may lead to a complex model with many parameters, increasing latency

-The noisier the data, the more post-processing steps required to filter low-quality predictions, increasing latency

• Lack of diversity in the dataset will mean the model struggles with unfamiliar inputs, potentially requiring additional processing so will increase latency
• Complex features such as complex patterns and dependencies can require more computational resources and therefore, increase latency

Question 6

Problem of Machine Learning Dependencies in feeding data

Answer 6

This is an issue in machine learning that arises from the fact that when feeding data from data sources into the machine learning model, there are a series of components/processes the data must travel through before reaching the model, and we cannot do these things in parallel. This creates latency.

These components/processes include:

Data source –> Ingestion/Integrated –> Data store -> Cleaning and pre-processing –> Machine Learning Model

Question 7

Critical Path Algorithm

Answer 7

This is used to mitigate the problem of machine learning dependencies when feeding data.

It is essentially a project management tool that will identify the longest sequence of dependent tasks (the critical path), and help in optimizing it by identifying bottlenecks to produce the lowest completion time for a project.

Question 8

Natural Language Understanding (NLU)

Answer 8

Structured sequence of machine learning models and algorithms that work together to gradually transform and understand user input.

Each step in the sequence, referred to as the pipeline, contributed to a deeper linguistic understanding of the input, allowing the chatbot to generate accurate and contextually appropriate responses.

Question 9

Natural Language Understanding 3 advantages

Answer 9

• Accuracy. Each process in the NLU pipeline enhances understanding of the input data leading to a better output
• Efficiency. Since the pipeline is modular, each section can be individually focused on for optimization.
• Adaptability. The modular nature of the pipeline allows for easy updates to individual components without having to make drastic changes to the entire system.