L9 - Systems for ML

Question 1

What are the three stages of ML ecosystem? (slide 4)

Answer 1

1. Model development
2. Training
3. Inference

Question 2

What performance metric is important for training?

Answer 2

throughput

Question 3

What performance metric is important for interference?

Answer 3

latency

Question 4

What are the two parts of model development?

Answer 4

1. Data part
2. Model part

Question 5

What needs to be done in data part?

Answer 5

short: data collection, cleaning and visualisation

1. identify sources of data
2. join data from multiple sources
3. clean data
4. plot trends and anomalies

Question 6

What resource is the bottleneck for preprocessing data?

Answer 6

CPU

Question 7

Why do we care about performance of preprocessing?

Answer 7

1. affects the end-to-end training time.
2. consumes significant CPU and power

Question 8

What needs to be done in model part?

Answer 8

short: feature engineering, model design; then training and validation

1. build informative features
2. design new model architectures
3. tune hyperparameters
4. validate prediction accuracy

Question 9

What are deep neural networks?

Answer 9

neural networks with multiple hidden layers

Question 10

What are the three steps of feature extraction and model search?

Answer 10

1. feature extraction
2. model search
3. hyperparameter tuning

Question 11

What are the three steps of DNN training?

Answer 11

1. forward pass: compute activations and loss
2. backward pass: compute gradients
3. update model weights: to minimise loss

there are iterated over the training dataset

Question 12

What are the characteristics of DNN training?

Answer 12

1. computationally intensive
2. error-tolerance
3. large training datasets

Question 13

What is meant under error tolerance?

Answer 13

trade-off some accuracy for large benefits in cost and/or throughput

Question 14

How can be large training datasets be a problem?

How is this solved?

Answer 14

problem: Data often does not fit in memory

solution: overlap data fetching and training computation

Question 15

How is single-node training done for DNNs?

Answer 15

1. data preprocessing done on the CPU
2. DNN training on GPUs/TPUs

Question 16

What two types of parallelism can be achieved via distributed training?

Answer 16

1. data parallelism
2. model parallelism

Question 17

What is data parallelism?

Answer 17

partition the data and run multiple copies of the model

synchronise weight updates

Question 18

Common ways to implement data parallelism?

Answer 18

1. parameter server
2. AllReduce (allows GPUs to communicate faster with each other)

Question 19

What is model parallelism?

Answer 19

partition the model across multiple nodes

note: more complicated

Question 20

What does each worker do when a parameter server is used?

Answer 20

Each worker has a copy of the full model and trains on a shard of the dataset. Each of them compute forward and backwards. At the end they send their gradients to the parameter server.

Question 21

What is the task of the PS itself?

Answer 21

It aggregates the gradients it received from each worker. Then computes the new model weight which is sent to each worker.

Question 22

What is the idea of AllReduce?

Answer 22

• GPU workers exchange updates directly via AllReduce collective
• gradient aggregation is done across workers

Question 23

List three steps of AllReduce.

Answer 23

1. workers computer forward/backward
2. reduce-scatter
3. all-gather

Question 24

Why is AllReduce possible? Seems like a lot of communication…

Answer 24

GPU-GPU interconnects are fast

Question 25

Why might one need model parallelism?

Answer 25

Some models do not fit on one device.

Question 26

What is the problem with traditional model parallelism?

Answer 26

low HW utilisation due to sequential dependencies note: look at slide 33

Question 27

How can we improve it?

Answer 27

with pipeline parallelism note: look at slide 34

Question 28

Can we combine it with data parallelism?

Answer 28

Yes

Question 29

There are two approaches to distributed training. What is synchronous training?

Answer 29

workers operate in lockstep to update the model weights at each iteration

Question 30

What about asynchronous training?

Answer 30

each worker independently updates model weights

Question 31

Which one offers better model quality?

Answer 31

synchronous

Question 32

At what cost?

Answer 32

lower throughput

Question 33

What is the main idea of PipeDream? (async pipeline parallelism)

Answer 33

- workers alternate between forward and backward passes - gradients are used to update model weights immediately note: slide 36

Question 34

What is the issue?

Answer 34

different workers do forward/backward passes on different versions of weights extra: weight stashing is a solution, gotta read a paper for this

Question 35

Is the output of model development the trained model?

Answer 35

No.

Question 36

What can't be done with just a trained model?

Answer 36

1. retrain 2. track data and code (for debugging) 3. capture dependencies 4. audit, e.g. data privacy

Question 37

What is the output of model development?

Answer 37

training pipeline

Question 38

What is done in the training step? (remember: second stage of ML ecosystem)

Answer 38

short: iterate through dataset, compute forward & backward pass on each batch of data to update model parameters 1. train models on live data 2. retrain on new data 3. validate accuracy 4. manage versioning

Question 39

What is done in the inference step? (remember: third and last stage of ML ecosystem)

Answer 39

short: compute forward pass on a (small) batch of data 1. server models with real data (in real-time) 2. embed model serving insides the end-user app 3. optimise for low latency 4. quantise model weights for memory efficiency

Question 40

What are the performance metrics for training?

Answer 40

throughput accuracy

Question 41

And for inference?

Answer 41

throughput accuracy latency

Question 42

What is a tensor?

Answer 42

a multi-dimensional array with elements having a uniform type

Question 43

What is a computation graph?

Answer 43

DAG where nodes are mathematical operations and edges are data (tensors) flowing between operators

Question 44

What is a static computation graph?

Answer 44

The framework first builds the computation graph, then executes it.

Question 45

And dynamic?

Answer 45

The framework builds the computation graph as it is being executed.

Question 46

Advantages of static?

Answer 46

more opportunities to optimise if you a complete view at the beginning

Question 47

Advantages of dynamic?

Answer 47

1. intuitive/familiar execution model for programmers - Can modify and inspect the internals of the graph during runtime, which is helpful for debugging 2. easies support of dynamic control flow

Question 48

Why is debugging important for ML?

Answer 48

1. Need to catch training problems early in a run, because runs are expensive 2. Detecting problems is difficult; bugs manifest as convergence issues, not program crashing

Question 49

What are the challenges for rewinding/replaying ML training?

Answer 49

1. many sources of non-determinism 2. hard to efficiently checkpoint all the states