7. Model Building

Question 1

What is data parallelism?

Answer 1

Data parallelism is when the dataset is split into parts and then assigned to parallel computational machines or graphics processing units (GPUs). A small batch of data is sent to every node, and the gradient is computed normally and sent back to the main node.

Question 2

What are the two strategies in data parallelism?

Answer 2

Synchronous Training: The model sends different parts of the data into each accelerator or GPU. Every GPU has a complete copy of the model and is trained solely on a part of the data.
Asynchronous Training: workers don’t have to wait for each other and all workers are independently training over the input data and updating variables asynchronously.

Question 3

What is “all-reduce sync” strategy good for?

Answer 3

It is great for Tensor Processing Unit (TPU) and one‐machine multi‐GPUs.

Question 4

What is tf.distribute.Strategy used for?

Answer 4

It is a TensorFlow API to distribute training across multiple GPUs, multiple machines, or TPUs.

Question 5

What are the TensorFlow distributed training strategies?

Answer 5

MirroredStrategy: Synchronous distributed training on multiple GPUs on one machine.
CentralStorageStrategy: Synchronous training but no mirroring meaning model variables are kept in the CPU.
MultiWorkerMirroredStrategy: Synchronous distributed training across multiple workers, each with potentially multiple GPUs or multiple machines.
TPUStrategy: Synchronous distributed training on multiple TPU cores.
ParameterServerStrategy: Some machines are designated as workers and some as parameter servers.

Hints: Monkeys Climb More Than Pandas.

Question 6

What is model parallelism?

Answer 6

In model parallelism, every model is partitioned into parts, just as with data parallelism. Each model is then placed on an individual GPU. It can be used to train a model such that it does not fit into just a single GPU.

Question 7

What is the best architecture for synchronous distribution training for TensorFlow?

Answer 7

All-reduce architecture

Question 8

What is the best architecture for asynchronous distribution training for TensorFlow?

Answer 8

Parameter server architecture

Question 9

What are the tools for deploying TensorFlow models?

Answer 9

tf.serving
TFLite for mobile devices
TensorFlow.js for browsers

Question 10

What is Convolutional Neural networks usually used for?

Answer 10

Image classification

Question 11

What is Recurrent Neural networks usually used for?

Answer 11

It is designed to operate upon sequences of data. It can be used for text classification or prediction of values in the sequence, e.g., long short-term memory network. It can also be used for time-series and speech recognition.

Question 12

What do you use to train a neural network?

Answer 12

Stochastic gradient descent

Question 13

What is the goal of training a neuron network?

Answer 13

To find a set of weights and biases that have low loss

Question 14

What is loss in neuron network used for?

Answer 14

The loss is used to calculate the gradients.
Gradients are used to update the weights of the neural network.

Question 15

What are the outputs of regression, binary classification and multiclass classification?

Answer 15

Numerical
Binary
Single label multclass

Question 16

What are the activation functions of regression, binary classification and multiclass classification?

Answer 16

One node with a linear activation unit
Sigmoid activation unit
Softmax activation function

Question 17

What are the loss functions of regression, binary classification and multiclass classification?

Answer 17

MSE
Binary cross-entropy, categorical hinge loss and squared hinge loss (Keras)
Categorical cross-entropy and sparse categorical cross-entropy

Question 18

When do you use sparse categorical cross-entropy and categorical cross-entropy?

Answer 18

Use sparse categorical cross‐entropy when your classes are mutually exclusive and categorical cross‐entropy when one sample can have multiple classes or labels.

Question 19

What is gradient descent?

Answer 19

The gradient descent algorithm calculates the gradient of the loss curve at the starting point.
The gradient of the loss is equal to the derivative (slope) of the curve.
The gradient has both magnitude and direction (vector) and always points in the direction of the steepest increase in the loss function.
The gradient descent algorithm takes a step in the direction of the negative gradient in order to reduce loss.

Question 20

What is learning rate?

Answer 20

Also known as Step size
gradient x learning rate = next point

Question 21

What is batch?

Answer 21

In gradient descent, a batch is the total number of examples you use to calculate the gradient in a single iteration.

Question 22

What is batch size?

Answer 22

Batch size is the number of examples in a batch.

Question 23

What is epoch?

Answer 23

An epoch means an iteration for training the neural network with all the training data. A forward pass and a backward pass together are counted as one pass.

Question 24

What are the examples of hyperparameter in neuron network?

Answer 24

Loss, learning rate, batch size, and epoch are hyperparameters

Question 25

What are the characteristics of different batch sizes?

Answer 25

Larger batch sizes take less time to train but are less accurate.
If batch size is too small, training will bounce around; if it’s too large, training will take a very long time.
Large batch size can lead to out of memory error while training neural networks.

Question 26

What do small and large learning rate mean?

Answer 26

If the learning rate is too small, training will take ages; if it’s too large, training will bounce around and ultimately diverge.

Question 27

What is transfer learning?

Answer 27

Transfer learning is an optimization to save time or get better performance. You can use an available pretrained model, which can be used as a starting point for training your own model.
Transfer learning can enable you to develop models even for problems where you may not have very much data.

Question 28

When do you use semi-supervised learning?

Answer 28

When you don’t have enough labeled data to produce an accurate model and you don’t have the resources to get more data, you can use semi‐supervised techniques to increase the size of your training data.

Question 29

What is the limitation of semi-supervised training?

Answer 29

If the portion of labeled data isn’t representative of the entire distribution, the approach may fall short.

Question 30

What is data augmentation?

Answer 30

To get more data to train the neural networks, you need to make minor alterations to your existing dataset such as flips or translations or rotations.

Question 31

What are the two types of data augmentations?

Answer 31

Offline: All augmentations are done before training
Online: augmentations are done on-the-fly and it is preferred for large datasets.

Question 32

What is bias?

Answer 32

Bias is the difference between the average prediction of our model and the correct value we are trying to predict. It is actually the error rate of the training data.

Question 33

What is variance?

Answer 33

The error rate of the testing data is called variance.
A model with high variance pays a lot of attention to training data and does not generalize on the data it hasn’t seen before.

Question 34

What is bias variance trade-off

Answer 34

You need to find the right balance without overfitting or underfitting the data. If your model is too simple and has very few parameters, then it may have high bias and low variance. If our model has a large number of parameters, then it’s going to have high variance and low bias.

Question 35

What is underfitting?

Answer 35

An underfit model fails to sufficiently learn the problem and performs poorly on a training dataset and does not perform well on a test or validation dataset.
An underfit model has high bias and low variance.

Question 36

What are the reasons for model underfitting?

Answer 36

Data used for training is not cleaned.
The model has a high bias.

Question 37

How to reduce underfitting?

Answer 37

Increase model complexity.
Increase the number of features by performing feature engineering.
Remove noise from the data.
Increase the number of epochs or increase the duration of training to get better results.

Question 38

What is overfitting?

Answer 38

The model learns the training data too well and performance varies widely with new unseen examples or even statistical noise added to examples in the training dataset. An overfit model has low bias and high variance.

Question 39

How to reduce overfitting?

Answer 39

Reduce overfitting by training the network on more examples.
Reduce overfitting by changing the complexity of network structure and parameters.

Question 40

How to avoid overfitting?

Answer 40

Regularization technique
Dropout: Probabilistically remove inputs during training.
Noise: Add statistical noise to inputs during training.
Early stopping: Monitor model performance on a validation set and stop training when performance degrades.
Data augmentation.
Cross‐validation.

Hints: Dolphins Never Eat Apple Cores.

Question 41

What is regularization?

Answer 41

It tunes the loss function by adding a penalty term that prevents excessive fluctuation of the coefficients, thereby reducing the chances of overfitting.

Question 42

What is L1 regularization?

Answer 42

L1 regularization shrinks the parameters toward 0.
Penalizes the sum of absolute values of the weights.
Built‐in feature selection
Robust to outliers
Reduce model size

Question 43

What is L2 regularization?

Answer 43

L2 (ridge) forces weights to be small
Penalizes the sum of squares of the weights
Doesn’t perform feature selection
Not robust to outliers
Improve generalization in linear models

Question 44

What are the common issues for neural network training?

Answer 44

Exploding gradients: Too large to converge. Use batch normalization and lower learning rate
Dead ReLU units: Once the weighted sum for a ReLU unit falls below 0, the ReLU unit can get stuck. Lowering the learning rate can help keep ReLU units from dying.
Vanishing gradients: The gradients for the lower layers (closer to the input) can become very small. When the gradients vanish toward 0 for the lower layers, these layers train very slowly or they do not train at all. The ReLU activation function can help prevent vanishing gradients.
Dropout regularization to prevent overfitting: This type of regularization is useful for neural networks. It works by randomly “dropping out” unit activations in a network for a single gradient step. The more you drop out, the stronger the regularization: 0.0 = No dropout regularization.
1.0 = Drop out everything. The model learns nothing.
Values between 0.0 and 1.0 = More useful.

Question 45

How to reduce training loss?

Answer 45

If the features don’t add information relative to existing features, try a different feature.
Decrease the learning rate.
Increase the depth and width of your layers
If you have lots of data, use held‐out test data.
If you have little data, use cross‐validation or bootstrapping.

Question 46

What are the possible reasons for model training not converging and bouncing around?

Answer 46

Features might not have predictive power.
Raw data might not comply with the defined schema.
Learning rate seems high, and you need to decrease it.
Reduce your training set to few examples to obtain a very low loss.
Start with one or two features (and a simple model) that you know have predictive power and see if the model overperforms your baseline.