Feedforward Neural Networks 2

Question 1

Node

Answer 1

• Processes input data and contributes to the network’s output
• can be tensor, matrix, vector, or scalar value
• Input can be from features from data or output value of other node

Question 2

Edge

Answer 2

• Represents a function argument
• Pointers to nodes
• Can be adjusted during optimization to make better predictions

Question 3

Epoch

Answer 3

• One full cycle through the entire training dataset
• Adjust parameters (weights and biases) based on the gradients of the loss function

Question 4

Training a Model Steps

Answer 4

1. Define a computation graph (FFNN class)
2. For each epoch:
   a. for each batch of data: compute loss, autograd to compute gradients, and take step with optimizer
   b. evaluate on validation set to avoid overfitting

Question 5

Shuffling the Training Data

Answer 5

• What if “I love you” at end of training set 1000 times
• Parameters will be inaccurately updated
• Randomly shuffling the order at each time step of before training epoch

Question 6

Early Stopping

Answer 6

• Prevents overfitting
• Stop training when performance starts to decline
• Return best parameters

Question 7

Training Tricks

Answer 7

1. Shuffling the training data
2. Early stopping
3. Parameter dropout

Question 8

Parameter Dropout

Answer 8

• Prevents overfitting
• Randomly setting a portion of the model’s parameters (weights) to zero
• Done only at training time

Question 9

Mini-batching

Answer 9

• Splitting the training set into smaller, manageable subsets
• Combines smaller operations into one big one (more efficient computation)

Question 10

Batch Data Loading

Answer 10

• Instead of processing one single sentence, process mini-batch of sentences
• Must perform sentence padding on mini-batches

Question 11

Sentence Padding

Answer 11

• done during batch data loading
• pad shorter sentences in batch to match lengths