Feedforward Neural Networks 1

Question 1

Neural Network

Answer 1

• Uses only one hidden layer
• Splits complex (non-linearly separable) feature spaces (can’t be done with linear functions)
• Used in simple problems where relationship between input/output is straightforward

Question 2

Why can’t we use Linear Model for everything?

Answer 2

Linear models can’t learn FEATURE COMBINATIONS.

Question 3

Linear Features

Answer 3

• treating each feature independently (not connected to any other features)
• in text classification, prob/weight of each word is calculated separately and later summed
• uses logistic regression

Question 4

Feature Combinations

Answer 4

• Process of combining two or more features to create NEW, MORE COMPLEX features
• More complex, involving non-linear interactions, polynomial features, or higher-order relationships

Question 5

XOR

Answer 5

• Simple non-linear function
• Demonstrates that linear models cannot solve non-linearly separable problems.

Question 6

Deep Neural Networks

Answer 6

• Uses two or more hidden layers
• Can split even more complex feature spaces
• Used for tasks requiring learning from large amounts of unstructured data

Question 7

Immediate Conjunctive Features

Answer 7

• Specific combination of two or more features to create a NEW feature
• Simple and direct combinations of existing features

Question 8

softmax()

Answer 8

• Converts raw scores from output layer into probabilities that sum to one
• Works by first exponentiating each element in the input vector and then normalizes them
• “Different weights, same feature”

Question 9

Feedforward Neural Network

Answer 9

• Type of neural network where connections between nodes do not form cycles (can’t go backwards)
• Neurons in each layer are fully connected to neurons in next layer
• Each neuron applies an activation function
• After producing the output, the network computes a loss function and backpropagates

Question 10

Loss Function

Answer 10

Quantifies the difference between the predicted output and the actual target

Question 11

Backpropagation

Answer 11

Gradients of the loss function are calculated and used to adjust the weights in the opposite direction

Question 12

Log Likelihood

Answer 12

• Simply the natural logarithm of the likelihood function.
• Probability distributions involve products of probabilities.
• Log transforms these into sums, simplifying the optimization process.

Question 13

Likelihood

Answer 13

• Quantifies how well model explains observed data given certain parameters
• Measures the probability of observing the data under the model

Question 14

Why maximize log likelihood?

Answer 14

• Helps to find neural network parameters that best explain the observed data
• Higher values indicate that model predicts observed data more accurately, making it useful for model evaluation
• Incorporates regularization to prevent overfitting

Question 15

Gradient of the Loss Function

Answer 15

• Optimization algorithm
• Provides info on how to adjust parameters to minimize the loss
• If gradient is positive, indicates that increasing parameters will increase loss
• If gradient is negative, increasing parameters will decrease loss