lecture 5 - neural networks

Question 1

What is the curse of dimensionality in the context of linear models?

Answer 1

As the number of features (or dimensions) increases, the amount of data needed to properly train a linear model grows exponentially, making data sparse in high-dimensional spaces.

Question 2

Why do linear models struggle in high-dimensional spaces?

Answer 2

They depend on predefined basis functions to represent the data, but in high-dimensional spaces, the number of basis functions explodes, making the model computationally expensive and prone to overfitting.

Question 3

What are the main solutions to the curse of dimensionality for linear models?

Answer 3

1. Support Vector Machines (SVMs)
2. Neural Networks (NNs)

Question 4

How do Support Vector Machines (SVMs) help address the curse of dimensionality?

Answer 4

SVMs define basis functions centered on relevant training points (support vectors), simplifying the model by using only relevant points to construct the decision boundary.

Question 5

How do Neural Networks (NNs) help address the curse of dimensionality?

Answer 5

NNs use basis functions (hidden layers) that are learned and adapted during training, allowing the model to capture complex relationships flexibly.

Question 6

How are the basis functions in Neural Networks structured and optimized?

Answer 6

The structure of basis functions (e.g., number of neurons and activation functions) is fixed, but their weights are adaptive and optimized during training to discover useful patterns in the data.

Question 7

What is the goal of a neural network?

Answer 7

The goal of a neural network is to build a more powerful representation of the data by applying multiple functional transformations in sequence.

Question 8

What is the first step in constructing a neural network?

Answer 8

Start with a linear model that combines input features using weights and biases to produce a linear output.

Question 9

How are new features (hidden units) created in a neural network?

Answer 9

New features are created by constructing linear combinations of the input features, followed by applying an activation function to introduce nonlinearity.

Question 10

Why are nonlinear activation functions applied in neural networks?

Answer 10

Nonlinear activation functions (e.g., ReLU or sigmoid) are applied to introduce nonlinearity, enabling the network to model complex relationships.

Question 11

How are additional layers added in a neural network?

Answer 11

After the first layer, the outputs of the hidden units are combined and passed through additional layers, each applying new weights and activation functions.

Question 12

How is the final output of a neural network computed?

Answer 12

The final output is computed by applying another activation function (e.g., sigmoid or softmax) to the output of the last layer.

Question 13

What is the purpose of using multiple layers in a neural network?

Answer 13

Multiple layers enable the network to learn hierarchical representations, with each layer capturing increasingly complex patterns in the data.

Question 14

Which activation function is typically used for regression in the output layer of a neural network?

Answer 14

The identity function is used for regression tasks, as it outputs continuous real-valued numbers.

Question 15

Which activation function should be used for binary classification in the output layer?

Answer 15

The sigmoid function is used for binary classification, as it maps outputs to probabilities between 0 and 1

Question 16

Which activation function is suitable for multi-class classification in the output layer?

Answer 16

The softmax function is used for multi-class classification, as it converts logits into probabilities across multiple classes.

Question 17

What happens when a linear function is used as the activation function in hidden layers?

Answer 17

It results in a linear model, which cannot handle complex relationships in the data, limiting the network’s capability.

Question 18

What is the role of the sigmoid function in hidden layers?

Answer 18

The sigmoid function smoothly squashes inputs to a range between 0 and 1, introducing nonlinearity.

Question 19

Why is the tanh function commonly used in neural networks?

Answer 19

The tanh function squashes inputs between -1 and 1, is centered around zero, and was often used in early neural networks for better gradient flow than the sigmoid function.

Question 20

What are the advantages of using ReLU as an activation function?

Answer 20

ReLU outputs zero for negative inputs and the input itself for positive inputs. It is computationally efficient and helps avoid the vanishing gradient problem.

Question 21

What is the difference between ReLU and Leaky ReLU?

Answer 21

Leaky ReLU allows small negative values for negative inputs, preventing dead neurons and improving gradient flow.

Question 22

output layer activation functions

Answer 22

1. identity (regression)
2. sigmoid (binary classification)
3. softmax (multi-classification)

Question 23

hidden layer activation functions

Answer 23

1. linear function (limited)

nonlinear functions

1. sigmoid
2. tanh
3. relu
4. leaky relu

Question 24

What does it mean when we say that neural networks are universal function approximators?

Answer 24

It means that neural networks can approximate any function given enough data, layers, and hidden units.

Question 25

What types of functions can neural networks approximate?

Answer 25

1. quadratic functions 2. sine waves 3. absolute value functions 4. step functions.

Question 26

What kind of shape does a neural network approximate for a quadratic function?

Answer 26

A neural network approximates a **parabolic** shape for a quadratic function.

Question 27

How does a neural network approximate a sine wave function?

Answer 27

It approximates the sine wave by learning the oscillatory pattern of the function.

Question 28

What is the behavior of a neural network when approximating an absolute value function?

Answer 28

The network approximates a **V-shaped** function when approximating an absolute value function.

Question 29

How does a neural network handle step functions?

Answer 29

Neural networks approximate step functions by learning discrete jumps between values, simulating thresholding behavior.

Question 30

Why might a simple linear decision boundary not work well for some datasets?

Answer 30

A simple linear decision boundary may not work well **when the data is not linearly separable**, requiring a more complex decision-making model like a neural network.

Question 31

What simple functions do neural networks use as building blocks?

Answer 31

Neural networks use simple functions like AND, OR, and other **logical gates** as building blocks to model complex decision boundaries.

Question 32

How does a neural network solve the XOR problem?

Answer 32

By combining AND and the inverted OR gates, and then applying an OR gate to both, a neural network can perfectly separate points into two classes, solving the XOR problem.

Question 33

What is the significance of solving the XOR problem in neural networks?

Answer 33

Solving the XOR problem demonstrates that neural networks can model complex, non-linear relationships, unlike simple linear models.

Question 34

What was the contribution of Pitts and McCulloch (1943) to neural networks?

Answer 34

They developed the first mathematical model of biological neurons, showing that Boolean operations could be implemented by neuron-like nodes, which became the origin of automaton theory.

Question 35

What is Hebb’s rule (1949) in the context of neural networks?

Answer 35

Hebb’s rule states that the connection strength between two neurons increases when both neurons are activated simultaneously.

Question 36

What was Rosenblatt’s contribution to neural networks in 1958?

Answer 36

Rosenblatt introduced the perceptron, a network of threshold nodes for pattern classification, and formulated the perceptron convergence theorem.

Question 37

How did neural networks experience renewed interest in the 1980s and 1990s?

Answer 37

New techniques like backpropagation, physics-inspired models (Hopfield net, Boltzmann machines), and unsupervised learning led to impressive applications, such as character recognition and speech recognition.

Question 38

What modern techniques have driven the "revolution" in neural networks since the 1990s?

Answer 38

Modern techniques include 1. support vector machines 2. kernel methods 3. ensemble methods (bagging, boosting, stacking) 4. deep learning 5. transparency methods like LIME and SHAP.

Question 39

What are weights space symmetries in neural networks?

Answer 39

Weights space symmetries refer to the number of possible configurations in the weight space that can arise due to the nature of activation functions and the architecture of the network.

Question 40

How does symmetry in the activation function create redundancy in the weight space?

Answer 40

Symmetric activation functions like tanh produce the same output when the input sign is flipped, allowing multiple weight configurations (e.g., positive and negative signs) to result in the same error during training.

Question 41

What is the impact of symmetry in hidden units on a neural network?

Answer 41

Hidden units can be reshuffled without changing the output of the network, creating permutational symmetries.

Question 42

Why do weights space symmetries pose a problem during optimization?

Answer 42

Weights space symmetries create a complex optimization landscape with many local minima, making it harder to find the global minimum during training.

Question 43

What is the total number of symmetries

Answer 43

- M!2^M - M! = number of ways M hidden units can be reshuffled - 2^M = accounts for flipping the sign of the weights

Question 44

What are the three main cases for choosing weights to minimize the error function?

Answer 44

1. Regression: Linear outputs and sum-of-squares error (SSE). 2. Binary classification: Logistic sigmoid and cross-entropy error. 3. Multiclass classification: Softmax outputs and multiclass cross-entropy error.

Question 45

What is the output type and error function used for regression?

Answer 45

1. Model the output using a Gaussian distribution. 2. Define the likelihood function. 3. Take the negative logarithm of the likelihood to get the error function. 4. Apply (stochastic) gradient descent to minimize the error function.

Question 46

How can error functions based on maximum likelihood (ML) be written?

Answer 46

Since data points are independent, the total error E(w) can be expressed as the sum of individual errors E_n(W) for each data point.

Question 47

What is stochastic gradient descent (SGD)?

Answer 47

SGD is a version of gradient descent that updates weights using only one data point at a time, rather than the entire dataset.

Question 48

How is the gradient of the error function with respect to weights calculated in a simple linear model?

Answer 48

- The gradient is computed by multiplying the difference between the predicted output and the target output with the corresponding input feature. - dE_n/dw_{ji} = (y_nj - t_nj) x_ni - dE_n/dw_{ji} = δ_j * a_i

Question 49

How do the concepts for training a simple linear model generalize to multiple layers in a neural network?

Answer 49

The error gradient is propagated backward through the layers, with each layer computing the gradient with respect to its weights using the chain rule.

Question 50

derivative of tanh function

Answer 50

1-tanh()

Question 51

What is the goal of the forward pass in backpropagation?

Answer 51

The goal is to compute activations (outputs) for each layer by applying an input vector and propagating it forward through the network.

Question 52

How is the error calculated during backpropagation?

Answer 52

- The error is calculated by evaluating the difference between the predicted output (y_k) and the target output (t_k) for each output unit, resulting in an error term (δ_k). - δ_k = (y_k - t_k)

Question 53

What is the purpose of backpropagating the error?

Answer 53

The purpose is to propagate the error backward through the network **using the errors from the next layer** and the weights connecting them to update the network’s weights accordingly.

Question 54

How is the error term for a hidden unit (δ_j) obtained during backpropagation?

Answer 54

- The product of the derivative of the activation function and the sum of the weighted errors from the next layer. - δ_j = h'(z_j) * SUM(w_kj * δ_k)

Question 55

How is the derivative with respect to the first set of weights calculated during backpropagation?

Answer 55

The derivative is calculated by multiplying the error term (δ_j) with the activation from the previous layer (a_i)

Question 56

What does the Jacobian matrix represent in neural networks?

Answer 56

- The Jacobian matrix represents the partial **derivatives of each output with respect to each input**, - This helps in sensitivity analysis of the network’s outputs to its inputs.

Question 57

What does the Hessian matrix represent in neural networks?

Answer 57

- The Hessian matrix represents the second-order partial **derivatives of the error with respect to the weights**, indicating the curvature of the error surface.

Question 58

What are some applications of the Hessian matrix in neural networks?

Answer 58

1. Non-linear optimization techniques. 2. Re-training feed-forward networks. 3. Pruning by removing less significant weights. 4. Laplace approximation for Bayesian networks.

Question 59

What happens when the number of hidden units (M) increases in a neural network?

Answer 59

Increasing the number of hidden units increases the flexibility of the model, **reducing bias but increasing variance**, leading to a higher risk of overfitting.

Question 60

How many weights are there in a neural network with one hidden layer?

Answer 60

The total number of weights is (D+1)M+(M+1)K - D is the number of input features. - M is the number of hidden units. - K is the number of output nodes.

Question 61

Why does increasing the number of layers and weights in a neural network lead to overfitting?

Answer 61

Increasing the number of layers and weights rapidly increases the number of parameters, making the model more prone to overfitting on the training data.

Question 62

Why is regularization necessary in neural networks?

Answer 62

Regularization is necessary because overfitting can occur early due to the large number of parameters, especially when the model has many hidden units or layers.

Question 63

How can the number of hidden nodes be evaluated using validation and random restarts?

Answer 63

By plotting a graph of performance using random initializations of weights with different numbers of hidden units, and **selecting the configuration with the smallest generalization error on the validation set**.

Question 64

Why is it important to use multiple random starts when evaluating a neural network?

Answer 64

Multiple random starts ensure that the performance of the network is not dependent on a specific random initialization, leading to more reliable results.

Question 65

What does low variation in the outcomes indicate when evaluating hidden units?

Answer 65

Low variation indicates high bias and low variance, meaning the model may underfit the data.

Question 66

What does high variation in the outcomes indicate when evaluating hidden units?

Answer 66

High variation indicates low bias and high variance, meaning the model may overfit the data.

Question 67

Why is it important for reproducible results to use a specific random seed?

Answer 67

Using a specific random seed ensures that the algorithm’s performance can be reproduced consistently, reducing variability due to random initialization.

Question 68

What is weight decay regularization in neural networks?

Answer 68

- Weight decay regularization modifies the error function by adding a penalty term for large weights - Higher λ implies stronger penalization of large weights.

Question 69

What are the main shortcomings of weight decay regularization?

Answer 69

- It is not invariant to scaling and translations. - When input data is scaled (e.g., multiplied by a constant) or shifted (e.g., adding a bias), the impact of weight decay changes unpredictably. - Simple weight decay applies the same regularization factor to all weights, which doesn't account for the different effects that transformations have on input and output, leading to biased results.

Question 70

What is the new regularization term introduced for neural networks?

Answer 70

- The new regularization term involves regularizing different weight groups separately - W_1 (e.g., input weights) and W_2 (e.g., output weights) are regularized independently with coefficients λ_1 and λ_2 to account for differences in their roles across layers.

Question 71

Why is the new regularization prior considered improper?

Answer 71

- Simple weight decay related to a Gaussian prior - Since the new prior cannot be normalized, it leads to difficulties in 1. Selecting regularization coefficients α_1 and α_2 2. Comparing different models effectively. 3. Requiring separate priors for bias weights.

Question 72

What is an alternative to regularization in neural networks?

Answer 72

1. Early stopping 2. Dropout

Question 73

How does early stopping work?

Answer 73

While training, the training error decreases continuously, but the validation error decreases initially and then increases when overfitting begins. Early stopping halts training at the point where validation error is minimized, ensuring good generalization.

Question 74

How does dropout work?

Answer 74

involves randomly deactivating neurons during training to prevent overfitting by reducing co-adaptation of neurons.

Question 75

What are activation functions in neural networks, and where are they used?

Answer 75

Activation functions are used to introduce non-linearity into the model, allowing it to learn complex patterns.

Question 76

What is the sigmoid activation function and its output range?

Answer 76

- The sigmoid function squashes the input z into a range between 0 and 1, making it useful for binary classification problems. - s-shaped graph

Question 77

What are the pros and cons of using the sigmoid function?

Answer 77

**pro** - easy to understand and implement **cons** - suffers from the vanishing gradient problem for large and small z - outputs are not zero-centered, which can lead to slower convergence during training

Question 78

What problem arises when scaling data centered around 0 with the sigmoid function?

Answer 78

Since the sigmoid function outputs values between 0 and 1, any negative values become positive after applying the function, losing the centeredness of the data.

Question 79

How does the hyperbolic tangent function differ from the sigmoid function?

Answer 79

The hyperbolic tangent function has an output range of [−1,1], making it zero-centered and better for optimization compared to the sigmoid.

Question 80

What are the pros and cons of the hyperbolic tangent function?

Answer 80

**pros** - outputs a wider range ([-1,1]) - zero-centred, aiding in gradient-based optimization **cons** - Suffers from the vanishing gradient problem for large or small inputs.

Question 81

What is an advantage of using the hyperbolic tangent function over the sigmoid function?

Answer 81

It better retains the **centeredness and scale** of the data, improving optimization during training.

Question 82

What is the drawback of the hyperbolic tangent function?

Answer 82

Despite its advantages, it involves a costly computation, requiring four exponentials.

Question 83

What is the ReLU activation function, and how does it behave?

Answer 83

- The ReLU function outputs z if z>0 and 0 otherwise. - It introduces sparsity in the network by setting many neurons to zero.

Question 84

What are the pros and cons of using ReLU?

Answer 84

**pros** - computationally efficient - helps mitigate vanishing gradient problem **cons** - Can suffer from the dying ReLU problem, where neurons output 0 for all inputs and stop updating during training.

Question 85

How does the leaky ReLU function differ from the standard ReLU?

Answer 85

Unlike standard ReLU, leaky ReLU allows a small slope (e.g., 0.01) for z<0 instead of outputting zero, addressing the dying ReLU problem.

Question 86

What are the pros and cons of leaky ReLU?

Answer 86

- pro: Addresses the dying ReLU problem by allowing small gradients for negative inputs. - con: Slightly more computationally expensive than standard ReLU.

Question 87

What issue remains with leaky ReLU?

Answer 87

It is still non-differentiable at z=0, though this rarely causes issues in practice.

Question 88

Why is initializing weights to zero a problem in neural networks?

Answer 88

- If all weights are initialized to zero (or the same value), every neuron in the same layer will compute the same output during forward propagation, leading to **identical gradients during backpropagation**. - This symmetry prevents the network from learning useful representations, reducing it to an ineffective structure.

Question 89

What is Xavier initialization used for?

Answer 89

Xavier initialization is used for networks with sigmoid or tanh activations. It ensures that activations are neither too small nor too large, avoiding the vanishing or exploding gradient problem.

Question 90

What is He initialization, and when is it used?

Answer 90

He initialization is preferred for ReLU activations. It ensures that activations are scaled appropriately for ReLU's non-linear nature, reducing issues like vanishing gradients.

Question 91

What is invariance in the context of neural networks?

Answer 91

Invariance refers to the property where a model produces **consistent outputs even when the input undergoes certain transformations**, such as rotation, scaling, or translation.

Question 92

Why is invariance important in classification problems?

Answer 92

Invariance ensures that models can classify inputs correctly regardless of transformations. Examples: 1. Handwriting recognition: Digits should have the same classification despite changes in position or size. 2. Speech recognition: Invariant to non-linear warping that preserves temporal ordering (e.g., speed or accent variations).

Question 93

What are the challenges in handling invariances using large datasets?

Answer 93

1. It requires a large sample set where all possible transformations are present, which is impractical because the **dataset size grows exponentially**. 2. Translating, rotating, and scaling every image drastically increases the number of training examples, making it **computationally inefficient**.

Question 94

What is an alternative to relying on larger datasets for invariance?

Answer 94

Instead of relying on large datasets, models can be designed to inherently exhibit the required invariances through appropriate design.

Question 95

four approaches to handling invariance

Answer 95

1. augment the training set 2. regularization-based invariance 3. preprocessing to extract invariant features 4. architectural design

Question 96

explain augmenting the training set to handle invariance

Answer 96

Transform the training data by applying variations (e.g., rotating, scaling, translating) to create examples with the desired invariances. - Example: Rotating an image of a digit “5” slightly should still result in the digit being classified as “5”.

Question 97

How does regularization-based invariance work?

Answer 97

Regularization terms are added to the loss function to penalize changes in the model’s output when inputs are transformed. - Example: If a small rotation in input causes a drastic change in output, the penalty ensures the model learns to reduce sensitivity to such changes.

Question 98

How can preprocessing be used to handle invariance?

Answer 98

Preprocessing can extract invariant features before feeding the data into the model. - Example: For images, preprocessing may include converting to grayscale, normalizing intensity, or using handcrafted features (e.g., edge detection).

Question 99

How does architectural design help achieve invariance?

Answer 99

Neural networks can be designed with built-in invariance properties. 1. Convolutional Neural Networks (CNNs): Convolutional layers detect patterns regardless of position, making them inherently invariant to translation. 2. Pooling layers: Enhance translation invariance by summarizing spatial information.

Question 100

hyperbolic tangent equation

Answer 100

[e^z - e^{-z}] / [e^z + e^{-z}]