Lecture 5

Question 1

Feature extraction

Answer 1

starts from an initial set of measured data and builds derived values (features) intended to be informative and non-redundant, facilitating the subsequent learning and generalization steps, and in some cases leading to better human interpretations.

Question 2

Neural Network approach to construct a non-linear classifier

Answer 2

Uses a large number of simpler activation functions

The functions are fixed (Gaussian, sigmoid, polynomial basis functions)

and optimization involves linear combinations of these fixed functions

Question 3

Artificial Neural Networks

Answer 3

Take inspiration from the brain

define functions that are computed by neurons (units)

and have both input and output units with hidden layers

Question 4

What’s the relationship between the number of hidden layers and the network’s capacity

Answer 4

the capacity of the network increases with more hidden units and hidden layers

Question 5

What are the components of neural networks

Answer 5

An input layer x; independent variable

an arbitrary amount of hidden layers

an output layer y (hat); dependent variable

a set of weights (coefficients) and biases at each layer

and a choice of activation functions for each layer

Question 6

Activation functions

Answer 6

Are applied on the hidden units and achieve non-linearity

Question 7

What are some popular activation functions?

Answer 7

Sigmoid, Tanh, and ReLU (Rectified Linear Unit)

Question 8

What are the components of neural network training?

Answer 8

A forward pass and a backward pass

Question 9

forward pass

Answer 9

performs inference

Question 10

backward pass

Answer 10

performs learning;

is a routine to compute the gradient and uses the chain rule of the derivative of the loss function ;

weights and biases can be changed to reduce error

Question 11

Back propagation

Answer 11

an efficient method for computing gradients needed to perform gradient-based optimization of the weights in a multi-layer network

Question 12

Deep neural network

Answer 12

a multilayer perceptron or a neural network with multiple hidden layers

Question 13

Which of the following introduces non-linearity to a neural network?

Rectified Linear Unit (ReLU) function, Convolution function, or
Stochastic Gradient Descent

Answer 13

Rectified Linear unit, it is a non-linear activation function.