Neural Network Basic Ingredienrs

Question 1

Activation functions are typically what type of functions?

Answer 1

Non linear functions

Question 2

List three example activation functions

Answer 2

Sigmoid, ReLU, tanh

Question 3

The softmax function is a generalization of what?

Answer 3

The logistic function, but to multiple dimensions

Question 4

the softmax function is also known as what?

Answer 4

Softargmax or normalized exponential function

Question 5

Softmax does what?

Answer 5

It converts a vector z of K real numbers (e.g either positive or negative(and they don’t all have to be the same sign (eg - or +))) into a vector of K probabilities s (aka a probability distribution), where the mapping between z and s is order preserving and where all the elements in s sum to one .

Question 6

What is forward propagation?

Answer 6

The process of a neural network generating an output for a given input

Question 7

Explain the specificity of the term back propagation

Answer 7

Backpropagation is actually only the process of calculating the gradient of the loss function with respect to the weights, iteratively using the chain rule. Formally speaking, It doesn’t refer to how the gradient is used. However, loosely speaking it is often used loosely to refer to the entire learning algorithm, including how the gradient is used (such as by stochastic gradient descent)

Question 8

Stochastic gradient descent

Question 9

Rprop

Answer 9

Resilient backpropagation

Question 10

Batch

Answer 10

In gradient descent, a batch is the total number of examples [by examples they mean training samples, right?] you use to calculate the gradient in a single iteration.

Question 11

Batch size for stochastic gradient descent

Answer 11

1

Question 12

Convex problems

Answer 12

have only one minimum; that is, only one place where the slope (of the loss function (where the independent variable is the weight(s)) is exactly 0

Question 13

Ln(1)

Answer 13

0

Question 14

Cross entropy

Answer 14

A measure for calculating how good a classifier is. More specifically when trying to classify the class of some item, we declare a p(x) to be the true probability distribution for if some item is class x in a discrete set of x ∈ X. This p(x) is usually expressed as a one hot vector <0 0 0 0 1.0 0 0> if we say the class set is of size 7. Our model is going to predict some distribution q(x)

Question 15

What does the graph of a logistic function look like?

Answer 15

Question 16

What are the domain and range of the sigmoid function?

Answer 16

Domain: -infinity to infinity
Range : 0 to 1

Question 17

Why is argsoftmax considered the generalization of the logistic function to multiple dimensions?

Answer 17

Logistic function takes a scalar of any real value and converts it into another scalar between the range of 0 and 1. Argsoftmax takes a vector (of d dimensions) of any real values and converts it into another vector (of d dimensions) whose values are all in the range of [0,1]

Question 18

When is the softmax function used?

Answer 18

is used as the activation function in the output layer of neural networks that are performing multi-class classification problems

Question 19

What is the formula for the Jacobian of the softmax function?

Answer 19

Question 20

Categorical cross entropy loss is also called what?

Answer 20

Softmax loss

Question 21

Classification problems can be subdivided into what two categories?

Answer 21

Multi-class classification and multi-label classification

Question 22

Explain the difference between multi-class and multi-label classification

Answer 22

multi-class classification - each sample belongs to only one class (mutually exclusive)

multi-label classification - each sample may belong to multiple classes (or to no class)

Question 23

Kronecker delta function

Answer 23

Question 24

List four benefits of ReLU neurons

Answer 24

1) helps prevent vanishing gradients 2) its idempotent
3) it creates sparse activations
4) it’s faster to compute in both forward and backward passes

Question 25

What is a downside of using ReLUs?

Answer 25

Dying ReLUs

Question 26

What would three consecutive sigmoid activations looks like?

Question 27

what would ReLU(ReLU(ReLU(2))) yield?

Answer 27

2

Question 28

Principal difference between sigmoid and tanh?

Answer 28

Sigmoid range is (0, 1). Tanh range is (-1,1)

Question 29

Sigmoid formula

Answer 29

1/(1+e^(-x))

Question 30

LU in ReLU stands for what?

Answer 30

Linear Unit

Question 31

FC layer stands for what?

Answer 31

Fully connected layer

Question 32

A unit with an identity activation function is equivalent to what?

Answer 32

Multiple linear regression

Question 33

A unit with a sigmoid activation function is equivalent to what?

Answer 33

Logistic regression

Question 34

ReLU stands for what?

Answer 34

Rectified linear unit

Question 35

Activation functions serve what two primary purposes?

Answer 35

1) help a model account for interaction effects
2) help a model account for non-linear effects

Question 36

What is an interactive (interaction) effect?

Answer 36

It is when one variable A affects a prediction differently depending on the value of B. For example, if my model wanted to know whether a certain body weight indicated an increased risk of diabetes, it would have to know an individual height.

Question 37

Give an example of how the output of of a node’s value is calculated

Answer 37

You might have input A and input B and bias b, and wA of 2 and wA of 3, and activation function F with value of ReLU. So the calculation would be ReLU(2A + 3B + b)

Question 38

Tanh range

Answer 38

-1 to 1

Question 39

A Sigmoid function is especially useful when we have to predict what?

Answer 39

The probability of something

Question 40

Why is the sigmoid function useful for predicting probability?

Answer 40

Because probability of something must be between 0 and 1 which is the the same range as sigmoid

Question 41

What layer is sigmoid mostly used in?

Answer 41

Activation layer

Question 42

Why is sigmoid mainly used on activation layer?

Answer 42

Because When used on other layers it can cause the network to get stuck during training time

Question 43

Explain the dying ReLu phenomenon

Answer 43

When the weights in the network lead to a pre-activation calculation for a neuron to be negative, that neuron will then have 0 value and can be called “dead” as it doesn’t affect downstream neurons

Question 44

Give two examples of saturating non linearities

Answer 44

Tanh (x) and sigmoid(x)

Question 45

ReLU formula

Answer 45

ReLU(x) = max(x, 0)

Question 46

Rectified networks

Answer 46

Networks that use the rectifier function (ReLU) for hidden layers

Question 47

Why is it useful that part of ReLU is linear?

Answer 47

Because ReLU préserves many of the useful features of linear models

Question 48

List two of the useful properties of linear activation functions

Answer 48

1) They’re easy to optimize with gradient based methods 2) they generalize well

Question 49

Is ReLu considered linear or nonlinear?

Answer 49

Nonlinear

Question 50

List two common learning rate schedules

Answer 50

1) constant learning rate
2) monotonically decaying rate after an optional warmup

Question 51

Mlp stands for?

Answer 51

Multi layer perceptron

Question 52

What is an MLP

Answer 52

A sequence of layers of neurons

Question 53

What is an intuitive understanding of what the bias is?

Answer 53

The bias of a neuron controls the trigger happiness of the neuron. The hire it is, the more likely it is to fire (if act function is ReLu) otherwise it just adds a scalar value to the output

Question 54

In rectified networks does every hidden layer use ReLUs?

Answer 54

No, not necessarily

Question 55

Are activation functions part of a neuron? Or are they applied after a neuron, I.e is there an entire activation function layer applied to the outputs of a layer?

Answer 55

Logically activation functions are part of a neuron - they convert the pre output of a neuron into the output of a neuron

Some frameworks or notations however might apply the activation function as a separate layer applied element-wise to a whole set of neuron outputs (i.e a layer’s output）