Basic definitions

Question 1

What is an activation function?

Answer 1

An activation function defines the output of a node given a set of inputs. There are many types of activation functions. The two most common used in neural networks are the logistic (sigmoid) and the hyperbolic tangent.

Question 2

What are the two most common activation functions and why?

Answer 2

The two most common activation functions used in neural networks are logistic (Sigmoid) and hyperbolic tangent.

This is true for two reasons:
First, they introduce non-linearity to a NN. This is problem because most problems that a NN solves are non-linear – i.e., they cannot be solved by separating classes with a straight line.
Second, they limit the output of a node to a certain range. Logistic produces output between 0, 1. Hyperbolic tangent produces output between -1, 1

Question 3

What is a neural network?

Answer 3

Neural Networks are one type of learning algorithm that is used within machine learning. NNs are composed of a large number of highly interconnected processing elements (nodes) working in parallel to solve a specific problem. A key feature of a NN is that it learns by example.

Question 4

What is a node?

Answer 4

A NN consists of multiple layers and each of these layers consists of one or more nodes. These nodes are all connected and work together to solve a problem.

Question 5

What is an edge?

Answer 5

A neural network consists of multiple layers and each layer consists of one or more nodes. These nodes are all connected via edges, which mimic the synapse connections found within the human brain.

Question 6

Target Output

Answer 6

Neural networks are supervised learning algorithms, which means that the NN is provided with a training set. This training set provides targets that the NN aims to achieve. Technically speaking, the target is the desired output for the given input.

Question 7

What is total error or global error?

Answer 7

The NN is successfully trained once it has minimized (to an acceptable level) the difference between its real or actual output and its target output. This difference called total error or global error. The total error is typically calculated using a cost function, such as mean squared error or root mean square error.

Question 8

What is the relationship between local error and total or global error?

Answer 8

The total error is the sum of all the local errors. A local error is the difference that occurs between the actual output of a single node and the target output that was expected. For example:
Actual Output: .75
Target Output: 1.0
Error: 0.25

Question 9

What is net input?

Answer 9

Net input refers to the sum of all inputs into a hidden or output nodes. It is calculated by adding together the multiplication of each input by its respective weight. This calculation is usually performed using the summation operator.

Question 10

Is a NN an algorithm?

Answer 10

Yes. A NN is a machine learning algorithm insofar as it is a set of instructions designed to perform a specific task. Apart from this, algorithms are used with a NN to successfully train the network. One of the most common algorithms used to accomplish this is backpropagation, which makes use of gradient descent to optimize and ultimately train the network.

Question 11

What is machine learning?

Answer 11

Machine learning is the science of getting computers to act without being explicitly programmed.

Question 12

What is batch training or “full batch”?

Answer 12

Batch training is a particular form of gradient descent, which is used in conjunction with back propagation to train a network. Batch training works by summing the gradients for each training set element and then updating the weights in one iteration.

Question 13

What is stochastic gradient descent?

Answer 13

Stochastic gradient descent is another form of gradient descent, which is used in conjunction with back propagation to train a network. SGD works by updating weights for each training set element, not their sums. This means it utilizes many more iterations than batch training.

Question 14

What is mini-batch training?

Answer 14

Another form of gradient descent, which is used in conjunction with back propagation to train a network. Mini-Batch training works by summing the gradients for multiple training set elements (but not all of them) and then updating the weights. The size of the Mini-Batch can be pre-set as a hyperparameter or randomly chosen by the algorithm. To continue with our example above, these elements would be n number of images from a training set of 10,000 images. Mini-Batch is one of the more popular and successful gradient descent methods.

Question 15

What is back propagation?

Answer 15

Backpropagation is the process of the neural network working backwards to adjust weights proportionally.

Question 16

What is the chain rule?

Answer 16

Here are two definitions:

1. The chain rule is a way to find a derivative of an equation that is a function found inside another function.
2. The chain rule is used to compute the derivative of the composition of two or more functions.

Within a NN, the chain rule is used to calculate the derivative across an entire NN. Practically speaking, this makes it possible to adjust the weights and hopefully succeed in training the NN.

Question 17

What is classification?

Answer 17

NNs excel at classification. Classification refers to sorting or classifying data into groups. For example, a neural network might be presented with a folder of various pictures: some are dogs, some are cats, some are pigs. In this instance the NN is required to learning how to classify these pictures into their three respective groups.

Question 18

What is a convolutional neural network?

Answer 18

A CNN is a specific type of feedforward NN. CNNs are typically used with image recognition and in many regards are the core of computer vision systems. They are also used in NLP.

Question 19

What does “converge” mean?

Answer 19

In ML, converging refers to the output moving closer and closer to the desired target value. Converge is the opposite of diverge.

Question 20

What does “diverge” mean?

Answer 20

The opposite of “converge” is “diverge”. This occurs when the output continues to oscillate (undergo fluctuations). In this circumstance, the output is not inching steadily toward the target, but is oscillating instead.

Question 21

What is the “curse of dimensionality”?

Answer 21

The curse of dimensionality is a ML phrase that refers to the difficulty of working in multiple dimensions.

Question 22

What is dropout?

Answer 22

Dropout is a form of regularization that helps a network generalize fittings and increase accuracy. It is often used with deep NNs to combat overfitting, which it accomplishes by occasionally switching off one or more nodes in the network.

A node that is switched off cannot have its weights updated, nor can it affect other nodes. This causes the other weights that are switched on to become more insensitive to the weights of other nodes and, eventually, begin to make better decisions on their own.

Question 23

What is deep learning?

Answer 23

Deep learning is a relatively new field within ML that refers to NNs with multiple hidden layers. The adjective “deep” refers to the amount of hidden layers that exist; the more layers, the deeper the learning.

Question 24

What is an epoch?

Answer 24

An epoch refers to one forward pass and one backward pass of ALL training examples in a neural network. In other words, an epoch describes the number of times a network sees an entire data set. For example, if you are training a network to recognize images and have a set of 10,000 images, these images are the training examples. An epoch will have occurred once all of the examples are passed through the network – “passing through the network” includes both a forward and backward pass through the network.

Question 25

What is a training example?

Answer 25

For example, if you are training a network to recognize images, and you have 10,000 images, these images are the training examples.

Question 26

What is a batch?

Answer 26

A batch refers to the total number of training examples in both a forward and backward pass. For example, 100 images.

Question 27

What is an iteration?

Answer 27

An iteration refers to the number of times a batch of data passes through the network. For example, if you have 500 images (training examples) and your batch size is 100, then it will take 5 iterations to complete one epoch.

Question 28

What is a gradient?

Answer 28

A gradient is the slope of the error function at a specific weight – or in other words, it is the individual error of a single weight in a neural network. Technically, it is a vector and arrived at by calculating the derivative of the slope at a specific point.

Question 29

Describe the ReLU activation function.

Answer 29

The Rectified Linear Unit (ReLU) has become very popular in the last few years. It computes the function ƒ(κ)=max (0,κ). In other words, the activation is simply threshold at zero.

In comparison to sigmoid and tanh, ReLU is more reliable and accelerates the convergence by six times.

Unfortunately, a con is that ReLU can be fragile during training. A large gradient flowing through it can update it in such a way that the neuron will never get further updated. However, we can work with this by setting a proper learning rate.

Question 30

Describe the sigmoid activation function.

Answer 30

The sigmoid non-linearity has the mathematical form σ(κ) = 1/(1+e¯κ). It takes a real-valued number and “squashes” it into a range between 0 and 1.

However, a very undesirable property of sigmoid is that when the activation is at either tail, the gradient becomes almost zero. If the local gradient becomes very small, then in backpropagation it will effectively “kill” the gradient.

Also, if the data coming into the neuron is always positive, then the output of sigmoid will be either all positives or all negatives, resulting in a zig-zag dynamic of gradient updates for weight.

Question 31

Describe the hyperbolic tangent (tanh) activation function.

Answer 31

Tanh squashes a real-valued number to the range [-1, 1]. Like sigmoid, the activation saturates, but — unlike the sigmoid neurons — its output is zero centered.