Predictive Models: Deep Neural Networks

Question 1

What is a Deep Neural Network and what does it consist of?

Answer 1

Deep Neural Network is a model inspired by the brain.

It is a network of interconnected nodes, with input and output nodes, with hidden layers in between making computations.

Question 2

What are Deep Neural Networks doing?

Answer 2

The DNN finds the correct mathematical manipulation to turn similar input into output based on its training, whether it be a linear relationship or a non-linear relationship.

Question 3

What is a node/neuron, and how are they connected

Answer 3

A node is a computational unit.

The node has one or more weighted input connections from and to other nodes.

Question 4

What does a node consist of?

Answer 4

A node consists of a sum function and an activation function.

Question 5

What is the weight in a DNN?

What are they usually denoted or referred to as?

Answer 5

The weight is the impact on a node, by a previous node.

e.g. Imagine the previous node outputs 10.
The weight to the next node is 0.1. This means it will impact the next nodes sum function with 1

Weight is usually denoted as theta (θ)

Question 6

Imagine a fully connected layer. (all nodes between layers are connected to each other).

There are 10 nodes in each layer.

How many connections do each node in the next hidden layer have?

Answer 6

All nodes have 10 connections from the previous layer.

There is one connection to each individual node, pr. node from the previous layer, a fully connected layer.

Question 7

What is the sum function?

Answer 7

The sum function takes the weighted sum of all the connections from other nodes and, sums them up.

Question 8

What is the activation function?

Answer 8

The activation function is a computation, that fires if the sum functions return a sum high enough (the threshold is met)

Different types of activation functions decide how much a node should fire given a threshold.

Question 9

What is the threshold?

Answer 9

A threshold is a certain value, at which, the activation function will fire or how much it will fire.

Question 10

What is the output layer?

Answer 10

The output layer consists of a or multiple nodes that contains the hypothesis function.

It produces a result from the inputs.

Question 11

What is a bias node?

Answer 11

A bias node is a node that can be in a hidden layer. Imagine a node at the bottom of each layer which simply adds a number (bias) of, for example, 1, to the next layer. or in the end just before the output node.

Question 12

Do the activation functions change between layers?

Answer 12

Yes, the activation functions can change between the layers.

Question 13

What is feedforward in a Neural Network?

Answer 13

It is the simplest form of a Neural Network. It means that the learning only travels forward, and not in e.g. a cycle.

You could say, information only moves one way, forward.

Question 14

What does it mean if a DNN is ‘shallow’?

Answer 14

That it has 1 or a low amount of hidden layers.

Question 15

Explain the ‘step’ activation function

Answer 15

This activation function fires when the weighted sum is above a threshold. Then it sends the exact same value to the next layer (weight is necessarily not the same)

Question 16

What is the problem with the step activation function?

Answer 16

Multiple nodes can take the value of 1 and fire. Then this could make it hard to classify/decide.

Question 17

Which different activation functions are there?

Answer 17

1. Sigmmoid
2. Tanh
3. ReLu
4. Leaky ReLu
5. binary step function

Question 18

A solution to the problem of the step function is the linear function, but what is the problem with that?

Answer 18

If we use the linear step function we would get a range of values as output. e.g y=ax, this represents not only one but a range of values.

The problem is that if we only use the linear function in a DNN, it could cause the output layer to become a linear function. This makes it impossible to map non-linear data.

Question 19

What is the Sigmoid activation function?

Answer 19

The Sigmoid activation function looks like a smooth step function. (S - shape)

The function works in such a way that between x -2 and 2 the y-value changes drastically, resulting in the tendency to results being brought to either end, almost like a binary result. Making it good for distinctions.

Another big advantage is that the output is always going to be between -1 and 1!

Question 20

What are the disadvantages to the sigmoid activation function?

Answer 20

Because of the slope in the ends, it is hard to make significant changes at x-values farther from 0. The gradient is very flat, this makes it so that the algorithm starts to learn extremely slow with almost no progress.

Question 21

What is the Tanh activation function?

Answer 21

This activation function is very similar to the sigmoid function. In fact, it is a scaled Sigmoid function.

The properties are the same, near 0,0 changes to x are significant for y. as we go to the sides the changes are minimal.

Question 22

What is the ReLu activation function?

Answer 22

The ReLu function gives an output if x is positive, and gives 0 otherwise.

When x is negative it is flat on the x-axis (y=0) - the line is horizontal.

When x is positive it is linear.

Question 23

Does the ReLu function have the same issues as a linear activation function?

Answer 23

No it can be stacked because it is not only linear. So combinations of ReLu is not linear

Question 24

What is the benefit you gain from ReLu?

Answer 24

When a network of neurons are initialised with random values almost 50 % of the network will yield 0. Because the output is 0 for negative values.

This means fewer neurons are firing (sparse activation) making the network lighter.

Question 25

What is the Dying ReLu Problem?

Answer 25

Because the gradient can go towards 0 for activations, the weights will not get adjusted during descent.

This means that the neurons that go into that state will stop responding to variations in error/input. Making a part of the network passive.

Question 26

What is the fix to the Dying ReLu Problem?

Answer 26

Leaky ReLu:
Make the horizontal line (where x is minus), non-horizontal.

The idea is that this makes the gradient non-zero so it can recover during training.

Question 27

Which is less computationally expensive? ReLu or tanh/sigmoid

Answer 27

ReLu because it involves simpler mathematical operations.

Question 28

Which activation function would you use for classification?

Answer 28

sigmoid or tanh

Question 29

What is an input node?

Answer 29

An input node does not compute, it only holds a value.

Question 30

What is a hidden layer?

Answer 30

It is a layer of nodes that does the computation in DNN.

Question 31

What is a bias node?

Answer 31

A bias node is a node that simply adds or subtracts a certain amount.

Question 32

What can the bias node do?

Answer 32

It can push the activation function to the left or right side of the graph.

Multiplying now makes the graph steeper
Adding/subtract shifts the function left/right

Question 33

When you train a model with a training set what are you essentially doing?

Answer 33

You change the weights (theta) so that the computations will result in an output that fits the training data.

Question 34

What is ‘Cost function’?

Also called ‘Loss function’

Answer 34

The cost function finds the difference between the predicted value and the real (labelled) value.

Question 35

What is the cost function for linear regression models?

Answer 35

RMSE: Root-Mean-Square-Error

Question 36

Which types of classification algorithms have we heard of?

Answer 36

Decision trees
Random Forests
K-Nearest Neighbor
Logistic Regression

Question 37

Which types of clustering algorithms have we heard of?

Answer 37

K-Means

K-Modes

Question 38

What is the decision boundary?

Answer 38

It is the threshold (y) where the value will be categorized as 1 or 0. So positive or negative.

Question 39

What is overfitting?

Answer 39

It is when the adjustments are too high for each iteration causing the model to oscillate over the optimal line

Question 40

What is backproporgation?

Answer 40

When you want to change the weights in your neural network to better predict.

Gradient descent optimizes the backpropagation.

Question 41

What does regularization do?

Answer 41

Regularization prevents overfitting but making small changes to the cost function algorithm. The goal is to make it perform better on unseen data.