Deep Learning

Question 1

Why do we need non-linear activation functions?

Answer 1

1. otherwise the whole network will collapse into a linear model.
2. helps to capture non-linear complexities.

Question 2

Is ReLU a better activation function than sigmoid?

Answer 2

in sigmoid graphs flattens out for large values of x thus causing vanishing gradients. ReLU prevents this from happening.

Question 3

What are the two problems with gradient descent?

Answer 3

1. local optima
2. saddle points: gradients is zero but is not local optimum
3. efficiency: can be solved using stochastic gradient descent.

Question 4

What are the main differences between CNNs and fully connected MLPs?

Answer 4

CNNs:
1. sparse connections:
outputs are connected to inputs within receptive field; less
weights; less overfitting
2. weight sharing:
less overfitting, regularization
3. location invariant