ann

Question 1

latent

Answer 1

existing but not yet developed; hidden, concealed

Question 2

simple perceptron

Answer 2

activation function = step function
1. perform (forward) inference
2. compute loss
3. update weights (remember learning rate)

Question 3

multilayer perceptron

Answer 3

use backpropagation and gradient descent to update weights

Question 4

output layer activation/loss

Answer 4

binary classification = sigmoid + binary cross-entropy
multiclass classification - non-mutually exclusive = sigmoid + binary cross-entropy on each output
multiclass classification - mutually exclusive = softmax (normalize output) + categorical cross-entropy
regression = no activation + MSE

Question 5

hidden layer activation

Answer 5

sigmoid and tanh - small and large values of z cause gradient to be 0
ReLU = max (0, z) - gradient = 0 for z negative
LReLU - nonlinear but piecewise linear, gradient != 0
LReLU(x) = x, x > 0
= ax, x < 0, a = configurable slope

Question 6

detecting over/underfitting

Answer 6

underfitting - training and validation set errors are both high => need higher ANN complexity
overfitting - training loss decreasing, validation loss increasing => IID, more data, decrease complexity, smaller magnitude weights (regularization)

Question 7

Universal Approximation Theorem

Answer 7

A MLP with a linear output layer, at least one hidden layer with any squashing activation function (sigmoid/tanh) can approximate any function, provided the network is given enough hidden neurons

Question 8

weight initialization

Answer 8

needs to be random to break the symmetry of the ANN
sample from normal distribution

Question 9

learning rate

Answer 9

high learning rate - faster, but can miss minimum
small learning rate - slower, but guaranteed to reach minimum

Question 10

mini batch gradient descent

Answer 10

combination between stochastic (fast, high variance) and batch (slow, low variance)
not one, not all training examples at a time, but some (4, 8, 16)

Question 11

training procedure

Answer 11

1. split into training, validation and test set
2. split into mini-batches, update all weights
3. after all mini-batches are done, one epoch is done, reshuffle, do 3-5 epochs
4. set some checkpoints for every epoch, compute training vs validation loss to prevent over/underfitting
5. tune hyperparameters on validation set
6. report performance results on test set with k-fold cross validation

Question 12

CNN uses

Answer 12

image classification, object detection, object segmentation

Question 13

CNN technology

Answer 13

apply convolutional filters to reduce height/width of feature map and increase depth => 1-dimensional array => plug into ANN
encoder-decoder / autoencoder

Question 14

CNN on levels

Answer 14

first layers - detect low lever features (vertical/horizontal lines)
deeper layers - detect higher level concepts (parts of face)
deepest layers - reconstruct entire image, highlight most important classification features (moustache)

Question 15

RNN

Answer 15

word2vec