04 neural networks*

Question 1

artificial neurons

Answer 1

input signals = independent variables
triggers when activated = activation functions
nerve impulse = output value

Question 2

perceptron

Answer 2

input layer = linear regression
perceptron = calculate weighted sum on input, bias factor
produces output value

Question 3

activation functions

Answer 3

threshold function
if x >= 0, output 1
if x < 0, output 0

Question 4

problems with perception

Answer 4

non linear curves, eg. XOR problem`

Question 5

feed forward neural network

Answer 5

one or more nodes in hidden layer
- node in one layer is linked to every single node on the second layer
- more than one activation functions can be used
- simulate thinking process of multiple input signals with different combinations

Question 6

neuron

Answer 6

non input node (hidden node) and output node
can be used with:
- sigmoid
- rectifier linear unit RELU
- hyperbolic tangent tanh

to account for non-linear

Question 7

sigmoid

Answer 7

1/ (1 + pow(e, -x))
0 to 1

Question 8

hyperbolic tangent

Answer 8

(pow(e, x) - pow(e, -x)) / (pow(e, x) + pow(e, -x))
-1 to 1

Question 9

rectifier linear unit

Answer 9

max(0, x)
0 to infinite

Question 10

training artificial neural network

Answer 10

1. initialise weights
2. set epochs (how many rounds)
   - forward propagation
   - predict results y
   - compute errors (mean squared or cross entropy)
   - back propagation (update weights)
3. repeat

Question 11

back propagation

Answer 11

• update weights to minimise loss/cost/error
• done with gradient descent
• plot sum of square errors (y) and intercept (x) (form a happy curve)
• by measuring the gradient, we can find out the point where error is the lowest, therefore selecting that parameter (gradient = 0 at the lowest point of the curve)
• each step to measure the next gradient is done using “learn rate”
• if learning rate too small, slow traing
• if learning rate too high, miss optimal point
• max step configured to usually be 1000

Question 12

overfitting

Answer 12

common in deep learning when high numbers of hidden nodes and layers
solutions:
- complexity
- dropout
- regularisation

Question 13

tuning dataset

Answer 13

split data into training and testing set

Question 14

machine learning vs deep learning

Answer 14

ML
input -> feature engineering -> classification -> output

Deep learning
input -> feature extraction + classification -> output

Question 15

padding

Answer 15

add 0s to the edge so that output and input is the same size

Question 16

why should we use pooling

Answer 16

1. reduce number of parameters and computation
2. avoid overfitting

Question 17

recurrent neuron networks ** how to draw

Answer 17

h(t) represent memory
y(t) represent output

output h is combined with new iteration to produce new output

Question 18

long short term memory ** explain and rmb how to draw

Answer 18

unique kind of RNN

x(t) represent inputs
c(t) represent
h(t) represent

1. first layer -> forget gate outputs f(t)
   prev output + current input + activation function
   combine with c(t-1),
   if c(t-1) = 0, discard,
   if c(t-1) = 1, keep
2. second layer -> input gate layer
   inputs are
   outputs are i(t) and ~c(t)
3. candidate cell
   update c(t) to a new value
   c(t) = f(t) * c(t-1) + i(t) * ~c(t)
4. output layer