3 - Backpropagation in computation graphs

Question 1

Computational graph

Answer 1

eg if f(x,y,z) = (x+y)*z

x = -2
+ (q =3)
y = 5
- (t=-12)
z = -4

(Imagine lines matching the correct operation)
Backpropagation goes right to left in these

Question 2

partial derivative

Answer 2

How much the output changes when one(?) input changes

Question 3

Chain rule

Answer 3

F(x) = f(g(x)) then F’(x) = f’(g(x))g’(x)

’ means derivative

Question 4

Derivatives: if q = x +y then…

Answer 4

derivative of q w/r to x = 1
derivative of q w/r to y = 1

Question 5

Derivatives: if f = qz then

Answer 5

derivative of f w/r q = z
derivative of f w/r z = q

Question 6

General concept for why chain rule is useful in computational graphs

Answer 6

To determine the “effect” of one input on the output, follow the chain from the output to the input

Ie, in the example, to find deriv f w/r x, we need deriv f w/r q multiplied by deriv q w/r x

Question 7

Is a computational graph the same as a neural network?

Answer 7

NO!
Computational graph is much bigger and shows operations

Question 8

Sigmoid derivative

Answer 8

(1-sig(x))(sig(s))

That’s
(1-σ(x))(σ(s))

Question 9

Patterns in backward flow: ADD gate

Answer 9

gradient distributor

Question 10

Patterns in backward flow: MAX gate

Answer 10

Gradient Router

Question 11

Patterns in backward flow: MUL gate

Answer 11

gradient switcher

EG
x*y where x=3 and y=-4 and the gradient is 2

would mean that x has -8 (-42) and y has 6 (32)

Question 12

Jacobian Matrix

Answer 12

The derivative of each element of z (output) w/r to each element of x

Question 13

Steps of training a simple net

Answer 13

Forward pass
Compute loss
Propagate loss backwards
Step the optimiser (ie update parameters)
reset all the gradients to 0