Chapter 2: The Math of Neural Networks

Question 1

core building block of neural network

Answer 1

layer

Question 2

Neural layers do what?

Answer 2

extract representations out of the data fed to them

Question 3

Chaining together successive neural layers creates what?

Answer 3

Progressive Data distillation

Question 4

Dense Neural layer

Answer 4

Fully connected neural layer

Question 5

Parts of the compilation step

Answer 5

1. loss function 2. an optimizer 3. metrics to monitor during data and training

Question 6

multidimensional NumPy arrays also called what?

Answer 6

Tensors

Question 7

What are tensors

Answer 7

Tensors are a generalization of matrices to an arbitrary number of dimensions. A container for data almost always numerical

Question 8

Alternate name for dimension in the context of tensors?

Answer 8

Axis

Question 9

Term for the number of axes of a tensor?

Answer 9

its rank

Question 10

Tensor that contains only 1 number

Answer 10

Scalar

Question 11

An array of numbers

Answer 11

Vector

Question 12

1D Tensor

Answer 12

Vector

Question 13

0D Tensor

Answer 13

Scalar

Question 14

How many Axes do vectors have?

Answer 14

1

Question 15

What is a vector’s dimension?

Answer 15

the number of entries along its axis. NOT the same as tensor dimension

Question 16

An array of vectors

Answer 16

Matrix

Question 17

2D Tensor?

Answer 17

Matrix

Question 18

3 Key attributes of a Tensor

Answer 18

1. Number of axes
2. Shape
3. Data type

Question 19

Tensor Shape

Answer 19

A tuple of numbers that describes how many dimensions it has along each axis

Question 20

Tensor Axis convention

Answer 20

Axis = 0 Sample axis
Axis = 1 Feature axis
Axis = 2 Time axis

Question 21

Element-wise operations

Answer 21

applied independently to each entry the tensors

Question 22

vectorized implementations

Answer 22

amenable to massively parallel implementation

Question 23

Geometric Interpretation of Deep Learning

Answer 23

Neural networks consist entirely of chains of tensor operations and all of these tensor operations are just geometric transformations of the input data

Question 24

Differentiable

Answer 24

Can be derived

Question 25

Gradient

Answer 25

The derivative of a tensor operation. the generalization of the concepts of derivatives to functions of multidimensional inputs

Question 26

Gradient and Loss reduction

Answer 26

by shifting weights slightly in the opposite direction of the gradient

Question 27

mini-batchStochastic gradient descent

Answer 27

the process of using random batches of a data set to train the weights of a neural network.

Question 28

batch stochastic gradient descent

Answer 28

using all of your data to train your neural network via stochastic g. d.

Question 29

What role does momentum play in SGD?

Answer 29

It addresses two issues 1. Convergence Speed 2. local minima

Question 30

What gives rise to the backpropagation algorithm?

Answer 30

applying the chain rule to the computation of the gradient values in a neural network allowing us to find the derivative

Question 31

backpropagation

Answer 31

starts with the final loss value and works backward from the top layers to the bottom layers applying the chain rule to compute the contribution that each parameter had in the loss value

Question 32

Where does the knowledge of network lie?

Answer 32

in the weight tensors (attributes of the layers)

Question 33

Dense neural network layers

Answer 33

Fully connected

Question 34

layer compatability

Answer 34

The idea that every layer will only accept input tensors of a certain shape and will return output tensors of a certain shape

Question 35

network space

Answer 35

the network space you choose constrains your space of possibilities (hypothesis space)

Question 36

Optimizer

Answer 36

determines how the network will be updated based of the loss function. It implements a specific variant of stochastic gradient descent

Question 37

Keras backend engines

Answer 37

currently just Tensorflow, Theano, and Microsoft Cognitive toolkit (CNTK)

Question 38

Keras and hardware

Answer 38

Can run on either CPU or GPU

Question 39

Keras on CPU

Answer 39

Tensorflow uses a library for tensor operations Eigen

Question 40

Keras on GPU

Answer 40

Tensorflow uses a library for deep-learning op. called NVIDIA CUDA Deep Neural Network Library

Question 41

Ways to define neural model

Answer 41

Sequential or Functional API