Class 8

Question 1

deep learning

Answer 1

broad family of techniques for ML in which the hypotheses take the form of a complex algebraic circuit with tunable connection strengths

Question 2

neural networks

Answer 2

networks trained by deep learning methods

Question 3

feedforward network

Answer 3

neural network with connections only in 1 direction – forms a DAG with designated input and output nodes

Question 4

recurrent network

Answer 4

neural network that feeds its intermediate or final outputs back into its own inputs

Question 5

universal approximation theorem

Answer 5

states that a network with just 2 layers of computation, 1st = nonlinear and 2nd = linear, can approximate any continuous function to an arbitrary degree of accuracy

Question 6

activation function

Answer 6

first layer in a network, the nonlinear one

Question 7

relu

Answer 7

rectified linear unit

Question 8

softplus

Answer 8

smooth version of ReLU

Question 9

vanishing gradient

Answer 9

error signals are extinguished as they are propagated back through the network

Question 10

automatic differentiation

Answer 10

applies rules of calculus in a systematic way to calculate gradients for any numeric program

Question 11

one hot encoding

Answer 11

non-numeric attributes (think strings) given a numeric expression

Question 12

convolutional neural network

Answer 12

neural network that contains specially local connections

Question 13

kernel

Answer 13

pattern of weights that is replicated across multiple local regions

Question 14

convolution

Answer 14

process of applying the kernel to the pixels of the image

Question 15

stride

Answer 15

size of the step that the kernel takes across an image

Question 16

receptive field

Answer 16

part of a neuron where the sensory input can affect the neurons activation

Question 17

pooling layer

Answer 17

layer in a neural network that summarizes a set of adjacent units from the preceding layer with a single value

Question 18

downsampling

Answer 18

process of making the stride larger – coarsens the resulting image

Question 19

tensors

Answer 19

multidimensional arrays of any dimension

Question 20

feature map

Answer 20

output activations for a given filter – created by tensors

Question 21

channels

Answer 21

dimension of the matrix information that carries information about features

Question 22

residual networks

Answer 22

neuralnetworks that avoid the problem of vanishing gradients by building a very deep network, has skip connections

Question 23

batch normalization

Answer 23

improves rate of convergence on SGD by rescaling the values generated by the internal layers of the network from examples within each minibatch

Question 24

neural architecture search

Answer 24

used to explore the state space of possible network architectures – using a neural network to find the best neural network

Question 25

weight decay

Answer 25

adding a penalty to the loss function (same as regularization for a neural network)

Question 26

dropout

Answer 26

technique for introducing noise at training time which forces the model to become more robust (similar to boosting), can randomly deactivate units (perceptrons)

Question 27

recurrent neural networks

Answer 27

neural network that are distinct from feed forward networks in that they allow cycles in the computation graph

Question 28

markov assumption

Answer 28

RNN’s assume the current state is based on a finite set of previous states

Question 29

gating units

Answer 29

vectors that control the flow of information in the LSTM via elementwise multiplication of the corresponding vector

Question 30

unsupervised learning

Answer 30

takes a set of unlabeled examples, may try learning a new representation like specific feature or image, might try to learn a generative model

Question 31

generator

Answer 31

network that maps values to produce samples from the distribution

Question 32

discriminator

Answer 32

network that classifies inputs as real (from the training set) or fake (from the genrator)

Question 33

generative adversarial network

Answer 33

pair of networks that combine to form a generative system

Question 34

transfer learning

Answer 34

occurs when experience with one learning task helps an agent learn better on another task

Question 35

multitask learning

Answer 35

form of transfer learning where we simultaneously train a model on multiple objectives

Question 36

deep reinforcement learning

Answer 36

field of research on multilayer computation graphs