Topic 2

Question 1

Perceptron neurons

Answer 1

A perceptron takes several binary inputs, , and produces a single binary output

Question 2

Weights

Answer 2

real numbers expressing the importance of the respective inputs to the output

Question 3

Threshold value

Answer 3

neuron’s output, or , is determined by whether the weighted sum is less than or greater than some threshold value

Question 4

Layer

Answer 4

The outputs of a first layer can feed a 2nd layer, and a 3rd layer a so on, creating more nuanced, abstract decisions.

Question 5

Bias

Answer 5

bias = -threshold, a measure of how easy it is to get the perceptron to output a . Or to put it in more biological terms, the bias is a measure of how easy it is to get the perceptron to fire.

Question 6

NAND gate

Answer 6

Any computation can be built using NAND gates, and perceptrons implement a NAND gate.

Question 7

Input layer

Answer 7

input layer perceptrons are really special units which are simply defined to output the desired values,

Question 8

Learning algorithms

Answer 8

can automatically tune the weights and biases of a network of artificial neurons. This tuning happens in response to external stimuli, without direct intervention by a programmer.

Question 9

Sigmoid neuron

Answer 9

more tunable than perceptrons, small changes to inputs cause small changes to outputs; also called logistic neurons

Question 10

Sigmoid function

Answer 10

sigma(z) = 1/ (1+ exp(-z))

Question 11

Activation function

Answer 11

the general form of neural net functions, of which perceptrons and sigmoid neurons are examples

Question 12

Input neurons

Answer 12

the neurons making up the input layer

Question 13

Output neurons

Answer 13

the neuron(s) making up the output layer

Question 14

hidden layer

Answer 14

layers between input and output layers

Question 15

Multilayer perceptrons

Answer 15

or MLPs, another name for multiple layer networks

Question 16

Feedforward neural networks

Answer 16

open loop design, all neurons feed in a single direction, no feedback loops

Question 17

Recurrent networks

Answer 17

feedback loops are possible; neurons fire for some limited duration of time, before becoming quiescent

Question 18

Cost function

Answer 18

quantifies how well algorithm is performing, goal is to minimize; also called “loss” or “objective” function

Question 19

Quadratic cost function

Answer 19

Also called Mean Square Error (MSE) function, smooth, monotonic cost functions make tuning easier than discrete functions

Question 20

Gradient descent algorithm

Answer 20

a minimization algorithm which seeks the minium by calculating derivatives (slope) and goes “downhill”

Question 21

Learning rate

Answer 21

asdf

Question 22

stochastic gradient descent

Answer 22

Much faster learning speed than full computation; select small number of randomly chosen training inputs and calculate cost and change in cost to get gradient of cost.

Question 23

learning rate

Answer 23

a small, positive parameter used to define step-size, or how quickly the algorithm can move along the gradient

Question 24

Mini-batch

Answer 24

random sample on inputs used in stochastic gradient descent

Question 25

Epoch

Answer 25

complete computation of a training set

Question 26

Validation set

Answer 26

data not used in training, to validate the algorithm hasn’t overfit and will work on unseen data

Question 27

hyper-parameters

Answer 27

parameters not directly-selected by the algorithm, e.g. learning rate

Question 28

deep-neural networks

Answer 28

many-layer structure - two or more hidden layers; a series of many layers, with early layers answering very simple and specific questions about the input image, and later layers building up a hierarchy of ever more complex and abstract concepts.