Neural Networks Flashcards
How to separate Test, Train and Validation Data
Separate base data set into 6 components
- Train Features
- Train Targets
- Validation Features
- Validation Targets
- Test Features
- Test Targets
Pic just shows training and test
What is logistic regression?
Outputs a probability that a given input belongs to a certain class
Explain the softmax function, why use e?
Turns scores(integers which reflect output of neural net) into probabilities
e turns negatives into positives
Explain why sigmoids are no longer used as an activation function
- Derivatives of sigmoid max out a .25, thus, during backprop, errors going back into the network will be shrunk by 75% -100% at every layer
- For models with many layers, as you get closer to layers near input, weight updates will be tiny and take a long time to train
- Not zero centered(More inconvenience) - feeds only positive value to next layer, in turn, gradient of weights of this input value X will be always positive or always negative, depending on the gradient of the whole expression f. This leads to undesirable zig-zagging of gradient updates for the weights
Explain why Relu is used as an activation function
- If max is positive, then derivative is 1, o there isn’t the vanishing effect you see on backpropagated errors from sigmoids.
- Leads to faster training
Drawbacks of Relu
- Large learning rate, coupled with a large gradient, may lead to respectively large negative adjustment of weights and biases(our step down)
- Adjustment may lead to a negative input into the Relu caculation. Negative = 0 for Relu. During backprop, the derivative of zero is zero, so chain rule leads to a zero update. This leads to a “dead” neuron which may waste computation and reduce learning
- Its very hard if not impossible to input a large positive adjustment(to counter the earlier), since we are moving down to a local minimum. Not sure if this would even matter mathimatically, but may help if taken in a batch view
Training Loss
Average cross-entropy loss
S = Softmax, D = Cross-Entropy, L = Loss
When performing numerical optimization, what are good things to do with variables
- Mean of zero
- Equal Variance
This minimizes search performed by optimizer
For images, how to you prepare your data for optimization
take each channel, subtract 128 and divide by 128. Doesn’t change the data, just makes it easer for numerical optimization
discuss gaussian distribution and sigma
- mean zero and standard deviation sigma.
- sigma determines order of magnitude for outputs at inital point of optimization
- Beaucse softmax sits ontop of sigma, the order of magnitude also determiness peakiness of inital probability distribution
- Large sigma = uncertain, large peaks
- Small sigma - Opinionated, small peaks -
For optimization, what are basic ideas on how to initialize weights and biases
start from gaussian distribution with small sigma. Small sigma means more certain.
What are the 3 sets of data used to measure performance and how are they used
- training - optimize loss
- validation - measure performance of training
- test - never use until final measurments
Explain Stochastic Gradient Descent
- Average loss for very small random fraction of training data
- This average loss is typically a bad estimate at first and may actually increase error
- Thus, you do it many times, taking small steps each time
Explain Momentum
Use running average of gradients as directio to take, instead of gradient in current batch
Explain learning rate decay
decreasing the learning rate over time during the training process(every time it reaches a plateau, exponential decay
What is ADAGRAD
- modification of SGD which implicity does momentum and learning rate decay by default
- Makes learning less sensitive to hyperparameters
Whats a scalar
single value that represents a zero dimension shape(1, 2,4, -0.3)
Whats a vector
a single dimesion shape with a certain length
What is a matrix and how do you describe it?
2 dimensional grid of values. If had 2 rows and 3 columns, its a 2x3 matrix
describe a vector as a matrix
1x len matrix.
describe indices of a matrix
row then column index
How to reshape vectors from horizontal to vertical
Less common way
More common way of reshaping data
Describe element-wise operations within Matrix
Example of element wise operation -normalizing an image
How to perform element wise operations between matrices
Must be the same size
How does matrix multiplication work
Take element wise operation for row and column pairs. Treat as separate horizontal and vertical vector pairs.
When can you NOT take a dot product of two matrices
if lengths of horizontal vector doesn’t match length of vertical vector. Since you take take element wise operation on these vector pairs, they must have same length
Important Reminders about Matrix Multiplication
- The number of columns in the left matrix must equal the number of rows in the right matrix. When viewing shape of two matrices side by side, if the value dimensions match on inside, your good. 2x3 and 3X2. Values on inside(3) match.
- The answer matrix always has the same number of rows as the left matrix and the same number of columns as the right matrix. (outside values, 2X2)
- Order matters(not commutative). Multiplying A•B is not the same as multiplying B•A.
- Data in the left matrix should be arranged as rows., while data in the right matrix should bearranged as columns.
What is the transpose of a matrix
switch the values in the rows to be values in the columns. If not a square, then 2x4 becomes 4x2, which may assist in matrix multiplication
When can you safely use a matrix transpose
How to call the transpose function in numpy
.T
If you modify a transposed matrix, what are implimcations
- it modifies both the transpose and the original matrix, too!
- They are sharing the same copy of data.
- Consider the transpose just as a different view of your matrix, rather than a different matrix entirely.
What is a gradient
- just a derivative generalized to functions with more than one variable
- think slope or direction of greatest ascent. Taking negative of gradient is direction of steepest descent
Explain local minima
area where loss is low, but not the lowest. Primarily caused by poor weight initialization
When calculating the error of a function, what are upsides of using SSE
- The squared error penalizes outliers more than small errors
- Makes values all positive
Brief explaination of chain rule
Think of the network as a bunch of dominos. Some dominos are weights, biases, dot products, activation values, error. The goal is to fine tune the dominos that contain weights and biases. In order to due so, we take the derivative of the final domino(the error domino). Then, if you want to find a weight update, you just find the derivative of each connecting dominos and multiply the values togheter. This is the chain rule.
What is the gradient of really small and large values
zero, so in turn steps are zero
How to do calculate weight update
- Calculate error term
- multiply by input x , then divide by number of records to get average delta weigt change
3.
Describe components of Output Error Term
Output Error * Derivative of Output(Post Activation Function)
Pic uses sigmoid and derivative of sigmoid.
Error term(output or hidden) is used to update weights that connect into layers.
For multilayer perceptron, how is the weight matrix written?
Just remember, in matrix notation, its always rows then columns
For backpropagation, how do you interpret the weights for each hidden layer
weights assign how impactful hidden and input layers are on the total error. Since inputs and hidden layers are all multiplied by weights, it only makes sense that the error stems from these weights.
What is Backpropagation
Basically flipping the network around once you found an output error, and using this output error as the input. The output error is feed in as input, multiplied by weights to identify hidden unit error terms(the amount of error caused by the hidden unit). These error terms are then multiplied by hidden unit gradients to identify magnitude and direction a network should move.
How to create dummy variables for categorical variables
for loop, select the columns from the data, specify a prefix name, then concat those columns to the data. The pd.get_dummies will identify unique values and add those after the prefix.
How do you scale variables to ensure they have zero mean and std of 1
take each variable, subtract the mean then divide by standard dev.
Code loops through features, calculates mean and std dev for each column, store these values in a dictionary as a tuple. Use data.loc[:,each] to perform calculation for every row of the looped column.
How to assign features and targets
create list of target fields, then use .drop method to remove these from data set, assigning dropped fields as targets and remaining fields as features
how to split the feature and target data into train_features, test_features, val_features, val_targets
Just slice the data and assign to variables. The pic is selecting various rows since there is no comma, just the last amount of rows or the first amount of rows
How do you line up the features with targets
zip the two data sets
How to implement SGD
create batches(which are just an array of index values from the data set) via np. random.choice.
Using this batch(list of index values), select rows(batch) and all columns from train_features and train_targets
During Training, how do you get a vector of features as input
During SGD, you create a batch which is an index of values used to select data from train_features and train_targets. well call the features data a matrix of 128 rows by 56 columns and the targets a vector of 128 values. In order to get a vector of feature values to input into the neural network, you zip the train_features with the train_targets and assign the result to an X, y variable. Now the X variable is a single vector of 56 features with a target of 1 value.
- Create batch of random index values
- Select data using batch
- Zip data with targets
Basic idea of K-Fold Cross Validation
- Avoid separating test data that is never used while at the same time not cheating(overfitting)
- Break data into k buckets, train k times and each time use a new bucket as the test bucket while training on the others
- Average results
Implement K-Fold using SKLearn
discuss confusion matrix
Code Accuracy
Two types of errors in machine learning(Basic)
Explain Model Complexity Graph
As models become more complex, errors become smaller and smaller. However, to esure model generalizes well and performs well on training data, look for point where validation errors increase while training errors decrease. Remember to use Validation Set, not Test Set
create a function which counts how many words are in a review
Code Softmax Function
How to mathimatically say a change in the layer 2 value causes a change to the model loss/cost
This is just a derivative/gradient.
So a node with a larger gradient with respect to the cost is going to contribute a larger change to the cost. In this way, we can assign blame for the cost to each node. The larger the gradient for a node, the more blame it gets for the final cost. And the more blame a node has, the more we’ll update it in the gradient descent step.
Conceptually(or mathimatically spoken), how do you update weights via gradient descent?
First find how much blame the layer value caused on the error, then how much blame the weights caused on the layer value. This is mathimatically spoken using the chain rule. Its a domino effect, chaning something in the network will domino all the way down.
C = Cost, L2 is final layer value, W2 is weights feed into L2
Explain the cross entropy calculation
multiply jth element of one hot encoded vector by natural log, multiply by jth element of softmax, sum, take negative value.
Best Practice for preparing inputs and weights
Basics of implementing mini-batching
- Just splitting the train_features data into batches
- Specify batch_size(typically 128)
- Divide batch size into length of features(the rows). This gives us total batches.
- Use for loop to create batches. Use range criteria to specify starting, ending and step size
Compare a perceptron and their inputs to a biological neuron
Inputs = dendrites
perceptron = neuron
If percetron activates(cacluates a 1), a positive value is processed. If neuron fires, the output signal is sent along the axon
For gradient descent, the error function must have what properties?
Continuous- small variations in position translate to small variations error
Differentiable - you can calculate the derivative of the function
Explain the sigmoid function
- For large positive numbers, you get values close to 1
- For large negative numbers, you get values close to 0
- For numbers close to 0, you get values close to .5
explain this diagram
example shows probability of points being blue. Going diagonally across the chart is a line(yx +b = 0), anything above line is blue, below line is red. This illustrates taking the sigmoid of the function WX + b. For values close to the line, you get a .5 probability, but as you move farther into blue, the probability gets higher.
Plug in coordinate values to confirm value of X. Input X into sigmoid function. You don’t need to actually calculate the formulate becuase you know if X=0, then prob will be .5(per definition of a sigmoid)
How to one-hot encode the picture
Describe the Maximum Likeihood Concept
- find probabilities for each occurrence
- multiply these probabilities together(JUST IN THIS EXAMPLE. PRODUCTS ARE BAD, SUMS ARE GOOD. Whatever this value is, we try to maximize to find the best model. In the picture, the model on the right displays a higher maximium probability(P |ALL) and thus is cosidered the superior model(model classifies all occurences better than other model)
Best way to calculate Maximum Likelihood -
Use Log function becuase it turns products into sums. Don’t want products because with many data points, changes to one value can alter equation drastically
Log(1)? Whats the implication for calculating maximum likelihood?
- The log of any probability will be negative, so we must take the negative log of the probability.
Explain Cross Entropy and how goal of Maximizing Probability has changed
If I have a bunch of events and bunch of probabilities, how likely are those events happen based on those probabilities. If likely, small CE and vice verse
- calculate probabilties of occurrences
- Take sum of negative logs of each probability,
low = good, high = bad
- Goal is to minimize cross entropy, not maximize probabilities
Explain this Cross Entropy Diagram
In the left chart, the two points with 2.3 and 1.6 -log values correspond to the incorrectly classified points(red dot is in blue region, blue dot is in red region.) You can see the values are larger in comparison to the correctly classified points(red in red region). In turn, the distance between correctly and incorrectly classified points is a measurable error that we try to minimize. The right chart has a lower cross entropy error becuase each point is correctly classified.
Describe this non-linear photo in terms of probability in regards to the line that separates the two regions
The white line represents a non-linear equation in which everything on the line has a probability of 50%. Anything in the blue space has a probability above 50% and red space is below 50%.
In the basic sense, we linearly combine probabilities for this point, and sense the value has to be between 0 and 1, we apply a non-linear activation function such as sigmoid. We do this for all points
How would these linear expression be diagramed as a neural net
How can you combine these two models using a neural net diagram
How to clean up this diagram
When you see this diagram of a neural net, what should you be thinking it represents?
it represents the non-linearly boundary in white.
How could you write this with the bias outside?
describe this photo
when you increase your variables(x1, x2, x3), you increase your dimensions. After 3 dimensions, it becomes very hard to visualize. But in 3 dimensions, now you just have planes in 3d space and the final product is some non linear plane.
describe this photo
combine linear model into a non-linear model. Then you combine these non-linear models to makea more complex non-linear model. This is a deep neural net which allows us to generate highly complex probability boundaries
describe this photo
for multi-class classification, we add a layer in the model(with n nodes that represent the items being classified, in this case 3 nodes for 3 animals) that identifies the probability for each animal.
How many parameters
weights = 28*28*10
+
Bias = 1*10
Best way of preventing overfitting
Early Termination - stop training when our validation set stops improving.
Another way to prevent overfitting(not early termination) - Just the basic concept
Regularization - applying artificial contraints to network that implicitly reduce number of free parameters while not making it more difficult to optimize ( think yoga pants, not hard to get into, but fit great)
L2 Regularization
Add L2 norm of weights(multiplied by small constant) to loss. The L2 norm is just the sum of squares of the individual elements in a vector.
Explain Dropout
Randomly setting activation values to zero. You are destroying up to half of your data flowing through the model. Then you randomly do it again.
Why use Dropout as a way to prevent overfitting?
Network can’t rely on any given activations to be present. Forced to learn redundant representations. Takes consensus over ensemble of networks. If dropout doesn’t work, you may need a bigger network.
create model using one hidden layer, relu and dropout
Gradient Descent - Describe how to find gradient
Calculate parital deriviates of error with respect to each input. Picture below has two inputs(two dimensional)
Gradient Descent - Weight and Bias step
Learning rate * error term(either the output error term or the hidden unit error term) * input(either a raw input value or the hidden unit activation value)
Gradient Descent - Write a function for weight and bias step
Gradient Descent - Why does error have to be continuous?
If it was discrete, you may not be able to tell small variations in error and thus give us an idea of what direction to take.
Gradient Descent - What do correctly and incorrectly classified points tell a line?
correctly classified = go farther away so my error is smaller, prediction closer to 1
incorrectly classified = come closer, error is smaller and prediciton closer to 1
Neural Nets - How to configure for multi-classification?
Add as many output nodes as classes
Chain Rule?
in the picture, if you want to find the partial derivative of B with respect to X, you just multiply the partial deriviative of B with respect to A by the partial derivative of A with respect to X.
Applies when you have functions of functions
Training Neural Nets - Early Stopping
Accounting for model complexity, where # of epochs indicates model complexity, the early stopping algo will stop the training once the testing error stops decreasig and starts to increase, while the training error continues to decrease
Training Neural Nets - Regularization
sigmoid of a larger number is closer to 1 and sigmoid of smaller number closer to zero.
Concept illustrate multiplying the activation formula by a scaler integer. For the sigmoid activation formula, this leads to steeper slopes and higher chance values either 1 or 0, nothing in between
Be weary of super accurate training models as they may overfit.
Training Neural Nets - Regularization. Whats the issue with steeper slopes?
Harder to do gradient descent with lower range of continuous values.
Harder to tune model to correct errors
Model on right is too certain.
Training Neural Nets - Regularization - Basic concept
Punish large coefficients/weights to avoid steep slopes
Add a term to error function
Training Neural Nets - Regularization - Difference between L1 and L2 regulariztaion
L1 = Lamda * sum of absolute value of weights
L2 = Lamda * sum of squares of weights
Training Neural Nets - Regularization - Why choose L1 or L2?
L1 = weights converted to 1 or 0.
Good for feature selection as 0 weights indicate features are not valuable in prediction. May have 100’s of features.
Good if you want less weights and end with small set
L2 = weights converted to small homogenuous set. Typically best for training models as weights can be tuned(versus on or off settign in L1)
example shows taking absolute value versus sum of squares.
Training Neural Nets - Dropout - What is it? Whats a good comparison?
probability that each node will be turned off during each epoch. Requires other nodes to pick up slack and prevent overfitting
Having a dominant right hand in basketball, and using left hand only during practice to better the overall skill.
Training Neural Nets - How to avoid local minimum?
Random Restarts - Start from a handful of different places and perform gradient descent from there.
Training Neural Nets - Describe Vanishing Gradients?
Using the sigmoid function as an example, when you approach values on the left or right side of the function, the gradient is really close to zero(because the slope is flat)
Training Neural Nets - How to mitigate vanishing gradients?
Use different activation functions which allow for wider range of gradients other than zero
Relu , Tanh
Training Neural Nets - Learning rate(high versus low)
lower typically better as smaller steps will lead to convergence. Large steps may skip over minimum.
Training Neural Nets - Momentum(basic concept), advantages
Use a weighted average of previous steps to avoid a local minimum. The previous step is weighted highest while decreasing for each previous step. May even bounce of the global minimum but not very much.
Beta is between 0 & 1
