Chapter 8 Evaluate The Performance of Deep Learning Models

Question 1

It is typical to use a simple separation of data into training and test datasets or training and validation datasets for performance evaluation. Keras provides two convenient ways of evaluating your deep learning algorithms this way, what are they? P 62

Answer 1

1. Use an automatic verification (validation) dataset.
2. Use a manual verification (validation) dataset.

Question 2

How can we use an Automatic Verification Dataset in Keras to separate a validation set for performance evaluation after each epoch? P 62

Answer 2

Keras can separate a portion of your training data into a validation dataset and evaluate the performance of your model on that validation dataset each epoch. You can do this by setting the “validation_split” argument on the fit() function to a percentage of the size of your training dataset.
model.fit(X, Y, validation_split=0.33, epochs=150, batch_size=10)

Question 3

How can we use a manual Verification Dataset in Keras to separate a validation set for performance evaluation after each epoch? P 62

Answer 3

1- Using train_test_split to create a test dataset
2- Setting the “validation_data” argument on the fit() function to (X_test,y_test)
model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=150, batch_size=10)

Question 4

The gold standard (for splitting data) for machine learning model evaluation is…. P 64

Answer 4

k-fold cross validation

Question 5

Cross validation is often used for evaluating deep learning models.True/False P 64

Answer 5

False. Cross validation is often not used for evaluating deep learning models because of the greater computational expense, Nevertheless, when the problem is small enough or if you have sufficient compute resources, k-fold cross validation can give you a less biased estimate of the performance of your model.

Question 6

define 10-fold cross validation test harness

What does the below code do? P 64
# MLP for Pima Indians Dataset with 10-fold cross validation
from keras.models import Sequential
from keras.layers import Dense
from sklearn.model_selection import StratifiedKFold
import numpy
# fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)
# load pima indians dataset
dataset = numpy.loadtxt(“pima-indians-diabetes.csv”, delimiter=”,”)
# split into input (X) and output (Y) variables
X = dataset[:,0:8]
Y = dataset[:,8]
from sklearn.model_selection import StratifiedKFold

kfold = StratifiedKFold(n_splits=10, shuffle=True, random_state=seed)
cvscores = []
for train, test in kfold.split(X, Y):
# create model
model = Sequential()
model.add(Dense(12, input_dim=8, activation= “relu” ))
model.add(Dense(8 , activation= “relu” ))
model.add(Dense(1, activation= “sigmoid” ))
# Compile model
model.compile(loss= “binary_crossentropy” , optimizer= “adam” , metrics=[ “accuracy” ])
# Fit the model
model.fit(X.iloc[train], Y.iloc[train], epochs=150, batch_size=10, verbose=0)
# evaluate the model
scores = model.evaluate(X.iloc[test], Y.iloc[test], verbose=0)
print(“%s: %.2f%%” % (model.metrics_names[1], scores[1]*100))
cvscores.append(scores[1] * 100)
print(“ (+/- {:.2f})”.format(numpy.mean(cvscores), numpy.std(cvscores)))

Answer 6

The example creates and evaluates 10 models using the 10 splits of the data and collects all of the scores. The verbose output for each epoch is turned off by passing verbose=0 to the fit() and evaluate() functions on the model. The performance is printed for each model and it is stored. The average and standard deviation of the model performance is then printed at the end of the run to provide a robust estimate of model accuracy.

🧨Notice that we had to re-create the model each loop to then fit and evaluate it with the data for the fold.