Chapter 1: Fundamentals of Deep Learning

Question 1

Concise AI Definition

Answer 1

The effort to automate intellectual tasks normally performed by humans

Question 2

Symbolic AI

Answer 2

The approach towards creating human-level artificial intelligence by having programmers handcraft a sufficiently large set of explicit rules for manipulating knowledge

Question 3

What was the dominant paradigm in AI from the 1950s to the late 1980s?

Answer 3

Symbolic AI

Question 4

What did Symbolic AI do well? and poorly?

Answer 4

did well with well-defined logical problems ex chess not well with fuzzy complex problems ex image classification

Question 5

What replaced Symbolic AI

Answer 5

Machine Learning

Question 6

What question gave rise to machine learning?

Answer 6

Could a computer go beyond what we know how to order it to perform? and learn on its own to perform a specified task?

Question 7

What is the classical programming approach?

Answer 7

Humans input rules (a program) and data to be processed according to these rules, yielding answers

Question 8

Machine Learning steps

Answer 8

Humans input data and the answers to the data and the computer outputs the rules that connect the two. These rules can then be applied to new data for original answers.

Question 9

Are machine learning systems programmed?

Answer 9

No, they’re trained on data in which it finds statistical structure.

Question 10

What is the relationship between Deep Learning, AI and Machine Learning?

Answer 10

(AI (ML (DL)))

Question 11

Predefined set of operations from which ML selects its transformation?

Answer 11

Hypothesis Space

Question 12

What is Deep Learning

Answer 12

an approach towards learning representations from data that puts an emphasis on learning successive layers of increasingly meaningful representations. Information-distillation via successive layers.

Question 13

The layered representations in Deep Learning are almost always learned via what models?

Answer 13

neural networks

Question 14

What determines the change a deep learning layer affects on its input?

Answer 14

Its weight, the change implemented by the layer is parameterized by its weights

Question 15

Loss Function of the network

Answer 15

measures how far an output is from what is expected, capturing how well the network does

Question 16

The central algorithm in Deep Learning

Answer 16

The Backpropagation algorithm

Question 17

What happens to a network’s loss score?

Answer 17

It is used by the optimizer to implement the backpropagation algorithm to adjust the weights of the layers

Question 18

What is the training loop for Deep Learning?

Answer 18

The process of starting with random weight values and through repeated feedback from the loss score adjusting layer weights

Question 19

What is the goal of the training loop?

Answer 19

to find the layer weights that minimize the loss score

Question 20

Probabilistic Modeling

Answer 20

An early form of Machine Learning. The application of the principles of statistics to data analysis

Question 21

What is the best-known algorithm in Probabilistic Modeling?

Answer 21

Naive-Bayes

Question 22

Naive Bayes

Answer 22

type of ML classifier based on applying Baye’s theorem while assuming that the features of in the input data area all independent (a strong/naive assumption)

Question 23

Logistic Regression

Answer 23

a CLASSIFICATION algorithm not regression. Closely related to Naive Bayes, a type of probabilistic modeling

Question 24

What held neural networks back for a long time?

Answer 24

The missing piece was an efficient way to train large neural networks

Question 25

What is the Backpropagation Algorithm

Answer 25

A way to train chains of parametric operations using gradient-descent optimization

Question 26

Kernel Methods

Answer 26

a group of classification algorithms, the best known is SVM

Question 27

Support Vector Machine (SVM)

Answer 27

A classification algorithm which solves classification problems by trying to find good decision boundaries between categories.

Question 28

How do SVMs decide on a decision boundary?

Answer 28

1. data mapped onto hyperplane 2. try to compute the decision boundary that will best “maximize the margin” or maximize the distance between the closest points from each class

Question 29

Kernel Trick

Answer 29

What gives kernel methods their name since otherwise they would often be computationally intractable. You dont have to compute coordinates of your points in new space just the distance between pairs of points; done via Kernel Function

Question 30

Decision Trees

Answer 30

Flowchart-like, successive questions, allow for input classification or predict output values given inputs

Question 31

Random Forest

Answer 31

A specific Decision tree algorithm involves building a large number of specialized decision trees and ensembling their outputs.

Question 32

Gradient Boosting Machines

Answer 32

a ML technique based on ensembling weak prediction models, generally decision trees. “gradient boosting’ = Iteratively training new models that specialize in addressing the weak points of the previous models

Question 33

Go-to algorithm for computer-vision tasks and generally all perceptual tasks

Answer 33

Deep Convolutional Neural networks

Question 34

What were the key factors in Deep learning’s success?

Answer 34

1. That it offered better performance on a wide array of problems. 2. Was much easier because it completely automates what used to be the most crucial step in ML workflow: Feature Engineering. in DL you learn all the features in one pass rather than having to reengineer them yourself

Question 35

Why can’t multiple shallow-layer ML methods be applied in sequence to replicate Deep learning?

Answer 35

Because the optimal first representation layer in a three-layer model isn’t the optimal first layer in a one-layer or two-layer model.

Question 36

What is transformative about deep learning?

Answer 36

it allows a model to learn al layers of representation JOINTLY, at the same time, rather than in succession (greedily)

Question 37

Deep Learning could also be called?

Answer 37

Joint Learning, layers adjust together as any layer changes

Question 38

2 essential characteristics of how Deep Learning learns from Data?

Answer 38

1. the incremental, layer by layer way in which increasingly complex representations are developed 2. that the intermediate incremental representations are learned jointly

Question 39

Two most important ML techniques to be familiar with today and their most effective uses

Answer 39

gradient-boosting machines for shallow-learning problems where structured data is available. Deep Learning for perceptual problems

Question 40

Dominant Deep Learning Library?

Answer 40

Keras

Question 41

Dominant gradient-boosting library?

Answer 41

XGBoost

Question 42

The fundamental algorithm for Deep Learning in time series

Answer 42

The Long Short-term Memory Algorithm (LSTM)

Question 43

Why are deep neural networks parallelizable?

Answer 43

Because they perform mostly many small matrix multiplications

Question 44

TPU

Answer 44

Tensor Processing Unit. A chip designed and built by google from the ground up specifically for deep neural networks.

Question 45

What algorithmic improvements allowed for better gradient-propagation circa 2010

Answer 45

1. better activation functions for neural layers
2. better weight-initialization schemes
3. Better optimization schemes (RMSProp and Adam)

Question 46

Theano and Tensorflow

Answer 46

two symbolic tensor-manipulation frameworks for python that support autodifferentiation simplifying the implementation of new models

Question 47

What makes Deep Learning Neural nets so scalable?

Answer 47

Highly amenable to parallelization, DL models trained by iterating over small batches of data, so they can be trained on data sets of arbitrary size

Question 48

What makes Deep Learning so much simpler than other Machine Learning techniques?

Answer 48

No need for feature engineering