Deep learning

Question 1

What is machine learning? How is it different from statistics?

Answer 1

What

• A ML system is trained rather than explicitly programmed. It’s presented with many examples relevant to a task and it finds statistical structure (i.e.patterns).
• ML discovers rules. To do ML we need three things:
  
  1. Input data points: images, for example
  2. Examples of the expected output: images of dogs and cats
  3. A way to measure the performance of the algorithm: what’s the distance between the algorithm’s current output and its expected output? This feedback adjusts the way the algorithm works. We call this adjustment step learning

Difference

• Unlike statistics, ML is able to manage large and complex datasets which classical statistics like Bayesian one would be impractical.
• ML (especially deep learning) exhibits comparatively little mathematical theory and is engineering oriented.
• Hands-on discipline in which ideas are proven empirically more often than theoretically

Question 2

What is deep learning?

Answer 2

• Deep Learning (DL) is a specific subfield of ML.
• Learning process here is taken even further to successive layers of increasingly meaningful representations
• The number of layers that contribute to a model is called the depth.
• In DL, the layered representation is commonly learned through models called neural networks

Question 3

What are neural networks?

Answer 3

• NNETs are processing devices (algorithms or actual hardware) that are loosely modeled after the neuronal structure of the mammalian cerebral cortex but on much smaller scales.
• NNETs are typically organised in layers which are made up of a number of interconnected nodes. Each node contains an activation function.
• Patterns are presented to the NNETs via the Input Layer which communicates to one or more Hidden Layers
• Hidden Layers are the processing units where the computations are done via a system of weighted connections
• Finally, results are linked to an Output Layer where the actual answer is output
• Each layer acts as a filter that purifies at each stage the original input data. That’s what DL is!

Question 4

How does the parameterization and evaluation process look like for NNETs?

Answer 4

• A neural network is parametrized by its weights
• The goal is to find the right values for these weights
• Weights are linked to each and every neuron and describe how the neurons are performing
• First weight inizialitation is random
• To control the output of a neural network we have to be able to measure how far the ouput is from what we expect
• The loss function of the network does exactly this job!
• DL uses the score given by the loss function as a feedback signal to adjust the value of the weights
• The direction of the adjustment is to lower the loss function score for the current example
• The adjustment step is carried out by an optimizer which implements the feedback mechanism called backpropagation algorithm
• Doing the adjustments means to train the network

Question 5

What are tensors?

Answer 5

• Most of computer programs can be reduced to a small set of binary operations on binary inputs like AND, OR, NOR, etc.
• The basic ingredient in NNETs is a mathematical object called tensor
• Almost any transformation learned by a neural network can be reduce to few tensor operations applied to tensors of numeric data
• Tensors are generalizations of vectors and matrices which we used to play with in linear algebra
• Tensor operations are generalization of standard linear algebra calculus
• How to interpret these complex tensor operations? Well, they are just geometric transformations of the input data

Question 6

How do activiation functions (a.f.) work? What different a.f. are there?

Answer 6

How

• The purpose of the Activation Function (a.f.) is to check the output value Y and decide whether other connections should consider this neuron as activated or not
• Again, this is a biological similarity: the brain works in the exact same way
• The a.f. introduces non-linearity which allows to capture complex problems which typically can’t be represented by linear functions.
• There is no predefined receipt to pick up the right activation function. For instance, if you know the function you are trying to approximate has given characteristics

Question 7

What types of neural networks exist? Explain them briefly

Answer 7

• Feedforward Neural Networks: no cycles. The information moves only in one direction from the input to the output without loops
  
  • Single-layer Perceptron
    
    • The basic unit of computation in a NNET is the neuron, also called node or unit
  • Multi-layer Perceptron (MLP)
    
    • Multiple layers of computational units
• Convolutional Neural Network (CNN)
  
  • Pooling Layers: they reduce the spatial dimension of the problem
    
    • In particular: they reduce the number of weights (parameters) up to 75% thus lessening the computational cost
  • Dropout Layers: they also take care of the overfitting problem
    
    • The idea is to make the network ”redundant” so that it should be able to provide the right classification even if some activations are dropped out
• Recurrent Neural Networks (RNN)
  
  • ​With respect to feedforward networks, RNN have feedback loops
  • They are generally applied to sequential tasks like handwriting and speech recognition, NLP