Deep Learning-based Deep Neural Networks and Applications Flashcards
What is DL/DNN?
a subfield of machine learning that involves training neural networks to learn from data and make predicitons
What fields are DNNs used in?
computer vision, NLP, speech recognition, etc
What are deep learning algorithms designed to do?
automatically learn features from raw data by stacking multiple layers of nonlinear transformations
What does the term “deep” refer to?
The number of layers which can range from a few to hundreds to thousands
What can NN based on deep learning be viewed as?
a special case of an already existing multi-layer neural network
What was the beginning of deep learning?
Neocognitron which had 7 layers
By who and when was Neocognitron developed?
Dr. Fukushina in early 1980s
What was Neocognitron used for?
handwritten japanese character recognition and pattern recognition
What did Neocognitron inspire?
CNNs (it just didn’t use backpropagation)
What did Hubel and Wiesel win the 1981 Nobel Prize in Physiology or Medicine for?
research on the visual system
What did Hubel and Wiesel discover?
feature detectors (neurons that respond selectively to lines or edges)
What kind of machine learning techniques are a majority?
SVM
When was deep learning started and why?
in 2006 because CPUs and GPUs have become able to handle more massive amounts of data quickly
How was deep learning different from existing multi-layer perceptrons (when it began)?
added pre-processing through learning, had up to 1000 hidden layers as opposed to 1 to 2
What algorithm solves the overfitting problem?
dropout algorithm
When did Hinton and his students win the Turing award for deep learning achievements?
2018
Since 2009 how has deep learning advanced?
has begun to surpass existing pattern recognition methods
What field is deep learning excelling in?
pattern recognition
What is the difference between classification methods in machine learning and deep learning?
machine learning has separate feature extraction and classification steps while deep learning does those simultaneously and automatically
What kind of data sets does deep learning learn from?
large scale data sets
When is deep learning superior to other learning models?
when there are larger data sets
What kind of operations does a CNN use?
convolutional operations
What is CNN used for?
image analysis and image recognition
How do CNNs learn?
using feature maps
What particular image recognition application are CNNs used for?
car systems
What are 3 major CNNs throughout history?
Neocognitron (1980), LeNet-5 (1989-1998), AlexNet (2012)
How does CNN perform object recognition?
measures the likelihood of an object’s identity in an image
What is a loose definition of a CNN?
a neural network created to mimic the way the eye processes image information
How does CNN compare to FNN?
its more complex but more effective for image processing
How does CNN borrow techniques from cat visual cells?
the neurons that are activated depend on where the “cat” is looking at the screen (only specific areas of neurons are activated at certain times)
How is the CNN designed based off of cat visual cells?
there’s a neural network that corresponds to neurons for each region on the screen
What is convolution?
an operation that multiplies the weights of surrounding pixel values and adds them then uses this as a new pixel
How do we implement a CNN to mimic the flow and processing of visual information in a human brain?
using convolutional filters
What is stride?
distance to apply the kernel (if stride=1 then move kernel 1 pixel at a time, stride=2 skip one pixel each move, etc)
If the original size is mm, kernel size is nn, and stride is s, what does the image size become?
(mn)/(s+1) * (mn)/(s+1) (aka 1/stride)
What are CNN weights?
numbers listed in the filter (correlates to the weight of FNN)
What is perceptron learning?
process of modifying weight (modifying numbers inside the filter) which allows features to be effectively extracted from the image
How do you use CNNs to mimic the human visual cortex?
stack CNNs like perceptrons allowing you to learn simple info in low layers and abstract info in high layers
What is a Hierarchical Feature Map?
the information processed at each layer of multiple stacked CNNs that is organized and expressed graphically
What is another word for a feature map?
activation map
What does a feature map show?
the result of extracting features from the input image
How do CNNs use extracted features?
for object recognition
True or false: As layers become deeper the original image becomes less abstract?
false, it becomes more abstract
How do CNNS build neural networks?
by connecting multiple layers
When and why would CNN use an FNN classifier attached to the back?
for video data, because CNN has excellent ability to compress video data but the output of CNN is only 2D so the FNN helps with that
What reduces the size of input data?
pooling aka subsampling
Why should we use pooling?
calculations are faster when the layer size is smaller and the probability of overfitting decreases with smaller neural network parameters
True or false: The higher the floor, the more information it contains?
true
What kind of features does classic computer vision use?
hand-crafted features
Why is computer able to have a certain level of performance?
because it is designed with human intuition
True or false: Computer vision has high performance on complex data like natural images?
false
What was the paradigm change in feature extraction brought by CNN?
learning feature extraction and classification is simultaneous
Why is CNN end-to-end learning?
since the entire process from input to output is learned at once
Which CNN layers are responsible for feature extraction?
convolution layer and pooling layer
Which CNN layer is responsible for classification?
FC layer
Why is CNN superior to traditional CV or FNN in performance?
end-to-end learning, feature learning, uses tens to hundreds of layers, maintains original structure of data, and partial connectivity and weight sharing
What does partial connectivity and weight sharing do in terms of performance?
dramatically reduces the number of operations
What is a RNN?
a recurrent neural network is a type of ANN that is capable of processing sequential data
What do RNNs have that is different from feedforward neural networks?
a feedback loop
What kind of data is RNN useful for?
sequential data such as time series, speech recognition, and handwriting recognition
What is sequential data?
ordered data (either temporal or spatial)
What are some examples of sequential data?
stock prices, text and audio data
What is necessary for processing sequential data and making accurate predictions?
to remember historical data
Why is there a need for RNN, aka a neural network to better handle sequential data?
standard neural networks don’t remember data from the distant past
What are the functions of RNNs?
ability to: handle variable-length inputs, track long-term dependencies, and keep information about the order
What is recurrent data?
data used to train recurrent neural networks
What is necessary to train and RNN?
cutting data to certain lengths to make multiple training sample
What serves as a kind of memory that carries information from the past into the future?
when past learning influences future learning outcomes
What are some applications for RNNs?
language modeling and generation, music composition, speech recognition, and time series anomaly detection
What is a classification model?
ability to recognize features or objects and recognize emotions based off of facial expressions, mainly focuses on discernment abilities
What are some examples of classification models?
multi-layer perceptron, CNN, RNN, etc
What is a generative model?
AI that imitates human handwriting, art styles, etc.
What are some examples of generative models?
HMM and deep learning based generation model
What is the key difference between classification and generative models?
classification models determine which classes inputs belong to, and generative models learn patterns from input data
Describe a generative model.
a neural network that automatically discovers patterns in the training data and creates new samples that resemble the probability distribution of the training data
What does a generative model do to produce output?
learn the latent space representation of training data
What is an auto encoder (AE)?
a neural network that learns input data and performs encoding and decoding to produce an output identical to the input
What are AEs used for?
feature learning, dimensionality reduction, representation learning
What are some applications of AEs?
anomaly detection, abnormal financial transactions, visualization and restoration of data, meaning extraction, and image search
What are the elements of an AE?
encoder, latent space, decoder, loss function
What does the encoder in an AE do?
encodes input into the latent space
What does the latent space in an AE do?
high-level features that compress and represent the original pattern
What does the decoder in an AE do?
unpacks the latent space and restores it to the input
What does the loss function in an AE do?
uses MSE of input and output images
Does modern deep learning need an AE to achieve high performance?
no
Which generation models perform better than AE?
GAN and cycleGAN
What is RBM?
Restricted Boltzmann Machine, a model proposed by Hinton and is an unsupervised generative model that captures higher-order correlations in the data
How does RBM automatically fin patterns in data?
extracts important features and combines features to form patterns
What are applications of RBM?
feature extraction, dimensionality reduction, e.g. spectrum analysis
What kind of neural network does an RBM use?
shallow neural network with only 2 layers (visible layer and hidden layer)
What is GAN?
Generative Adversarial Network
When was GAN introduced and by who?
Ian Goodfellow and others in 2014
How is GAN implemented?
2 competing neural networks in a zero-sum game where one aims to fool the other by producing realistic candidates
What can GAN do?
create fake images or videos that look real
What are the 2 GAN NN called?
the generator (DCNN) and the discriminator (CNN)
What does the generator do?
creates new data instances and is trained to produce data that is similar to the training set
What does the discriminator do?
evaluates the data instances for authenticity and is trained to distinguish between the generated data and real data
What process trains the two NN together?
adversarial training
How does adversarial training work?
when is begins, the generator doesn’t know what to generate so random noise is supplied as input, and it makes random noise output, then the discriminator has an easy time distinguishing between real and fake, but as training goes on, the replicas and distinguishing gets better
Why should we use random noise in generators?
using average pixel value calculation we can produce images but they will be the same every times, so generative models need stochastic random elements to influence the output, essentially the random noise acts as a seed
When did Google train a neural network to recognize cats?
2012
How did Google train the cat recognition model?
expose it to 10 million randomly chosen youtube video thumbnails over 3 days
What kind of learning did the cat recognition model use?
unsupervised learning to identify relevant parts of a photo based on data patterns
What kind of accuracy did the cat recognition network have?
70%
How many connections and computers did the cat recognition network have?
1 billion connections and 16,000 computers
How is deep learning used in computer games?
adjust difficulty, personalize, and improve immersion
How do computer games use deep learning to adjust game difficulty?
based on the player’s performance it adjusts the difficulty to balance the difficulty and player skill level in order to improve player engagement
How do computer games use deep learning to personalize the game?
analyze player data and behavior to create tailored experiences and content recommendations
How do computer games use deep learning to improve immersion?
improve text to speech, speech to text, and speech synthesis
How does cycleGAN differ from GAN?
GAN uses 2 adversarial neural networks, and cycleGAN uses 4 (2 generators and 2 discriminators)
What are some global companies that are using deep learning technology?
google, MS, facebook, twitter, and baidu
How has google used deep learning?
voice recognition, translation, cat image recognition, youtube recommendations, automatic tagging, etc.
How did google contribute to the deep learning research field?
released TensorFlow source code
How has MS used deep learning?
detecting breed of dog from photo, simultaneous interpretation technology for Cortana and Skype
What is MS’s main deep learning project?
Adam Project
How has Facebook used deep learning?
creating DeepFace to recognize human faces, faces can be recognized from various angles or lighting based on user-uploaded pictures, translator tool
What kind of accuracy does DeepFace have?
97%
How has Twitter used deep learning?
photo analysis
How did Baidu break into deep learning?
Prof. Andrew Ng, the leader of the Brain project at Google, moved to Baidu in 2014
Which domestic (Korean) companies use deep learning?
Naver, Daum Kakao, Samsung, LG, etc
How does Naver use deep learning?
voice recognition, news summaries
What is evolving through deep learning?
image recognition
How has image recognition improved?
accuracy increased and recognition time decreased
What kind of deep learning technology will emerge in the near future?
XAI (Explainable AI) which means in addition to recognition results, the reason is also explained
Where are some current limitations of deep learning?
in text, video, and voice recognition
What are the current expectations of deep learning?
learning on large scale data, range getting wider, accuracy improving, scope of application is expanding (many high expectations for the 4th industrial revolution)
Why are GPUs important for deep learning?
they can performs multiple computations simultaneously
Why are GPUs optimized for training deep learning models?
they have thousands of cores
How much faster is a GPU than a CPU?
3 times faster when processing parallel tasks
What is crucial for speeding up deep learning neural network training and why?
parallelism because it requires millions of calculations
What hardware was developed for the parallel processing that is needed by deep learning algorithms?
NPU (neural processing unit)
What is NPU?
a processing unit specifically designed to speed up neural networks also called IPUs (intelligent processing unit)
How does an NPU differ from a GPU?
NPU is specialized for AI tasks and GPU is versatile
What range of tasks is a GPU good for?
graphics rendering, scientific simulations, and computation heavy tasks
What market is rapidly increasing?
the AI semiconductor market
What hardware is currently being developed for AI?
next generation NPUs
Why is TensorFlow used for ML and DL?
it has convenient software kits that returns results when you put information into the library
When was TensorFlow released and by who?
2015 by Google
What is TensorFlow?
a deep learning framework implemented in C++ with multiple interfaces that can be accessed from multiple languages through Python
How does TensorFlow visualize various tasks?
dataflow graphs
What language is Keras written in?
Python
What is the most popular backend to use with Keras?
TensorFlow
What kind of neural networks does Keras use?
feedforward, convolutional, and recurrent neural networks and combinations of them
What kind of hardware processing units can Keras run on?
CPU and GPU
What model type is used in Keras as the core data structure?
sequential linear stack model
How is Google utilizing TensorFlow?
search, ads, google maps, street view, translation, YouTube, etc
How is TensorFlow utilized for ML and DL?
recognition or cursive letters or numbers, voice, image, NLP and machine translation
Who was PyTorch developed by?
Facebook’s AI team
What is PyTorch?
a Python-based open source ML library
When was PyTorch released?
2016, then stabilized in April 2019
what is PyTorch used for?
NLP
129-133
