18 in on the paper. Class 19

Question 1

What are autoencoders for? What is undercomplete and overcomplete autoencoders?

Answer 1

The general idea of the autoencoders is, to learn a representation of the data by using the identity function x = g(f(x))
It is generally used to learn compressed or sparse code h for input x
Identity functiopn does not need to learn the representation for the whole input space.
Undercomplete Autoencoders
h is smaller than x
we can think of those autoencoders as a limitation on hardware, the system can not learn the identity function
Overcomplete autoencoders
h is larger than x, therefore the system becomes complex, there is the need to use regularization, we can think of the overcomplete autpencoders as, limitations on software because the system is able to learn the representation but not the constraints

Question 2

Linear autoencoders?

Answer 2

linear autoencoders are undercomplete autoencoders which means that they have smaller number of hidden units compared to the units in input and output. Generally it uses PCA or SVD. those methods are trying to apply dimensionality reduction and tries to find the meaningfull representation of the data.

Question 3

Shallow Autoencoders ?

Answer 3

Shallow autoencoders are undercomplete, they have smallerr number of units in the hidden space compared to the input and output units. The input and the otuput are exatcly the same in shallow autoencoders which means the inputs are also the outputs.

Question 4

Deep Autoencoders?

Answer 4

With deep autoencoders, after shallow and linear autoencoders, we finallyhave non linearity. Deep autoencoders are also

Question 5

Deep Autoencoders?

Answer 5

With deep autoencoders, after shallow and linear autoencoders, we finallyhave non linearity. Deep autoencoders are also undercomplete autoencoders. It does a mirroring affect in the encoder and the decoder part, it starts from a certain number of units and it decreases in the hidden space, then when it comes to the encoder part, it does a mirroring, and it starts to decrease again.

Question 6

What are the overcomplete autoencoders?

Answer 6

These are regularized autoencoders. 
Sparse Autoencoders
Denoiseing autoencoders
Contractive autoencoders
Autoencoders with dropout on the hidden layer

Question 7

Sparse Autoencoders?

Answer 7

limit the capacity of the autoencoder by adding a regularization term with the cost function, penalizing the entries because h the hidden space is greater than the input space.
L(x,g(f(x)) + saçmaişşaret(h)

Question 8

Denoising Autoencoder?

Answer 8

Instead of learning an identity function what we want is to learn how to remove the noise from the data.
we take the original data, we corrupt it with noise and we give it to out autoencoder, network learns how to project the corrupted inputs into the original ones

Question 9

Representing joint probability?

Answer 9

most variables influence each other
Most variables do not influence every variabe directly.
Edges represents direct influence
Paths represents influence
Directed models have a flow going from one direction to another(casual flow)

a-> b -> c ->
p(a). p(b|a). p(c|a,b)
The last one means that, c is consitionally independent from a, given b which actually means p(c|b)
Undirected models have no clear flow
a-> s -> b when s is observed it blocks the flow between a and b
d-seperation is used in directed models and more complicated