College 3

Question 1

Define: Occam’s razor

Answer 1

Prefer the simplest hypothesis that fits the data.

‘all thing being equal, the simplest solution tends to be the best one’

Question 2

x1= 1
x2= 2
x3=3

w0= 0.7,	
w1=  0.5	
w2= -0.5 	
w3 = 1

what will be the output of the perceptron?

Answer 2

0.5 -1 + 3 +0.7 = 3.2 > 0 so 1

Question 3

What are the main drivers of the breakthrough in deep learning?

Answer 3

• data
• computation
• machine learning

Question 4

What are flaws in deep learning?

Answer 4

Adding carefully crafted noise to a picture can create a new image that people would see as identical, but which a DNN sees as utterly different. In this way, any starting image can be tweaked so a DNN misclassifies it as any target image a researcher chooses.

Question 5

What are strengths in deep learning?

Answer 5

• ability to integrate information from huge heterogeneous sources
• ability to predict
• ability to detect / recognise
• ability to discover patterns

Question 6

What are weaknesses in deep learning?

Answer 6

• DL algorithms lack ‘common sense’
• DL cannot put events into their context
• DL depends critically on the quality of the underlying statistics/data
• DL is opaque
• DL is ill-understood

Question 7

Define: Amara’s law

Answer 7

We tend to overestimate the effect of a technology in the short run and underestimate the effect in the long run.

Question 8

How to deal with AI?

Answer 8

• ignore the hype, focus on the concrete impact for your domain
• without human, AI algorithms are of no value
• Many applications will see a performance boost
• Many domains/jobs will transform dramatically (also in science)
• AI algorithms offer tremendous powerful tools but lack proper understanding.

Question 9

What does Tishby’s information plane say?

Answer 9

The first hidden layer has a lot of information about the input as well as the output and the last hidden layer has little information on both (at initialisation)

Question 10

What are the two phases in deep learning mechanics?

Answer 10

1. mapping input to output (fast).

2. getting rid of noise (slow) compressing representations

Question 11

What is the relationship between deep learning and gaussian processes.

Answer 11

Dropout – a technique that’s been in use in deep learning – can give us principled uncertainty estimates. These uncertainty estimates basically approximate those of our Gaussian process.

Question 12

Define: lottery ticket hypothesis

Answer 12

If you want to win the lottery, you should buy all the tickets
There are many pathways from input to output, there is always one winning ticket, so try all the options.
This is an explanation why DL is successful and why you use so many parameters.

Question 13

How does network pruning work (standard approach)?

Answer 13

1. Train the network (randomly initialise weights: Wr)
2. Remove superfluous network (i.e., small weights)
3. Fine-tune the network

option: repeat steps 2 and 3

Question 14

What is step 3 in the alternative approach to network pruning?

1. Train the network
2. Remove superfluous network (i.e., small weights)
3. ?

Answer 14

Retrain the network by initialising with Wr

Question 15

After pruning, subnetworks can do the job. What is the advantage of using subnetworks?

Answer 15

1. The subnetworks are typically between 1% - 15% of the original size
2. They require considerably less training time.
3. they perform at or around the same level

Question 16

Why is overparameterization useful?

Answer 16

You need to keep the original initialisation when pruning because random reinitialisation destroys the advantage of pruning.

Question 17

What does new work say about overparameterization?

Answer 17

Given a randomly initialised overparameterized network, find subnetworks that perform that task. Such subnetworks (composed of random weights can even outperform trained ones.

Question 18

What are the conclusions?

Answer 18

1. deep learning networks are overparameterized.
2. Their learning dynamics consist of two phases
3. large depth may provide a computational, rather than a representational advantage
4. Overparameterization may provide a sampling rather than a representational advantage
5. random subnetworks may outperform trained dense ones.