Lecture 18- Build a neural network for language

Question 1

What do we need to build a neural network?

Answer 1

• Something that can activate (fire) with a threshold.
• Inputs and outputs to that something
• These inputs and outputs are the connections
• The connections can be excitatory or inhibitory

Question 2

How do you build into a model the concept of neural firing being excitatory or inhibitory?

Answer 2

• Weighted connections

- Range from -1 (inhibitory) to +1 (excitatory)

Question 3

What is the model more commonly used in neural networks as opposed to all or none threshold activation?

Answer 3

-A sigmoid (gradual) curve

Question 4

How do you work out how much input something is getting?

Answer 4

• For each neuron firing to it times by weighting
• Add all of this up
• This gives you the output (if it has reached threshold

Question 5

What is the simplest way we might model language with a neuron network? What is this method called?

Answer 5

• One input/ neuron= One word

- A localised representation

Question 6

What is a more realistic way we might model language with a neuron network? What is this method called?

Answer 6

• One input/neuron= one letter
• So a word means spelling out the letters in the word
• Called a distributed representation

Question 7

What are the steps to ‘teach’ a neural network to get words from letters?

Answer 7

1: Activate the input
2: Get some random output
3: Trigger an error
4: Adjust weights so connections to correct answer get more weight and connections to incorrect ones get less.
5: Repeat training over and over (thousands) so network learns lots of words

Question 8

Who where the researchers who did lots of research in this area and what did they teach their network to do?

Answer 8

• Rumelhart & McLellan
• Input was the word and the goal was to teach the network to give the past tense
• Follow the exact same process as before

Question 9

In Rumelhart and McLellan’s neural network what problem presents with verbs like Kiss?

Answer 9

Network too simple to have two S

Question 10

After a while (many repetitions) what does the network learn to do instead of just memorizing?

Answer 10

• Learn the pattern
• For the verbs this means that it automatically puts ed in as it knows that this is the past tense in English
• Can test that the network has learnt the pattern by entering in a new word that has not been trained on, should be able to make it into the past tense without being taught explicitly

Question 11

What problem presents when the pattern for making verbs into the past tense has been learnt?

Answer 11

• There are irregular verbs that don’t follow the rule
• The network will get these wrong due to over generalizing
• Rumelhart and McLellan found it took longer for the network to learn these verbs with accuracy (have to memorize them one by one)