Lecture 19- Learning and Structure in Neural Networks Flashcards
What are the two functions of neurons?
- An input function
- An activation function
What is the architecture of a neural network?
How many neurons there are, and how they connect
List the four components of a neural network…
- Neurons
- Architecture
- Weights
- Training
What components of a neural network are relatively constant in their design?
- Weights
- Neurons
What are the two types of architecture what is the difference between them?
- Classifier= tons of inputs, one outcome
- Transformation= lots of inputs, lots of outputs
What is information compression?
-Squeeze data into smaller size
What are some uses of compression in terms of technology?
Built around unchanging pixels:
-In an image, pixels right next to each other are
often about the same
-In a movie, the next frame is usually about the
same as the last frame (especially in areas)
Built around how common patterns are:
-If something is very common, give it a very brief
shorthand
-If something is uncommon, give it a longer
shorthand
How does compression work in terms of language?
Find patterns then just need to code the unique part and say add the common part.
E.g. if everything starts with “the noun”
Just transfer the noun part which will be different each time
How does compression work in terms of children learning grammar?
-Learn to pick up the pattern in what they are hearing i.e in the “the noun” example only a small number of words (class determiners) can go in the “the” spot
What is classifying architectures generally used for?
Word recognition
put in order…
learning
compression
pattern finding
- Compression
- Pattern finding
- Learning
How can you being learning once you recognize patterns?
Pattern finding is essentially learning as if you can find the pattern you are no longer memorizing you can take things to novel situations
What are two ways learning can go wrong?
- Overfitting: A network can just memorize every
answer. So no pattern is ever learned
-Underfitting: A network might lose too much
information and not be able to answer
How is capacity invovled in choosing an architecture?
How can this problem be fixed?
- If capacity far bigger than problem, no learning can occur just memorizing.
- Few hidden nodes requires learning
What are the two key points from this lecture?
- The network structure imitates the problem to solve
- Nothing in the system indicates what it is about. This is decided from outside.
