Reverse Engineering Human Visual Intelligence Flashcards
What does “Reverse Engineering” mean?
Forward engineering models within wisely chosen constraints coming from brain measurements, test deviations from such measurements and adjust the engineering if the deviations are growing further away from the biological system.
What are the steps of reverse engineering?
- Step 1: Define and operationalize a domain of interest
- Step 2: Get the measurements of the internal system relevant to the domain of interest
- Step 3: Engineer a model under the constraints of such measurements
- Go back to Step 2 to get more measurements and repeat Step 3 under the constraints of new measurements.
- Repeat until you end up with a model that encapsulates Understanding of the domain of interest.
What is the measure of the center of gaze in humans?
Approximately 10 degrees.
What problems in object recognition machines were not able to handle in 2009?
Recognition of the objects that have high uncertainty in object view.
What is the unit of information transmission for vision?
spikes
What is the basic architecture of the model of primate visual system?
It is a deep neural network, largely feedforward, with layers corresponding to the biological visual system: RGC, LGN, V1, V2, V4, IT
What is the mean firing rate of the site to the image?
It is an averaged measure used to model the response of neurons in IT. IT neuronal spiking responses comprise different patterns of activity for different images. The spikes are recorded from multiple sites in IT and then averaged over time and repetitions for each site into one metric: “the mean firing rate of the site”.
What do neural population “feature vectors” encode and how are they used in the model of visual system?
The “feature vectors” encode the averaged response of IT neurons to an image. This vector can be further mapped into the neural population state space where it will represent the response to a certain image. Mapping multiple such responses and building a linear classifier on this population space allows further prediction of the animal’s responses to images through decoding the data from IT.
What are the constraints for building a model of ventral stream neural processing?
- deep architecture
- spatially local, linear filters in each layer
- threshold non-linearities
- non-linear pooling
- normalization
- convolution
What are the layers of HMO model?
4 layers: V1, V2, V4, IT
Does HMO have initial neural fitting?
HMO has no initial neural fitting, but is just performing the object recognition task. Only after the task the responses of the HMO’s artificial neurons are mapped to the same linear space as the responses of biological neurons in IT. This allows to find the linear mapping from HMO’s neurons to IT neurons and further make predictions of IT response based on HMO’s response.
Responses in what layer(s) in the ventral stream did HMO predict best?
HMO’s layer 3 predicts V4, with prediction accuracy about 50%;
HMO’s top layer predicts IT, with prediction accuracy about 50%.
What is RDM and what does it measure?
RDM - representational dissimilarity matrix that represents the distances in neural responses to different images. It is used as yet another metric to compare model representations and brain representations.
Is optimizing object recognition performance of the neural network the right strategy for engineering the model that explains responses in ventral stream?
It is an important part of the strategy, but not sufficient for building the model that explains responses in IT. As machine vision networks advanced, so did their IT response prediction accuracy, but only up to a certain point after which it began to decline. Therefore, we need more constraints from neuroscience to adjust the model.
What is meant by non-invasive neural control?
Find control stimuli - “controller image” - that will drive a neuron to a certain state. Further by showing these stimuli to the animal we can non-invasively manipulate the neural response by driving the neurons into a desired state.
