Weel 9 - Object Recognition and Categorisation

Question 1

What is Indexing with Local Features

Answer 1

Each patch/region surround a point of interest has a descriptor; some high-dimensional feature space (e.g., SIFT)
Close points in feature space have similar descriptors, indicating similar local content
Important for 3d reconstruction and
Retrieving images of similar objects
(try match target image features to descriptors)

Question 2

How do we efficiently find the relevant features of a new image

Answer 2

Using the idea of an inverted file index

Question 3

What is Inverted File Index

Answer 3

for text docs, use an index to find pages where word occurs
We want to find all images(pages) in which a
feature(word) occurs
we want to map our features to ‘visual words’

Question 4

Visual words: Main idea

Answer 4

Extract some local features from a number of images and map them into the 128-dimension space (if using SIFT)
Each point in the space is a local descriptor (SIFT vector)

Question 5

How do we match visual words

Answer 5

When we see close points in feature space, we have similar descriptors (similar content)
Content is close enough to assume it is the same

Question 6

How do we use clusters for visual words

Answer 6

We can create clusters to reduce the complexity (millions of points) to far fewer clusters which are considered the same
“quantize via clustering”

Question 7

What are the cluster centres

Answer 7

the prototype “words”

Question 8

How do we create the inverted file for visual words

Answer 8

-database of images
-run sift, find interest points and encode descriptors
-cluster descriptors
-Create our list of visual words (cluster centres)
-we pass all images through the visual words
-for each word, we have the list of images where this visual word occurs

Question 9

How does the inverted file index handle new images

Answer 9

extract visual words it contains (sift)
map image to relevant words in the index
find all the other images that contain the same words
then compare word counts
(images will have the same similar visual words in common)

Question 10

What is Spatial Verification

Answer 10

Sometimes, non similar images will have high visual word similarity (eg buildings with lots of similar windows)

spatial verification is used to check the images are actually the same
only some of the matches are mutually consistent

Question 11

What is the spatial verification strategy

Answer 11

Use the generalised hough transform
Let each matched feature cast a vote on location, scale, orientation of the model object
(uses encoded information about the position, scale, and orientation of each feature match)
Verify parameters with enough votes

Question 12

What is the Video Google system

Answer 12

eg find all scenes in film where actor is wearing blue tie
1.Collect all words within query region
2.Inverted file index to find relevant frames
3.Compare word counts
4.Spatial verification

Question 13

What are the issues with visual vocabulary formation

Answer 13

• Sampling strategy: where to extract features eg blobs or corners…?
• Clustering / quantization algorithm
• Unsupervised vs. supervised(external labels or annotations are used to guide the clustering process)
• What corpus provides features (universal vocabulary?)
• Vocabulary size, number of words(too small-> may not capture visual content)

Question 14

What is a good sampling strategy to find specific, textured objects

Answer 14

Sparse sampling at interest points

Question 15

What is a good sampling strategy for object categorisation

Answer 15

Dense sampling

Question 16

What is a good sampling strategy for more image coverage

Answer 16

Multiple complementary interest operators

Question 17

What are 4 main sampling strategies

Answer 17

Randomly
Multiple interest operators
Dense
Sparse

Question 18

What are some clustering/quantisation methods

Answer 18

k-means (typical choice)
agglomerative clustering
mean-shift

Question 19

What is a query region

Answer 19

Eg want to find specific object in images
pull out only the SIFT descriptors whose positions are within the relevant polygon

Question 20

What is object categorisation

Answer 20

Instead of recognising a specific dress
how can we recognise any dress
(a category)

Question 21

What is the task description of object categorisation

Answer 21

Given a small number of training images of a category, recognize a-priori unknown instances of that category and assign the correct category label

Question 22

What are Visual object categories

Answer 22

humans tend to find ‘basic-level categories’
EG
abstract level = animal, mammal
basic level = dog, cat, cow
individual level = dobermann, “Gary”

Question 23

What is a-priori unknown

Answer 23

not known or seen by the system during the training phase

Question 24

What is a functional category

Answer 24

“Something i can sit on”
“something i can eat”

Question 25

what is an ad-hoc category

Answer 25

“something you can find in an office environment”

Question 26

What are challenges to robustness of object categorisation

Answer 26

illumination
object pose
clutter
occlusions
intra-class appearance
viewpoint

Question 27

what is the scale of supervision we can use

Answer 27

less: unlabeled, multiple objects
medium: classes labelled, some clutter
more: cropped to object, parts within object are labelled

Question 28

What features must our visual word representation have for object categorisation

Answer 28

robust to intra-category variation
robust to deformation, articulation
…
still discriminative

Question 29

What is the loose v strict BoW definition

Answer 29

Looser: independent features
Stricter: independent features with histogram representation (how frequently each word appears in image)

Question 30

BoW overview

Answer 30

1. feature detection and representation (SIFT)
2. create codewords dictionary (visual word index)
3. Image representation via bag of codewords

Question 31

BoW: Feature detection and representation

Answer 31

creating regular grid:
- eg histogram
Interest point detector:
- Use state-of-the-art interest point detector
- represent features using SIFT

Question 32

BoW: Image representation

Answer 32

for each image, we have frequency histogram
with each visual word and the count of how many times it appears

Question 33

How do we compare BoWs

Answer 33

We can use many methods with histogram frequency data
Eg euclidean distance, normalised scalar product

Question 34

Recognition with BoW histograms

Answer 34

BoWs representation means we can describe
the unordered set of points with a single vector (of fixed dimension across image examples)
this provides an easy way to use distribution of feature types

Question 35

what are the two method types for recognition

Answer 35

generative methods
discriminative methods

Question 36

What is a discriminative method for recognition

Answer 36

Learn a decision boundary/ rule (classifier) assigning bag of features representation of images to difference classes
Zebra/non-zebra

Question 37

What is a generative method for recognition

Answer 37

Use probability
p(image|zebra)
p(image|no zebra)
look at likelihood values

Question 38

Example discriminative: knn classification

Answer 38

map histogram in graph space, with boundaries between images
when we have a new image, build histogram which maps to a point in the graph
find the k nearest histograms
if the average says (positive eg yes zebra) then positive (negative -> negative)

Question 39

nearest neighbour classification pros

Answer 39

Simple to implement
Flexible to feature / distance choices
Naturally handles multi-class cases
Can do well in practice with enough representative data

Question 40

nearest neighbour classification cons

Answer 40

Large search problem to find nearest neighbors
Storage of data
Must know we have a meaningful distance function

Question 41

what are some other types of discriminative classifiers

Answer 41

boosting
SVMs

Question 42

Example generative: the naive bayes model

Answer 42

Assume that each feature is conditionally independent
p(w1,…,w|c) = ∏p(wi|c)
want to maximise:
c* = argmaxc p(c)∏p(wi|c)
if we know nothing about data assume uniform prior p(c)

Question 43

What is p(c) in naive bayes

Answer 43

Prior prob. of the object classes

Question 44

What is p(wi|c) in naive bayes

Answer 44

Likelihood of i-th visual word given the class
Estimated by empirical frequencies of visual words in images from a given class

Question 45

what is ∏

Answer 45

multiply them together

Question 47

How can we improve spatial information of BoW model

Answer 47

-visual phrases: frequently co-occurring words
-semi-local features: describe configuration, neighbourhood
-let position be part of each feature

Question 48

BoW pros

Answer 48

• Flexible to geometry / deformations / viewpoint
• Compact summary of image content
• Provides vector representation for sets
• Empirically good recognition results in practice

Question 49

BoW cons

Answer 49

• Basic model ignores geometry – must verify afterwards, or encode via features.
• Background and foreground mixed when bag covers whole image
• Interest points or sampling: no guarantee to capture object-level parts.
• Optimal vocabulary formation remains unclear.