Week 8 - Image stitching Flashcards
What is Panorama stitching
An application of SIFT
taking different images with different rotations and stitch them together
What is the issue with just directly stitching two images
There will be illumination changes eg dark corners that will show up
What is the first step of image stitching
Finding corresponding points
What is image alignment
Fitting a model M to a transformation between pairs of features (matches) in two images
Find transformation T that minimises
Σresidual (T(xi),x’i)
residual = residual errors
xi is the set of points
2D transformation models: Euclidean
translation + rotation
2D transformation models: Affine
translation + rotation + scaling + shear
(shear = stretching
eg rectangle to rhombus)
2D transformation models: Similarity
translation + rotation + scaling
2D transformation models: projective
homography
includes out of plane rotations
what is x,y in a grayscale function
the domain
what is the pixel intensity I in the grayscale function
the range
What does filtering/convolution do in terms of pixels
changes the pixel values (range)
G(x) = h{F(x)}
What does warping do in terms of pixels
changes the pixel locations (domain)
G(x) = F(h{x})
What are examples of global warping/transformation
translation
rotation
scaling and aspect
affine
perspective
barrel
What is transformation T
A coordinate changing machine
p’ = T(p)
What does it mean when T is global
is the same for any point p (applied to all)
can be described by just a few numbers (parameters)
What is uniform scaling
same scaling for each components
S = 2x2 = [sx 0 0 sy]
What is Rotation matrix around θ
2x2 = [cosθ -sinθ sinθ cosθ]
How do we represent translation in matrix form
Not possible
translation is not a linear operation on 2D coordinates
What are all 2D linear transformations a combination of (4 types)
scale
rotation
shear
mirror
what are the properties of linear transformations
■ Origin maps to origin
■ Lines map to lines
■ Parallel lines remain parallel
■ Ratios are preserved
■ Closed under composition
What is Closed under composition
apply multiple linear transformations successively by multiplying the transformation matrices then applying
What are homogeneous coordinates
represent 2d point with a 3d vector up to a defined scale
move from 2d to 3d
(x,y,w)
how to convert back from homogeneous coordinates
(x, y,w) -> (x/w, y/w)
What are basic affine transformations
scale
shear
translation
2d in-plane rotation
What is always the matrix form for basic affine transformations
any 3x3 matrix with bottom row:
0 0 1
what is the only difference between properties of affine and linear transformations
affine transformations do not necessarily map to origin
What are all Affine transformations a combination of
linear transformations and
translations
If we stay in same place and rotate camera is it an affine transformations
No
What happens if the bottom row of 3x3 transformation matrix is not 001
Plane projective transformations or
Homographies
Projects one plane onto a different plane via a particular point of projection
What are black areas in homographies
Where there is no pixel available from original image to map new perspective projection
What are homographies a combination of
affine transformations and projective warps
what is the difference between linear transformations and homographies
origin does not necessarily map to origin
parallel lines do not necessarily remain parallel
ratios are not preserved
(still: Lines map to lines, Closed under composition)
How do we solve homographies
Defines a constrained least squares problem
What do we assume about the length of vector when solving homographies
lenght of vector (h00…h22) is 1 (we normalise it)
so we only have 8 unknowns
Need 4 pairs of (x,y) to solve
What do we generally assume about feature extraction
That we have extracted a set of correct correspondences between the two images (ground truths)
generally, this is not the case
Image alignment: outliers
(incorrect match)
image alignment does not work well with outliers
least squares does not find a good line of best fit with outliers
What are inliers
correct matches
How do we deal with outliers
- Given a hypothesized line
- Count the number of points that “agree” with the line - they are on the line
■ “Agree” = within a small distance of the line
■ I.e., the inliers to that line - For all possible lines, select the one with the largest number of inliers
What is RANSAC
(RANdom SAmple Consensus)
How does RANSAC work
RANSAC loop:
1. Randomly select a s sample matches
s == minimum sample size that lets you fit a transformation model
2. Compute transformation (find a model that fits) from sample group
3. Find inliers to this transformation
4. If the number of inliers is sufficiently large (model is good fit), re-compute least squares estimate of transformation on all of the inliers
This refine the inliers to improve accuracy
5. Repeat N times
6. Keep the transformation with the largest n
(needs amount inliers to be larger that outliers to work)
How many times do we runs RANSAC
related to how many outliers we expect
and probability of success we would like
What is forward warping
we have source and target image and T homography matrix
send each pixel x,y to its corresponding x’,y’ = T(x,y)
What is the issue of forward warping
if a pixel lands between two pixels
solve by adding a contribution
normalise later (splitting)
But we can still get holes
What is Inverse warping
Take each pixel from target image and find inverse transform
move from x’y’ to x’y’
using x,y = T-1(x’,y’)
What is the issue with inverse warping
if the pixel again comes between two pixels
we can resample colour values from interpolated source image (we have all the information)
What are the steps of creating panoramas
detect features SIFT
match features
compute a homography using RANSAC
combine images together (image blending)
What is alpha blending
Simplest method of blending
close to the seam
interpolate using an α parameter
Iblend = αIleft + (1-α)Iright
What is the effect of window size on image blending
if window is too big: ghosting effect, can see the other image underneath
if window is too small: aggressively joined together with no smoothing
what is the optimal window for image blending
To avoid seams:
■ window = size of largest prominent feature
To avoid ghosting:
■ window <= 2*size of smallest prominent feature
