Video Processing

Question 1

Video Processing:
What is Video?

Answer 1

• sequence of frames captured over time
• image data is a function of space and time

Question 2

(Motion Analysis) What information can be extracted from time varying sequences of images?

Answer 2

– Camouflaged objects easily visible when
moving
– Size and position of objects are more easily
determined when the objects move
– Even simple image differencing provides edge
detector for objects moving over any static
background

Question 3

Into how many stages can analysis of visual motion be divided?

Answer 3

2
– measurement of the motion
– use of motion data to segment scene into objects and
to extract information about shape and motion.

Question 4

How many types of motion are there to consider?
– movement in the scene = static camera,
– movement of the camera = ego motion.
* should be the same (motion is relative) but not always true -
if scene moves illumination, shadow and specular effects
need to be considered

Answer 4

2
– static camera=movement in the scene
– ego motion = movement of the camera
It should be the same (motion is relative) but not always true -
if scene moves:
-illumination, shadow and specular effects
need to be considered

Question 5

In what are we interested in? (Optical Flow and Motion)?

Answer 5

We are interested in finding the movement of scene objects from time-varying images (videos).

Question 6

Which are the uses of motion analysis?

Answer 6

-Track object behavior
-Align images (mosaics)
-3D shape reconstruction
-Correct for camera jitter (stabilization)
-Special effects

Question 7

What is the Motion field? And

Answer 7

-projection of the 3d scene motion into the image
- length of flow vectors inversely proportional to depth Z of 3D point (points closer to the camera move quickly across the image plane

Question 8

What is the Optical flow?

Answer 8

• apparent motion of brightness patterns (or colors) in the image

Question 9

Ideally, optical flow should be the same as the motion field?

Answer 9

Yes

Question 10

How can apparent motion be caused?

Answer 10

It can be caused by lighting changes without any actual motion.

Question 11

What is necessary to estimate pixel motion from image?

Answer 11

We have to solve the pixel correspondence problem.

Question 12

What is the pixel correspondence problem?

Answer 12

Given a pixel in frame t, look for nearby pixels
with same characteristics (colour, brightness, …)
in frame t − 1.

Question 13

How to estimate pixel motion from image H to image I? (pixel correspondence problem)

Answer 13

We need to find pixel correspondences:
- Given a pixel in H, look for nearby pixels of the same
color in I

Question 14

What are the key assumptions in order to solve the pixel correspondence problem?

Answer 14

-color constancy: a point in H looks “the same” in image I
(for example: In grayscale images, this is brightness constancy)
– small motion: points do not move very far

Question 15

What is the Lucas Kanade method (most popular Optical
Flow Algorithm) main goal? And how do you achieve it?

Answer 15

To get more equations for a pixel
The basic idea is to: impose additional constraints

Question 16

What is the most common Lucas Kanade aditional constraint?

Answer 16

It is to assume that the flow field is smooth locally by pretending the pixels neighbors have the same (u,v).
But If we use a 5x5 window, that gives us 25 equations per pixel!

Question 17

What is the problem that comes from the most common Lucas Kanade aditional constraint?

Answer 17

Having more equations than unknowns.

Question 18

How to solve the problem that comes from the most common Lucas Kanade aditional constraint?

Answer 18

Solve least squares problem.
-The sum are over all pixels in the K x K window

Question 19

When is the problem that comes from the most common Lucas Kanade aditional constrain solvablet?

Answer 19

A^T * A should be :
-invertible
- not to small due to noise (λ1 and λ2 should not be too small)
- well-conditioned (λ1 and λ2 should not be too large (λ1 = larger)

Question 20

What is the Algorithms goal in Optical Flow?

Answer 20

They try to approximate the true motion field of the image plane.

Question 21

What is background subtraction?

Answer 21

-it allows looking at video data as a spatio-temporal volume.
- if camera is stationary, each line through time corresponds to a single ray in space

Question 22

When are Background subtraction techniques commonly used for?

Answer 22

They are commonly used for segmenting out objects of interest in a static camera scene:
- surveillance
- robot vision
- object tracking
- traffic applciations
- human motion capture
- augmented reality

Question 23

How does the background subtraction allow the segmentation of objects of interest in a static camera scene?

Answer 23

Trough the foreground mask (binary image) that it creates, containing moving objects in static camera setups:
- subtracting the observed image from the estimated image and thresholding the result

Question 24

What is Foreground detection?

Answer 24

How the object areas are distinguished from the background

Question 25

What is Background maintenance?

Answer 25

How the background is maintained over time

Question 26

What is Post-processing?

Answer 26

How the segmented object areas are detected

Question 27

What is the generic algorithm for a Static Background?

Answer 27

• create an image of the stationary background
• subtract current frame and known background frame - motion detection algorithms - these only work if the camera is stationary and the objects are moving against a fixed background

Question 28

What is the generic algorithm with Frame differencing?

Answer 28

• background is estimated to be the previous frame
• depending on the object structure, speed, frame rate and global - threshold may be useful (usually is not)

Question 29

What is another approach is to model the background?

Answer 29

It is by using a running average. A pixel is marked as foreground.
The thresholding (th) is predefined and often followed by morphological closing with a 3x3 kernel and the discarding of small regions

Question 29

What is another approach is to model the background?

Answer 29

It is by using a running average. A pixel is marked as foreground.
The thresholding (th) is predefined and often followed by morphological closing with a 3x3 kernel and the discarding of small regions

Question 30

In the background update why is α kept small?

Answer 30

In onder to prevent artificial tails forming behind moving object

Question 31

What is Tracking?

Answer 31

• crucial issue in CV
• could compute optical flow from one to another but flow only reliable for small motions

Question 32

Where can tracking be apllied?

Answer 32

-Body pose tracking, activity recognition
-Censusing a bat population
-Video-based interfaces
-Medical apps
-Surveillance

Question 33

In more than just a pair of frames, why can’t we use optical flow from one to the other?

Answer 33

Because flow is only reliable for small motions, and we may have:
-occlusions;
-texture less regions that yield bad estimates

Question 34

What is the difference between diference and tracking?

Answer 34

• Detection: we detect the object independently in each frame and can record its position over time
• Tracking: we use image measurements to estimate the position of the object but also incorporate position predicted by dynamics

Question 35

What defines the Kalman filter?

Answer 35

• hidden state consists of the true parameters we care about
• the measurement is our noisy observation
• at each step, state changes and we get a new observation

Question 36

What form do the predicted/corrected state distributions have?

Answer 36

Gaussian

Question 37

What are the only parameters that need to be maintained in the Kalman Filter?

Answer 37

The mean and covariance

Question 38

What is the method for tracking linear dynamical models in gaussian noise?

Answer 38

The Kalman Filter