Motion Estimation: Advancements and Applications in Computer Vision

By Fouad Sabry

What is Motion Estimation

In computer vision and image processing, motion estimation is the process of determining motion vectors that describe the transformation from one 2D image to another; usually from adjacent frames in a video sequence. It is an ill-posed problem as the motion happens in three dimensions (3D) but the images are a projection of the 3D scene onto a 2D plane. The motion vectors may relate to the whole image or specific parts, such as rectangular blocks, arbitrary shaped patches or even per pixel. The motion vectors may be represented by a translational model or many other models that can approximate the motion of a real video camera, such as rotation and translation in all three dimensions and zoom.

How you will benefit

(I) Insights, and validations about the following topics:

Chapter 1: Motion_estimation

Chapter 2: Motion_compensation

Chapter 3: Block-matching_algorithm

Chapter 4: H.261

Chapter 5: H.262/MPEG-2_Part_2

Chapter 6: Advanced_Video_Coding

Chapter 7: Global_motion_compensation

Chapter 8: Block-matching_and_3D_filtering

Chapter 9: Video_compression_picture_types

Chapter 10: Video_super-resolution

(II) Answering the public top questions about motion estimation.

(III) Real world examples for the usage of motion estimation in many fields.

Who this book is for

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of Motion Estimation.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: May 12, 2024

Book preview

Chapter 1: Motion estimation

Estimating the amount of motion between two 2D images, typically from successive frames in a video sequence, is known as motion estimate. Movement occurs in three dimensions, but the images are projections of the three-dimensional scene onto a two-dimensional plane, making this an ill-posed problem. Motion estimation can be performed on the entire image (global motion estimate) or on smaller regions or individual pixels. The translational model is just one of many that can be used to represent the motion vectors, which can include things like rotation, translation in all three dimensions, and zooming in and out.

Motion estimation and optical flow are often used interchangeably but mean the same thing. Image registration and stereo correspondence are related in this way as well. In reality, each of these expressions describes a method for locating matching features in successive still photos or video frames. Points in two photographs (frames) of the same real-world scene or item are typically the same point in space and time. The matching metric, a measure of how similar two image points are, must be defined before motion estimation can begin. There is no universally correct answer here; rather, the optimal matching metric will depend on the purpose of the estimated motion and the optimization approach employed during the estimation phase.

Based on the location of the same or a comparable macroblock in a different image, known as the reference picture, a motion vector is created to represent the macroblock in question in the original image.

According to the H.264/MPEG-4 AVC standard, motion vector is defined as:

The coordinates of the decoded image are offset with respect to the coordinates of the reference image via a two-dimensional vector called a motion vector, which is utilized for inter prediction.

Both pixel-based (direct) and feature-based (indirect) approaches can be used to locate motion vectors (indirect). Two papers, one from each side of a well-known controversy, were written in an effort to reach a resolution.

Algorithm for matching blocks

Techniques based on phase correlation and the frequency domain

Pixel recursive algorithms

Optical flow

Indirect techniques employ features like corner detection to find matches in adjacent frames, typically by the application of a statistical function over a small or large region. The statistical function's goal is to filter out erroneous motion matches.

RANSAC is one example of a useful statistical function.

One may argue that defining the matching criteria is a necessary step in practically any approach. The only real distinction is between comparing each pixel first (like with squaring the difference) and then summarizing over a local image region (like with feature based approaches) or vice versa (block base motion and filter based motion). One new method of matching criteria works by first summarizing a local image region at each pixel location (using some feature transform like Laplacian transform), then comparing the two sets of summaries. While some matching criteria can eliminate pairs of data items that don't really belong together while having a high matching score, other matching criteria can't.

In computer vision and image processing, affine motion estimation is used to calculate a rough approximation of the relative motion of two pictures or frames. Assumes the motion can be represented by a linear transformation followed by a translation and rotation (affine transformation).

Motion compensation is the process of using an image's motion vectors to generate a new transformation.

Motion estimation and compensation are essential components of video compression because they allow for the use of temporal redundancy. Almost all video encoding standards, including the most current HEVC, use block-based motion estimation and compensation.

Reconstructing a scene's 3D geometry from moving camera images is what simultaneous localization and mapping is all about.

{End Chapter 1}

Chapter 2: Motion compensation

When predicting the next frame of a video based on the frames before and after it, motion compensation is an algorithmic technique used to account for camera and/or object motion. It's used to create MPEG-2 files and other video compression formats through encoding video data. Using the transformation from a reference image to the current image, motion compensation provides a description of the image. The temporal period of the referred-to image is not limited. The compression efficiency is enhanced when new images can be synthesized from old ones with high accuracy.

In video coding standards, motion compensation and the discrete cosine transform are the two most important video compression techniques (DCT). Block motion compensation (BMC) or motion-compensated DCT (MC-DCT) hybrid coding is commonly used in most video coding standards like the H.26x and MPEG formats (MC DCT).

Motion correction takes advantage of the fact that the camera or an object in the frame changing position is often the only difference between successive frames of a movie. This means that the data used to represent one frame in a video clip is very likely to be identical to data used to represent the following frame.

When using motion compensation, a video stream will have a few complete (reference) frames, and the remaining frames will only contain the data necessary to turn one into the next.

The following is a simplified graphical description of how motion compensation works. Two consecutive stills from the film Elephants Dream were taken. Because it includes less information, the bottom (motion compensated) difference between two frames compresses far better than the other images. Therefore, the data needed to encode a compensated frame will be significantly less than that needed to encode a difference frame. This also means that it is also possible to encode the information using difference image at a cost of less compression efficiency but by saving coding complexity without motion compensated coding; as a matter of fact that motion compensated coding (together with motion estimation, motion compensation) occupies more than 90 percent of encoding complexity.

Images in MPEG are anticipated either backwards from earlier frames (P frames) or forwards from both earlier and later frames (B frames). Because the next frame is