Optical Flow: Exploring Dynamic Visual Patterns in Computer Vision

By Fouad Sabry

What is Optical Flow

Optical flow or optic flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene. Optical flow can also be defined as the distribution of apparent velocities of movement of brightness pattern in an image.

How you will benefit

(I) Insights, and validations about the following topics:

Chapter 1: Optical flow

Chapter 2: Least squares

Chapter 3: Fourier optics

Chapter 4: Image segmentation

Chapter 5: Lucas-Kanade method

Chapter 6: Horn-Schunck method

Chapter 7: Digital image correlation and tracking

Chapter 8: 3D reconstruction

Chapter 9: Visual odometry

Chapter 10: Harris corner detector

(II) Answering the public top questions about optical flow.

(III) Real world examples for the usage of optical flow in many fields.

Who this book is for

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of Optical Flow.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: May 13, 2024

Book preview

Chapter 1: Optical flow

When an observer moves relative to a scene, the observed objects, surfaces, and edges appear to move in a specific pattern known as optical flow or optic flow.

In the 1940s, American psychologist James J. Gibson introduced the concept of optical flow to describe the visual stimulus provided to animals in motion.

Ordered image sequences can be used to estimate motion in the form of either continuous image velocities or individual image displacements. To compare the efficacy of various optical flow methods, John L. Barron, David J. Fleet, and Steven Beauchemin present a comprehensive analysis. The precision and density of measurements are emphasized.

The optical flow methods try to calculate the motion between two image frames which are taken at times t and t+\Delta t at every voxel position.

Differential techniques are so-called because they approximate the image signal with local functions using Taylor series; that is, To do this, they take partial derivatives in space and time.

For a (2D + t)-dimensional case (3D or n-D cases are similar) a voxel at location (x,y,t) with intensity I(x,y,t) will have moved by \Delta x , \Delta y and \Delta t between the two image frames, and the following limitation on the fluctuation of light intensity can be given:

I(x,y,t) = I(x+\Delta x, y + \Delta y, t + \Delta t)

Assuming the shift is negligible, the image constraint at I(x,y,t) with Taylor series can be developed to get:

{\displaystyle I(x+\Delta x,y+\Delta y,t+\Delta t)=I(x,y,t)+{\frac {\partial I}{\partial x}}\,\Delta x+{\frac {\partial I}{\partial y}}\,\Delta y+{\frac {\partial I}{\partial t}}\,\Delta t+{}}

higher-order terms

Since a linearization is accomplished through truncating the higher order terms, it follows that:

\frac{\partial I}{\partial x}\Delta x+\frac{\partial I}{\partial y}\Delta y+\frac{\partial I}{\partial t}\Delta t = 0

or, dividing by \Delta t ,

{\displaystyle {\frac {\partial I}{\partial x}}{\frac {\Delta x}{\Delta t}}+{\frac {\partial I}{\partial y}}{\frac {\Delta y}{\Delta t}}+{\frac {\partial I}{\partial t}}{\frac {\Delta t}{\Delta t}}=0}

which results in

\frac{\partial I}{\partial x}V_x+\frac{\partial I}{\partial y}V_y+\frac{\partial I}{\partial t} = 0

where V_x,V_y are the x and y components of the velocity or optical flow of I(x,y,t) and \tfrac{\partial I}{\partial x} , \tfrac{\partial I}{\partial y} and \tfrac{\partial I}{\partial t} are the derivatives of the image at (x,y,t) in the corresponding directions.

I_{x} , I_y and I_t can be written for the derivatives in the following.

Thus:

I_xV_x+I_yV_y=-I_t

or

{\displaystyle \nabla I\cdot {\vec {V}}=-I_{t}}

Since there are two variables missing from this equation, it is intractable. The aperture problem is a common issue in optical flow algorithms. The optical flow can be calculated with a different set of equations that are determined by an extra constraint. The estimation of actual flow requires additional assumptions made by all optical flow methods.

Phase correlation - the inverse of the cross-power spectrum in normalized form

Minimizing the sum of squared differences or the sum of absolute differences, or optimizing the normalized cross-correlation, are all examples of block-based methods.

Partial derivatives of the image signal and/or the sought flow field, as well as higher-order partial derivatives, can be used in differential methods to estimate optical flow:

The Lucas-Kanade approach, which uses patched images and an affine model of the flow field, The Horn-Schunck technique involves maximizing a functional that takes into account residuals from a brightness constancy constraint and a specific regularization term that characterizes the desired smoothness of the flow field.

The Buxton-Buxton technique is predicated on an edge-motion model applied to a series of images.

Coarse Optical Flow by Correlation, as in the Black-Jepson Method

Various modifications and extensions of Horn-Schunck that make use of additional data terms and smoothness terms constitute the broader category of general variational methods.

Using discrete optimization techniques, we first quantify the search space, then tackle image matching by labeling each pixel so that the resulting deformation minimizes the distance between the source and target images. KITTI and Sintel are two additional widely used benchmark datasets.

One of the most important areas of study in optical flow is motion estimation and video compression. In spite of its superficial resemblance to a dense motion field derived from motion estimation techniques, optical flow is the study of not only the determination of the optical flow field but also of its use in estimating the 3D nature and structure of the scene, as well as the 3D motion of objects and the observer relative to the scene, with the vast majority of these estimations relying on the image itself. Jacobian.

Think of a five-frame sequence in which a ball travels from the bottom left to the top right of the screen. Using motion estimation methods, we can deduce that the ball is traveling in a vertical and lateral direction by analyzing the frames in the sequence. The sequence has been described as thoroughly as is necessary for video compression (such as MPEG). In machine vision, however, knowing whether the ball or the observer is moving to the right is crucial but unknown information. Even if a fixed, patterned background were present in all five images, we still wouldn't be able to say for sure that the ball was traveling in a rightward direction, because the pattern could be infinitely far away from the camera.

There are a number of optical flow sensor designs available. An image sensor chip coupled with a processor running an optical flow algorithm is one possible setup. A vision chip is an alternative setup; it's an integrated circuit that contains both the image sensor and the processor on the same die. An optical mouse with a generic optical mouse sensor is a good illustration of this type of device. To achieve fast optical flow computation with low current consumption, the processing circuitry is sometimes implemented with analog or mixed-signal circuits.

Optical flow sensors could benefit from recent developments in neuromorphic engineering, which are used to create circuits that react to optical flow. Inspiration for these circuits could be found in biological neural circuitry that also reacts to optical flow.

As the primary sensing component for tracking mouse movement across a surface, optical flow sensors find widespread application in computer optical mice.

In robotics applications, optical flow sensors are typically used to measure visual motion or relative motion between the robot and other objects in its immediate vicinity. Another active area of study is the integration of optical flow sensors into unmanned aerial vehicles (UAVs) for use in maintaining flight stability and navigating around obstacles.

{End Chapter 1}

Chapter 2: Least squares

The method of least squares is a standard approach in regression analysis that is used to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns). This is accomplished by minimizing the sum of the squares of the residuals made in the results of each individual equation. A residual is the difference between an observed value and the fitted value provided by a model.

The most significant use is found in the field of data fitting. When the problem has substantial uncertainties in the independent variable (the x variable), simple regression and least-squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares. [Case in point:] when the problem has substantial uncertainties in the independent variable (the x variable), simple regression and least-squares methods have problems.

There are two types of problems that come under the heading of least squares: linear or ordinary least squares, and nonlinear least squares. The distinction between the two types is based on whether or not the residuals are linear in all unknowns. In statistical regression analysis, one of the problems to be solved is called the linear least-squares issue, and it has a closed-form solution. The iterative refinement method is often used to solve the nonlinear issue. During each iteration, the system is approximately modeled after a linear one, and as a result, the fundamental calculation is the same for both scenarios.

The variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve are both described by polynomial least squares.

When the observations come from an exponential family with identity as its natural sufficient statistics and mild-conditions are satisfied (for example, for normal, exponential, Poisson, and binomial distributions), standardized least-squares estimates and maximum-likelihood estimates are the same. This is the case for all exponential families with identity as their natural sufficient statistics. The technique of least squares is capable of being