Least Squares: Optimization Techniques for Computer Vision: Least Squares Methods

By Fouad Sabry

What is Least Squares

The method of least squares is a parameter estimation method in regression analysis based on minimizing the sum of the squares of the residuals made in the results of each individual equation.

How you will benefit

(I) Insights, and validations about the following topics:

Chapter 1: Least squares

Chapter 2: Gauss-Markov theorem

Chapter 3: Regression analysis

Chapter 4: Ridge regression

Chapter 5: Total least squares

Chapter 6: Ordinary least squares

Chapter 7: Weighted least squares

Chapter 8: Simple linear regression

Chapter 9: Generalized least squares

Chapter 10: Linear least squares

(II) Answering the public top questions about least squares.

(III) Real world examples for the usage of least squares in many fields.

Who this book is for

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of Least Squares.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: May 11, 2024

Book preview

Chapter 1: Least squares

The method of least squares is a standard approach in regression analysis that is used to approximate the solution of overdetermined systems (sets of equations in which there are more equations than unknowns). This is accomplished by minimizing the sum of the squares of the residuals made in the results of each individual equation. A residual is the difference between an observed value and the fitted value provided by a model.

The most significant use is found in the field of data fitting. When the problem has substantial uncertainties in the independent variable (the x variable), simple regression and least-squares methods have problems; in such cases, the methodology required for fitting errors-in-variables models may be considered instead of that for least squares. [Case in point:] when the problem has substantial uncertainties in the independent variable (the x variable), simple regression and least-squares methods have problems.

There are two types of problems that come under the heading of least squares: linear or ordinary least squares, and nonlinear least squares. The distinction between the two types is based on whether or not the residuals are linear in all unknowns. In statistical regression analysis, one of the problems to be solved is called the linear least-squares issue, and it has a closed-form solution. The iterative refinement method is often used to solve the nonlinear issue. During each iteration, the system is approximately modeled after a linear one, and as a result, the fundamental calculation is the same for both scenarios.

The variance in a prediction of the dependent variable as a function of the independent variable and the deviations from the fitted curve are both described by polynomial least squares.

When the observations come from an exponential family with identity as its natural sufficient statistics and mild-conditions are satisfied (for example, for normal, exponential, Poisson, and binomial distributions), standardized least-squares estimates and maximum-likelihood estimates are the same. This is the case for all exponential families with identity as their natural sufficient statistics. The technique of least squares is capable of being developed in its own right as the method of moments estimator.

The reasoning that follows is couched almost entirely in terms of linear functions; nonetheless, the usage of least squares is not only acceptable but also feasible for families of functions that are more generic. Also, the least-squares approach may be used to fit an extended linear model by iteratively applying local quadratic approximation to the likelihood (using the Fisher information). This is possible when using the Fisher information.

Adrien-Marie Legendre is credited as being the one who first developed and published the least-squares technique (1805), During the Age of Discovery, scientists and mathematicians strove to give answers to the issues of traversing the Earth's waters using the concept of least squares. This approach emerged out of the sciences of astronomy and geodesy as a result of their efforts. The precise description of the behavior of celestial bodies was the key to allowing ships to travel in wide seas, where sailors could no longer depend on land sightings for navigation. This was the case since shore sightings were no longer available.

The approach represented the zenith of a number of developments that had taken place during the course of the eighteenth century:

Aggregation is the process of combining several observations in order to arrive at the most accurate estimate possible of the actual value; mistakes tend to reduce rather than grow as a result of this process, which was perhaps originally articulated by Roger Cotes in 1722.

The process of combining many observations made under the same circumstances, as opposed to just making an effort to observe and record a single observation in the most precise manner possible. The strategy was often referred to as the technique of averages. Tobias Mayer, who was investigating the librations of the moon in 1750, and Pierre-Simon Laplace, who was working on explaining the variations in motion of Jupiter and Saturn in 1788, were two notable individuals who employed this technique in their respective research.

The combining of a number of separate observations made under a variety of circumstances. The name method of least absolute deviation was given to the technique through time. In 1757, Roger Joseph Boscovich used it in his work on the shape of the earth, and Pierre-Simon Laplace used it in 1799 for the same issue. Both men are known for their contributions to the field.

The construction of a criteria that can be examined to identify whether the solution with the least amount of error has been obtained is what is referred to as criterion development. Laplace attempted to propose an approach to estimating that would result in the least amount of estimation error as well as provide a mathematical form of the probability density for the mistakes. Laplace modeled the error distribution by using a symmetric two-sided exponential distribution, which we now refer to as the Laplace distribution. He utilized the sum of absolute deviation as the error of estimate. He believed that these were the most straightforward assumptions he could make, and he had wished to attain the arithmetic mean as the most accurate estimate he could. Instead, he relied on the posterior median as his estimate.

In 1805, the French mathematician Legendre provided the first explanation of the technique of least squares that was both comprehensive and clear. This, of course, resulted in a disagreement on the precedence with Legendre. However, it is to Gauss's credit that he went beyond Legendre's work and was able to successfully combine the technique of least squares with the laws of probability and the normal distribution. This is an accomplishment that is worthy of praise. He had been successful in completing Laplace's program, which required him to specify a mathematical form of the probability density for the observations, depending on a finite number of unknown parameters, and define a method of estimation that minimizes the error of estimation. In addition, he had specified a mathematical form of the probability density for the observations depending on a finite number of unknown parameters. Gauss demonstrated that the arithmetic mean is in fact the best estimate of the location parameter by modifying both the probability density and the technique of estimation. He did this in order to prove that the arithmetic mean is the best estimate. He then took the issue in a new direction by questioning what shape the density should take and what technique of estimation should be used in order to get the arithmetic mean as an estimate of the location parameter. This brought the issue full circle. As a result of this endeavor, he came up with the normal distribution.

The power of Gauss's approach was first put on display when it was put to use to determine where the recently found asteroid Ceres would end up in the future. This was one of the first examples of the method's usefulness. Giuseppe Piazzi, an Italian astronomer, made the discovery of Ceres on January 1, 1801. He was able to observe its motion for a period of forty days until it became obscured by the brightness of the sun. Astronomers wanted to identify the position of Ceres once it emerged from behind the sun, but they did not want to solve Kepler's difficult nonlinear equations of planetary motion in order to do so. Using least-squares analysis, the only forecasts that were accurate enough for the Hungarian astronomer Franz Xaver von Zach to successfully relocate Ceres were those that were done by the 24-year-old Gauss.

In 1810, after reading Gauss's work, Laplace proved the central limit theorem and then utilized it to establish a large sample justification for the technique of least squares and the normal distribution. This was done after Gauss's work had been translated into French. In the year 1822, Gauss was able to demonstrate that the least-squares method of conducting regression analysis is the most effective method available. This was accomplished by demonstrating that in a linear model in which the errors have a mean