Random Optimization: Fundamentals and Applications

By Fouad Sabry

What Is Random Optimization

Because it does not require the gradient of the problem to be optimized, random optimization (also known as RO) is a family of numerical optimization methods that can be used to functions that are neither continuous nor differentiable. These kinds of optimization approaches are also referred to as direct-search methods, derivative-free methods, and black-box methods.

How You Will Benefit

(I) Insights, and validations about the following topics:

Chapter 1: Random optimization

Chapter 2: Mathematical optimization

Chapter 3: Gradient

Chapter 4: Continuous function

Chapter 5: Differentiable function

Chapter 6: Normal distribution

Chapter 7: Evolution strategy

Chapter 8: Unimodality

Chapter 9: Limit (mathematics)

Chapter 10: Probability distribution

(II) Answering the public top questions about random optimization.

(III) Real world examples for the usage of random optimization in many fields.

(IV) 17 appendices to explain, briefly, 266 emerging technologies in each industry to have 360-degree full understanding of random optimization' technologies.

Who This Book Is For

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of random optimization.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: Jul 1, 2023

Book preview

Chapter 1: Random optimization

Random optimization (RO) refers to a family of numerical optimization techniques that do not need the gradient of the problem to be improved. As a result, RO may be used to functions that are neither continuous nor differentiable. Random optimization was first developed in the 1960s. These kinds of optimization approaches are also referred to as direct-search methods, derivative-free methods, and black-box methods.

Matyas, who presented an early presentation of random optimization combined with fundamental mathematical analysis, is often credited with coining the term random optimization. RO operates by shifting to better places in the search space on an iterative basis. These better positions are sampled using methods such as a normal distribution that surround the present position.

Let f: ℝn → ℝ be the fitness or cost function which must be minimized.

Let x ∈ ℝn designate a position or candidate solution in the search-space.

Following is an explanation of the fundamental RO algorithm:

Set the initial value of x to a place in the search space that is completely random.

Repeat the steps below as many times as necessary until one of the termination criteria is satisfied (for example, when an appropriate level of fitness has been attained):

Take a representative sample at a new place y by appending a random vector with a normal distribution to the current position x.

If the condition f(y) > f(x) is met, then proceed to the next location by changing x to equal y.

At this time, x is at the best-found position.

This technique is equivalent to a (1+1) evolution strategy that maintains a constant step size.

Matyas demonstrated that the fundamental form of RO converges to the optimal value of a straightforward unimodal function by employing a limit-proof. This proof demonstrates that convergence to the optimal value will definitely take place if a potentially infinite number of iterations are carried out. However, this demonstration is not applicable in the real world since there is a limit to the number of iterations that may be carried out. In point of fact, such a theoretical limit proof will also demonstrate that selecting samples from the search space at random will always provide results that are arbitrarily near to the optimal.

Mathematical analyses are also done by Baba, who used the optimizer variants of Baba and Dorea on two real-world problems, demonstrating that the optimum is approached very slowly and, furthermore, that the methods were actually unable to locate a solution of adequate fitness, unless the process was started sufficiently close to the optimum to begin with. This was shown by the fact that the optimum was approached very slowly.

{End Chapter 1}

Chapter 2: Mathematical optimization

The selection of a best element, with reference to some criteria, from some collection of accessible alternatives is the goal of mathematical programming as well as mathematical optimization, which may also be written as the alternate spelling optimisation.

An optimization issue is defined under the more general approach as the process of either maximizing or reducing the value of a real function. This is accomplished by methodically selecting input values from within an acceptable set and calculating the value of the function. A significant portion of the field of applied mathematics is dedicated to the application of optimization theory and methods to many different formulations. In a broader sense, optimization refers to the process of determining the best available values of a particular objective function when that function is applied to a defined domain (or input). This process can be applied to a wide variety of different types of objective functions and a wide variety of different types of domains.

Depending on whether the variables being optimized are continuous or discrete, optimization issues may be broken down into two distinct types:

A discrete optimization problem is an optimization problem that has discrete variables. In a discrete optimization problem, an object such an integer, permutation, or graph has to be located from a set that has a countable size.

The process of extracting an optimum value from a continuous function is referred to as continuous optimization, and it is used to describe a kind of issue that involves continuous variables. Constrained difficulties and multimodal problems are two examples of these types of issues.

The following is one possible representation of a problem requiring optimization::

Given: a function f : A → ℝ from some set A to the real numbers

Sought: an element x0 ∈ A such that f(x0) ≤ f(x) for all x ∈ A (minimization) or such that f(x0) ≥ f(x) for all x ∈ A (maximization).

A issue with such a formulation is referred to as an optimization problem or a mathematical programming problem (a phrase that is not directly connected to computer programming but is still in use, for example in linear programming; see the section on History below for more information). This overarching paradigm lends itself well to the modeling of a wide variety of theoretical and practical challenges.

Considering that the following is true

{\displaystyle f(\mathbf {x} _{0})\geq f(\mathbf {x} )\Leftrightarrow -f(\mathbf {x} _{0})\leq -f(\mathbf {x} ),}

It is sufficient to merely address issues involving minimization. On the other hand, the alternative viewpoint, which is to address solely situations involving maximizing, is also a legitimate one.

In the domains of physics, problems that are posed using this method may refer to the technique as energy minimization. When doing so, they may talk of the value of the function f as reflecting the energy of the system that is being represented. When it comes to machine learning, it is always required to continually assess the quality of a data model by making use of a cost function. In this function, a minimum suggests a set of possible ideal parameters with an optimal (lowest) error.

Typically, A is some subset of the Euclidean space ℝn, often governed by a predetermined list of restrictions, requirements of equality or inequality that must be met by the members of group A.

The domain A of f is also referred to as the choice set or the search space, while the components of A are often referred to as candidate solutions or plausible solutions.

Depending on the context, the function denoted by the letter f may be referred to as an objective function, a loss function or cost function (functions that minimize losses or costs), a utility function or fitness function (functions that maximize benefits), or, in some contexts, an energy function or energy functional. An optimum solution is referred to as a viable solution that optimizes (or maximizes, depending on whether or not that is the aim) the objective function.

Conventional optimization issues in mathematics are often phrased in terms of minimization.

A local minimum x* is defined as an element for which there exists some δ > 0 such that

{\displaystyle \forall \mathbf {x} \in A\;{\text{where}}\;\left\Vert \mathbf {x} -\mathbf {x} ^{\ast }\right\Vert \leq \delta ,\,}

the expression f(x*) ≤ f(x) holds; That is to say, on some portion of the area around x*, every value of the function is either higher than or equal to the value at that element. Local maxima are defined similarly.

A global minimum is at least as excellent as any element that is even remotely possible, as contrast to a local minimum, which is just at least as good as any adjacent components. In a minimization issue, it is possible for there to be more than one local minimum, unless the objective function being minimized is convex. In a problem that is convex, if there is a local minimum that is interior (meaning that it is not on the edge of the set of feasible elements), then it is also the global minimum; however, in a problem that is nonconvex, there may be more than one local minimum, and not all of those local minima need to be global minima.

Many of the proposed algorithms for solving nonconvex problems, as well as the vast majority of the solvers that are available for purchase, are incapable of differentiating between locally optimal solutions and globally optimal solutions, and as a result, they will only consider the former to be true answers to the initial problem. This is true even when the algorithms are designed to solve nonconvex problems. Global optimization is the area of applied mathematics and numerical analysis that focuses on the creation of deterministic algorithms that are able to guarantee convergence to the actual optimal solution of a nonconvex problem within a finite amount of time. This area of study is related to the study of convex optimization.

Notation that is unique is often used when expressing optimization challenges. Here are several instances:

Take into account the note that follows::

{\displaystyle \min _{x\in \mathbb {R} }\;\left(x^{2}+1\right)}

This denotes the minimum value of the objective function x² + 1, when choosing x from the set of real numbers ℝ.

In this scenario, one is the absolute lowest possible value, occuring with x equal to zero.

In the same vein, the notation

\max _{x\in \mathbb {R} }\;2x

requests the value that gives the highest benefit from