Semi-empirical Neural Network Modeling and Digital Twins Development

By Dmitriy Tarkhov and Alexander Nikolayevich Vasilyev

Semi-empirical Neural Network Modeling presents a new approach on how to quickly construct an accurate, multilayered neural network solution of differential equations. Current neural network methods have significant disadvantages, including a lengthy learning process and single-layered neural networks built on the finite element method (FEM). The strength of the new method presented in this book is the automatic inclusion of task parameters in the final solution formula, which eliminates the need for repeated problem-solving. This is especially important for constructing individual models with unique features. The book illustrates key concepts through a large number of specific problems, both hypothetical models and practical interest.

• Offers a new approach to neural networks using a unified simulation model at all stages of design and operation
• Illustrates this new approach with numerous concrete examples throughout the book
• Presents the methodology in separate and clearly-defined stages

• Language: English
• Publisher: Academic Press
• Release date: Nov 23, 2019
• ISBN: 9780128156520

Dmitriy Tarkhov

Dmitry Tarkhov, born 14 Jan 1958 in St. Petersburg. In 1981 graduated with honors from the faculty of physics and mechanics of Leningrad Polytechnic Institute, majoring in Applied Mathematics and entered graduate school at the Department “Highermathematics”. After graduation, heworked at the Department as an assistant, then associate Professor and continues to work presently as a Professor. In 1987 he defended the thesis “the Straightening of the trajectories on the infinite dimensional torus”, for which he was awarded the degree of Ph.D. of physical and mathematical Sciences. In 1996, while working as a part-time chief systems analyst at the St. Petersburg Futures exchange he began studying neural networks. He has published more than 200 scientific papers on this topic. In 2006 he defended doctoral thesis “Mathematical modeling of technical objects on the basis of structural and parametrical adaptation of artificial neural networks”, for which he was awarded the degree of doctor of technical Sciences.

Book preview

Semi-empirical Neural Network Modeling and Digital Twins Development

First Edition

Dmitriy Tarkhov

Professor, Department of Higher Mathematics, Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russian Federation

Alexander Vasilyev

Professor, Department of Higher Mathematics, Peter the Great St. Petersburg Polytechnic University, St. Petersburg, Russian Federation

Table of Contents

Cover image

Title page

Copyright

About the authors

Preface

Acknowledgments

Introduction

1: Examples of problem statements and functionals

Abstract

1.1 Problems for ordinary differential equations

1.2 Problems for partial differential equations for domains with fixed boundaries

1.3 Problems for partial differential equations in the case of the domain with variable borders

1.4 Inverse and other ill-posed problems

2: The choice of the functional basis (set of bases)

Abstract

2.1 Multilayer perceptron

2.2 Networks with radial basis functions—RBF

2.3 Multilayer perceptron and RBF-networks with time delays

3: Methods for the selection of parameters and structure of the neural network model

Abstract

3.1 Structural algorithms

3.2 Methods of global non-linear optimization

3.3 Methods in the generalized definition

3.4 Methods of refinement of models of objects described by differential equations

4: Results of computational experiments

Abstract

4.1 Solving problems for ordinary differential equations

4.2 Solving problems for partial differential equations in domains with constant boundaries

4.3 Solving problems for partial differential equations for domains with variable boundaries

4.4 Solving inverse and other ill-posed problems

5: Methods for constructing multilayer semi-empirical models

Abstract

5.1 General description of methods

5.2 Application of methods for constructing approximate analytical solutions for ordinary differential equations

5.3 Application of multilayer methods for partial differential equations

5.4 Problems with real measurements

Index

Copyright

Academic Press is an imprint of Elsevier

125 London Wall, London EC2Y 5AS, United Kingdom

525 B Street, Suite 1650, San Diego, CA 92101, United States

50 Hampshire Street, 5th Floor, Cambridge, MA 02139, United States

The Boulevard, Langford Lane, Kidlington, Oxford OX5 1GB, United Kingdom

© 2020 Elsevier Inc. All rights reserved.

No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions.

This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices

Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods, professional practices, or medical treatment may become necessary.

Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility.

To the fullest extent of the law, neither the Publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data

A catalog record for this book is available from the Library of Congress

British Library Cataloguing-in-Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-0-12-815651-3

For information on all Academic Press publications visit our website at https://www.elsevier.com/books-and-journals

Publisher: Mara Conner

Acquisition Editor: Chris Katsaropoulos

Editorial Project Manager: Mariana L. Kuhl

Production Project Manager: Nirmala Arumugam

Cover Designer: Greg Harris

Typeset by SPi Global, India

About the authors

Alexander Vasilyev was born in St. Petersburg (Leningrad) on 10 August 1948. After graduating mathematical school №239 with a gold medal and the Physics Faculty of Leningrad State University (LSU) with honors, he defended the Ph.D. thesis: New boundary value problems for ultrahyperbolic and wave equations, in the Leningrad branch of the Steklov Mathematical Institute in 1978. Working since 1980 at the Department of higher mathematics of Peter the Great St. Petersburg Polytechnic University as an Associate Professor and since 2007 as a Professor, he read advanced courses and electives in various areas of modern mathematics, led seminars. He prepared and in 2011 defended his doctoral thesis: Mathematical modeling of systems with distributed parameters based on neural network technique, for the degree of Doctor of Technical Sciences in the specialty 05.13.18 – Mathematical modeling, numerical methods, and software. Professor Vasilyev’s scientific interests are in the field of differential equations, mathematical physics, ill-posed problems, meshless methods, neuro-mathematics, neural network modeling of complex systems with approximately specified characteristics, heterogeneous data, digital twins, deep learning, global optimization, evolutionary algorithms, big transport system, environmental problems, educational projects. He is the author and co-author of three monographs, chapters in the reference book, chapters in two collective monographs, textbook (with the Ministry of Education stamp) – in Russian; he published about 180 works devoted to neural network modeling; he has Honors Diploma of the Ministry of education of the Russian Federation, the Diploma and Awards of Polytechnic University Board. Professor Vasilyev is the Chairman, the member of the Organizing Committee of the conferences; he is the head, the central executive and participant of projects supported by grants of RF, a member of the editorial board of the Journal Mathematical Modeling and Geometry. He is fond of painting and graphics.

Dmitry Tarkhov, born 14 Jan 1958 in St. Petersburg. In 1981 graduated with honors from the faculty of physics and mechanics of Leningrad Polytechnic Institute, majoring in Applied Mathematics and entered graduate school at the Department Higher mathematics. After graduation, he worked at the Department as an assistant, then associate Professor and continues to work presently as a Professor. In 1987 he defended the thesis the Straightening of the trajectories on the infinite dimensional torus, for which he was awarded the degree of Ph.D. of physical and mathematical Sciences. In 1996, while working as a part-time chief systems analyst at the St. Petersburg Futures exchange he began studying neural networks. He has published more than 200 scientific papers on this topic. In 2006 he defended doctoral thesis Mathematical modeling of technical objects on the basis of structural and parametrical adaptation of artificial neural networks, for which he was awarded the degree of doctor of technical Sciences.

Preface

The fourth industrial revolution is taking place before our eyes; it poses urgent problems for modern science. The adoption of cyber-physical systems in manufacturing requires an adequate reflection of the physical world in cybernetic objects. In computing nodes that control technological lines, robots, complex technical objects (airplanes, cars, ships, etc.), twins of the corresponding objects should function. Such twins cannot be laid into the computing node once unchanged, as the simulated object changes during operation. The virtual twin of the real object should change according to the information coming from the sensors. Algorithms of such changes should be implemented in the mentioned computational node.

Industry 4.0, the transition to which is currently underway, is not possible without solving the above problems. However, presently accepted methods of mathematical modeling are poorly adapted to their solution. In our opinion, the creation of suitable mathematical and algorithmic tools to solve uniformly a wide range of problems arising in the transition to Industry 4.0 is a crucial step in the development of Industry 4.1. By Industry 4.1, we mean an industry that has the same production processes as Industry 4.0, but the implementation of these processes will be based on unified and cheaper technologies.

We believe that one of these key technologies is the neural network technique. Currently, neural networks are actively used in the problems of Big Data, image processing, pattern recognition, complex system control, and other tasks of artificial intelligence. The issue of creating cyber-physical systems requires adequate means of mathematical modeling. We offer neural networks as such tools. Quite a lot of publications is devoted to the application of neural networks to mathematical modeling problems; a small review of them we have led in the introduction to this book. At the moment, it is necessary to move from solving individual problems to a single methodology for solving them, to which we have devoted this monograph.

Our methodology is based on three simple steps. The first step is to characterize the quality of the mathematical model in the form of a functional. The first chapter demonstrates how to do this on a large set of problems. The second step is to choose the type of neural network that is most suitable for solving the problem. In the second chapter, we described the most useful from our point of view types of neural networks and gave recommendations concerning the choice of a particular type of neural network depending on the characteristics of the problem solved. The third step is to train the neural network, by which we mean minimizing the target functional. Algorithms of such training we have considered in the third chapter. A significant part of the algorithms involves the simultaneous adaptation of the neural network parameters and the selection of its structure. In the fourth chapter, we presented the results of computational experiments on the problems formulated in the first chapter.

The fifth chapter is devoted to the construction of approximate multilayer solutions of differential equations based on classical numerical methods. To understand our approach, we must point out that the transition to Industry 4.1 requires a paradigm shift in mathematical modeling. Traditional mathematical modeling of a real object is performed in two steps. The first step is to describe the object by a differential equation or a system of such equations (ordinary or partial ones) with accompanying conditions. The second step is the numerical solution of these equations with the maximum possible accuracy, the construction of a control system based on a differential model, etc. This numerical calculation is the basis for the design of the object, the choice of its optimal parameters, conclusions about its performance, etc. If subsequent observations of the object, measurements of its parameters come into conflict with the calculations done, then an additional study of the processes taking place in the object is carried out. According to the results of these studies, the differential model of the object is refined. Then the computational studies of this model are repeated.

Such the approach requires a lot of time and intellectual resources. To improve the efficiency of mathematical modeling, we propose to change the point of view on the differential model of the object. We consider it not as accurate initial information for subsequent studies, but as approximate information about the object along with the measurement data. According to this information, we make a set of mathematical models of the object. At the same time, we foresee and provide the possibility of changing the parameters of these models in the operation of the object. We can choose from a set of models the model best suited to the object at this stage of its life cycle.

Thus, we use known formulas of numerical methods for solving differential equations not to generate tables of numerical solutions, but to create a set of adaptive functional solutions. We have a number of problems with real measurements for which our models reflect the object more accurately than the exact solutions of the original differential equations—some similar problems we gave in the fifth chapter.

Acknowledgments

Dmitry Tarkhov; Alexander Vasilyev

We want to express our gratitude for the useful discussions and support of our colleagues – the Professors and Experts of Peter the Great St. Petersburg Polytechnic University, the Moscow State University, the Moscow Aviation Institute, and other Institutions and Organizations: Evgeniy Alekseev, Alexander Belyaev, Elena Budkina, Vladimir Gorbachenko, Vladimir Kozlov, Boris Kryzhanovsky, Evgenii Kuznetsov, Tatiana Lazovskaya, Sergey Leonov, Nikolai Makarenko, Galina Malykhina, Yuri Nechaev, Vladimir Osipov, Alexander Petukhov, Dmitry Reviznikov, Vladimir Sukhomlin, Sergey Terekhov, Valery Tereshin, Yuri Tiumentsev, Tatiana Shemyakina, Lyudmila Zhivopistseva.

We would also like to express our gratitude to our students at Polytechnic University for their assistance. Among them, we wish to mention Yaroslav Astapov, Anastasia Babintseva, Alexander Bastrakov, Serafim Boyarskiy, Dmitry Dron, Daniil Khlyupin, Kirill Kiselev, Anna Krasikova, Polina Kvyatkovskaya, Ilya Markov, Danil Minkin, Arina Moiseenko, Anastasia Peter, Nikita Selin, Roman Shvedov, Gleb Poroshin, Dmitry Vedenichev.

We express our sincere gratitude to the Elsevier Team that worked on our book for their valuable support and assistance: our Senior Acquisitions Editor, Chris Katsaropoulos, our Senior Editorial Project Manager, Mariana Kühl Leme, our Project Manager, Nirmala Arumugam, and our Copyrights Coordinator, Kavitha Balasundaram.

This book is based on research carried out with the financial support of the grant of the Russian Science Foundation (project №18-19-00474).

Introduction

At the moment, the transition to the use of a simulation model (digital twin) at all stages of the work of a complex technical object has matured. At the first stage—during the design process—structure and parameters of the object are determined according to the requirements using a mathematical model. Next, a series of problems is solved to test the operation of the future object in various situations. After its manufacture, it is necessary to clarify the model by measurements made with a real object and evaluate its performance. During operation, the object is exposed to external influences, wear, which leads to changes in its characteristics. These characteristics need to be refined by measurements and observation of the object in the course of its work, making the necessary changes in its operation modes. The simulation model allows predicting the possibility of destruction of the object and the need to stop its operation for the repair or end of the life cycle. Using a single model greatly simplifies the conduct of the entire simulation cycle. This book is devoted to the methods of creating models of this sort (digital twins).

The classical approach to the mathematical modeling of real objects consists of two stages. At the first stage, a mathematical model is made on the basis of studying physical or other processes, most often in the form of differential equations (ordinary or partial derivatives) and additional conditions (initial, boundary, etc.). At the second stage, the model is examined—numerical solutions of the indicated equations are built, etc. During the modeling of complex technical objects, computer-aided engineering (CAE) packages are usually used, based on distinct variants of finite element method (FEM)—ANSYS, ABAQUS, etc. However, modeling a real object through them encounters several principled difficulties. Firstly, to apply FEM, it is essential to know the differential equations, describing the behavior of an object. Precise information on the equations usually lacks due to the complexity of describing the physical processes which occur in the simulated object. Secondly, to apply FEM, it is necessary to know the initial and boundary conditions, information on which for the real objects is even less accurate and complete. Thirdly, during the operation of the real object, its properties and details of physical processes taking place may vary. This functioning requires appropriate adaptation of the model, which is difficult to implement with a model built based on FEM.

We believe that another approach seems to be more promising when at the second stage, an adaptive model is built, which can be refined and rearranged in accordance with the observations on the object. The monograph is devoted to the presentation of our methodology for constructing such a model, illustrated with numerous examples. As the main class of mathematical models, the class of artificial neural networks (ANN) is used, that have proven themselves to behave well in complex data processing tasks.

At the moment, the neural network technique is one of the most dynamically developing spheres of artificial intelligence. It has been successfully used in various applied areas, such as:

1.Forecasting various indicators of financial market and economic indicators (exchange rates and stocks, credit, and other risks, etc.).

2.Biomedical applications (diagnosis of different diseases, identification of personality).

3.Control systems.

4.Pattern recognition, identity recognition.

5.Geology (prediction of the availability of mineral resources).

6.Ecology and environmental sciences (weather forecast and various cataclysms).

Before describing the features of neural network modeling, we formulate the task of mathematical modeling in the most general form. By the mathematical model of the researched object (system), we will understand the mapping F : (X, Z) ↦ Y, that establishes the relationship between the set of input data X, which defines the conditions for the functioning of the object, the set of state parameters Z, which characterizes the condition of the elements (components) of the model, and the set of output data Y. The mapping F in this approach is characterized by the structure (the type of the element and the connections between them) and the set of parameters. Most often, researchers are limited development of the model to the adjustment of the parameters with a fixed mapping structure, which is selected based on the physical considerations, experimental data, etc., but the simultaneous selection of the structure and parameters of the model seems more adequate. Such a selection of parameters (or parameters and structure) is performed based on the minimization of a certain set of functionals {J} of an error, quality, etc., which determine the degree to which the model fulfills its purpose.

In data processing tasks generally, a finite set of input parameters X and a set of corresponding parameters Y is given. In this case, the error functional shows by how much does the output of the model for a given input differs from the output Y known from the experience.

In the problems, where the construction of the mathematical model is carried out based on differential data, the functional shows by how much does the desired function satisfy the differential equation, which is assumed to be known. In such problems, additional ratios are traditionally used in the form of specified boundary and initial conditions, although our approach allows us easily to consider the problems in which, in addition to the differential equation, approximately known, generally speaking, replenished experimental data are specified. Meanwhile, additional relationships related to the nature of the described object or reflecting the model features could be considered: symmetry requirements, conservation laws, solvability conditions for the arising problem, etc. In fact, the methods we offer for solving these problems differ little from each other; we only need to properly select a set of functionals {J} and a set of functions F, in which the model is selected. The proposed approach to modeling in this wording is consonant with the idea expressed by the great L. Euler: Every effect in the universe can be explained as satisfactorily from final causes, by the aid of the method of maxima and minima, as it can from the effective causes.

In this book, models are sought in several standard functional classes, usually referred to as neural networks. Currently, both in Russia and abroad, a wealth of experience has been gained in applying certain types of neural networks to numerous practical tasks. The need to establish a unified methodology for the development of algorithms for the construction and training of various types of neural networks, as applied to solve a wide class of modeling problems, has matured.

There is a fairly wide range of tasks in mathematical modeling (related generally to the description of systems with distributed parameters), which lead to the studies of the boundary value problems for partial differential equations (or integro-differential equations).

The main methodological mistake in the building of mathematical models of real objects is that the partial differential equation (along with the boundary conditions) is taken as the modeling object from which its approximate model is built—the solution found through one or another numerical method. It is more accurate to look at the differential equations (all together with accompanying initial and boundary conditions) as an approximate model containing information about the modeled object, from which one can move to a more convenient model (for instance, functional), using equations and other available information. Even more accurate is the consideration of the hierarchy of models of different accuracy and area of applicability, which can be refined as new information becomes available.

Only a small number of problems in mathematical physics, usually possessing symmetry, allows an exact analytical solution. Existing approximate solution methods either allow us to obtain only a pointwise approximation likewise to grid methods (obtaining a certain analytical expression from the pointwise solution is a separate problem) or impose specific requirements on a set of approximate functions and require solving an important auxiliary problem of partition of the original region, just as in the finite element method.

The existing neural network approaches for solving problems of mathematical physics are either highly specialized or use variants of the collocation method with fixed neural network functions, which can lead to notable errors between the nodes.

Among the publications devoted to the topic of the use of neural networks for solving partial differential equations (generally, these are special type networks with inherent adjustment method), we are going to note some.

Our goal was not to present an exhaustive review of the literature on the application of artificial neural networks for building approximate solutions of differential equations. Any review of this sort quickly becomes outdated. This goal is more consistent with the review publications in scientific journals. Here we indicate those works that we consider important for the indicated research direction, those publications that impressed us and helped us in determining our approach towards the construction of mathematical models of real objects from heterogeneous data.

In his fundamental work [1], devoted to the 20th anniversary (1968–88) of the multiquadric-biharmonic method (MQ-B), R. Hardy not only addresses the history of the origin and development of this original method but also summarizes the main ideas and advantages associated with its application.

Failures in use in the topography of different types of modifications of trigonometric and polynomial series for constructing curves or surfaces on the scattered data led to the emergence of a new approach. To approximate a function, for example, of two independent variables instead of piecewise approximators, expansion (in Hardy’s notation)

by multiquadrics in the form Qj(x, y; xj, yj, Δ)=[(x−xj)²+(y−yj)²+Δ²]¹/² is used.

The advantages of expansions are noted: their greater efficiency in comparison, for instance, with expansions in spherical harmonics; multiquadrics Qj, and, consequently H type functions are infinitely differentiable, which makes it possible to fit the values of functions, derivatives, etc. An overview of the applications of this approach is given in geodesy, geophysics, cartography, and terrestrial survey; in photogrammetry, remote sensing, measurement, control and recognition, signal processing; in geography and digital terrain model; in hydrology.

A number of significant results in the application of neural networks with radial basis functions, called RBF, for solving differential and integral equations were obtained by Edward Kansa. We would highlight articles [2–5]. In the online publication [4], he gives an overview of different approaches and motivation to use RBF for solving partial differential equation, outlining prospects, and identifying problem areas. Kansa notes that the numerical solution of partial differential equations is mainly prevailed by the finite differences, elements or volume methods, using local interpolation schemes. For local approximations, these methods require the construction of a grid, which in the case of large dimensions represents a nontrivial problem.

An asymmetric collocation method for partial differential equations is considered using the example of elliptic equations. In the case of parabolic and hyperbolic equations, the use of RBF expansions is carried out only for spatial coordinates, whilst the time dependence is considered in the method of straight lines (which leads to the consideration of regular differential equations). In the elliptic case, differential equations and boundary conditions, for simplicity, are assumed to be linear, and the boundary value problem is properly posed. The action of the operators comes down to the effect upon the basis functions, and the collocation problem arises, leading to the solution of the linear system for the coefficients of expansion in the basis functions.

Kansa points out that more interesting problems for the partial differential equations require a well-thought-out arrangement of nodes to cover a fairly wide range of examples for important physical phenomena. The use of MQ for solving Burgers’ equation with viscosity revealed that with an increase in Reynolds number, the adaptive selection of the subset nodes was required to describe the discontinuity front adequately.

The most important place in Kansa’s work is his remark on the problem definition from the perspective of global optimization, bypassing collocation methods, which are essentially ill-conditioned. The works by Yefim Galperin et al. [6, 7], developing this approach, are distinguished, in which the initial conditions, boundary conditions and the equations themselves are introduced through functionals, the weighted sum of which defines the final global functional. The