Feedforward Neural Networks: Fundamentals and Applications for The Architecture of Thinking Machines and Neural Webs

By Fouad Sabry

What Is Feedforward Neural Networks

A feedforward neural network, often known as a FNN, is a type of artificial neural network that does not have connections that form a cycle between its nodes. Therefore, it is distinct from its offspring, which are known as recurrent neural networks.

How You Will Benefit

(I) Insights, and validations about the following topics:

Chapter 1: Feedforward neural network

Chapter 2: Artificial neural network

Chapter 3: Perceptron

Chapter 4: Artificial neuron

Chapter 5: Multilayer perceptron

Chapter 6: Delta rule

Chapter 7: Backpropagation

Chapter 8: Types of artificial neural networks

Chapter 9: Learning rule

Chapter 10: Mathematics of artificial neural networks

(II) Answering the public top questions about feedforward neural networks.

(III) Real world examples for the usage of feedforward neural networks in many fields.

Who This Book Is For

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of feedforward neural networks.

What Is Artificial Intelligence Series

The Artificial Intelligence eBook series provides comprehensive coverage in over 200 topics. Each ebook covers a specific Artificial Intelligence topic in depth, written by experts in the field. The series aims to give readers a thorough understanding of the concepts, techniques, history and applications of artificial intelligence. Topics covered include machine learning, deep learning, neural networks, computer vision, natural language processing, robotics, ethics and more. The ebooks are written for professionals, students, and anyone interested in learning about the latest developments in this rapidly advancing field.
The Artificial Intelligence eBook series provides an in-depth yet accessible exploration, from the fundamental concepts to the state-of-the-art research. With over 200 volumes, readers gain a thorough grounding in all aspects of Artificial Intelligence. The ebooks are designed to build knowledge systematically, with later volumes building on the foundations laid by earlier ones. This comprehensive series is an indispensable resource for anyone seeking to develop expertise in artificial intelligence.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: Jun 24, 2023

Book preview

Chapter 1: Feedforward neural network

A feedforward neural network, often known as a FNN, is a kind of artificial neural network that does not include connections that create a cycle between its nodes. As a result of this, it is distinct from its descendent in the following ways: neural networks with recurrent connections.

The feedforward neural network was the very earliest and most basic kind of artificial neural network that was ever developed.

A linear neural network, which is the simplest kind of feedforward neural network, consists of a single layer of output nodes; the inputs are supplied straight to the outputs through a sequence of weights. Each node performs a calculation that determines the total sum of the products of the weights and the inputs. By making an adjustment to the weights, the mean squared errors that occur between these estimated outputs and a set of predetermined target values may be reduced to a greater extent. The method of least squares, often known as linear regression, has been used for more than two centuries and is the name commonly given to this methodology. For the purpose of predicting the movement of planets, Legendre (1805) and Gauss (1795) utilized it as a method for obtaining a decent, approximate linear fit to a group of points.

The single-layer perceptron is a kind of neural network that combines a threshold function with a linear neural network. If the output value is greater than a certain threshold, which is normally set at zero, the neuron fires and takes the activated value, which is typically one; if the threshold is not met, the neuron takes the deactivated value, which is often negative one. Neurons that have this particular kind of activation function are often referred to as linear threshold units. In the scientific literature, the word perceptron is often used to refer to networks that include just a single instance of these units. Both Ernst Ising and Wilhelm Lenz, working in the 1920s for the Ising model, and Warren McCulloch and Walter Pitts, working in the 1940s, described neurons in physics that were quite similar to one other.

The active and deactivated states of a perceptron may each have any value that the designer chooses, so long as the threshold value falls somewhere in the middle of the two.

It is possible to train perceptrons using a simple learning technique that is often referred to as the delta rule. In order to perform a kind of gradient descent, it first computes the errors that occur between the estimated output and the sample output data, and then utilizes this information to generate an adjustment to the weights.

Single-layer perceptrons are only capable of learning linearly separable patterns. In 1969, Marvin Minsky and Seymour Papert demonstrated in a classic book titled Perceptrons that it was impossible for a single-layer perceptron network to learn an XOR function. Despite this, it was known that multi-layer perceptrons, also known as MLPs, are capable of constructing every boolean function that may possibly exist. For instance, Shun'ichi Amari published his first book in 1967.

A neural network with a single layer has the ability to calculate a continuous output rather than a step function. The so-called logistic function is an option that is often selected:

{\displaystyle f(x)={\frac {1}{1+e^{-x}}}}

If this option is used, the single-layer network will be exactly the same as the logistic regression model, which is utilized extensively in statistical modeling. One of the families of functions known as sigmoid functions includes the logistic function. This is due to the fact that their graphs take the form of an S, which is similar to the lowercase last letter of the Greek letter Sigma. Due to the fact that it has a continuous derivative, it may be used in the process of backpropagation. Additionally, this function is favored due to the simplicity with which its derivative may be computed:

{\displaystyle f'(x)=f(x)(1-f(x)).}

(The fact that f satisfies the differential equation above can easily be shown by applying the chain rule.)

If the activation function of a single-layer neural network is modulo 1, then this network is able to solve the XOR problem with only a single neuron.

{\displaystyle {\begin{aligned}f(x)&=x\mod 1\\f'(x)&=1\end{aligned}}}

This category of networks is made up of several layers of computing units, which are often linked to one another in a feed-forward fashion. Every neuron in one layer is connected, through directed connections, to every neuron in the layer below it. A sigmoid function is used as an activation function for the units that make up these networks in many different applications. However, because learning the vanishing gradient issue, sigmoidal activation functions are not effective in deep neural networks. This is because these functions have extremely tiny derivative values outside of a limited range.

According to the universal approximation theorem for neural networks, every continuous function that maps intervals of real numbers to some output interval of real numbers can be approximated arbitrarily closely by a multi-layer perceptron with just one hidden layer. This theorem was developed for neural networks and states that this can be done for any function that maps intervals of real numbers to some output interval of real numbers. This finding is valid for a broad variety of activation functions, including the sigmoidal functions, amongst others.

Learning strategies of all kinds are implemented in multi-layer networks. Alexey Grigorevich Ivakhnenko and Valentin Lapa are credited with the publication of the first deep learning MLP in the year 1965. In order to do this, the network performs a calculus on the error function's derivative with regard to the network weights. It then adjusts the weights in such a way that the error is reduced (thus going downhill on the surface of the error function). Because of this, back-propagation can only be used on systems that have many distinct activation functions.

In general, the challenge of teaching a network to perform well even on examples that were not used as training samples is a very nuanced one that necessitates the use of additional methods. This is of the utmost importance in situations when only a relatively small number of training samples are available. The risk is that the network will try to match itself too closely to the training data and will not be able to accurately capture the real statistical process that is producing the data. The focus of computational learning theory is on developing methods for training classifiers using just a subset of available data. A straightforward heuristic that deals with neural networks and goes by the name early stopping helps to guarantee that the network will generally perform well with data that was not part of the training set.

Other issues that often arise with the backpropagation method are the rate at which it converges and the risk that it will arrive at a solution that is lower than the global minimum for the error function. Back-propagation in multi-layer perceptrons is now the tool of choice for many machine learning problems because to the existence of practical techniques that make it possible.

One further option is to utilize a number of separate neural networks that are then regulated by an intermediate, which is a behavior that is analogous to what occurs in the brain. These neurons are capable of performing independently and can do a significant amount of work; the outputs may then be merged.

In a broader sense, a feedforward network may be constructed out of any directed acyclic graph by designating some nodes (that do not have any children or parents) as inputs and other nodes (that do not have any parents or children) as outputs. One way to think of them is as multilayer networks, with certain edges skipping layers and the layers being counted either backwards starting from the outputs or forwards starting from the inputs. It is possible to make use of a wide variety of activation functions, and there may also be links between the weights, as seen in convolutional neural networks.

Radial basis function networks are one kind of feedforward network. These networks utilize a different activation function than conventional feedforward networks.

In certain contexts, the term multi-layer perceptron may be used in a generic sense