Long Short Term Memory: Fundamentals and Applications for Sequence Prediction

By Fouad Sabry

What Is Long Short Term Memory

Long short-term memory, often known as LSTM, is a type of artificial neural network that is utilized in the domains of deep learning and artificial intelligence. LSTM neural networks have feedback connections, in contrast to more traditional feedforward neural networks. This type of recurrent neural network, commonly known as an RNN, is capable of processing not only individual data points but also complete data sequences. Because of this property, LSTM networks are particularly well-suited for the processing and forecasting of data. For instance, LSTM can be used to perform tasks such as connected unsegmented handwriting identification, speech recognition, machine translation, speech activity detection, robot control, video game development, and healthcare.

How You Will Benefit

(I) Insights, and validations about the following topics:

Chapter 1: Long short-term memory

Chapter 2: Artificial neural network

Chapter 3: Jürgen Schmidhuber

Chapter 4: Recurrent neural network

Chapter 5: Vanishing gradient problem

Chapter 6: Sepp Hochreiter

Chapter 7: Gated recurrent unit

Chapter 8: Deep learning

Chapter 9: Types of artificial neural networks

Chapter 10: History of artificial neural networks

(II) Answering the public top questions about long short term memory.

(III) Real world examples for the usage of long short term memory in many fields.

Who This Book Is For

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of long short term memory.

What Is Artificial Intelligence Series

The Artificial Intelligence book series provides comprehensive coverage in over 200 topics. Each ebook covers a specific Artificial Intelligence topic in depth, written by experts in the field. The series aims to give readers a thorough understanding of the concepts, techniques, history and applications of artificial intelligence. Topics covered include machine learning, deep learning, neural networks, computer vision, natural language processing, robotics, ethics and more. The ebooks are written for professionals, students, and anyone interested in learning about the latest developments in this rapidly advancing field.
The artificial intelligence book series provides an in-depth yet accessible exploration, from the fundamental concepts to the state-of-the-art research. With over 200 volumes, readers gain a thorough grounding in all aspects of Artificial Intelligence. The ebooks are designed to build knowledge systematically, with later volumes building on the foundations laid by earlier ones. This comprehensive series is an indispensable resource for anyone seeking to develop expertise in artificial intelligence.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: Jun 26, 2023

Book preview

Chapter 1: Time-inhomogeneous hidden Bernoulli model

For the purpose of automated voice identification, the time-inhomogeneous hidden Bernoulli model (TI-HBM) is a possible replacement for the hidden Markov model (HMM).

In contradiction to HMM, TI-HBM does not have a Markov-dependent process for the process of transitioning between states, Rather, it is a Bernoulli process that may be extended (one that is independent).

The TI-HBM decoding process is simplified as a result of this distinction since it eliminates the need for dynamic programming at the state level.

Thus, the computational complexity of TI-HBM for probability evaluation and state estimation is {\displaystyle O(NL)} (instead of {\displaystyle O(N^{2}L)} in the HMM case, where N and L are number of states and observation sequence length respectively).

The TI-HBM is capable of modeling the duration of acoustic units (such as.

phoneme/word length) via the use of a built-in metric known as survival probability.

In a job involving phoneme identification, the TI-HBM is both easier to use and quicker than the HMM, However, its functionality is similar to that of HMM.

For information, read [1] or [2].

{End Chapter 1}

Chapter 2: Artificial neural network

Computing systems that are modeled after the biological neural networks that make up animal brains are known as artificial neural networks, or ANNs for short. These networks are more often referred to as neural networks, or just NNs.

An ANN is built on top of a network of interconnected units or nodes that are referred to as artificial neurons. These neurons are meant to roughly imitate the neurons that are found in a biological brain. Each connection, similar to the synapses that are found in real brains, has the ability to send a signal to other neurons. An artificial neuron is one that first receives signals, then analyzes those signals, and then sends messages to other neurons that it is linked to. The signal at a connection is a real number, and the output of each neuron is calculated by some non-linear function of the sum of its inputs. The signal at a connection is referred to as the signal at the connection. Edges are another name for the connections. The weight of neuronal dendrites and edge connections often changes as a function of ongoing learning. The intensity of the signal at a connection may be increased or decreased depending on the weight. It's possible that neurons have a threshold, and that only when the total signal exceeds that threshold will the neuron send out a signal. In most cases, groups of neurons are organized into layers. It's possible for each layers to make unique changes on the data that they receive. Signals make their way from the first layer, which is known as the input layer, to the final layer, which is known as the output layer, perhaps after traveling through the layers more than once.

Processing instances, each of which has a known input and output, allows neural networks to learn (or be trained) by creating probability-weighted connections between the two, which are then stored inside the data structure of the net itself. This is how neural networks learn. The process of training a neural network using an existing example often involves calculating the difference between the processed output of the network (typically a prediction) and a target output. This is done in order to determine how well the network has learned from the example. This distinction is where the mistake lies. The network will then make any necessary adjustments to its weighted associations in accordance with a learning rule and making use of this error value. The output that is generated by the neural network as a result of successive changes will become progressively similar to the output that is intended. After a sufficient number of these changes, the training may be ended depending on specified conditions after having been completed. This kind of learning is referred to as supervised learning.

These kinds of systems learn to carry out tasks by thinking about examples, in most cases without being programmed with rules that are relevant to the activities themselves. For instance, in image recognition, they might learn to recognize images that contain cats by analyzing example images that have been manually labeled as cat or no cat, and then using the results to recognize cats in other images. In this way, they would learn to identify images that contain cats by analyzing images that have been manually labeled as cat or no cat. They accomplish this despite the fact that they have no previous knowledge about cats, such as the fact that cats have hair, tails, whiskers, and faces that are similar to cats. Instead, they develop identifying traits in a completely automated fashion from the instances that they analyse.

It was Warren McCulloch and Walter Pitts who made the discovery that basic perceptrons were unable to process the exclusive-or circuit and that computers did not have the capability to process neural networks that were helpful.

1970 saw the publication of the general approach for automated differentiation (AD) of discrete linked networks of nested differentiable functions that had been developed by Seppo Linnainmaa. in terms of criteria such as the identification of traffic signs (IJCNN 2012).

ANNs were first conceived as an effort to mimic the structure of the human brain in order to solve problems that were difficult for traditional algorithmic approaches to solve. They quickly shifted their focus to enhancing the empirical outcomes, largely giving up on efforts to stay faithful to the biological origins of their products. Neurons are linked to one another in a variety of different configurations, which enables the output of some neurons to become the input of other neurons. The network has the shape of a graph that is directed and weighted.

Artificial neural networks (ANNs) are made up of artificial neurons, which are essentially drawn from biological neuronal networks. Each artificial neuron receives inputs and generates a single output that is capable of being distributed to several additional neurons. The feature values of a sample of external data, such as photos or papers, may serve as the inputs. Alternatively, the outputs of other neurons can serve in this capacity. The job, such as identifying an item in a picture, is completed by the outputs of the last output neurons of the neural net.

In order to determine what the output of the neuron is, we must first calculate the weighted sum of all of the neuron's inputs. This sum must be computed while taking into account the weights of the connections that go from the inputs to the neuron. We are going to add a bias term to this total.

In deep learning, the neurons are often arranged in numerous layers, since this kind of learning emphasizes complexity. Neurons in one layer can only make connections with other neurons in the layer immediately before and the layer immediately succeeding it. The layer that takes