Applied Deep Learning: Design and implement your own Neural Networks to solve real-world problems (English Edition)

By Dr. Rajkumar Tekchandani and Dr. Neeraj Kumar

Deep Learning has become increasingly important due to the growing need to process and make sense of vast amounts of data in various fields. If you want to gain a deeper understanding of the techniques and implementations of deep learning, then this book is for you.

The book presents you with a thorough introduction to AI and Machine learning, starting from the basics and progressing to a comprehensive coverage of Deep Learning with Python. You will be introduced to the intuition of Neural Networks and how to design and train them effectively. Moving on, you will learn how to use Convolutional Neural Networks for image recognition and other visual tasks. The book then focuses on localization and object detection, which are crucial tasks in many applications, including self-driving cars and robotics. You will also learn how to use Deep Learning algorithms to identify and locate objects in images and videos. In addition, you will gain knowledge on how to create and train Recurrent Neural Networks (RNNs), as well as explore more advanced variations of RNNs. Lastly, you will learn about Generative Adversarial Networks (GAN), which are used for tasks like image generation and style transfer.

• Language: English
• Publisher: BPB Online LLP
• Release date: Apr 29, 2023
• ISBN: 9789355513700

Book preview

Chapter 1

Basics of Artificial Intelligence and Machine Learning

Introduction

Artificial intelligence is the demonstration of intelligent processes by machines, especially computer systems similar to human intelligence. The process of artificial intelligence includes learning, reasoning, and self-correction. This chapter covers the basic concepts related to artificial intelligence, such as patterns, classification, and regression problems. In this chapter, we briefly describe the basic concept of machine learning algorithms along with their types, such as deep learning and shallow learning techniques.

Structure

In this chapter, we will cover the following topics:

Patterns

Intra-class variability

Inter-class similarity

Pattern analysis tasks

Data and its types in machine learning

Machine learning feature set

Types of Regression

Classification

Clustering

Agglomerative clustering

Divisive clustering algorithms

Clustering applications

Hands-on session on AI and machine learning basics

Objectives

After going through this chapter, you will be able to understand the basic concepts of machine learning and its types. You will understand the types of classification and regression problems. You will get a brief idea about clustering algorithms and will be able to perform clustering algorithms in R.

Patterns

In the last few years, pattern analysis has been one of the emerging trends in the research community. A pattern can be termed as a type of theme of repeating events or elements of a set of objects. The elements repeat themselves in a manner which can be predictable. In the digital world, a pattern is almost everything. For example, the color of clothes, speech patterns, and so on. It can either be observed mathematically or physically by using some type of algorithm.

A pattern can be a proven solution in a specified manner for a common problem. In 1979, Alexander quoted Each pattern is a three-part rule, which expresses a relation between a certain context, a problem, and a solution. According to the Gang of Four, a pattern can be summarized into four different parts: a context where the pattern can be useful, the issues which are addressed by the patterns, the forces used for forming a solution and the solution that resolves those forces.

Example 1: Consider the following example and try to complete these patterns.

1,2,3,4,5,6,…,24,25,26,27,28.

1,3,5,7,9,11,…,25,27,29,31,33

2,3,5,7,11,13,…,29,31,37,41,43

1,4,9,16,25,36,…,121,144,169,196

1,2,4,8,16,32,64,…,1024,2048,4096,8192

1,1,2,3,5,8,13,…,55,89,144,233,377

1,1,2,4,7,13,24,…,81,149,274,504,927

3,5,12,24,41,63,….., 201,248,300,357,419

2,7,12,17,22,27,32,…..,42,47,52,57,62

1,6,19,42,59,…,95,117,156,191,?

As per the preceding example, it is easy to fill the patterns initially, but as we progress downwards, it becomes a little bit complex. So, the pattern is defined as any regularity or structure in data and pattern analysis is the automatic discovery of patterns in data.

Pattern representation

In the field of computer science, a pattern is represented by using vector feature values. The feature is any distinct characteristic, quality, or aspect. Features can be numeric (for example, width and height) or symbolic. Suppose if there are d features of an object; then the combination of these d features can be represented as a column vector of dimension-d known as a feature vector. Space which is defined by the d-dimension feature vector, is called a feature space. Then, the objects are represented as some points in the feature space, and that representation is termed as a scatter plot. Figure 1.1 shows various pattern representations:

fig1.jpgfig1.jpg

Figure 1.1 (a)–(c): Pattern representation, dimension space of vector, and different classes of patterns.

Figure 1.1(a)–(c) represents the vector of size d, the dimension space of the vector and classes of patterns.

Analysis of patterns

It is a phase of pattern recognition that uses the existing knowledge present in data to uncover patterns using techniques of data analysis. Pattern analysis deals with the detection of patterns of the data from the same source automatically, making predictions of the data coming from the source. The information coming from the source can be of any form, such as text, images, family trees, records of commercial transactions, and so on.

The identification of the patterns from a finite data set has very distinctive and different challenges. So, in order to design an effective pattern analysis algorithm, one should consider three key features as follows.

Robustness

The first challenge for designing an effective pattern analysis algorithm is the fact that when it comes to real-time applications, data can be demolished by noise because of the randomness of the wireless channel or by virtue of human errors. So, while designing the algorithm, it must be kept in mind that the algorithms must identify the approximate patterns and can handle noisy data smoothly such that it should not affect the output of patterns or data analysis techniques. The algorithms that possess this property are considered robust.

Computational efficiency

As the amount of data is increasing enormously day by day, the designed algorithm must be able to handle larger datasets due to the enormous increase in data with time. So, if the algorithm works well for small objects, it should also work well for large datasets. Basically, computationally efficient algorithms are those whose resources scale increases polynomially with an increase in the size of the data.

Statistical ability

This property is the most basic property an algorithm should have, which states that the patterns that are identified by the algorithm are genuine and have an accidental relation with the data set attributes. We can define this property as if we apply the algorithm to the new data coming from the same source so that it should be able to identify a similar type of pattern. Thus, the output of the algorithm should be sensitive to the data source and not to the particular dataset. So, if the algorithm is sensitive only to a particular dataset, it can be termed as stable for short, and if the algorithm gives similar types of patterns from all the datasets coming from the same source, then such an algorithm can be termed as statistically stable.

Pattern classes

A pattern class can be defined as a collection of similar types of objects that may not be identical. A class consists of exemplars, prototypes, paradigms, and learning/training samples. Figure 1.2 represents different types of class variabilities:

fig3.jpg

Figure 1.2 (a): Low inter-class and high intra-class variability. (b) Low intra-class and high inter-class variability.

There are the following two types of variability and one type of similarity in the pattern class:

Intra-class variability

Inter-class variability

Machine learning algorithms have to deal with these variabilities and similarities.

Intra-class variability

It refers to the deviations in the particular class score for a specific object which is not a part of the systematic difference. So, basically, it is within the class variability. It can be used to map different types of objects. If we talk about land covers, then bare soil, forests, and rocks can be mapped using intra-class variability. It is the variation that exists between all the samples of a particular class that are used to learn the machine.

Intra-class variability uses a tool named as feature space which predicts the patterns according to their features by using spectral signatures. So, according to the specific requirement, an appropriate feature space can be chosen.

Inter-class variability

Inter-class variability means the variability among the different types of classes in a dataset. It can be used in cases where one needs to separate different classes that exist in the dataset. Also, the accuracy of the classification of objects depends upon inter-class variability. Figure 1.2(a) represents the patterns having low inter-class and high intra-class variability, whereas Figure 1.2(b) represents the patterns having low intra-class and high inter-class variability. For an ideal clustering algorithm, the intra-class variability should be minimum, whereas inter-class variability should be maximum. This is an important criterion for getting classes of different patterns. In classification, there exists a labeled dataset, so inter and intra-class variability does not exist in supervised learning. The inter-class and intra-class variability is dependent on a distance metric.

Inter-class similarity

In inter-class similarity, the data is similar, almost nearest to each other and belongs to the same cluster. The outcomes are calculated as the ratio of the total summation distance to the summation of all the distances within one cluster of the dataset. Inter-class similarity can be seen in figure 1.3:

fig2.jpg

Figure 1.3: Inter-class similarity

If you observe I1, both I and 1 look similar, but they are actually different. Here, I represent an alphabet, and 1 represents a number. So, this shows the inter-class similarity. The interclass similarity shows the similarity with other classes.

Pattern analysis tasks

When we discuss the importance of patterns in data, the main aim of pattern analysis is the frequent prediction of data features as the function of the values of the other features. So, it is expected that the tasks associated with pattern analysis isolate a common feature, that is, their prediction intention. The training data arrives as (X, Y), where X is the feature set called as vector and represented as X= { x1, x2…. xn}. A function f’’ is required to predict the class label Y’ on the basis of feature set X’. Basically, there are three types of pattern analysis tasks namely supervised, semi-supervised, and unsupervised tasks. We will discuss all these types of tasks in detail.

Pattern classification—supervised tasks

The supervised tasks are pattern analysis tasks where there is an incorporated label for each output. For such types of tasks, a pattern can be sought in the form of the following:

f(x, y) = L(y, g(x))

Where denotes the loss function and g is the prediction function. It gives the variation between the true value y and the output of the prediction function, so when the pattern is detected, it is expected that the loss is zero. Figure 1.4 shows the process of classification using feature extraction:

Figure 1.4: Pattern classification

For example, if we take the label as Y = Wild Animals, that is, a class, then the feature set is X= {Canines, strips, carnivores}. If a particular object, say, Tiger has these features, then our prediction algorithm f predicts this object Tiger in Y. Figure 1.4 depicts the pattern classification process where X represents a feature vector.

Pattern clustering-unsupervised tasks

Unsupervised tasks do not have labels on them. These types of tasks are only available as the training data and labels should be predicted for the use of test data. Basically, there are two methods used for unsupervised tasks named as cluster analysis and a principal component. Cluster analysis is a sub-branch of machine learning that groups the unlabeled or unclassified data.

Cluster analysis is used for unsupervised tasks in the grouping of datasets with other attributes. Cluster analysis identifies the common identities in the data sample present and then further reacts on the basis of the absence or presence of such common entities from the new data coming from the same data source.

Semi-supervised tasks

In semi-supervised tasks, the distinct label or feature is known partially. For example, consider a case of ranking where we only have relative ordering in the training set. But our objective is to provide the same ordering of the new data coming from the same source. For these types of issues, a value function is assumed, and suggestion for its value is made in the process of training data. So, new data from the same data source is assessed by using the output of the value function.

In another type of situation, there can be a transduction case in which partial information is present regarding the labels. In such type of cases, only some data arrives with the incorporated label, so the task becomes to predict the label for the data that is not labeled. The main aim here is to have the combination of the querying cost and generalization error.

Data and its types in Machine Learning

Data can be defined as a set of values with respect to quantitative or qualitative variables. Nowadays, almost everything can be represented as data. Understanding the different data types is essential for machine learning models. From a machine-learning perspective, mostly the data can be divided into four basic types as follows:

Numerical data

Text data

Time series data

Categorical data

Figure 1.5 shows different types of datasets used in machine learning:

fig34.jpg

Figure 1.5: Types of data

Numerical data

Numerical data is a type of data in which data points are numerical values. It can also be termed quantitative data. For example, the number of cars sold in the past month. Numerical data can be divided into discrete or continuous data. Discrete data is a set of distinct values, whereas continuous data is any value within a specified range.

For example, a student’s score marks as 10, 20, 30, 40, 50, and 60. Then, this is referred to as numeric data within the range of 0 to 100.

Time series data

Time series data can be defined as an order of the numbers collected at regular intervals of time. It has application in the case of finance. There is a temporal value attached to the time series. So, it can be something like a timestamp that comes as trends in time.

For example, measuring the environmental sensor data after every month is a time-series data. The difference between numerical and time series data is that time series follow an implied ordering, whereas numeric data may consist of a bunch of values, but it does not have any ordering sequence.

Text data

Basically, words are text data. While using machine learning techniques, we convert the words into numbers by using appropriate functions such as word formulation function such as word2vec. For example, while reading this book, you are actually reading the text data.

Categorical data

Categorical data is a set of variables that consist of a large number of label values instead of numeric values. In categorical data, the total number of values is represented by a fixed set.

Categorical variables can also be termed as nominals. For example:

A color variable contains the values: green, blue, and red

A pet variable contains the values: cat and dog

T-shirt size as S, M, L, and XL

Figure 1.6 shows a bar graph of categorical data:

fig27.jpg

Figure 1.6: Categorical data representation

Figure 1.6 shows a bar graph of categorical data in which fruit classes are divided into five categories, such as apple, banana, and so on and the number of pieces is shown for each fruit. Here, each value represents a different category.

Issues with the categorical data

Some machine learning algorithms work fine with categorical data. For example, the use of a decision tree algorithm, such as split theory based on information gain, is suitable for learning from the categorical data without performing any data transformation. However, many machine learning algorithms do not work on labeled data as they require all the input and output variables to be in the numeric state.

This can be viewed as a constraint for clustering-based algorithms, so the categorical data have to be converted into numerical data. Also, if the output variable is categorical, then there is a need for converting the predictions into a categorical form so that it can be used for a particular type of application.

Hence, now the question arises as to how to convert categorical data into numeric data.

The conversion of the categorical data into numeric data can be done by using the following encoding techniques:

Integer encoding

One-hot encoding

Both these techniques are explained as follows.

Integer encoding

Integer encoding is the first step for the conversion of categorical data into numeric data. Here, each unique object or class is assigned a value different from other classes. For example, we have three objects, namely, apples, banana, and orange. Table 1.1 shows an example of integer encoding:

Table 1.1: Integer encoding

Integer encoding can be easily reversible as 1 → apple, 2 → banana and 3 → orange.

The advantage of integer encoding is that the integer values have a numerically ordered relationship among themselves which machine learning algorithms can easily understand.

One-hot encoding

In machine learning, one hot encoding includes a collection of bits that represent the data using a unique 1-bit representation for a unique class (high/ON) and the rest of the bits having a value as 0 (low/OFF). It is used for multi-class classification. Apart from this, the representation of the data in which there is only a single 0 bit is present, and the rest of the bits are 1 is called as one-cold. Table 1.2 represents the one hot and binary representation of a number from 0 to 9:

Table 1.2: Binary and one hot representation

One-hot encoding is generally used to represent the different states of the state machine. Moreover, in binary or gray code, the decoder helps in determining the state of the machine. But in the case of the one-hot state machine, there is no need for the decoder.

Table 1.3 represents the label encoding method in which the objects are divided categorically; for example, Apple is labeled into category 1:

Table 1.3: Label encoding

Table 1.4 represents one-hot encoding where each object has only a single 1-bit (high).

Table 1.4: One-hot encoding

Machine learning feature set

In pattern recognition and machine learning, a feature can be defined as a distinct computable characteristic or property of a phenomenon. Choosing effective, informative, and discriminative features are considered as an important role in classification, regression, and pattern recognition. Different types of features are discussed as follows.

Handcrafted features

Handcrafted features refer to the derived properties for use in different algorithms in which the data is present in the image itself. For example, edges and corners are the two simple features which can be extracted from images. A simple edge detection algorithm such as Laplacian based on differentiation (rate of change) works by determining the regions in the image where the intensity of the image changes quickly.

To understand it in simple language, we know that a usual image is nothing but a 2D matrix (or multiple matrices when there are multiple channels like RGB (Red, Green, and Blue). Figure 1.7 represents an image in gray-scale with intensity values from 0 to 255:

grayscale.jpg

Figure 1.7: Gray-scale image

In the case of a gray-scale image (8-bit) (or a black and white image), the image is usually a matrix of two-dimension with pixel values ranging from 0 to 255, where 255 denotes complete white and 0 denotes complete black and the range from 0 to 255 represents grey-scale image as shown in figure 1.7.

Spatio temporal features

Space-time features capture the distinctive shapes in a video. It provides a self-determined representation of events that occurred with time during different types of motion in a scene. Spatiotemporal features are generally extracted from the video directly and avoid the possible failures of pre-processing techniques such as tracking and motion segmentation.

So, the motion and space features are combined over a finite time interval to form the spatiotemporal features. These types of features are commonly used for human action recognition systems, as shown in figure 1.8:

fig30.jpg

Figure 1.8: Use of spatio-temporal features

Discriminative features

Discriminative features are used mostly in face recognition, as shown in figure 1.9:

fig32.jpg

Figure 1.9: Facial image¹

When feature learning is applied to these images, they give output as discriminative features. Figure 1.10 shows the discriminative features of a dataset consisting of the parameters such as height and time:

desc..jpg

Figure 1.10: Discriminative features

Artificial intelligence

In computer science, artificial intelligence is also called machine intelligence. It is the intelligence demonstrated by machines, such as humans display natural intelligence. AI includes the process of reasoning, self-correction, and learning. It can be classified as either strong or weak. Strong AI is also termed artificial general intelligence and is basically an AI system that consists of comprehensive human cognitive capabilities. This type of AI system can find a solution without the use of human involvement, even when given with an unfamiliar task. Moreover, weak AI, known as narrow AI, is an AI system that is trained and intended for a specific task.

For example, Siri (Apple’s virtual personal assistant) is considered as a weak AI. Figure 1.11 represents an image of AI:

artificial-intelligence-confusion-719504626-1068x601.jpg

Figure 1.11: Artificial intelligence ²

Machine learning and its goals

Machine learning can be defined as the scientific study of statistical models and algorithms, which computer systems use to execute an explicit task without using categorical instructions. Thus, it depends upon the inference and patterns instead. It can be observed as a subset of artificial intelligence (AI). A good generalization ability is the main objective of machine learning algorithms.

The following are the goals of machine learning algorithms:

Regression

Binary and multi-class classification

Clustering

Recommendation systems

Supervised and unsupervised anomaly detection

These are explained as follows:

Regression

Regression is an elegant word to describe that a model will allocate a continuous value (response) to a data observation different from a discrete class. We will discuss Regression in detail in the upcoming section.

Classification

Problems related to classification involve the insertion of a data point in a pre-defined category of class. In some cases, the classification problem performs the assignment of a class in an observation, whereas in other cases, the main objective is to predict the chances that the observation, which addresses the given classes. We will discuss Classification in detail further in the Classification section.

Figure 1.12 represents the classification based on different scenes:

Figure 1.12: Image classification based on different scenes ³

Clustering

Clustering is defined as an unsupervised technique for determining the alignment and organization of a given data set. It is a method of collecting information into clusters or groups. Each cluster is considered by an enclosed set of data points and the centroid of the cluster. The centroid of the cluster is the average of all the data points. We will discuss this in detail in the Clustering section.

Figure 1.13 shows the image clustering example based on scenes without any labels:

Figure 1.13: Image clustering example based on scenes.

Figure 1.12 consists of various labels, and figure 1.13 does not consist of any labels. That is the exact difference between supervised (classification) and unsupervised learning (clustering). In the case of supervised learning, we have prior knowledge about labels in which a new object is to be classified, but in the case of unsupervised learning, we do not have any prior knowledge of labels. We have to assign labels by ourselves on the basis of inter-class, intra-class variabilities and inter-class similarity.

Consider figure 1.13 and try to cluster the scenes into clusters of seven wonders of the world.

Recommendation systems

Another type of problem that can be addressed is a recommendation system, or even called as a recommendation engine. These systems are kind of data filtering systems, and they make recommendations for different types of applications, such as books, products, articles, restaurants, music, movies, and so on. In recommendation systems, two common methods are collaborative and content-based filtering.

The use of recommendation systems can be easily seen on Amazon and Netflix. Netflix makes recommendation systems to keep its viewers busy and intends to have plenty of content to see. So, in order to keep their users watching Netflix, they provide these types of recommendations such as Top Picks for Alex, Because you watched …, and Suggestions for you.

Amazon also does the same thing in order to increase its sales by up-selling, maintaining sales by user engagement, and so on. They provide recommendations such as Related to items you viewed, Customers who bought this item also bought, and More items to consider.

Anomaly detection

Another type of problem is anomaly detection. We think that data coming from a particular source is sensible and well-behaved, but actually, this is not the case in most of the cases. Sometimes, there exists irrelevant data because of errors or faults in the measurement and also because of fraud. Sometimes, it can be due to the measurements that are anomalous and revealing the deteriorating piece of electronics or hardware.

Anomalies can even denote the real problems sometimes and cannot be easily described. For example, consider a defect in manufacturing. In such type of cases, the detection of anomalies offers a degree of control of quality and also the perception that whether the steps taken to minimize the defects have actually worked or not. In both cases, there are possible cases when it can be helpful to determine the values of the anomaly, and specific machine learning algorithms can be applied carefully in such cases.

Machine learning techniques

The machine learning techniques are divided into two broad learning techniques, namely, deep learning and shallow learning, and further classification of the two techniques of machine learning is shown in figure 1.14:

fig20c.jpg

Figure 1.14: Types of machine learning techniques

These are all the various types of machine learning techniques, which are discussed as follows.

Deep learning techniques

Deep learning is a subsection of machine learning that consists of neural networks that can automatically train the classifiers for the purpose of classification. It is beneficial when we have a multi-class classification problem. Deep learning can also be referred to as a deep neural network or deep neural learning. Figure 1.15 shows hidden layers of neurons forming a neural network:

nn.jpg

Figure 1.15: Neural network

Now, let us discuss different types of machine-learning techniques.

Discriminative learning techniques

Working with discriminative models, if one wants to predict the label Ci for a given object D, then according to Naïve Bayes Theorem:

(1.1)

Where P(A) is the probability of occurrence of event A. P(B) is the probability of occurrence of event B. P(B|A) is the probability of event B, given that event A has occurred. P(A|B) is the probability of event A, given that event B has occurred

Example 2: We want to find out a patient’s probability of having cancer disease if they are alcoholic. ABC is the test to diagnose cancer disease.

A denotes the event Patient has cancer disease. Previous data tells us that 10% of patients coming to the clinic have cancer disease. P(A) = 0.2.

B denotes the event of the litmus test that the Patient is an alcoholic; 5% of the clinic’s patients are alcoholics. P(B) = 0.08.

In total, 7% are alcoholics among those patients who are diagnosed with liver disease. This is B|A: the probability that a patient is alcoholic, given that they have cancer disease, is 5%.

Applying the Naïve Bayes theorem, we get,

P(A|B) = (0.05 * 0.2)/0.08 = 0.125.

Therefore, there is a 12.5% probability that a patient has cancer if he is an alcoholic.

Let "Ci represents the label/class, and d represent the object. So, by applying Naïve Bayes theorem, we get, whether an object d belongs to a particular class Ci or not.

(1.2)

where

P(Ci) = Probability of the hypothesis Ci.

P(d) = Probability of the object d

P(dCi) = Probability of occurrence of d given Ci.

P(Ci|d) = Probability of occurrence of Ci given d.

In general cases, we need the most probable label given the training dataset. So, in such a case, the maximum a posteriori (MAP) label function is written as follows:

(1.3)

We can divide the discriminative learning techniques into two parts as follows:

Linear regression

Logistic regression

These are two types of regressions that will be discussed in the next sections.

Types of Regression

The problems based on supervised learning are divided into two parts: (i) regression and (ii) classification. Both can be used in building a concise model that can use the attribute variables and can find the values of the dependent attribute.

A problem is said to be a regression problem when the input variable is a continuous or real value. For example, weight, salary, and so on.

There are various models of regression. Figure 1.16 shows the types of regression models:

fig10.jpg

Figure 1.16: Types of regression models

For a proper understanding of regression, let us take the following examples.

Example 3: There are four sentences given here. We have to identify which one is a regression task:

Predicting the age of your friend

Predicting the weight of your friend

Prediction of the nationality of people

Predicting if the stock market will increase tomorrow

Any guesses? So, let us discuss on these four sentences now. Prediction of the age and weight of a person is a type of regression task as it is a real value while predicting the nationality comes under categorical tasks, and changes in the stock market is a discrete value, say yes or no.

Example 4: There are four sentences given here. We have to identify which one is a regression task and which one is a classification task:

Predicting the height of a person

Predicting the gender of the person based on his/her handwriting

Predicting the BMI of a person

Prediction of the nationality of the person

Predicting height and BMI is a reg0ression task and predicting the gender and nationality of a person is a classification task.

Linear regression

Linear regression is a type of regression which fits information with the most suitable hyper-plane. It can be defined as a linear approach for modeling the relationship between independent variables and dependent variables. Figure 1.17 shows the pattern obtained using linear regression:

linearreg.jpg