Statistical Classification: Fundamentals and Applications

By Fouad Sabry

What Is Statistical Classification

In the field of statistics, the problem of classification refers to the task of determining which of a number of categories (sub-populations) an observation belongs to. Assigning a particular email to the "spam" or "non-spam" class is one example; another is providing a diagnosis to a patient on the basis of observed features of that patient.

How You Will Benefit

(I) Insights, and validations about the following topics:

Chapter 1: Statistical classification

Chapter 2: Supervised learning

Chapter 3: Support vector machine

Chapter 4: Naive Bayes classifier

Chapter 5: Linear classifier

Chapter 6: Decision tree learning

Chapter 7: Generative model

Chapter 8: Feature (machine learning)

Chapter 9: Multinomial logistic regression

Chapter 10: Probabilistic classification

(II) Answering the public top questions about statistical classification.

(III) Real world examples for the usage of statistical classification in many fields.

(IV) 17 appendices to explain, briefly, 266 emerging technologies in each industry to have 360-degree full understanding of statistical classification' technologies.

Who This Book Is For

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of statistical classification.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: Jun 23, 2023

Book preview

Chapter 1: Statistical classification

The challenge of determining which category (sub-population) an observation (or series of observations) belongs to in statistics is known as classification. Examples include classifying an email as spam or non-spam, and determining a patient's diagnosis based on the patient's symptoms (sex, blood pressure, presence or absence of certain symptoms, etc.).

A set of quantifiable traits, also referred to as explanatory variables or features, are frequently derived from the analysis of the individual data. These characteristics can be categorical (such as A, B, AB, or O for blood type), ordinal (such as big, medium, or small), integer-valued (such as the frequency of a specific word in an email), or real-valued (e.g. a measurement of blood pressure). Other classifiers compare observations to prior observations using a distance or similarity function.

A classifier is an algorithm that implements classification, particularly in a practical implementation. The mathematical function carried out by a classification algorithm, which assigns input data to a category, is sometimes referred to as a classifier on occasion.

Term usage varies a lot between fields. The characteristics of observations are known as explanatory variables (or independent variables, regressors, etc.) in statistics, where classification is frequently done with logistic regression or a similar procedure, and the categories to be predicted are known as outcomes, which are considered to be possible values of the dependent variable. In machine learning, the various categories that can be predicted are referred to as classes, the observations are frequently referred to as instances, and the explanatory variables are referred to as features (grouped into a feature vector). Different terminology may be used in other fields: For instance, the term classification in community ecology typically refers to cluster analysis.

Examples of the more broad problem of pattern recognition, which is the assignment of some form of output value to a given input value, include classification and clustering. Other instances include parsing, which assigns a parse tree to an input sentence detailing the grammatical structure of the sentence, regression, which assigns a real-valued output to each input, sequence labeling, which assigns a class to each member of a sequence of values, etc.

Probabilistic classification is a typical classification subclass. These kinds of algorithms use statistical inference to determine which class is appropriate for a certain instance. Probabilistic algorithms produce a probability that the instance belongs to each of the potential classes, in contrast to other algorithms that just return the best class. Then, in most cases, the class with the highest probability is chosen. However, compared to non-probabilistic classifiers, such a method provides a number of advantages:

It can produce a confidence value corresponding to its selection (in general, a classifier that can do this is known as a confidence-weighted classifier).

Consequently, it can refrain if it isn't confident enough to select a specific output.

Probabilistic classifiers can be more efficiently incorporated into more complex machine learning tasks in a way that partially or entirely eliminates the issue of error propagation because of the probabilities that are generated.

Fisher started the statistical classification process by developing a number of classification criteria based on various Mahalanobis distance modifications, with a new observation being allocated to the group whose center has the smallest adjusted distance from the observation.

Contrary to frequentist methods, Bayesian classification techniques offer a natural way to incorporate all available data regarding the relative sizes of the various categories within the total population.

In some Bayesian techniques, group membership probabilities are calculated; this results in a more informative result than just assigning a single group-label to each new observation.

Binary classification and multiclass classification can be seen as two distinct challenges in classification. Multiclass classification entails placing an object in one of multiple classes, whereas binary classification, a simpler operation, just includes two classes. Multiclass classification frequently necessitates the simultaneous use of multiple binary classifiers because many classification techniques have been created particularly for binary classification.

The majority of algorithms specify a specific instance whose category is to be predicted using a feature vector containing specific, quantifiable attributes of the instance. Each characteristic is referred to as a feature, often known as an explanatory variable in statistics (or independent variable, although features may or may not be statistically independent). Features can be categorical (such as A, B, AB, or O, for blood type), ordinal (such as big, medium, or small, or integer-valued (such as the number of times a specific word appears in an email), binary (such as on or off), or real-valued (e.g. a measurement of blood pressure). If the instance is an image, the feature values may be the image's pixels; if it is a piece of text, the feature values may be the frequency with which various words occur. Some algorithms only operate on discrete data and necessitate the grouping of real-valued or integer-valued data (e.g. less than 5, between 5 and 10, or greater than 10).

Many classification techniques can be expressed as a linear function that uses a dot product to combine the feature vector of an instance with a vector of weights to award a score to each of the k possible categories. The category with the highest score is the one that was anticipated. The following generic form describes the linear predictor function, a particular kind of score function:

{\displaystyle \operatorname {score} (\mathbf {X} _{i},k)={\boldsymbol {\beta }}_{k}\cdot \mathbf {X} _{i},}

where Xi is the feature vector for instance i, βk is the vector of weights corresponding to category k, and score(Xi, k) is the rating given when categorizing instance I under category k.

Theorem of Discrete Choice, where options are situations, and individuals are persons, The utility connected to person I selecting category k is represented by the score.

These fundamentally arranged algorithms are referred to as linear classifiers. The method used to establish (train) the ideal weights and coefficients, as well as how the result is interpreted, set them apart.

Some examples of these algorithms include

Statistical model for a binary dependent variable using logistic regression

Regression with more than two discrete outcomes is known as multinomial logistic regression.

Regression using only two possible values for the dependent variable is known as probit regression.

The perceptron algorithm

a group of techniques for supervised statistical learning called the support vector machine

A technique used in statistics, pattern recognition, and other disciplines is linear discriminant analysis.

A vast toolset of classification algorithms has been developed since no single type of classification is suitable for all types of data sets. The most often employed include:

A computational model for machine learning based on connected, hierarchical functions is called an artificial neural network.

Boosting (meta-algorithm) is a machine learning technique.

Machine learning algorithm using decision trees

Machine learning method for ensembles based on binary search trees called random forest

The practice of encoding computer programs as a collection of genes is known as genetic programming.

Algorithm using gene expression programming that uses evolution

Multi expression programming

An example of a genetic programming algorithm is linear genetic programming.

Window function for kernel estimation

A non-parametric classification technique is k-nearest neighbor.

Learning vector quantization

Machine learning's linear classifier for statistical classification

Fisher's linear discriminant: A technique used in pattern recognition, statistics, and other disciplines

Statistical model for a binary dependent variable using logistic regression

Probabilistic categorization algorithm: Naive Bayes

Binary classifiers can be learned under supervision using the perceptron algorithm.

In machine learning, a quadratic classifier is