Bayesian Inference: Fundamentals and Applications

By Fouad Sabry

What Is Bayesian Inference

Bayesian inference is a type of statistical inference that updates the probability of a hypothesis based on new data or information using Bayes' theorem. This way of statistical inference is known as the Bayesian method. In the field of statistics, and particularly in the field of mathematical statistics, the Bayesian inference method is an essential tool. When conducting a dynamic analysis of a data sequence, bayesian updating is an especially useful technique to utilize. Inference based on Bayes' theorem has been successfully implemented in a diverse range of fields, including those of science, engineering, philosophy, medicine, athletics, and the legal system. Bayesian inference is strongly related to subjective probability, which is why it is frequently referred to as "Bayesian probability" in the field of decision theory philosophy.

How You Will Benefit

(I) Insights, and validations about the following topics:

Chapter 1: Bayesian Inference

Chapter 2: Likelihood Function

Chapter 3: Conjugate Prior

Chapter 4: Posterior Probability

Chapter 5: Maximum a Posteriori Estimation

Chapter 6: Bayes Estimator

Chapter 7: Bayesian Linear Regression

Chapter 8: Dirichlet Distribution

Chapter 9: Variational Bayesian Methods

Chapter 10: Bayesian Hierarchical Modeling

(II) Answering the public top questions about bayesian inference.

(III) Real world examples for the usage of bayesian inference in many fields.

(IV) 17 appendices to explain, briefly, 266 emerging technologies in each industry to have 360-degree full understanding of bayesian inference' technologies.

Who This Book Is For

Professionals, undergraduate and graduate students, enthusiasts, hobbyists, and those who want to go beyond basic knowledge or information for any kind of bayesian inference.

• Language: English
• Publisher: One Billion Knowledgeable
• Release date: Jul 1, 2023

Book preview

Chapter 1: Bayesian inference

A statistical inference method known as Bayesian inference uses the Bayes' theorem to update a hypothesis' probability when new data or information becomes available. In statistics, particularly in mathematical statistics, Bayesian inference is a crucial method. When analyzing a sequence of data in a dynamic manner, Bayesian updating is very crucial. Numerous fields, including science, engineering, philosophy, medicine, sport, and law have used Bayesian inference. Subjective probability, often known as Bayesian probability, and Bayesian inference are closely connected concepts in the philosophy of decision theory.

The posterior probability is generated via Bayesian inference from two antecedents: the prior probability and the likelihood function, which is derived from a statistical model of the observed data. According to Bayes' theorem, Bayesian inference calculates the posterior probability:

{\displaystyle P(H\mid E)={\frac {P(E\mid H)\cdot P(H)}{P(E)}},}

where

H stands for any hypothesis whose probability may be affected by data (called evidence below).

There are frequently opposing hypotheses, and the duty is to decide which is most likely.

P(H) , the earlier likelihood, is the estimate of the probability of the hypothesis H before the data E , the evidence at hand, is observed.

E , the evidence, corresponds to fresh information that wasn't used to calculate the previous probability.

{\displaystyle P(H\mid E)} , the post-hoc likelihood, is the probability of H given E , i.e, after E is observed.

The probability of a hypothesis given the observed evidence is what we are interested in learning.

{\displaystyle P(E\mid H)} is the probability of observing E given H and is called the likelihood.

As a function of E with H fixed, It indicates if the evidence is consistent with the stated theory.

The evidence is a function of the likelihood function, E , while the function of the hypothesis is the posterior probability, H .

P(E) is sometimes termed the marginal likelihood or model evidence.

This factor is the same for all possible hypotheses being considered (as is evident from the fact that the hypothesis H does not appear anywhere in the symbol, unlike for all the other criteria), and hence does not affect how likely each hypothesis is in comparison to the others.

For different values of H , only the factors P(H) and {\displaystyle P(E\mid H)} , in the numerator and, affect the value of {\displaystyle P(H\mid E)} – the posterior probability of a hypothesis is proportional to its prior probability (its inherent likeliness) and the newly acquired likelihood (its compatibility with the new observed evidence).

Bayes' rule can be expressed in the following way:

{\displaystyle {\begin{aligned}P(H\mid E)&={\frac {P(E\mid H)P(H)}{P(E)}}\\&={\frac {P(E\mid H)P(H)}{P(E\mid H)P(H)+P(E\mid \neg H)P(\neg H)}}\\&={\frac {1}{1+\left({\frac {1}{P(H)}}-1\right){\frac {P(E\mid \neg H)}{P(E\mid H)}}}}\end{aligned}}}

because

{\displaystyle P(E)=P(E\mid H)P(H)+P(E\mid \neg H)P(\neg H)}

and

{\displaystyle P(H)+P(\neg H)=1,}

where {\displaystyle \neg H} is not H , the logical negation of H .

Utilizing the law of multiplication is one quick and simple method for remembering the equation:

{\displaystyle P(E\cap H)=P(E\mid H)P(H)=P(H\mid E)P(E).}

Bayesian updating is popular and practical from a computing standpoint. But it's not the only updating rule that might be deemed logical.

Ian Hacking pointed out that conventional Dutch book defenses did not require Bayesian updating, leaving open the possibility that Dutch books could be avoided by other updating principles. Hacker stated: The dynamic assumption is not implied by the Dutch book argument or any other proof of the probability axioms in the personalist arsenal. None of them include Bayesianism. The dynamic assumption must therefore be Bayesian according to the personalist. It is true that a personalist may reject the Bayesian model of experience-based learning if they were to maintain consistency. Salt could start to taste bland.

Since the publication of Richard C. Jeffrey's rule, which applies Bayes' rule to the situation when the evidence itself is given a probability, there are non-Bayesian updating procedures that also avoid Dutch books (as addressed in the literature on probability kinematics).

Bayesian inference can be thought of as acting on this belief distribution as a whole if evidence is utilized to update belief over a collection of exclusive and exhaustive propositions.

Suppose a process is generating independent and identically distributed events {\displaystyle E_{n},\ n=1,2,3,\ldots } , nonetheless, the probability distribution is not known.

Let the event space \Omega represent the current state of belief for this process.

Each model is represented by event M_{m} .

The conditional probabilities P(E_{n}\mid M_{m}) are specified to define the models.

P(M_{m}) is the degree of belief in M_{m} .

Prior to the initial inference stage, \{P(M_{m})\} is a set of initial prior probabilities.

They must add up to 1, but else are random.

Suppose that the process is observed to generate {\displaystyle E\in \{E_{n}\}} .

For each M\in \{M_{m}\} , the prior P(M) is updated to the posterior P(M\mid E) .

Using the Bayes theorem:

{\displaystyle P(M\mid E)={\frac {P(E\mid M)}{\sum _{m}{P(E\mid M_{m})P(M_{m})}}}\cdot P(M).}

This process may be repeated in the event that new evidence is observed.

For a sequence of independent and identically distributed observations \mathbf {E} =(e_{1},\dots ,e_{n}) , By induction, it can be demonstrated that repeatedly using the information above is equivalent to

{\displaystyle P(M\mid \mathbf {E} )={\frac {P(\mathbf {E} \mid M)}{\sum _{m}{P(\mathbf {E} \mid M_{m})P(M_{m})}}}\cdot P(M),}

where

{\displaystyle P(\mathbf {E} \mid M)=\prod _{k}{P(e_{k}\mid M)}.}

The conviction in all models may be revised in a single step by parameterizing the space of models. Therefore, it is possible to conceptualize the distribution of belief over the parameter space as a distribution of belief over the model space. As is customary, the distributions in this section are expressed as continuous and are represented by probability densities. However, discrete distributions can also use the method.

Let the vector {\boldsymbol {\theta }} span the parameter space.

Let the initial prior distribution over {\boldsymbol {\theta }} be {\displaystyle p({\boldsymbol {\theta }}\mid {\boldsymbol {\alpha }})} , where {\boldsymbol {\alpha }} is a set of parameters to the prior itself, or hyperparameters.

Let \mathbf {E} =(e_{1},\dots ,e_{n}) be a sequence of independent and identically distributed event observations, where all e_{i} are distributed as {\displaystyle p(e\mid {\boldsymbol {\theta }})} for some {\boldsymbol {\theta }} .

Bayes' theorem is applied to find the posterior distribution over {\boldsymbol {\theta }} :

{\displaystyle {\begin{aligned}p({\boldsymbol {\theta }}\mid \mathbf {E} ,{\boldsymbol {\alpha }})&={\frac {p(\mathbf {E} \mid {\boldsymbol {\theta }},{\boldsymbol {\alpha }})}{p(\mathbf {E} \mid {\boldsymbol {\alpha }})}}\cdot p({\boldsymbol {\theta }}\mid {\boldsymbol {\alpha }})\\&={\frac {p(\mathbf {E} \mid {\boldsymbol {\theta }},{\boldsymbol {\alpha }})}{\int p(\mathbf {E} \mid {\boldsymbol {\theta }},{\boldsymbol {\alpha }})p({\boldsymbol {\theta }}\mid {\boldsymbol {\alpha }})\,d{\boldsymbol {\theta }}}}\cdot p({\boldsymbol {\theta }}\mid {\boldsymbol {\alpha }}),\end{aligned}}}

where

{\displaystyle p(\mathbf {E} \mid {\boldsymbol {\theta }},{\boldsymbol {\alpha }})=\prod _{k}p(e_{k}\mid {\boldsymbol {\theta }}).}

x , a piece of general info.

It's possible that this is a vector of values.

\theta , the distributional parameter of the data point, i.e, x\sim p(x\mid \theta ) .

This could be a parameter vector.

\alpha , the parameter distribution's hyperparameter, i.e, \theta \sim p(\theta \mid \alpha ) .

The hyperparameters in this might be a vector.

\mathbf {X} is the sample, a set of n observed data points, i.e, x_{1},\ldots ,x_{n} .

{\tilde {x}} , a brand-new data point with a forecasted distribution.

The distribution of the parameter(s) before any data are observed is known as the prior distribution, i.e.

p(\theta \mid \alpha ) .

There may be difficulties in determining the prior distribution; In this scenario, One option might be to obtain a previous distribution using the Jeffreys prior method before updating it with subsequent observations.

The observed data's distribution, conditional on its parameters, is known as the sampling distribution, i.e.

p(\mathbf {X} \mid \theta ) .

Additionally known as the likelihood, in particular when seen as a function of the parameter (s), sometimes written \operatorname {L} (\theta \mid \mathbf {X} )=p(\mathbf {X} \mid \theta ) .

The distribution of the observed data minimized over the parameter(s), also known as the marginal likelihood or evidence, is.

{\displaystyle p(\mathbf {X} \mid \alpha )=\int p(\mathbf {X} \mid \theta )p(\theta \mid \alpha )d\theta .}

It measures the degree of agreement between facts and expert judgment in a precise geometric meaning.

The distribution of the parameter(s) after accounting for the observed data is known as the posterior distribution. Bayes' rule, the foundation of Bayesian inference, determines this:

{\displaystyle p(\theta \mid \mathbf {X} ,\alpha )={\frac {p(\theta ,\mathbf {X} ,\alpha )}{p(\mathbf {X} ,\alpha )}}={\frac {p(\mathbf {X} \mid \theta ,\alpha )p(\theta ,\alpha )}{p(\mathbf {X} \mid \alpha )p(\alpha )}}={\frac {p(\mathbf {X} \mid \theta ,\alpha )p(\theta \mid \alpha )}{p(\mathbf {X} \mid \alpha )}}\propto p(\mathbf {X} \mid \theta ,\alpha )p(\theta \mid \alpha ).}

The phrase posterior is proportional to likelihood times prior or posterior = likelihood times previous, over evidence are used to convey this.

In practice, practically all sophisticated Bayesian models utilized in machine learning, the posterior distribution {\displaystyle p(\theta \mid \mathbf {X} ,\alpha )} is not obtained in a closed form distribution, mainly because the parameter space for \theta can be very high, or the Bayesian model retains certain hierarchical structure formulated from the observations \mathbf {X} and parameter \theta .

In these circumstances, We have to use approximation methods.

A fresh data point's distribution that has been marginalized across the posterior is known as