SVM

Table of Contents

Bayesian Network

Chapter 7 Bayesian Algorithms

7.1. Overview

This chapter introduces Bayesian methods, which are also called Bayesian algorithms, Bayesian machine learning, and probabilistic machine learning in the literature. Because such Bayesian methods were initially proposed for and closer to classification tasks by nature, the use of the 'Bayesian classifier' is also very common or even predominant in some technical publications. However, it is worthwhile to mention that such Bayesian classifiers, which belong to parametric machine learning methods, can also be extended to regression tasks. Other nonparametric Bayesian methods like Bayesian process are proposed more for regression and may appear much different from traditional Bayesian classifiers.

In this chapter, we will first provide a general background for statistics-based machine learning, which contains statistical inference adopted by both frequentists and Bayesians. The frequentists' inference method, i.e., maximum likelihood estimation, is used by many other machine learning methods like artificial neural networks, while Bayesians' method, i.e., Bayesian estimation, is adopted as the basis of Bayesian methods in this chapter. Then, major parametric Bayesian methods, e.g., Naive Bayes classifier, Bayesian networks, and Markov processes, will be discussed. Next, one nonparametric Bayesian method, i.e., Gaussian Process, will be introduced.

This chapter differs from the other chapters in a few ways, which will need close attention to avoid possible confusion. First, the diverse, incomplete, and conflicting statistics contents in different machine learning literature, especially Bayesianism-related topics, can cause severe learning difficulties. We will start the chapter with an introduction to extra statistics knowledge considering the overlap between machine learning and applied statistics. You can skip this section if you just want to know how specific Bayesian methods work. Second, a few other criteria for classifying machine learning methods like discriminative vs generative and parametric vs. nonparametric can also be hard to understand. These will also be explained when appropriate as we move through the technical content.

7.2. Statistics Background for Machine Learning

7.2.1. Statistics and Machine Learning

As can be found in the Appendix for statistics, the role and significance of statistics in the general context of machine learning can vary depending on the background, goal, and focus of machine learning practitioners. Despite this fact, we need to be aware that, for some machine learning methods, especially the Bayesian methods in this chapter, a strong background in statistics is required in order to better understand and utilize such methods. Due to this consideration, statistics knowledge is presented in this chapter in addition to what is provided in the appendices. As a result, this section will offer a big picture for the role of statistics in machine learning, which is both needed in this chapter and desired or helpful for other machine learning topics.

7.2.2. Frequentists and Bayesians

In statistics, there are two mainstreams: frequentism and Bayesianism. Frequentists build the probability from past events. In this way, the probability does not depend on our beliefs, but instead, objective measurements of what have occurred. For example, if we toss a coin 100 times and heads show up 38 times, then the frequency/probability of heads is

38 / 100 = 0.38

. By contrast, Bayesians view probability as a measure of belief, and thus, the probability is subjective,
and refers more to the future. Due to this reason, the workflow of Bayesians starts with a belief, or called a prior, then obtains some data to update our belief for an outcome called a posterior. As more data is obtained, the posterior becomes a new prior, and this process can be repeated to improve the model with more data.

When using statistics to address problems, such as classification and regression problems in machine learning, we essentially try to understand how different samples are distributed in the sample space, which determines their discrete (classification) or continuous (regression) labels/targets. We can understand the learning (or narrowly, training) process as an effort to search for the best statistical model to describe the data, so that such a model can be used for future predictions. Such a model can be defined with parameters represented by

\vec{θ} = [θ_{1}, θ_{2}, \dots]

. So machine learning for both classification and regression essentially aims to infer the parameters

\vec{θ}

for defining the model.

The difference between the two ways of treating probability also leads to the reality that frequentists mostly use the Maximum Likelihood Estimation, while Bayesians adopt Bayesian Estimation. In the following, these two statistical inference methods will be introduced first. The former may be brought up again when introducing other machine learning methods. The introduction to the latter will recall and further elaborate on some statistical concepts introduced in the Appendix for statistics as well as other basic statistics literature.

Here, let us first recall Bayes' theorem, which will be used in different ways in this chapter.

\begin{matrix} (7.1) & P (A ∣ B) = \frac{P (B ∣ A) P (A)}{P (B)}, if P (B) \neq 0 \end{matrix}

where,

A

and

B

are events and

P (B) \neq 0

P (A ∣ B)

is a conditional probability: the probability of event

A

occurring given that

B

is true. It is also called the posterior probability of

A

given

B

P (B ∣ A)

is also a conditional probability: the probability of event

B

occurring given that

A

is true. It can also be interpreted as the likelihood of

A

given a fixed

B

P (A)

and

P (B)

are the prior probabilities (also called marginal probabilities) of observing

A

and

B

, respectively.
If we substitute the law of total probability into the Bayes' theorem, we can obtain the format of the Bayes' theorem that we typically use in Bayesian classifiers.

\begin{matrix} (7.2) & P (A ∣ B) = \frac{\sum_{n = 1}^{N} P (B_{n} ∣ A) P (A)}{P (B)} \end{matrix}

where

{B_{n} : n = 1, 2, 3, \dots, N}

is a finite or countably infinite partition of a sample space (in other words, a set of pairwise disjoint events whose union is the entire sample space), in which each event

B_{n}

is measurable, and

A

is an event in the same probability space.

7.2.3. Overview of Statistical Inference

Statistical inference aims to learn characteristics of a population, i.e., the underlying probability distribution of all the possible data, from observed data sampled from the population [59]. The goal of statistical inference or development of statistical models in machine learning, no matter with Maximum Likelihood Estimation (MLE), Maximum A Posteriori Estimation (MAP), or Bayesian inference, is to infer the optimal model parameterized by

(\vec{θ})

from data

(\bar{X})

The Bayes' theorem can be reformulated as follows for statistical inference.

\begin{matrix} (7.3) & P (\vec{θ} ∣ \bar{X}) = \frac{P (\bar{X} ∣ \vec{θ}) \cdot P (\vec{θ})}{P (\bar{X})} \end{matrix}

P (\bar{X})

is something we generally cannot compute. However, the value of

P (\bar{X})

does not matter that much because it is just a normalization constant. When comparing models, we are mainly interested in expressions containing

\vec{θ}

, because

P (\bar{X})

stays the same for each model.

To explain MLE, MAP, and Bayesian using the above equation, it is better to first recall the fact that an algorithm is usually selected prior to an estimation. This algorithm defines a hypothesis space as explained in previous chapters. In other words, this algorithm provides a collection (a space) of possible models (hypothesis). Next, let us use the following analogy to easily understand the thing we are trying to do. What we have is some data (

\bar{X}

) generated by one or multiple
machines (analogous to models). Our aim is to look for such a machine(s), related to or defined by

\vec{θ}

, so that we can best reproduce the same data. In this analogy, selecting an algorithm is analogous to finding a manufacturer who has a collection of such machines.

MLE aims to maximize the likelihood as follows:

\begin{matrix} (7.4) & {\vec{θ}}^{*} = \arg max_{\vec{θ}} P (\bar{X} ∣ \vec{θ}) \end{matrix}

Thus, MLE tries to identify one model parameterized by

\vec{θ}

that mostly likely generates

\bar{X}

. That is, the probability of generating

\bar{X}

under the condition of

\vec{θ}

is maximized in MLE.

MAP differs from MLE in that MAP includes prior knowledge. In the same analogy, we can understand this as that different machines have different probabilities of delivering the claimed performance.

P (\vec{θ})

can be viewed as prior knowledge provided by the manufacturer about such probabilities. If we include this term in the estimation, we can obtain the following equation based on the MLE equation (Eq. 7.4) and the above Bayes's equation (Eq. 7.3):

\begin{matrix} (7.5) & {\vec{θ}}^{*} = \arg max_{\vec{θ}} P (\bar{X} ∣ \vec{θ}) P (\vec{θ}) = \arg max_{\vec{θ}} P (\vec{θ} ∣ \bar{X}) \end{matrix}

As can be seen, the major difference between MLE and MAP is whether to include prior knowledge

P (\vec{θ})

. In a more strict description,

P (\vec{θ})

is (prior knowledge about) the probability of the model parameterized by

\vec{θ}

in the hypothesis space. This difference actually explains why MLE is more susceptible to overfitting than MAP on datasets with outliers. This is because MLE is more inclined to fit to outliers without prior knowledge for excluding such outliers. Two simple algorithms that represent MLE and MAP are logistic regression with and without L2 regularization. Without regularization, logistic regression uses MLE and would contain all data points including outliers. With regularization, logistic regression can filter out outliers that do not comply with the regularization term.

Bayesian inference is much different because it searches for a possible combination of different models, or ensemble, instead of one model. Therefore, instead of the optimal model

{\vec{θ}}^{*}

, Bayesian methods aim to obtain the possibilities of different models

P (\vec{θ} ∣ \bar{X})

, or more accurately,

P ({\vec{θ}}_{t} \bar{X})

. The number of models can be large or even infinite. In the latter case, we can perform a certain number of sampling, e.g.,

T (t \in {1, \dots, T})

. Then, the prediction made by the overall model (or ensemble) is the average weighted by

P ({\vec{θ}}_{t} \bar{X})

Compared with MLE and MAP, Bayesian methods have the following advantages.

Bayesian methods perform better on small datasets. This is because, as mentioned, they are by nature ensemble learning. Due to the same reason, they are less susceptible to overfitting.
It is more convenient to incorporate prior knowledge into the Bayesian estimate. Therefore, extra constraints or assumptions about data or model distributions can be easily integrated.
Bayesian methods can better handle uncertainty. MLE and MAP only obtain $\vec{θ}$ via learning, e.g., weights of linear models and ANNs. Such deterministic predictions may not be adequate or reasonable in many conditions. For example, a classification model of this type outputs labels corresponding the classes with the highest probabilities. However, no information is available about the confidence level of such a prediction. Bayesian can provide such confidence information, which can prompt us to intervene when the confidence level is too low.

In the following, we will first provide more information about the use of MLE. After that, the general Bayesian estimation will be introduced, followed by detailed parametric and nonparametric Bayesian Methods.

7.2.4. Maximum Likelihood Estimation (MLE)

Suppose we have a training dataset

\bar{X}

, which has

I

independent and identically distributed (called i.i.d. in many places) samples. Let us assume the distribution of the samples can be described using a statistical model characterized by

\vec{θ}

. The basic idea of MLE in frequentism is that

\vec{θ}

is unknown but exists objectively, and

\vec{θ}

as a constant(s) can be estimated by ensuring that a given dataset (for training) has the maximum likelihood to appear (or be observed). Following this understanding, we can obtain the mathematical formulation of MLE:

\begin{matrix} (7.6) & θ_{M L E} = \arg max_{\vec{θ}} O (\bar{X} ∣ \vec{θ}) = \arg min_{\vec{θ}} ℓ (\bar{X} ∣ \vec{θ}) \end{matrix}

where

P ({\vec{x}}_{i} ∣ \vec{θ})

is the probability that the occurrence of sample

{\vec{x}}_{i}

is predicted by the model parameterized by

\vec{θ}

The MLE function

O (\bar{X} ∣ \vec{θ})

is the objective function constructed as the total probability of all the samples. The loss function

ℓ

is the opposite of the objective function

O

. These equations can be constructed as follows if we assume the attributes are independent of each other:

\begin{matrix} (7.7) & O (\bar{X} ∣ \vec{θ}) = ℓ (\bar{X} ∣ \vec{θ}) = P (\bar{X} ∣ \vec{θ}) = P ({\vec{x}}_{1}, \dots, {\vec{x}}_{I} ∣ \vec{θ}) = P ({\vec{x}}_{1} ∣ \vec{θ}) \dots P ({\vec{x}}_{i} ∣ \vec{θ}) \dots P ({\vec{x}}_{I} ∣ \vec{θ}) = \prod_{i = 1}^{I} P ({\vec{x}}_{i} ∣ \vec{θ}) \end{matrix}

In a general multiclass classification problem, what we try to maximize is the expectation of the total probability. When different samples and different attributes are not equally weighted, the objective is calculated using

O (\vec{x} ∣ \vec{θ}) =

\sum_{j = 1}^{J} λ_{i, j} P (x_{i j} ∣ \vec{θ})

, in which

λ_{i, j}

is used to represent the weights. Accordingly, we obtain

\begin{matrix} (7.8) & θ_{M L E} = \arg max_{\vec{θ}} O (\bar{X} ∣ \vec{θ}) = \arg max_{\vec{θ}} [\sum_{j = 1}^{J} λ_{i, j} P (x_{i j} ∣ \vec{θ})] \end{matrix}

In practice, the value of

P (x_{i} ∣ \vec{θ})

can be very small because

J

can be large in a large dataset. The product of such small values may cause numerical issues. To allow for such problems, we usually use the log of the above function as the loss function, i.e.,

ℓ \leftarrow \ln (ℓ)

, for optimization. Then, the mathematical formulation of MLE excluding weights

λ_{i, j}

becomes

\begin{matrix} (7.9) & ℓ = \sum_{i = 1}^{I} \ln [- P ({\vec{x}}_{i} ∣ \vec{θ})] \\ (7.10) & θ_{M L E} = \arg max_{\vec{θ}} \ln [P (\bar{X} ∣ \vec{θ})] \end{matrix}

Let us employ a simple example to understand MLE. Here, we have a bag containing black and white balls, for which we do not know the ratio of black balls to white balls. We took a ball from the bag and put it back. We performed 100 ball-taking operations, and we got 75 white balls in total. What is the proportion of white balls?

MLE can be used to estimate the above parameters

\vec{θ}

, that is, the proportion of white balls

P

in this example.

\begin{matrix} (7.11) & O (\bar{X} ∣ \vec{θ}) = P ({\vec{x}}_{1} ∣ \vec{θ}) \cdot P ({\vec{x}}_{2} ∣ \vec{θ}) \dots P ({\vec{x}}_{100} ∣ \vec{θ}) = P^{75} \cdot (1 - P)^{25} \end{matrix}

Then the natural log of the MLE likelihood function is

\begin{matrix} (7.12) & \ln P (\bar{X} ∣ \vec{θ}) = - 75 \cdot \ln P - 25 \cdot \ln (1 - P) \end{matrix}

The maximum value appears when the derivative of the above function is 0 :

\begin{matrix} (7.13) & \frac{d (\ln P (\vec{x} ∣ \vec{θ}))}{d P} = - \frac{75}{P} + \frac{25}{1 - P} = 0 \end{matrix}

Hence, we can get

P = 0.75

.
This method is relatively simple. However, its validity lies in a strict assumption that the dataset can represent the sample space. In other words, the distribution of the training samples needs to be the same or close to the distribution in the sample space.

7.2.5. Bayesian Estimation

As mentioned, Bayesian estimation is different from MLE in that it assumes the models or model parameters, i.e.,

\vec{θ}

, have their probability distribution and need to be estimated for given data. Therefore, the estimation of the model parameters can differ in different training data. Due to the same reason, the model estimation can also be updated as more data is obtained.

In the following, we can use classification tasks as an example to illustrate how Bayesian estimation models for classification, or Bayesian classifiers, are built up.

Let us consider a multiclass classification problem, in which we need to classify a dataset

\bar{X}

consisting of

I

samples

{\vec{x}}_{1}, {\vec{x}}_{2}, \dots, {\vec{x}}_{i}, \dots, {\vec{x}}_{I}

and each sample has

J

attributes

x_{i 1}, x_{i 2}, \dots, x_{i j}, \dots, x_{i J}

. We will need to classify the samples
into

K

categories:

y_{i} \in {c_{1}, c_{2}, \dots, c_{k}}

. The goal is to predict the probability of any sample

{\vec{x}}_{i}

belonging to the

k

th category:

P (c_{k} ∣ {\vec{x}}_{i})

Next, we slightly modified the above Bayes' theorem to illustrate its use in Bayesian estimation for classification purposes. We will replace

\vec{θ}

with

c_{k}

, which represents "belonging to class

c_{k} "

", and replace

\bar{X}

with sample "

{\vec{x}}_{i j}

". Thus, we are now considering individual models and samples instead of all the models (ensemble) and all the samples. Also, to maintain a consistent understanding with the previous subsection, we can understand

c_{k}

as a model that solely generates or predicts Class

k

samples.

\begin{matrix} (7.14) & P (c_{k} ∣ {\vec{x}}_{i}) = \frac{P ({\vec{x}}_{i}, c_{k})}{P ({\vec{x}}_{i})} = \frac{P ({\vec{x}}_{i} ∣ c_{k}) P (c_{k})}{P ({\vec{x}}_{i})} = \frac{P (x_{i 1}, x_{i 2}, \dots, x_{i J} ∣ c_{k}) P (c_{k})}{P ({\vec{x}}_{i})} \end{matrix}

where the joint probability

P (x_{i 1}, x_{i 2}, \dots, x_{i J} ∣ c_{k})

is the probability that all the attribute values of sample

{\vec{x}}_{i}

appear simultaneously. That is, each attribute of Sample

{\vec{x}}_{i}

has multiple attribute values, and the value taken by Sample

{\vec{x}}_{i}

will have a probability. Such probabilities, as well as the joint probability, can be estimated using the training data.

The above formulation of Bayes' theorem casts a theoretical basis for all the Bayes classifiers. In fact, in a more general sense, it presents a framework for understanding all the classification methods or even all the supervised learning methods. Based on the way of using the above equation, algorithms can be classified into discriminative models and generative models.

Discriminative models assume a function (implicit or explicit) for

P (c_{k} ∣ {\vec{x}}_{i})

directly. A majority of the classic machine learning algorithms like linear models, support vector machines, traditional ANNs, and KNNs are discriminative models. By contrast, generative models assume some function forms for

P ({\vec{x}}_{i}, c_{k})

and

P ({\vec{x}}_{i})

- estimating the parameters of

P ({\vec{x}}_{i}, c_{k})

and

P ({\vec{x}}_{i})

directly from training data - to calculate

P (c_{k} ∣ {\vec{x}}_{i})

indirectly. Bayesian methods like Naive Bayes and Bayesian networks, as well as less common approaches like Markov random fields and Hidden Markov Models ( HMM ), produce generative models.

Intuitively, we can see that discriminative models are more straightforward and thus can be simpler than generative models due to the above fact. However, generative models may provide better insights into the data. Using classification tasks as an example, we can simply understand that discriminative models identify boundaries in the data space, whereas generative models attempt to model how data is distributed throughout the space.

7.3. Parametric Bayesian Methods

7.3.1. Naive Bayes Classifier

One major difficulty in applying the above framework (Eq.7.14) for tasks like classification is that the joint probability, i.e.,

P (x_{i 1}, x_{i 2}, \dots, x_{i J} ∣ c_{k})

, is not easy to deal with. This is because it is hard to tell the relationship between attributes or their corresponding random variables. In particular, some of these variables may be mutually dependent. In that case, the calculation of

P (x_{i 1}, x_{i 2}, \dots, x_{i J} ∣ c_{k})

will be really challenging because it requires us to identify such inter-dependencies first.

The Naive Bayes classifier simplifies the problem by assuming that all the attributes (the corresponding random variables) are independent of each other. As a result, the joint probability becomes the product of the probabilities of all the attributes:

P (x_{i 1}, x_{i 2}, \dots, x_{i J} ∣ c_{k}) = \prod_{j = 1}^{J} P (x_{i j} ∣ c_{k})

. Substituting this into Eq. 7.14 , we can obtain the major equation for the Naive Bayes classifier:

\begin{matrix} (7.15) & P (c_{k} ∣ {\vec{x}}_{i}) = \frac{\prod_{j = 1}^{J} P (x_{i j}, c_{k})}{P ({\vec{x}}_{i})} = \frac{\prod_{j = 1}^{J} P (x_{i j} ∣ c_{k}) P (c_{k})}{P ({\vec{x}}_{i})} \end{matrix}

where

P (x_{i j} ∣ c_{k})

is the probability of the attribute value of the

j

th attribute of Sample

{\vec{x}}_{i}

in all the samples in Class

c_{k}

. Thus,

x_{i j}

only provides a specific attribute value out of all possible

V

values for this attribute, i.e.,

a_{v}

, in which

v \in {1, 2, \dots, V}

It is noted that, in some literature, the subscripts

i

and

k

are omitted to simplify the use of the Bayes' theorem in Bayesian estimation for multiclass classification problems. Accordingly,

\vec{x}

is a vector representing any sample, and

c

is any class. This equation is presented in the following but will not be adopted to ensure a more accurate in the description.

\begin{matrix} (7.16) & P (c ∣ \vec{x}) = \frac{\prod_{j = 1}^{J} P (x_{j} ∣ c) P (c)}{P (\vec{x})} = \frac{P (c)}{P (\vec{x})} \prod_{j = 1}^{J} P (x_{j} ∣ c) \end{matrix}

The use of Eq. 7.15 requires us to find a way of calculating

P (x_{i j} ∣ c)

and

P (c_{k})

. For

P (c_{k})

, it can be easily computed as the percentage of Class

c_{k}

samples in all the samples. Thus, the following equation can be used:

\begin{matrix} (7.17) & P (c_{k}) = \frac{| D_{c_{k}} |}{| D |} \end{matrix}

where

| D_{c_{k}} |

and

| D |

are the numbers of the samples in Class

c_{k}

and in the whole dataset, respectively.
The calculation of

P (x_{i j} ∣ c_{k})

will be much different for discrete (non-numeric) and continuous (numeric) attribute values. When we have a finite number of discrete values for an attribute, then

P (x_{i j} ∣ c_{k})

can be calculated as the ratio of samples with the attribute value

x_{i j}

, i.e.,

D_{c_{k}, x_{i j}}

, to the number of all the samples in this class,

D_{c_{k}}

\begin{matrix} (7.18) & P (x_{i j} ∣ c_{k}) = \frac{| D_{c_{k}, x_{j}} |}{| D_{c_{k}} |} \end{matrix}

where

| D_{c_{k}, x_{j}} |

is the number of samples with attribute value

j

in Class

c_{k}

. Thus, it is actually not related to Sample

{\vec{x}}_{i}

.
When we have continuous attribute values such as float numbers, the probability

P (x_{i j} ∣ c_{k})

can be computed by assuming that the attribute (random variable) follows a probability distribution. A predominant assumption is the Gaussian distribution:

P ({\vec{x}}_{i j} ∣ c_{k}) \sim N (μ_{k, j}, σ_{k, j}^{2})

. Based on this assumption,

P (x_{i j} ∣ c_{k})

can be computed as

\begin{matrix} (7.19) & P (x_{i j} ∣ c_{k}) = \frac{1}{\sqrt{2 π} σ_{k, j}} \exp [- \frac{{(x_{i j} - μ_{k, j})}^{2}}{2 σ_{k, j}^{2}}] \end{matrix}

The above equation can be used to calculate the probability of any attribute of any sample. Parameters about the variable distribution, i.e.,

μ_{k, j}

and

σ_{k, j}

, can be obtained from the training data. For example, we can find all the samples in Class

c_{k}

and assess the distribution of the values of attribute

j

to compute the mean and the standard deviation for the probability density function:

μ_{k, j}

and

σ_{k, j}

When all the attributes have continuous values, we will need to use a Gaussian distribution for each of them. The Naive Bayes classifier, in this case, is also called the Gaussian Naive Bayes classifier in some places.

The implementation of the Naive Bayes classifier is very straightforward. What can be confusing is that the training and testing processes are lumped together in some way. Let us assume that we want to predict the classification of a sample

{\vec{x}}_{i}

based on a given labeled training dataset

\bar{X}

. Then we will just need to calculate

P (c_{k})

and

P (x_{i j} ∣ c_{k})

with the training data using Eq. 7.17 , Eq. 7.18 , and Eq. 7.19. In particular,

P (x_{i j} ∣ c_{k})

can be calculated directly for discrete attribute values, while for continuous attribute values, the computation of

μ

and

σ

needs to be carried out with the training data before the computation of the probability. Next, we just need to use Eq. 7.15 for predicting the labels of new samples.

As can be seen, Naive Bayes classifiers need us to assess the distribution of the data. Thus, to ensure a smooth implementation, we need enough data to compute the probabilities. For those probabilities calculated using percentages, we will need to have samples in the numerators and denominators or the ratios. Usually, we modify the equations for calculating

P (c_{k})

and

P (x_{i j} ∣ c_{k})

for discrete attribute values. The following Laplace correction is commonly adopted to address this issue, which can help smoothen the predicted probability distributions.

\begin{aligned} (7.20) & P (c_{k}) & = \frac{| D_{c_{k}} | + 1}{| D | + K} \\ (7.21) & P (x_{i j} ∣ c_{k}) & = \frac{| D_{c_{k}, x_{j}} | + 1}{| D_{c_{k}} | + N_{j}} \end{aligned}

where

K

is the number of classes in the training dataset and

N_{j}

is the total number of attribute values in Attribute

j

.
For probabilities calculated using distribution functions, i.e.,

P (x_{i j} ∣ c_{k})

for discrete attribute values, we will need to make sure we have enough training samples to obtain a distribution function. This can help avoid the unintended removal of information about attributes due to inadequate or missing attribute values.

The following code shows how to train a regression model using the Gaussian Naive Bayes algorithm explained in this subsection. The Iris dataset from Scikit-learn was adopted for training. The performance of the trained model was compared with that obtained using the Gaussian Naive Bayes from Scikit-learn.

from sklearn.datasets import load_iris
              from sklearn.model_selection import train_test_split
              import numpy as np
              from sklearn.naive_bayes import GaussianNB

# ------------------- Load data ---------------------- #
              x, y = load_iris(return_x_y=True)
              x_train, x_test, y_train, y_test = train_test_split(x, y, random_state=20190308, test_size=0.3)
              # Calculate the probabilities of every attribute value P(x_j|c) for P(x|c)
              def gaussion_pdf(x_test, x):
                return np.exp(-(x_test-x.mean(0))**2 / (2 * x.std(0)**2)) / np.sqrt(2 * np.pi * x.std(0)**2)
              # -------------- Define Naive Bayes Model --------------- #
              classes = np.unique(np.concatenate([y_train, y_test], 0))
              pred_probs = []
              # Loop over all classes to obtain the probabilities of all testing samples belonging to different
                classes
              for i in classes:
                idx_i = y_train == i # idx_i stores all indices of samples in Class c (y_train == i)
                p_c = len(idx_i) / len(y_train) # Calculate P(c)
                p_x_c = np.prod(gaussion_pdf(x_test, x_train[idx_i]), 1) # Use Gaussian to calculate P(x|c) for
                continuous attribute values
                prob_i = p_c * p_x_c # Joint probability prob_i is the probability of sample i belonging to
                Class c
                pred_probs.append(prob_i)
              # Array for probabilities of samples belonging to different classes: N_samples x N_classes
              pred_probs = np.vstack(pred_probs).T
              # Index of the class with the highest probability as the classification for each sample
              label_idx = pred_probs.argmax(1)
              y_pred = classes[label_idx] # Select the class based on the index
              score = np.count_nonzero(y_test == y_pred)/len(y_test) # Accuracy: correct when the predicted class
                is the same as the actual label
              print('The accuracy of the self-developed Gaussian Naive Bayes is: ',score)
              # ------------------ Apply Gaussian Naive Bayes from Scikit-learn --------------------- #
              model = GaussianNB()
              model.fit(x_train, y_train)
              print('The accuracy of Gaussian Naive Bayes from Scikit-learn is: ', model.score(x_test, y_test))

The execution of the above code produced the following results:
The accuracy of self-developed Gaussian Naive Bayes is: 0.9333333333333333
The accuracy of Gaussian Naive Bayes from Scikit-learn is: 0.9333333333333333

7.3.2. Semi-Naive Bayesian Classifier

Naive Bayesian classifiers are fairly simple to understand and implement. However, the independence assumption of attributes adopted by Naive Bayesian classifiers may be difficult to hold in many cases. To address this issue, semi-naive Bayesian classifiers are proposed by relaxing the assumption of the conditional independence of attributes to some extent. This "semi-relaxation" of the independence requirement explains the origin of "semi-naive".

Semi-naive Bayesian classifiers still consider the relatively strong attribute dependencies but overlook weak interdependence to reach a compromise between accuracy and convenience. The different ways of selecting dependencies lead to different semi-naive Bayesian classifiers. In the following, common semi-naive Bayesian classifiers, i.e., One-Dependent Estimator (ODE), Tree Augmented Naive Bayes (TAN) [60], and Averaged ODE (AODE) [61], are introduced.

One-Dependent Estimator (ODE)

ODE is one of the most popular categories of semi-naive Bayesian classifiers. ODE is distinct in that it assumes each attribute depends on at most one another attribute:

\begin{matrix} (7.22) & P ({\vec{x}}_{i}, c_{k}) = P (c_{k}) \cdot \prod_{j = 1}^{J} P (x_{i j} ∣ c_{k}, p a_{j}) \end{matrix}

where

p a_{j}

is the attribute that the

j

th attribute of Sample

{\vec{x}}_{i}

, i.e.,

x_{i j}

, depends on. The above equation is formulated for Sample

{\vec{x}}_{i}

, so

i

is carried in the corresponding terms. However, such dependency is not related to the sample being

Chapter 7

Bayesian Algorithms

7.1. Overview

7.2. Statistics Background for Machine Learning

7.2.1. Statistics and Machine Learning

7.2.2. Frequentists and Bayesians

7.2.3. Overview of Statistical Inference

7.2.4. Maximum Likelihood Estimation (MLE)

7.2.5. Bayesian Estimation

7.3. Parametric Bayesian Methods

7.3.1. Naive Bayes Classifier

7.3.2. Semi-Naive Bayesian Classifier

One-Dependent Estimator (ODE)

Sample Code from AI Engineering

Linear Models