This chapter presents essential knowledge about the Artificial Neural Network (ANN), or more specifically, shallow ANNs. We will first check the structure of biological neurons to understand how ANN can mathematically imitate biological NN's behavior. Next, the mathematical formulation of the well-known M-P neuron model and other significant concepts, such as activation function, will be discussed. Then, we will see how perceptron, as one of the simplest and most typical NNs, works from a mathematical perspective. In particular, the training of such a network via backpropagation will be detailed. The yielded equations will be generalized for NNs with more than three layers to guide implementations. Based on that, extra skills about NN implementations, the illustration of the implementation procedure with an example, and the transformation of data as it moves through a network will be discussed in detail. Finally, a few other shallow ANN issues will be briefly mentioned.
8.2. Basics of Artificial Neural Networks
8.2.1. From Biological Neural Network to ANN
The Artificial Neural Network (ANN) was inspired by the way that biological neural networks work [66]. It conceptualizes such working mechanisms and then mimics the functions with mathematical formulations [67]. Thus, prior to an introduction to ANN, let us first take a look at the structure of a biological neural network, especially how different units, i.e., neurons and their components, work with each other so that information can be processed.
As one of the two basic types of cells, neurons serve the role of transmitting information using electrical impulses and chemical signals. As illustrated in Fig. 8.1, a typical neuron consists of a cell body and two extensions: an axon and several (e.g., 5-7) dendrites. A dendrite has the shape of tree branches and receives information from another cell. An axon has a long-tail shape and transmits information out of the cell. Information is transmitted and processed within the cell as electrical impulses. Neighboring neurons communicate with each other via chemicals across a small contact region called a synapse, which is between an axon and a dendrite of the next neuron.
In biological neural networks, every neuron is connected to other neurons. When the electrical potential of a neuron exceeds a threshold value, it will be activated. Once activated, this neuron transmits chemicals to the connected neurons to change their electrical potential. In 1943, McCulloch and Pitts conceptualized the above model [9]. This model is the well-known M-P model, which is widely adopted for building ANNs in machine learning. Mathematically, the M-P model for a neuron can be described as
where the output yy is formulated as a function of the input x_(j)x_{j} in which jj is one of the JJ input variables (or attributes), w_(j)w_{j} is the corresponding weight, and theta\theta is the threshold for the neuron.
Thus, as shown in Fig. 8.2, a sample or the input (column) array for Neuron kk, which is marked as vec(x)_(i)\vec{x}_{i}, will first be added up according to the corresponding weights: vec(w)_(k)^(T)* vec(x)_(i)=sum_(j=1)^(J)w_(kj)x_(ij)\vec{w}_{k}^{T} \cdot \vec{x}_{i}=\sum_{j=1}^{J} w_{k j} x_{i j}. The weighted sum as the result of the transfer function marked as " sum\sum " in the figure will be treated by the activation function f(*)f(\cdot) to produce the output y_(k)y_{k}.
Figure 8.1: Architecture and working mechanisms of neurons
Figure 8.2: M-P formulation of the working mechanism of neurons
8.2.2. Activation Function
The function ff in Eq. 8.1 is the activation function. The ideal activation function is the step function. But we usually adopt continuous and smooth functions such as the Sigmoid and tanh functions as the activation function. The major purpose of using an activation function is to squash the input, which can span over a wide range of values, into a small range, e.g., 0 to 1 , as the range for output. As a result, an activation function is also called a squashing function. The neural network as a network of connected neurons thus can be viewed as the combination of many functions such as y=f(sum_(j=1)^(J)w_(j)x_(j)-theta)y=f\left(\sum_{j=1}^{J} w_{j} x_{j}-\theta\right).
The Sigmoid function in the following is a classic activation function.
This function converts any value into the range between 0 and 1 . In addition to squeezing the input into the designated range of output, this function has another significant property that promotes its widespread adoption in the backpropagation for training neural networks. That is, the derivative of the Sigmoid function can be conveniently expressed using this function itself.
This function squeezes the input into a range between -1 and 1 . Similar to the Sigmoid function, the derivative of tanh\tanh can also be expressed using the function itself as
Perceptron is one of the simplest and most typical neural networks. It consists of two layers of neurons. One layer is for input, while the other layer consists of functional neurons for output. Perceptron can only learn linear problems due to its simple architecture. Otherwise, oscillation can occur in the learning process. To consider nonlinear problems, multiple layers of function neurons need to be adopted.
8.2.4. Multiple Layer Feedforward Neural Network
A feedforward neural network is an ANN wherein connections between the nodes do not form a cycle. As such, it is different from its descendant: recurrent neural networks. The most common ANNs are multiple-layer feedforward neural networks, which can be obtained by stacking multiple layers of perceptrons. In such a model, every layer of neurons is fully connected with the neurons in the next layer. Neurons from the same layer do not connect with each other. In a typical setting, we have an input layer, one to multiple hidden layers, and an output layer. While the input layer only takes the input (no other math operations), hidden layers and the output layer are composed of functional neurons like the above M-P neuron.
8.3. Training with Backpropagation
8.3.1. Concepts
Backpropagation is the most common learning algorithm for training ANNs. In a simple way, the weights, which determine the behavior of the network, are updated based on the difference between the prediction and true solution (labels). The algorithm thus involves two steps: forward propagation (forward pass) and backward propagation (backward pass or backpropagation). Forward propagation refers to the calculation and storage of the outputs of individual neuron layers, layer by layer from the input layer to the output layer.
Backward propagation, which will be termed backpropagation thereafter for consistency, strictly refers only to the calculation of the gradients and the updates on the weights using the obtained gradients. In each backward pass, the gradients are first obtained from the difference between the predicted and true solution in the output layer and then
propagated backward layer by layer to the first hidden layer. Such gradients or corrections, as the derivatives of the difference between predicted and actual labels with respective to different weights, can be viewed as the contributions of these weights to the errors. Therefore, the weights are updated according to their contributions to the errors. For example, weights or their corresponding neurons that cause an overestimate of the output will be modified so that they will lead to lower predictions in the next forward pass.
The term "backpropagation" is often used loosely to refer to the entire training process or algorithm, including both of the two passes and how the gradient is used, such as by stochastic gradient descent. In this book, to avoid confusion, "forward pass" and "backward pass" will be used for the two specific processes with different directions, while "backpropagation" will be used to refer to the algorithm including both passes and the optimization method (solver).
We can view the threshold theta\theta as a dummy node. However, more frequently, theta\theta is counted as part of the weights, i.e., one extra ww. Then, in each step of the backward pass, the weights w^(')sw^{\prime} s and theta^(')s\theta^{\prime} s will be updated as
As mentioned above, the increment or correction, Deltaw_(j)\Delta w_{j}, is determined with the difference between the predicted and actual values, which is included in the loss function ℓ\ell :
where eta\eta is the learning rate. One backward pass consists of updating all the weights associated with connections between any two neurons in the network. It usually takes place from the last layer of neurons connecting to the output, then propagates backward to the previous layers, and ultimately to the input layer. That is why it is called backpropagation. A learning process consists of multiple rounds of alternate forward and backward processes. The forward process obtains the predictions, and the backward process calculates the error as the difference between predictions and labels, based on which backpropagation is carried out. The whole learning process is, therefore, an optimization process whose goal is to minimize the difference between the predictions and labels.
8.3.2. Backpropagation in a 3-Layer Network
In the following, we will use a three-layer feed forward network as an example to show the detailed process of backpropagation. Equations for calculating the increments in weights will be derived, which are essential to coding a neural network. Illustrated in Fig. 8.3 is a representative neural network consisting of three layers of neurons, in which the bottom is the input layer containing JJ neurons. Therefore, the forward direction in the figure is the upward direction, which converts the input vec(x)\vec{x} (or x_(j)x_{j} ) into the output vec(y)\vec{y} (or y_(k)y_{k} ). Except for the input layer, all the layers have input and output together with weights and their associated math operations. Taking the output layer, for example, the input is beta\beta and the output is yy. The weights include ww 's and theta\theta 's between the hidden and output layer, and their associated math operations are the linear combination and activation. In this network, any neuron in the hidden layer, l in{1,cdots,L}l \in\{1, \cdots, L\}, is connected to all the neurons in the previous layer (J)(J) including Neuron jj and all the neurons in the next layer (K)(K) including Neuron kk. A weight w_(kl)w_{k l} connects Neuron ll in the hidden layer to Neuron kk in the next (output) layer.
As shown in Fig. 8.3, four types of operations are involved in converting the input of the network to its output. Among them, two generates alpha\alpha and beta\beta as linear combinations of of weights:
Thus, alpha\alpha and beta\beta can be viewed as intermediate weights. In fact, alpha\alpha and beta\beta are inputs for the hidden and output layers, respectively. The other two types of operations convert the input of each layer into the output of that layer. For the hidden layer, that is
Next, let us differentiate the predictions of the networks, tilde(vec(y))\tilde{\vec{y}}, from the labels, vec(y)\vec{y}. There are KK output nodes, thus both tilde(vec(y))\tilde{\vec{y}} and vec(y)\vec{y} should have KK elements. For example, tilde(vec(y))\tilde{\vec{y}} should be [ tilde(y)_(1), tilde(y)_(2),cdots, tilde(y)_(k),cdots, tilde(y)_(K)]\left[\tilde{y}_{1}, \tilde{y}_{2}, \cdots, \tilde{y}_{k}, \cdots, \tilde{y}_{K}\right]. Each of the elements can also be arrays with one or more dimensions. After each forward pass, tilde(vec(y))\tilde{\vec{y}} is calculated. Then, the error can be calculated with the given labels. Most commonly, the mean square root is used for constructing the error or loss:
Then, the backward pass can start from the above loss/error function. As introduced above, the purpose is to minimize the above error using gradient descent. There are four types of weights in the above neural network. Therefore, we need to calculate the four types of gradients in each iteration: Delta w,Delta theta,Delta v\Delta w, \Delta \theta, \Delta v, and, Delta gamma\Delta \gamma. Let us see how to derive equations for calculating each of them using primarily the chain rule. First, Delta w\Delta w is the derivative of the error function ℓ\ell with respect to ww :
Please note that the above deduction assumed the use of the Sigmoid activation function, whose derivative is f^(')=f*(1-f)f^{\prime}=f \cdot(1-f) (Eq. 8.3).
We can use a new symbol as follows to simplify the formulation:
When switching to a different activation function, e.g., tanh, we just need to replace all the [ tilde(y)_(k)(1- tilde(y)_(k))]\left[\tilde{y}_{k}\left(1-\tilde{y}_{k}\right)\right] with (1- tilde(y)_(k)^(2))\left(1-\tilde{y}_{k}^{2}\right) and replace all the [b_(l)(1-b_(l))]\left[b_{l}\left(1-b_{l}\right)\right] with (1-b_(l)^(2))\left(1-b_{l}^{2}\right).
8.3.3. Backpropagation in Neural Networks with 3+ Layers
The expressions for the weights in the above 3-layer network can be easily extended to obtain expressions for the gradients of weights in a neural network consisting of more layers, e.g., N > 3N>3. To simplify the deduction, we will adopt the tensor notation rather than the index notation. We will also need to start from the weights connecting to the last layer of neurons. As shown in Fig. 8.4, we will get the following equation similar to deduction for the 3-layer network:
where ox\otimes is the external (or tensor) product. In the latest version of NumPy, the operator for the external product is "numpy.tensordot" without any contraction (of axes); while '@' or "numpy.dot" is for the dot/inner product, which contracts the neighboring axes of the two tensors. You can see that we use superscripts to represent the number of the layer. This is much different from the subscripts, which were used in the above equations for node numbers. The subscripts now can be excluded because of the use of the tensor notation. vec(O)^(N-1)\vec{O}^{N-1} is the tensor for the output of the second last layer (i.e., Layer N-1N-1 ), which corresponds to vec(b)\vec{b} in the above 3-layer model. vec(g)\vec{g} is written as follows when Sigmoid is used as the activation function.
where o.\odot is the element-wise product, and II is the tensor of input of the last layer. In NumPy, we simply use the symbol '*' for the element-wise product.
Then, we can move to the second last layer, and the weight tensor bar(w)^(N-1)\bar{w}^{N-1} is expressed as
where vec(g)^(N-1)\vec{g}^{N-1} is used to replace e_(l)e_{l} in the deduction for the 3-layer network. This is because the g_(k)g_{k} and e_(l)e_{l} in the deduction for the 3-layer network have a similar meaning: the derivative of the loss function with respect to the input of the corresponding layer. Therefore, vec(g)^(N-1)\vec{g}^{N-1} is formulated as
If we continue the deduction to the next layer, we will see that the equations for Layer N-2N-2 will be the same as those for Layer N-1N-1. The only difference is the substitution of NN with N-1N-1, and as a result, N-1N-1 becomes N-2N-2. That is to say, any hidden layer nn, i.e., a layer between Layer 2 and Layer N-1N-1 (including these two layers), has the following equations for weight updates.
The above three equations can be used in a loop to update the weights in all the hidden layers.
8.4. Implementation
The theory of ANN is not difficult to understand. However, the implementation of a shallow neural network, especially the implementation of the forward and backward passes could be trivial and susceptible to mistakes even with the details offered in the previous section. What is particularly troublesome is the treatment of the dimensions or sizes of data, in the form of arrays or tensors, as the data moves through the network layer by layer. This can be viewed as a flow of tensors as coined in "TensorFlow".
This section will provide enough information for you to quickly put together your own code for implementing ANNs. For this purpose, we will first get familiar with some skills and concepts that have been widely used for training ANNs. Next, a typical procedure illustrated by pseudo-code will be provided to show the detailed steps for ANN training. Finally, based on one example, more details will be presented about the tensor flow, especially how data changes in shape as it moves through a typical network.
8.4.1. Practical Skills
We will need to understand how data moves through the neural network as well as several critical skills and concepts before we can implement the training of ANNs and understand the relevant literature well. The input, vec(x)\vec{x} is the data without labels while the output, vec(y)\vec{y}, is the corresponding label. Each time, one sample or multiple samples are fed into the
network in the format of an array. This data array moves forward through the network to obtain a prediction that can be compared with the corresponding label. The difference between the predicted and actual labels is propagated backward in terms of gradients to adjust the weights. The new weights are used in the forward pass of the next iteration. Therefore, the model represented by the weights of the network can be updated every forward-backward process. This process is called an iteration of the training process.
The real implementation can be more complicated due to the use many extra skills and terms. The first is about the use of batches, epochs, and related optimizers. If one sample is used for each (gradient descent) iteration, then the training method (optimizer) is called stochastic gradient descent. Here, "stochastic" is used because of the fact that updating the model after processing each sample usually leads to high randomness/stochasticity due to the variability in data points (samples). In this case, each update will take a relatively short time, but lots of iterations may be needed to identify a good direction of optimization. The other extreme is to process the whole training set for each iteration, and updates on the model are made in the backward pass with gradients of all the samples. This is called batch gradient descent.
More frequently, it is not possible or efficient to process the whole dataset within one iteration. Thus, we frequently break the training dataset into multiple "mini batches" of samples. Then, updates on the model are made in each iteration for one mini batch. This is called the mini batch gradient descent, which is also loosely called batch gradient descent in many places. We can calculate the loss or gradient for samples within one mini batch one by one before adding the loss or gradient up for updating the model. Or alternatively, we can treat all samples together as one array to obtain gradients for the model update. The use of mini batches also prompts us to use another concept, "epoch". One epoch means one dataset or the same number of samples as one dataset have been processed once, depending on the sampling method.
The selections of batch size and epoch number (or iteration number) need to consider the available resources and goals and to perform a trial in some cases. As the batch sizes increase, the use of memory and the consequent calculation time for each batch will increase dramatically. Compared with the stochastic and mini-batch methods, batch gradient descent method is more susceptible to the traps of local optima. This is because the randomness associated with small batch sizes adds noise to the training process, which helps the training to get out of local optima. But such randomness may also make it hard to find a good direction within an affordable amount of time.
8.4.2. Procedure for An Example
To simply illustrate the implementation of the neural network, we will start with a shallow neural network. Let us use the Boston House Prices (BHP) dataset from Scikit-learn as the example data. The dataset consists of 506 samples; each sample has 13 numbers (attribute values) as the input data and 1 number as the label/output. The 13 numbers of each sample will be fed as input to the network, processed layer by layer, and finally outputted as the prediction for that sample. The difference between the prediction and the label is the error, which needs to be minimized. To avoid confusion and facilitate coding, we will not count the input layer as one layer, because its input and output are identical. Accordingly, the first hidden layer will be viewed as the first layer. The following is the pseudo-code for implementing the neural network. The BHP dataset and a 3-layer (excluding the input layer) network are used to mark the sizes of arrays in the training process to better present the implementation details.
Simple Neural Network Training
Prepare data, e.g., get bar(X)\bar{X} and vec(y)\vec{y} from the BHP dataset.
Define network architecture, e.g., use a list [4,8,1][4,8,1] to represent a net with 3 layers, which have 4,8 , and 1 neurons. The number of neurons in the last layer is determined by the size of the label for each sample, i.e., 1 for BHP data.
Set up hyperparameters, e.g., eta\eta, and variables to be stored during training, e.g., bar(w), vec(theta), vec(O)\bar{w}, \vec{\theta}, \vec{O}, and loss, etc.
Repeat a certain number of iterations or until a certain rule is satisfied (e.g., loss improvements are smaller than a tolerance).
Initialize the network, e.g., filling the weight arrays of bar(w)\bar{w} and vec(theta)\vec{\theta} with random numbers.
Repeat for every sample in a batch.
Repeat for every layer in a network in the forward pass using equation vec(O)^(n)=f( bar(w)^(n)@ vec(O^(n-1))+ vec(theta^(n)))\vec{O}^{n}=f\left(\bar{w}^{n} @ \overrightarrow{O^{n-1}}+\overrightarrow{\theta^{n}}\right).
Repeat for every layer in a network in the backward pass using equations vec(g)^(n)= vec(g)^(n+1)@ bar(w)^(n+1)*[ vec(O)^(n)*(1- vec(O)^(n))]\vec{g}^{n}=\vec{g}^{n+1} @ \bar{w}^{n+1} \cdot\left[\vec{O}^{n} \cdot\left(1-\vec{O}^{n}\right)\right], Delta bar(w)^(n)=-eta* vec(g)^(n)ox vec(O)^(n-1),Delta vec(theta)^(n)=eta* vec(g)^(n)\Delta \bar{w}^{n}=-\eta \cdot \vec{g}^{n} \otimes \vec{O}^{n-1}, \Delta \vec{\theta}^{n}=\eta \cdot \vec{g}^{n}. Here, the weights of the whole batch are updated but will not be used until the next batch.
Calculate the loss.
The following code shows the training of a 3-layer feedforward NN following the above procedure. Please note that only the key steps in the forward and backward passes are given.
# Main loop for training
while True:
# Forward
Alpha = V @ X_train.T #X_train[i].reshape(y_train[i].size,1)
B = 1/(1 + np.exp(-(Alpha - Gamma)))
Beta = W @ B
Y_hat = 1/(1 + np.exp(-(Beta - Theta)))
y_hat = Y_hat.T
Ek = 0.5*np.sum((y_hat - y_train)**2,axis=1)
# Backward
G = (y_train.T-Y_hat)*Y_hat*(1-Y_hat)
Delta_W = eta * G @ B.T /y_train.shape[0]
Delta_Theta = np.sum(eta*G,axis=1) /y_train.shape[0]
Delta_Theta = Delta_Theta.reshape(Delta_Theta.shape[0],1)
E = -(W.T @ G) * B * (1-B)
Delta_V = eta * E @ X_train /y_train.shape[0]
Delta_Gamma = - eta * np.sum(E,axis=1) /y_train.shape[0]
Delta_Gamma = Delta_Gamma.reshape(Delta_Gamma.shape[0],1) # Change the size from (Node,) to (
Node,1)
# Improve with the learning rate and the overall descent direction
W = W + Delta_W
Theta = Theta + Delta_Theta
V = V + Delta_V
Gamma = Delta_Gamma
error = np.sum(Ek)/np.sum(y_train**2)
Iteration = Iteration + 1
print("Iteration:",Iteration,"Error is", error)
if error < tol:
break
# Prediction
Alpha = V @ X_test.T
B = 1/(1 + np.exp (-(Alpha - Gamma)))
Beta = W @ B
Y_hat_test = 1/(1 + np.exp(-(Beta - Theta)))
y_hat_test = Y_hat_test.T
Ek_test = 0.5*np.sum((y_hat_test - y_test)**2)
y_pred = y_hat_test
print('The loss (score) of training is', Ek_test)
8.4.3. *Shape and Arrangement of Arrays for Data
To better understand what the data, i.e., different arrays for input, weights, thresholds, and output, looks like, we first write out these arrays using the way that we used in the other chapters of the book.
