As introduced in the previous section, PCA is one of the most important dimensionality reduction methods. In fact, it is one of the most popular dimensionality reduction algorithms. Despite the fragmented nature of dimensionality reduction knowledge, PCA is viewed as a must-know by most people. It has wide applications in data compression, redundancy removal, and noise reduction. In the following, the concept and basic idea of PCA will be introduced first. Then, the theoretical basis, especially the deduction of major equations for PCA's implementations will be introduced. Next, the procedure and guidelines for implementing PCA will be presented. Lastly, kernel PCA as a way to extend PCA from a linear dimensionality reduction method into nonlinear ones will be discussed.

From the name of PCA, i.e., primary component(s) analysis, we can guess that the major idea of PCA is to find out the major components of data and represent the data only with these major components for the purpose of dimensionality reduction. In a more strict mathematical description, we attempt to implement dimensionality reduction for data with $J$$J$JJ$J$ dimensions. Because the data is unlabeled, each data point or called instance will have $J$$J$JJ$J$ features (or attributes). Please be aware that the data will have multiple data points, say $I$$I$II$I$, which is the number of instances. This number of data points cannot be confused with the dimensionality of the data, though both can determine the total size of the data. Our goal of dimensionality reduction is to reduce the dimension $J$$J$JJ$J$ rather than the number of instances $I$$I$II$I$. Specificially, our goal is to reduce the dimensionality of the data, $J$$J$JJ$J$, into a small one, $J^{\prime }(J^{\prime }<J)$$J^{\prime }\left(J^{\prime }<J\right)$J^(')(J^(') < J)J^{\prime}\left(J^{\prime}<J\right)$J^{\prime }(J^{\prime }<J)$. This dimensionality reduction is conducted in the hope that the data after dimensionality reduction can still represent the original data. That is, we hope the loss of essential information can be minimized while reducing the data dimensions.

Next, let us use a simple example to illustrate the above concepts and understand what it means by reducing dimensionality and finding major components. In the example shown in Fig. 12.1, we hope to reduce the dimensionality of the data represented by the points. The data is plotted in a two-dimensional space, i.e., have two axes, and accordingly, each data point has two elements (coordinates ${\overrightarrow{e}}_1$${\overrightarrow{e}}_1$vec(e)_(1)\vec{e}_{1}${\overrightarrow{e}}_1$ and ${\overrightarrow{e}}_2$${\overrightarrow{e}}_2$vec(e)_(2)\vec{e}_{2}${\overrightarrow{e}}_2$ ). Obviously, we can and only can reduce the data dimensionality from 2 to 1 . Thus, the purpose of PCA, in this case, is to find a direction or an axis to which the data can be projected. Therefore, a key to PCA is how to find this direction/axis as the new dimension, which can also be understood as the principal component in this PCA example.

Instead of revealing which direction is the best directly, let us see two candidate directions, i.e., ${\overrightarrow{w}}_1$${\overrightarrow{w}}_1$vec(w)_(1)\vec{w}_{1}${\overrightarrow{w}}_1$ and ${\overrightarrow{w}}_2$${\overrightarrow{w}}_2$vec(w)_(2)\vec{w}_{2}${\overrightarrow{w}}_2$, which represent the directions of the two straight lines pass through the center the data points. A comparison of the two directions could give us some hints for identifying better directions for projection. Many of us, in fact, can point out that ${\overrightarrow{w}}_1$${\overrightarrow{w}}_1$vec(w)_(1)\vec{w}_{1}${\overrightarrow{w}}_1$ is a better direction: when data is projected onto this direction or axis, more information of the original data can be retained. But why?

There are two explanations. The first explanation is that the data points are closer to this axis. In a more strict description, we can state that the total distance between all the points and this axis is smaller. The second explanation is that the projected points on ${\overrightarrow{w}}_1$${\overrightarrow{w}}_1$vec(w)_(1)\vec{w}_{1}${\overrightarrow{w}}_1$ are more separated from each other. That is, the variance of the projected data is higher, which can facilitate the differentiation between the projected data. In fact, these two explanations can be generalized as the criteria for dimensionality reduction with PCA. But, in a more general case, we attempt to find $J^{\prime }(1⩽J^{\prime }<J)$$J^{\prime }\left(1⩽J^{\prime }<J\right)$J^(')(1 <= J^(') < J)J^{\prime}\left(1 \leqslant J^{\prime}<J\right)$J^{\prime }(1⩽J^{\prime }<J)$ directions (axes) for projection so that the criteria can be met in the best way. In the later deduction, we will also show that the two criteria are equivalent: we can use either of them to find the same best directions for projections (or understood

Figure 12.1: Main idea of dimensionality reduction
as the best components).

As mentioned above, there are two criteria for selecting the best projection direction(s) or the principal components. Let us first see how to establish the theoretical basis based on the criterion of minimum distance. In order to obtain a general deduction, we will use generalized data and space. The 2D data and space in Fig. 12.1 may still be used to assist us in understanding the process. However, please be aware that the distance and the axis/plane to which the data is projected will be generalized into the Euclidean distance and hyperplane, which are not easy to visualize as those lines and planes in 1D and 2D spaces.

We will start with data consisting of $I$$I$II$I$ data points (instance), and each data point has $J$$J$JJ$J$ dimensions (attributes). Accordingly, we can use a matrix $\overline{X}$$\overline{X}$bar(X)\bar{X}$\overline{X}$ as follows to represent the data

where a data point ${\overrightarrow{x}}_i$${\overrightarrow{x}}_i$vec(x)_(i)\vec{x}_{i}${\overrightarrow{x}}_i$ has $J$$J$JJ$J$ attributes and is formulated using a column array as ${[x_{i1},x_{i2},\cdots ,x_{iJ}]}^T$${\left[x_{i1},x_{i2},\cdots ,x_{iJ}\right]}^T$[x_(i1),x_(i2),cdots,x_(iJ)]^(T)\left[x_{i 1}, x_{i 2}, \cdots, x_{i J}\right]^{T}${[x_{i1},x_{i2},\cdots ,x_{iJ}]}^T$.
It should be noted that PCA takes zero-centered data as the input. In general, data like that in Fig. 12.1 is not zero-centered. If that is the case, we will need to take the following action for zero-centering.

After this preprocessing operation, the result is that the origin of the coordinate system will be moved to the center of the data. In Fig. 12.1, the origin is moved to the data center marked by the intersection of ${\overrightarrow{w}}_1$${\overrightarrow{w}}_1$vec(w)_(1)\vec{w}_{1}${\overrightarrow{w}}_1$ and ${\overrightarrow{w}}_2$${\overrightarrow{w}}_2$vec(w)_(2)\vec{w}_{2}${\overrightarrow{w}}_2$. The following deduction will be performed with the zero-centered data. For such data, the following condition is met:

To generate the low-dimensional space, we first rotate the current coordinate system to obtain a new one ${\overline{W}}_0=$${\overline{W}}_0=$bar(W)_(0)=\bar{W}_{0}=${\overline{W}}_0=$ ${[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_J]}_{(J\times J)}$${\left[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_J\right]}_{(J\times J)}$[ vec(w)_(1), vec(w)_(2),cdots, vec(w)_(J)]_((J xx J))\left[\vec{w}_{1}, \vec{w}_{2}, \cdots, \vec{w}_{J}\right]_{(J \times J)}${[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_J]}_{(J\times J)}$, in which ${\overrightarrow{w}}_j={[w_{1j},w_{2j},\cdots ,w_{Jj}]}^T$${\overrightarrow{w}}_j={\left[w_{1j},w_{2j},\cdots ,w_{Jj}\right]}^T$vec(w)_(j)=[w_(1j),w_(2j),cdots,w_(Jj)]^(T)\vec{w}_{j}=\left[w_{1 j}, w_{2 j}, \cdots, w_{J j}\right]^{T}${\overrightarrow{w}}_j={[w_{1j},w_{2j},\cdots ,w_{Jj}]}^T$. Each vector ${\overrightarrow{w}}_j$${\overrightarrow{w}}_j$vec(w)_(j)\vec{w}_{j}${\overrightarrow{w}}_j$ represents an axis, which can also be understood as a direction or a dimension. ${\overrightarrow{w}}_j$${\overrightarrow{w}}_j$vec(w)_(j)\vec{w}_{j}${\overrightarrow{w}}_j$ has $J$$J$JJ$J$ components, which are the projections of the unit vector along this
axis onto the $J$$J$JJ$J$ axes of the original coordinate system. This new coordinate system also has $J$$J$JJ$J$ dimensions, which are represented by an orthonormal basis consisting of $J$$J$JJ$J$ unit vectors that are orthogonal to each other. Therefore, we should have the following relationships for any pair of vectors from this basis: ${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_j=\Vert {\overrightarrow{w}}_j\Vert =1$${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_j=‖{\overrightarrow{w}}_j‖=1$vec(w)_(j)^(T)* vec(w)_(j)=|| vec(w)_(j)||=1\vec{w}_{j}^{T} \cdot \vec{w}_{j}=\left\|\vec{w}_{j}\right\|=1${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_j=\Vert {\overrightarrow{w}}_j\Vert =1$ and ${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_{j^{\prime }}=0$${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_{j^{\prime }}=0$vec(w)_(j)^(T)* vec(w)_(j^('))=0\vec{w}_{j}^{T} \cdot \vec{w}_{j^{\prime}}=0${\overrightarrow{w}}_j^T\cdot {\overrightarrow{w}}_{j^{\prime }}=0$ when $j\ne j^{\prime }$$j\ne j^{\prime }$j!=j^(')j \neq j^{\prime}$j\ne j^{\prime }$.

However, our purpose is not to obtain this new coordinate system. Instead, we aim to find the best projection directions in a lower-dimensional space. That is, we attempt to reduce the dimensionality from $J$$J$JJ$J$ to $J^{\prime }(J⩽J^{\prime })$$J^{\prime }\left(J⩽J^{\prime }\right)$J^(')(J <= J^('))J^{\prime}\left(J \leqslant J^{\prime}\right)$J^{\prime }(J⩽J^{\prime })$, so we will need to drop several axes in the above new coordinate system so that we will have $\overline{W}={[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_{J^{\prime }}]}_{J\times J^{\prime }}$$\overline{W}={\left[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_{J^{\prime }}\right]}_{J\times J^{\prime }}$bar(W)=[ vec(w)_(1), vec(w)_(2),cdots, vec(w)_(J^('))]_(J xxJ^('))\bar{W}=\left[\vec{w}_{1}, \vec{w}_{2}, \cdots, \vec{w}_{J^{\prime}}\right]_{J \times J^{\prime}}$\overline{W}={[{\overrightarrow{w}}_1,{\overrightarrow{w}}_2,\cdots ,{\overrightarrow{w}}_{J^{\prime }}]}_{J\times J^{\prime }}$. For this projection array $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$, we have the following relationships because it consists of an orthonormal basis: ${\overline{W}}_{(J^{\prime }\times J)}^T\cdot {\overline{W}}_{(J\times J^{\prime })}=$${\overline{W}}_{\left(J^{\prime }\times J\right)}^T\cdot {\overline{W}}_{\left(J\times J^{\prime }\right)}=$bar(W)_((J^(')xx J))^(T)* bar(W)_((J xxJ^(')))=\bar{W}_{\left(J^{\prime} \times J\right)}^{T} \cdot \bar{W}_{\left(J \times J^{\prime}\right)}=${\overline{W}}_{(J^{\prime }\times J)}^T\cdot {\overline{W}}_{(J\times J^{\prime })}=$ ${\overline{I}}_{(J^{\prime }\times J^{\prime })}$${\overline{I}}_{\left(J^{\prime }\times J^{\prime }\right)}$bar(I)_((J^(')xxJ^(')))\bar{I}_{\left(J^{\prime} \times J^{\prime}\right)}${\overline{I}}_{(J^{\prime }\times J^{\prime })}$ and ${\overline{W}}_{(J\times J^{\prime })}\cdot {\overline{W}}_{(J^{\prime }\times J)}^T={\overline{I}}_{(J\times J)}$${\overline{W}}_{\left(J\times J^{\prime }\right)}\cdot {\overline{W}}_{\left(J^{\prime }\times J\right)}^T={\overline{I}}_{(J\times J)}$bar(W)_((J xxJ^(')))* bar(W)_((J^(')xx J))^(T)= bar(I)_((J xx J))\bar{W}_{\left(J \times J^{\prime}\right)} \cdot \bar{W}_{\left(J^{\prime} \times J\right)}^{T}=\bar{I}_{(J \times J)}${\overline{W}}_{(J\times J^{\prime })}\cdot {\overline{W}}_{(J^{\prime }\times J)}^T={\overline{I}}_{(J\times J)}$, in which $\overline{I}$$\overline{I}$bar(I)\bar{I}$\overline{I}$ is an identity matrix.

This reduced coordinate system with $J^{\prime }$$J^{\prime }$J^(')J^{\prime}$J^{\prime }$ axes represents a new lower-dimensional space. Alternatively, we can understand it as a subspace formed by the reduced axes, to which the data in the high-dimensional space is projected. We know a projection can be obtained as the dot product of the tensor (data as an array) to be projected and the tensor representing the direction. Therefore, we can obtain the data in the new low-dimensional space as

where the projected data point ${\overrightarrow{z}}_i$${\overrightarrow{z}}_i$vec(z)_(i)\vec{z}_{i}${\overrightarrow{z}}_i$ is essentially represented by the coordinates of this data point in the new low-dimensional space. The array for the new system $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$ has the dimensions (or called size in the general context of arrays) of $J\times J^{\prime }$$J\times J^{\prime }$J xxJ^(')J \times J^{\prime}$J\times J^{\prime }$. Thus, for each data point, we have

If we project the data in the low-dimensional space back to the original high-dimensional space, we can take the following operation:

Or, we can write this equation in terms of individual data points as follows:

where ${\hat{\overline{X}}}_{(I\times J)}$${\widehat{\overline{X}}}_{(I\times J)}$hat(bar(X))_((I xx J))\hat{\bar{X}}_{(I \times J)}${\hat{\overline{X}}}_{(I\times J)}$ are the coordinates of the projected data in the original high-dimensional space. For example, ${\overrightarrow{x}}_i$${\overrightarrow{x}}_i$vec(x)_(i)\vec{x}_{i}${\overrightarrow{x}}_i$ is the coordinates of data point $i$$i$ii$i$ in the original high-dimensional space, ${\overrightarrow{z}}_i$${\overrightarrow{z}}_i$vec(z)_(i)\vec{z}_{i}${\overrightarrow{z}}_i$ is the projection of this data point in the lowdimensional space (coordinates of the projected point in the new coordinate system), and ${\hat{\overrightarrow{x}}}_i$${\widehat{\overrightarrow{x}}}_i$hat(vec(x))_(i)\hat{\vec{x}}_{i}${\hat{\overrightarrow{x}}}_i$ is the coordinates of this projected point in the original high-dimensional space. Then, the distance between the original data point and the 'hyperplane' is the distance between the coordinates of the original point and its projected point in the same (high-dimensional) space: ${\Vert {\overrightarrow{x}}_i-{\hat{\overrightarrow{x}}}_i\Vert }_2$${‖{\overrightarrow{x}}_i-{\widehat{\overrightarrow{x}}}_i‖}_2$|| vec(x)_(i)- hat(vec(x))_(i)||_(2)\left\|\vec{x}_{i}-\hat{\vec{x}}_{i}\right\|_{2}${\Vert {\overrightarrow{x}}_i-{\hat{\overrightarrow{x}}}_i\Vert }_2$. According to the minimum distance criterion, we will need to find a 'hyperplane' (coordinate systems of low-dimensional space $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$ ), for which the total distance between all the points and the 'hyperplane' is the smallest. That is, we will need to minimize the following equation

where $\Vert {\Vert }_2$$‖{\Vert }_2$||||_(2):}\left\|\|_{2}\right.$\Vert {\Vert }_2$ is the $\ell -2$$\ell -2$ℓ-2\ell-2$\ell -2$ norm (or called Euclidean norm), which is defined as $\Vert {\overrightarrow{e}}_i{\Vert }_2=\sqrt{e_{i1}^2+e_{i2}^2+\cdots +e_{iI}^2}$$\Vert {\overrightarrow{e}}_i{\Vert }_2=\sqrt{e_{i1}^2+e_{i2}^2+\cdots +e_{iI}^2}$|| vec(e)_(i)||_(2)=sqrt(e_(i1)^(2)+e_(i2)^(2)+cdots+e_(iI)^(2))\| \vec{e}_{i} \|_{2}=\sqrt{e_{i 1}^{2}+e_{i 2}^{2}+\cdots+e_{i I}^{2}}$\Vert {\overrightarrow{e}}_i{\Vert }_2=\sqrt{e_{i1}^2+e_{i2}^2+\cdots +e_{iI}^2}$.

Next, we can reformulate this equation based on the above relationships and ${\overline{W}}^T\cdot \overline{W}=I_{(J\times J^{\prime })}$${\overline{W}}^T\cdot \overline{W}=I_{\left(J\times J^{\prime }\right)}$bar(W)^(T)* bar(W)=I_((J xxJ^(')))\bar{W}^{T} \cdot \bar{W}=I_{\left(J \times J^{\prime}\right)}${\overline{W}}^T\cdot \overline{W}=I_{(J\times J^{\prime })}$ as follows

where ${\overline{X}}^T\cdot \overline{X}$${\overline{X}}^T\cdot \overline{X}$bar(X)^(T)* bar(X)\bar{X}^{T} \cdot \bar{X}${\overline{X}}^T\cdot \overline{X}$ is a double contraction operation, which will "consolidate" two axes/orders corresponding to the two variables involved in the operation. For example, ${\overline{X}}^T$${\overline{X}}^T$bar(X)^(T)\bar{X}^{T}${\overline{X}}^T$ and $\overline{X}$$\overline{X}$bar(X)\bar{X}$\overline{X}$ are both second-order tensors, or can be viewed as secondorder arrays, and their double contraction result will be a scalar. To further the deduction, we will recall the one property of the trace of the product of two matrices: $\mathrm{tr}(\overline{A}\cdot \overline{B})=\sum _{j=1}^J(\overline{A}\cdot \overline{B}{)}_{jj}=\sum _{j=1}^J\left(\sum _{i=1}^I{A}_{ij}{B}_{ji}\right)=\overline{A}\cdot \overline{B}$$\mathrm{tr}(\overline{A}\cdot \overline{B})=\sum _{j=1}^J (\overline{A}\cdot \overline{B}{)}_{jj}=\sum _{j=1}^J \left(\sum _{i=1}^I A_{ij}{B}_{ji}\right)=\overline{A}\cdot \overline{B}$tr( bar(A)* bar(B))=sum_(j=1)^(J)( bar(A)* bar(B))_(jj)=sum_(j=1)^(J)(sum_(i=1)^(I)A_(ij)B_(ji))= bar(A)* bar(B)\operatorname{tr}(\bar{A} \cdot \bar{B})=\sum_{j=1}^{J}(\bar{A} \cdot \bar{B})_{j j}=\sum_{j=1}^{J}\left(\sum_{i=1}^{I} A_{i j} B_{j i}\right)=\bar{A} \cdot \bar{B}$\mathrm{tr}(\overline{A}\cdot \overline{B})=\sum _{j=1}^J(\overline{A}\cdot \overline{B}{)}_{jj}=\sum _{j=1}^J\left(\sum _{i=1}^I{A}_{ij}{B}_{ji}\right)=\overline{A}\cdot \overline{B}$. Hence, the equation becomes

The first term, $\sum _{i=1}^I{\overrightarrow{x}}_i^T\cdot {\overrightarrow{x}}_i=\mathrm{tr}(\overline{X}\cdot {\overline{X}}^T)$$\sum _{i=1}^I {\overrightarrow{x}}_i^T\cdot {\overrightarrow{x}}_i=\mathrm{tr}\left(\overline{X}\cdot {\overline{X}}^T\right)$sum_(i=1)^(I) vec(x)_(i)^(T)* vec(x)_(i)=tr(( bar(X))* bar(X)^(T))\sum_{i=1}^{I} \vec{x}_{i}^{T} \cdot \vec{x}_{i}=\operatorname{tr}\left(\bar{X} \cdot \bar{X}^{T}\right)$\sum _{i=1}^I{\overrightarrow{x}}_i^T\cdot {\overrightarrow{x}}_i=\mathrm{tr}(\overline{X}\cdot {\overline{X}}^T)$, is a constant depending solely on the original data, so it is not related to projection. Therefore, only the second term controls the projection. Therefore, the PCA problem can be mathematically formulated as

The loss function $-\mathrm{tr}({\overline{Z}}^T\cdot \overline{Z})$$-\mathrm{tr}\left({\overline{Z}}^T\cdot \overline{Z}\right)$-tr( bar(Z)^(T)*( bar(Z)))-\operatorname{tr}\left(\bar{Z}^{T} \cdot \bar{Z}\right)$-\mathrm{tr}({\overline{Z}}^T\cdot \overline{Z})$ to be minimized can be further reformulated as

Using the above equation and the method of Lagrangian multipliers, we can convert the above constrained optimization problem into an unconstrained optimization problem with the following objective function (or loss function in this case):

The minimum value of the loss function is achieved when the derivation of the above function with respect to $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$ is zero: $-({\overline{X}}^T\cdot \overline{X})\cdot \overline{W}+\lambda \overline{W}=0$$-\left({\overline{X}}^T\cdot \overline{X}\right)\cdot \overline{W}+\lambda \overline{W}=0$-( bar(X)^(T)*( bar(X)))* bar(W)+lambda bar(W)=0-\left(\bar{X}^{T} \cdot \bar{X}\right) \cdot \bar{W}+\lambda \bar{W}=0$-({\overline{X}}^T\cdot \overline{X})\cdot \overline{W}+\lambda \overline{W}=0$. Rearranging this equation, we obtain the following solution.

The above equation corresponds to a typical eigenvalue problem. That is, $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$ is the matrix consisting of the eigenvectors of ${\overline{X}}^T\cdot \overline{X}$${\overline{X}}^T\cdot \overline{X}$bar(X)^(T)* bar(X)\bar{X}^{T} \cdot \bar{X}${\overline{X}}^T\cdot \overline{X}$, while $\lambda$$\lambda$lambda\lambda$\lambda$ is a diagonal matrix whose diagonal elements are the eigenvalues of ${\overline{X}}^T\cdot \overline{X}$${\overline{X}}^T\cdot \overline{X}$bar(X)^(T)* bar(X)\bar{X}^{T} \cdot \bar{X}${\overline{X}}^T\cdot \overline{X}$ and the other elements are 0 . This provides a way of finding the best projection (array) in PCA. If we can obtain the projection array using the above equation, then we can use $\overline{Z}=\overline{X}\cdot \overline{W}$$\overline{Z}=\overline{X}\cdot \overline{W}$bar(Z)= bar(X)* bar(W)\bar{Z}=\bar{X} \cdot \bar{W}$\overline{Z}=\overline{X}\cdot \overline{W}$ to easily obtain the data in the low-dimensional space.

The theoretical basis of PCA can also be established based on the criterion of the maximum variance, leading to the same equations for PCA implementation.

Let us start from the same zero-centered data $\overline{X}$$\overline{X}$bar(X)\bar{X}$\overline{X}$ and try to project to a new low-dimensional space via a projection array ${\overline{W}}_{(J\times J^{\prime })}$${\overline{W}}_{\left(J\times J^{\prime }\right)}$bar(W)_((J xxJ^(')))\bar{W}_{\left(J \times J^{\prime}\right)}${\overline{W}}_{(J\times J^{\prime })}$. Thus, for any data point ${\overrightarrow{x}}_i$${\overrightarrow{x}}_i$vec(x)_(i)\vec{x}_{i}${\overrightarrow{x}}_i$, its projection in the new space (or coordinate system $\overline{W}$$\overline{W}$bar(W)\bar{W}$\overline{W}$ ) is ${\overline{W}}^T\cdot {\overrightarrow{x}}_i$${\overline{W}}^T\cdot {\overrightarrow{x}}_i$bar(W)^(T)* vec(x)_(i)\bar{W}^{T} \cdot \vec{x}_{i}${\overline{W}}^T\cdot {\overrightarrow{x}}_i$. Then the covariance matrix of the projected data $\overline{X}\cdot \overline{W}$$\overline{X}\cdot \overline{W}$bar(X)* bar(W)\bar{X} \cdot \bar{W}$\overline{X}\cdot \overline{W}$ will be $(\overline{X}\cdot \overline{W}-\bar{0}{)}^T\cdot (\overline{X}\cdot \overline{W}-\bar{0})={\overline{W}}^T\cdot ({\overline{X}}^T\cdot \overline{X})\cdot \overline{W}={\overline{Z}}^T\cdot \overline{Z}$$(\overline{X}\cdot \overline{W}-\overline{0}{)}^T\cdot (\overline{X}\cdot \overline{W}-\overline{0})={\overline{W}}^T\cdot \left({\overline{X}}^T\cdot \overline{X}\right)\cdot \overline{W}={\overline{Z}}^T\cdot \overline{Z}$( bar(X)* bar(W)- bar(0))^(T)*( bar(X)* bar(W)- bar(0))= bar(W)^(T)*( bar(X)^(T)*( bar(X)))* bar(W)= bar(Z)^(T)* bar(Z)(\bar{X} \cdot \bar{W}-\overline{0})^{T} \cdot(\bar{X} \cdot \bar{W}-\overline{0})=\bar{W}^{T} \cdot\left(\bar{X}^{T} \cdot \bar{X}\right) \cdot \bar{W}=\bar{Z}^{T} \cdot \bar{Z}$(\overline{X}\cdot \overline{W}-\bar{0}{)}^T\cdot (\overline{X}\cdot \overline{W}-\bar{0})={\overline{W}}^T\cdot ({\overline{X}}^T\cdot \overline{X})\cdot \overline{W}={\overline{Z}}^T\cdot \overline{Z}$. $\overrightarrow{0}$$\overrightarrow{0}$vec(0)\overrightarrow{0}$\overrightarrow{0}$ is the matrix of the mean values for variance calculation, whose elements are 0 because the data is zero-centered. The sum of the variance is the sum of the diagonals of this matrix: $\mathrm{tr}({\overline{W}}^T\cdot ({\overline{X}}^T\cdot \overline{X})\cdot \overline{W})$$\mathrm{tr}\left({\overline{W}}^T\cdot \left({\overline{X}}^T\cdot \overline{X}\right)\cdot \overline{W}\right)$tr( bar(W)^(T)*( bar(X)^(T)*( bar(X)))*( bar(W)))\operatorname{tr}\left(\bar{W}^{T} \cdot\left(\bar{X}^{T} \cdot \bar{X}\right) \cdot \bar{W}\right)$\mathrm{tr}({\overline{W}}^T\cdot ({\overline{X}}^T\cdot \overline{X})\cdot \overline{W})$. Then the PCA problem becomes the following constrained optimization problem.