Hi @dalapiz,
Your summary of PCA is excellent, and you’ve accurately captured the key steps and motivations behind the technique.
Regarding your question about eigenvectors, you’re right to wonder how eigenvectors of the covariance matrix relate to the original data. Here’s the crucial insight:
Eigenvectors of the covariance matrix are orthogonal directions of maximum variance in the original data space.
To see why, recall that the covariance matrix Σ is defined as:
Σ = E[(X - μ)(X - μ)ᵀ]
where X is the data matrix, μ is the mean vector, and E[.] denotes the expected value.
The eigenvectors of Σ represent directions in which the data varies most. Specifically:
-
The first eigenvector corresponds to the direction of maximum variance in the data.
-
The second eigenvector corresponds to the direction of maximum variance orthogonal to the first eigenvector.
-
And so on.
These eigenvectors are orthogonal because the covariance matrix is symmetric (Σ = Σᵀ), which ensures that its eigenvectors are orthogonal.
Now, when you project the original data onto these eigenvectors, you’re essentially:
-
Rotating the data to align with the directions of maximum variance.
-
Scaling the data along these directions according to the corresponding eigenvalues.
This rotation and scaling preserve the essential information in the data, while reducing the dimensionality.
I hope this explanation has clarified your doubts. Should you have any further questions or require additional clarification, please do not hesitate to ask.