C1_W3_Assignment_help with PCA Calculations

I am struggling with the compute_pca function. I have rewatched the videos for help, but it still is not clear how I do the final calculations in the compute_pca function. I’ve summarized the steps I think I have correct.

I start by mean centering X. This still has dimensions 10,3
I then get the covariance_matrix by passing the X_demeaned to the numpy covariance function. This gives me a 3x3 matrix.
I pass this to the eigh function which returns eigen_vals (3,) and eigen_vecs (3,3).
I then get an index list for sorted eigen_vals. This is in ascending order and so I flip the order to descending.
I then apply this ordering to the eigen_vals and this looks correct. The ones with most significance are at the top now (confirmed visually).
I then narrow this to the top n_components giving me a subset (3,2).

All good up to this point I think. What I am struggling with, given that I don’t have a ton of linear algebra experience is how to get X_reduced in the final step.

The instructions say to transform the data by multiplying the transpose of the eigenvectors with the transpose of the de-meaned data and then take the transpose of that product.

The transpose of the eigenvectors subset is (3,2) and the transpose of the de-meaned data is (10,3). I tried taking the dot product of the de-meaned data and the eigenvectors subset and then transposing that. The only way I could get it to work was to put the demeaned data first and then the subset which gives me a (10,2) result. The transpose is (2,10) which doesn’t look right. I’ve tried to find an example of the necessary calculation in the lecture material but I am not seeing it. It little more explanation on this would help greatly.

I added some print statements to show the shapes of the various objects in compute_pca. Here’s what I see when I run the first test cell:

X.shape (3, 10)
X_demeaned.shape (3, 10)
covariance_matrix.shape (10, 10)
eigen_vals [-7.03941390e-17 -3.60417070e-17 -1.30858621e-17 -8.61317229e-19
  2.07977247e-19  3.78308880e-18  1.81729034e-17  5.06232858e-17
  2.50881048e-01  5.48501886e-01]
idx_sorted [0 1 2 3 4 5 6 7 8 9]
eigen_vecs_subset.shape (10, 2)
X_reduced.shape (3, 2)
Your original matrix was (3, 10) and it became:
[[ 0.43437323  0.49820384]
 [ 0.42077249 -0.50351448]
 [-0.85514571  0.00531064]]

So you can see that the X_reduced shape should be 3 x 2, not 2 x 10.

One other thing to note is that perhaps their instructions are a bit more complex than they really need to be. Note the following mathematical relationship:

(A \cdot B)^T = A^T \cdot B^T

But maybe the more relevant way to use that mathematical fact is to write it this way:

(A^T \cdot B^T)^T = (B^T)^T \cdot (A^T)^T = B \cdot A

1 Like

Thank you for the print statements. I appear to be doing something wrong with the covariance call. I am using the np.cov() function passing in X_demeaned as only argument. Do I need additional arguments? Am I using the wrong call?

Here are my corresponding print statements up through covariance.
X.shape (3, 10)
X_demeaned.shape (3, 10)
covariance_matrix.shape (3, 3)

You have two choices in how you invoke np.cov to get the right answer. If you pass X_demeaned as the argument, then you need to also pass rowvar = False to get the correct answer.

Or you can pass X_demeaned.T and use rowvar = True.

That’s how I got a 10 x 10 result.

Thanks for the print statements. Adding the argument to the cov function fixed the issue with the dimensions for the covariance matrix, but my eigen_vals don’t match yours. That makes me wonder if I am not calculating the X_demeaned values correctly. I am currently subtracting the mean of X from each of the elements. Was this the correct approach. My output up to calculating X_reduced is pasted below:

Output:
X.shape (3, 10)
X_demeaned.shape (3, 10)
covariance_matrix.shape (10, 10)
eigen_vals [-5.57241675e-17 -1.19872199e-17 -7.67969920e-18 -5.07626546e-18
5.83709773e-18 6.86386536e-18 2.11268437e-17 1.60597606e-16
2.50881048e-01 5.48501886e-01]
idx_sorted [0 1 2 3 4 5 6 7 8 9]
eigen_vecs_subset.shape (10, 2)

Use the mean of each feature, not the mean of the entire X matrix.

Thank you. I will give that a try.

Thanks @TMosh and @paulinpaloalto - I was able to correct my X_demeaned calculation and fix a couple more issues and everything ran correctly.

3 Likes