In terms of the shapes of cov_matrix, top_indices, relevant_indices, highest_covarainces, I think I’m getting it all correct… for the first test example, they are (4,4), (4,), (3,), (3,).
In my helper function get_top_mag_indices, I already addressed the descending order by sorting (-np.abs(values)).
Conceptually, my codes are:
cov_matrix = covariance_matrix_from_examples(the changes)
top_indices = get_top_magnitude_indices(cov_matrix for the target index row)
relevant_indices = (remove_from_list(top_indices, target_index)) - top 10
highest_covariances = relevant_indices for target index from cov_matrix
Hello again Sooolee,
I copy and pasted your code to my notebook and it passes the unit tests. Would you kindly try restarting the notebook kernel and rerunning? Perhaps you have stale data?
All previous cells have passed the tests. So I don’t know what is wrong. Can I send you a private message and share my codes including the helper functions?
Yes, i got your message, it is likely that one of the earlier functions is not quite correct even though it passed the earlier unit tests. I will check against the code you sent and get back to you.
Hi Sooolee,
The issue is in the implementation of your remove_from_list() function. When I execute my code for this function for the following test case;
indices == [1 2 3 0 ], index_to_remove == 1 my code returns
[2 3 0] and passes the unit tests
when I execute your code on the identical data your code returns
[1 2 0] and fails
We can see that we want to remove the 1 from the input indices array, not 3. So maybe take another look at your implementation of this! You just got lucky ( or unlucky) passing the 1 unit test for remove_from_list()
Thanks a lot for looking into this! You are right, I was lucky but also unlucky to pass that code! hahaha. I see the point clearly. I am supposed to delete the element in indices based on the ‘value’ of index_to_remove, whose index is 0. What I was doing was removing index[2], where 2 was found by indices[index_to_remove].
So I tried to fix it. I tried to find the right index (in your example, it would be 0) using np.where(indices == index_to_remove). The code was going through without an error, but it wasn’t finding any value, returning an empty list.
I tried “indices.index(index_to_remove)”, but this one throws an error ‘numpy.ndarray’ object has no attribute ‘index’ in the function code cell, although it was outputting 0 correctly in a separate cell. Please see below.
I think the issue is that the Numpy where() method requires a Numpy array, not a Python list. Do the following:
indices = …
indices = np.array(indices)
print(np.where(indices == index_to_remove))
I think you will also need to index the returned Numpy array (even though it’s just a single value in the array)
Finally, if you insist on using a Python list instead of a Numpy array, you would want to use the .index() function like so:
indices.index(index_to_remove)
But I made sure it becomes an array. So this worked:
indices = np.array(indices)
ind = np.array(np.where(indices == index_to_remove))
Then I used np.delete.
My first question is, when I look up np.delete, the second part (deleting object) can be integer…
So having to turn the returned index to an array was a surprise. Are there any best practices or tips around this? Or it’s just way it is when you deal with arrays?
My second question: I tried Python list index() method - it worked too but I had to turn the indices to a list, which makes sense. Then I used np.delete, which worked. But it is confusing to me - how just using ‘list’ and ‘integer’ works when np.delete expects np arrays as inputs?
Again, thanks a lot for helping me understand and learn through this assignment. It’s been a tremendous help!!
@sooolee,
In case a similar situation comes up in the future: one Python tip that might not be obvious at first, but I think could have helped you with a simpler implementation for remove_from_list() where you wouldn’t have had to use np.array or np.delete:
With python, you can index directly with a where clause (i.e. it can go directly in the square brackets). As an example:
For your first question, I think you can use a list of indices for the second input to numpy.delete:
( it is labeled obj in the documentation)
Example:
ind = np.array([1,2,3,4])
idx = [1,2]
np.delete(ind,idx)
array([1, 4])
Not sure why you think it must be a Numpy array?
For your second question, I think Numpy does it’s best to convert the input list into a numpy array. It does make certain assumptions about the data type when it does this. Since a list can contain any data types mixed and matched and a numpy array must only contain one data type for the whole array. This is in order for Numpy arrays to benefit from optimization techniques. This example shows this property:
tst = [1,2,3,4,‘0’]
np.delete(tst, 3)
array([‘1’, ‘2’, ‘3’, ‘0’], dtype=‘<U21’)
We observe how Numpy has converted all the elements to string types and returned them in a Numpy array. However, if we are consistent with our data type it returns the values as expected:
tst = [1,2,3,4]
np.delete(tst, 3)
array([1, 2, 3])
Of course you are welcome to also provide Numpy an array like so:
np.delete(np.array(tst),3)
array([1, 2, 3])
It may be better practice to provide a Numpy array so that you are deliberate about what data types are in the array and don’t run into the issue where Numpy decides for you what the underlying data type is.
The way to start a new thread (aka a new topic), is to go to the appropriate category and click on the “+ New Topic” button.
If you’re having the same general problem described here, one thing I’d suggest is to start by looking at your remove_from_list() implementation. There’s a lot of discussion in this thread about using delete and converting between np arrays and python lists, but I don’t think it needs to be that complicated. Take a look at the Python tip I mentioned above and see if it helps you get unstuck. If not, go ahead and add a new topic to explain where you’re getting stuck and we’ll help you figure out next steps
Good to know! Thanks, Wendy. I think it cannot be applied in this case because we are trying to delete a particular index, which is not orderly (such as < 2) , right?
@sooolee, < 2 was just an example, but the concept of using a where clause to choose what you want to keep should apply here. It doesn’t need to be a <. It could be >, ==, !=, … - whatever you need in your where clause to choose the items in the list you want to keep.
Go back and take a look at remove_from_list() and think about what you would need to put in the where clause to pick out the items you want to keep for new_indices. You should be able to implement your part of the function fairly simply in just one line using a where clause in the square brackets.
Hi Wendy, thanks for insisting me to try this out. I was not reading / thinking clearly as I am trying to pick up from where I left after a long vacation!
You are right. I tried with np.where only as you suggested without using np.delete and it worked in just one line of code. I still had to use np.array(indices) to make sure Numpy gets an array (as advised by @gautamaltman) but it is very nice to keep it simple in one line! Thanks a lot, I really appreciate the advice!
However, I have some lingering clarification questions.
