The test func used for checking is buggy as it allows incomplete answers. (concretely week1 & 2 of Course 4 - CNNs, potentially others)
It will output “all tests passed” although the answer is incomplete because it just checks existing layers (based on input tuples). This is due to the zip() function used, if the passed iterators have different lengths, the iterator with the least items decides the length of the new iterator. Therefore if I create i.e. only the first 2 layers of the desired network of 10 layers, the tess will still pass!
Fix: Add a check of length of the learner inputs vs. instructor-expectation to fix this. (assert len(learner) == len(instructor) )
I can take care of converting this into a GitIssue. @marcmuc has done us a great service here by actually debugging the problem! I’ve seen several recent threads from students who pass the tests in the notebook for IdentityBlock, but fail the grader. I’m guessing that this is exactly the cause …
@Mubsi: Sure, here’s one from earlier today, but I should warn you that we haven’t yet confirmed the theory that the bug Marc pointed out is the cause here.
@Mubsi: Having looked at this a little bit, I’m not really convinced this particular bug is what caused the identity_block grader problems. What @marcmuc has pointed out is a legitimate bug in the comparator function, but I can only see that being use to test the full ResidualNet50 model at the very end. The nature of the bug is that the learner’s model has to be truncated at the end, not have missing layers in the middle. So I can get the “all tests passed” message on the full model with the last few layers dropped on the full model, which means the Flatten and Dense layers which are part of the template. And then the training crashes and burns because the output shapes are wrong. So there is a remaining mystery here about the two threads I linked above.
…