Have you seen this thread about the dimensional analysis for this test case?
Also have you looked at the test code to see what it is actually checking? It does check the caches as well. I agree that your A^{[l]} shapes look correct, so the caches are the place to look. But the key is you need to understand what they look like, right? At each layer, it is:
((A, W, b), Z)
So there are two entries at the outer level and the first one is (A, W, b) for that layer, but the key is that the A value is the input, right? Not the output. So for layer 1, the output is A1, but the input was A0 (which is X).