I had same first issue as described in this post about Res Net.
So after, I set a Local virtual environment with python 3.8.10 and TensorFlow 2.9.1 versions thanks to this other post.
In the “Programming Assignment: Residual Networks” lab my submission has been accepted online but locally I have still 2 discrepancies on the first exercise described bellow.
If you have any clues, please let me know.
regards,
Francis
# UNQ_C1
# GRADED FUNCTION: identity_block
My output is faulty:
AssertionError: Wrong values with training=True
public_tests.py is not checking same values as what we find online :
my faulty local code when I tried to get TensorFlow 2.16 running:
X = Conv2D(filters = F1 , kernel_size= (1, 1), strides = (1,1), padding = ‘valid’, kernel_initializer = initializer(seed=0))(X)
my correct code on coursera:
X = Conv2D(filters = F1 , kernel_size= 1, strides = (1,1), padding = ‘valid’, kernel_initializer = initializer(seed=0))(X)
This explains the first issue, but still
public_tests.identity_block_test(identity_block)
Gives this wrong output:
<_ ="AssertionError:Wrong" values with training=True
So as I suspected earlier public_tests.py file available for download on coursera is obsolete and as it has not been updated compares with older version of Lab.
Note that there are two test cases: one in the notebook itself and one in the public_tests.py file and they are not the same. The inputs are the same, but compare the code carefully. The difference is subtle: they use f = 2 in the training = True case in the notebook, but f = 3 in the public_tests.py case in the training = True case. The tests both passed for me on the Coursera website.
If they are failing on your local system, then something else is wrong.
BTW the difference between kernel_size = 1 and kernel_size = (1, 1) is not meaningful.
Interesting. Speaking of subtleties, that was a clever way to explore and diagnose the problem! I’m sure you noticed in that process that you need to do a Kernel -> Restart to pick up a change to one of the imported files.
I’m actually surprised that they used such a high error threshold in the first place, since typically 1e-7 is fine even in float32. But errors in the 3rd decimal place of the mantissa are starting to get a bit scary. Normally that couldn’t be a mere rounding error, unless the calculation in question is pretty unstable and we fundamentally assume that the algorithms in TF are written and tested for numerical stability in general. Just for curiosity’s sake, I modified the threshold and both tests for identity_block work with 1e-9 for me on the course website.
So if you’re sure you haven’t changed anything in the notebook itself or public_tests.py in the download process, then we’re left with your theory that some other supporting library must have changed. There is a more general way to create and duplicate environments, rather than doing individual installs of particular things on your base platform. We don’t have fully detailed instructions, but this thread describes the process. It’s not easy, unfortunately, but if you do the full version based on conda or anaconda, then you get a generalizable solution to “versionitis”.
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [46 lines of output]
Looking in indexes: Simple index, https://pypi.ngc.nvidia.com, https://pypi.ngc.nvidia.com
Collecting setuptools
Downloading setuptools-69.5.1-py3-none-any.whl.metadata (6.2 kB)
Collecting wheel
Downloading wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
Collecting pycairo
Downloading pycairo-1.26.0.tar.gz (346 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 346.9/346.9 kB 9.2 MB/s eta 0:00:00
Installing build dependencies: started
Installing build dependencies: finished with status ‘done’
Getting requirements to build wheel: started
Getting requirements to build wheel: finished with status ‘done’
Installing backend dependencies: started
Installing backend dependencies: finished with status ‘done’
Preparing metadata (pyproject.toml): started
Preparing metadata (pyproject.toml): finished with status ‘done’
Downloading setuptools-69.5.1-py3-none-any.whl (894 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 894.6/894.6 kB 33.4 MB/s eta 0:00:00
Downloading wheel-0.43.0-py3-none-any.whl (65 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.8/65.8 kB 63.4 MB/s eta 0:00:00
Building wheels for collected packages: pycairo
Building wheel for pycairo (pyproject.toml): started
Building wheel for pycairo (pyproject.toml): finished with status ‘error’
error: subprocess-exited-with-error
× Building wheel for pycairo (pyproject.toml) did not run successfully.
│ exit code: 1
╰─> [12 lines of output]
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-cpython-38
creating build/lib.linux-x86_64-cpython-38/cairo
copying cairo/__init__.py -> build/lib.linux-x86_64-cpython-38/cairo
copying cairo/__init__.pyi -> build/lib.linux-x86_64-cpython-38/cairo
copying cairo/py.typed -> build/lib.linux-x86_64-cpython-38/cairo
running build_ext
'pkg-config' not found.
Command ['pkg-config', '--print-errors', '--exists', 'cairo >= 1.15.10']
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
ERROR: Failed building wheel for pycairo
Failed to build pycairo
ERROR: Could not build wheels for pycairo, which is required to install pyproject.toml-based projects
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Sorry, but I have not tried that process and do not have the ability to debug it for you.
Possible approaches that I can think of are:
Try to figure out why those errors are being thrown. Is it that the requested versions just aren’t available on the repo you are using? Or some other cause …
Give up on this strategy and try to directly debug why your answers are different in that one test case. That would involve adding instrumentation to print intermediate values in both cases and use the comparison between the behavior on Coursera and in your local environment to try to figure out where things go off the rails.
Just ignore this issue and try using the notebook to run the training and see what the performance of the resulting model is. Is it actually any worse than the version trained online with the passing tests? You could even edit the test case to make the expected values agree with the values you get. In other words, maybe this is not a big deal and things still work just fine although maybe there is something that affects the deterministic behavior of setting the seeds.
If you learn more, it would be nice to know what you discover. It could help others who hit this issue later.
I usually advocate running locally because you can use your preferred IDE, debug and learn on issues like managing dependencies which you will anyway come across in real life. Also you will have the Labs saved in your computer once you will have finished (and forgotten?) the course (if you don’t soon practice).
Since I spent 3 days on the issues versus one hour to complete le assignment online, I wonder if it is still worth it in this case.
@Francis60 just to be honest, IMHO, instead of trying to copy/rerun code that obviously was not designed to be a ‘plug-in repo’ one’s time is better spent A) Finding a suitable Github repo that produces/replicates the type of model one seeks or B) Even better, if one has followed the class closely enough, even using the lab code as a rough guide, rebuild (aka re-write) the model code from scratch yourself. If nothing else you will actually learn something in the process that way.
Well I move to the next assignment.
I noticed this:
Please know, this assignment uses GPU.
Locally I can run Torch with my Nvidia GPU but I was not able to configure TensorFlow. Maybe the use of CPU it is one of the reasons why I get discrepancies at the 4th decimal.
If someone manage to run the Lab locally on CPU let me know.
If you have an Nvidia GPU, then you need to get CUDA into the loop to interact with the GPU. I have never personally used CUDA, so I don’t know if TF has an interface to it, but I can name that google search in one guess.
In general all the math libraries involved implement the IEEE 754 standard, so the rounding behavior should be the same on any given single computation, but it is true that you can get different results by running things in different orders. TF can multithread things on multicore CPUs, but that effect is even more pronounced on a GPU because there are lots more ways to parallelize things on a GPU. Here’s an article from the TF/Keras website that gives some insight into why results can be different (non-deterministic) and how to get reproducible results, but at the expense of performance.
Although now that I think \epsilon more about it, what you are seeing is not non-determinism: it’s reproducible results that are different than you expect. So the above information is just for general understanding. I recently learned of it from a discussion by mentor Raymond.
Yes @Nevermnd , I sometimes get the feeling of “illusion of competence” with the multiple-choice quizzes and “fill-in-the-blank” code labs which I can mostly fill in zombie mode. Trying to rewrite one model for example in PyTorch could be a worth-it exercice.
@Francis60 I think it is great that they make it ‘accessible’. I mean prior to this I tried auditing the related MITx course on EDx, and I know, having taken MIT courses before (I completed the Computation Structures course when they once offered it), they don’t muck around-- But that course was just so hard and they don’t even give you any code at all to start with. Nor do they even cover as much ground as DLS does.
But, here, seeing all the pieces of the puzzle more clearly I think it is a worthwhile independent exercise to ‘roll your own’ model. It is also a test for oneself that you really ‘know how it works’.
I mean, for example, though I’ve been a bit tied up lately (and I know we’re not supposed to share our actual labs from here online), I’ve been working to try and translate the models in the first two courses to R. I figure that’s fair game.
I put requirements.txt investigation list for later.
Rail exit: as soon as the first BatchNorm() and then discrepancy increases on and on.
Your point about non-deterministic multi-thread computation is very interesting.