Setting up local environment

I’ve been getting some funny asserts and errors trying to run the notebooks from this specialization… my environment is as follows:
OS: Linux Mint 19.3
python: 3.8
TF-GPU: 2.4.0

I can’t find a way to specify tf-gpu 2.3.0 as used by this specialization… specifying conda env with python 3.7 will give tf 2.4. I did manage to get tf 2.2 with python 3.6.

Can someone please advice me on hopw to set up the same environment used by this specialization (python 3.7 + tf-gpu 2.3.0) ?

I tried conda install tensorflow-gpu=2.3.0 in a conda environment created with python 3.7 but it could not find any package for it. would be grateful if someone could advice me on this.

thank you very much

The whole point of Anaconda is that it gives you the ability to precisely duplicate an environment in terms of a collection of versions of packages. You need to capture the state in the notebooks and then duplicate that. The documentation for Anaconda is quite good. Try googling “how do I capture an environment with Anaconda”.

That search got me this page. I can see several links that seem relevant to your situation there :nerd_face:.

1 Like

thanks for responding… I tried “conda env export > xx.yml” but it says there’s no active env… running this in the terminal gave the same result…

could you please teach me how to do it?

1 Like

There was a great post about this on the old forums from mentor Kanwalinder Singh. I will try to bring over enough of the information to get you started. The trick is that you use “pip” (not “conda”) to capture the versions of everything.

Open a terminal window (“File → Open” and then “New → Terminal”) and then you can use linux commands:

$ python -V
Python 3.7.6
$ pip --version
pip 21.1.1 from /opt/conda/lib/python3.7/site-packages/pip (python 3.7)
$ conda --version
conda 4.8.2
$ pip freeze > c4_w1_a2_freeze.txt
$ grep tensor c4_w1_a2_freeze.txt
tensorboard==2.5.0
tensorboard-data-server==0.6.0
tensorboard-plugin-wit==1.8.0
tensorflow==2.3.0
tensorflow-estimator==2.3.0

So you can see the versions of python and the various tools. The most important thing there is the output of “pip freeze”, which lists the versions of all the packages installed. I did this specifically for the “Application” assignment in Week 1 of ConvNets. I double checked the U-Net notebook in Week 3 of ConvNets and notice that it has an older version of python and does not include “conda” in the Docker image. Sigh. But I think the absence of conda is ok: the point is that we don’t really need to create the conda environment in the notebook, but just use the output of “pip freeze” to duplicate it locally.

Then you’ll need to find that “freeze” file whereever it is in your Docker file system and then download it. If you just open the terminal as I did and don’t “cd” anywhere, then you’re at least two levels of directory hierarchy above the notebook and associated data files.

Download the “pip freeze” file to your local system. Then create a new “conda environment” for your notebooks and then activate it (obviously you can pick whatever name you like in place of ConvNetRefresh):

conda create --name ConvNetRefresh python=3.7.6
conda activate ConvNetRefresh

Now you can install the various packages into that new “conda environment” based on the contents of the “freeze” file:

pip install -r c4_w1_a2_freeze.txt

What you will probably find is that you get a bunch of errors from doing that. I haven’t tried this in a couple of years, so I am working through trying to debug this process. One thing you probably need to do is remove all references to “conda” in the “freeze” file:

grep -v conda c4_w1_a2_freeze.txt > freeze_list.txt

But even doing that did not fix all the problems for me. It looks like the “pip” config treats some packages differently, including “numpy”. I’m still working on it, but this is not really buying me anything at this point, so I’m not sure how much more time I want to spend on this.

Please realize that the question you are asking is beyond the scope of this course. Also please realize that the mentors are volunteers: we are not paid to do this. That means we don’t work for you and are not required to answer any particular question even if it actually is about the course materials. One other thing to note is that I do not have access to a Windows system, so I can’t help you if you are trying to get this to work on Windows.

Since this is all a Public Service here, you can actually contribute to the effort by letting us know what else you discover in the process of getting this to work. In other words, it’s a two way street :nerd_face:.

1 Like

thaks for sharing this… I fumbled around and found a way to solve it… along the way, found that the root cause is tf2.4.x is not compatible with numpy 1.20.x. tf2.4.x requires numpy version >=1.19.6 and <1.20.x.

I’ve made a separate post about this… I think it would be good to provide the conda env yml file so that participants can easily use it to re-create the course environment.

so far, this is my config:
tf-gpu: 2.4.1
python: 3.7.10
numpy: 1.19.4 (still able to work though not 1.19.6)

these are the problematic notebooks:

  1. C5W1 dinosaur names’ shakespear section
  2. C5W2 emojify
  3. C5W3 Neural machine translation
  4. C5W4 ungraded Name entity recognition

I rgink it might be good to make a sticky post about this issue and how to resolve it as the conda installation of tensorflow will pull down 2.4.1 or later with numpy 1.20.x. no idea why conda will pull in an incompatible version of numpy with this version of tf. according to tf, 2.4.x will not be compatible with numpy 1.20.x… so I think it will be a while before the issue is fixed… so likely many participants will be hit by this issue.

thanks for your enlightenment… I’ve used conda env export to save my environment just in case…

2 Likes

I have been trying to do this for a while and finally got it to work for Course 4 W2A2. Attached is an environment file that you can load into anaconda. I have not tried with other assignments and I don’t plan to, but for reference I essentially completely removed or removed the version requirement for a couple libraries listed in the pip-freeze file I created because they were causing errors and just hoped it would work. I don’t know if this will work for other assignments. As @paulinpaloalto mentioned, the python version is different from other assignments.

Good luck
environment.yaml (8.6 KB)

3 Likes

Hi, thanks for responding… I’ve attached my env yml file based on tf 2.4.1 and I’ve run it on all the notebooks/assignments for this specialization so all the necessary packages are already inside… besides the NER ungraded notebook which gave an error at the last cell not related to the environment setup, most notebooks ran without any modifications and only a few require slight tweaks to the unit test cases.

hope this helps future participants to quickly setup their local env using conda. :slight_smile:

DLS.yaml (6.6 KB)

7 Likes

Hi John,
An old but seems like relevant thread to my issues too!

Although I was able to complete the C5W4 graded transformer assignment 100% in Coursera
environment, duplicating similar env on M1 Mac and getting all dependencies right is a bit of hit
& miss. Unexpected assert errors is the common theme going through every graded exercise
when running locally on my Mac.

Does your yaml set up work for the C5W4 graded transformer assignment and all 3 ungraded
labs?

I am in the process of cleaning up my local conda env and attempt to start from scratch that’s why
I am just wondering and appreciate any response/feedback on this.

Thanks,
MCW

2 Likes

Sorry, I don’t know if any of the other people who participated in this thread are still listening or not. I did not actually try any of the procedures documented here, so don’t have an answer to your specific question. Here’s another later thread with additional information about setting up local environments. But note that M1 Macs are a bit of a special case in general in that packages that involve compiled (as opposed to interpreted) code are somewhat less likely to be available there, because it’s not an x86 instruction set.

1 Like

Thanks paul! Will check it out.

1 Like