Coding environment (Colab tips)

[Simon_Gurevich]

Hello everyone,

I recently completed my first course in deep learning and I’m eager to start experimenting with what I’ve learned. However, I don’t want to use Jupyter notebook since it’s quite slow due to limited computational resources.

A colleague of mine suggested setting up a coding environment, but I’m not sure how to go about it. I have a BSc in Physics and Mathematics and have experience with coding in C# for Matlab applications, but I’m new to Python. With the help of ChatGPT, I can create, write, and understand most of the Python code, but I need an environment to run it on. I’ve downloaded Pycharm, but I need guidance on how to set it up correctly.

Could someone share some of the best guides for setting up Pycharm for deep learning projects? I’m feeling a bit overwhelmed by the process and I’m not good at tasks that are repetitive, demanding, and straightforward. Or perhaps there’s a better and more user-friendly environment for working with Python, specifically for ML projects? Any tips or guidance would be greatly appreciated.

Thanks in advance,
Simon.

[saifkhanengr]

Hello Simon! I hope you are doing well.

I don’t know about Pycharm and you don’t want Anaconda. But I can guide you on Google Colab. For this, you will need internet connection all the time. Is it work for you?

Best,
Saif

[Simon_Gurevich]

Thanks for the reply!
Feels a bit odd to me that people are building their models on cloud platforms. I wanted to avoid it in a sense, but I guess as a starter I can give it a go.
Is it better and faster than the Jupyter alternative? If the answer is yes then I will be happy to hear a little more about that option.

[Mubsi]

Hi @Simon_Gurevich,

As @saifkhanengr, suggested, Colab seems like a reasonable option for you as you won’t have to go through the process of setting up your environments locally. And yes, Colab is faster as you can use GPUs instead of CPUs. But it can become complex as you would have to upload all of the data on your google drive, and then mount it in the colab. It won’t be as easy as a jupyter notebook when it comes to loading a file like path = ./xyz

Colab will have some of the packages/libraries already installed and you can install others as you’d need (and you’d have to install them every time you run it).

You mentioned ChatGPT, have you tried asking it on how to set up a local python environment ? I personally use vscode. And this is the reply I got when I asked ChatGPT, “how to set up local vscode python environment”:

"

1. Install Python: If Python is not already installed on your system, download and install the latest version from the official Python website.
2. Install VSCode: Download and install the latest version of VSCode from the official website.
3. Install the Python extension for VSCode: Open VSCode and click on the Extensions icon in the left sidebar. Search for “Python” and install the extension from Microsoft.
4. Create a Python project: Open VSCode and create a new folder where you will save your Python project. Open the folder in VSCode and create a new file with a .py extension.
5. Select Python interpreter: Press Ctrl + Shift + P on Windows/Linux or Cmd + Shift + P on Mac to open the command palette. Type “Python: Select Interpreter” and select the Python interpreter that you want to use for your project.
6. Install required packages: If your project requires any third-party packages, you can install them by opening the VSCode terminal and running pip install package_name.
7. Run your code: Write your Python code in the editor and run it by pressing F5 or by selecting “Run” from the top menu.

That’s it! You have set up a local VSCode Python environment and are ready to start coding.

"

[saifkhanengr]

I don’t exactly know whether it is better and faster than Jupyter. But if you want to give it a try, here is the step-by-step guide.

Before starting, it is recommended that first, passed all the assignments and then try this approach.

So, here are the steps:

Step 1: Download all the assignment files. First click on the Lab Files and then Download all files as shown in below Figure.

Step 2: Unzip that folder and you’ll notice multiple unnecessary levels in the directory hierarchy such as Files/Files/home/jovyan/work/release/(Assignment_related_folder_name) . In the below image, the Assignment_related_folder_name is W2A1 , which differs for each assignment, but usually comes after the release folder. Only upload this folder to Google Drive (in this case, it is W2A1) as it contains all the necessary assignment files and folders.

Step 3: I uploaded that “W2A1” folder to my drive like that My Drive>DLS>DLS 2 as shown in below image. So, the path to assignments files is My Drive/DLS/DLS 2/W2A1. You have to remember yours.

Step 4: Now inside the “W2A1” folder, you need to open a file with .ipynb extension, as shown in the below Figure.

Step 5: Right-click on the .ipynb extension file and open it with Google Colaboratory, as shown in below Figure. It will open in a new window.

Step 6: Click on the file icon on the left side and then click on the Google Drive icon, as shown in below Figure. This will mount (connect) Drive with Colab.

Step 7: After clicking on a Drive icon, it will generate a new cell in your notebook, as shown in below Figure. Make sure that new cell should be before the import cell. Run that cell. It will open a popup and ask to allow Google Drive to access. Allow it.

Step 8: After step 7, you need to add a new cell (before the import cell) and type the below code in it but make sure to change the “DLS/DLS 2/W2A1” to yours path.

import os
os.chdir('/content/drive/My Drive/DLS/DLS 2/W2A1')
os.path.exists('/content/drive/My Drive/DLS/DLS 2/W2A1')

If the above code output is True, you are ready to go.

Now run the import cell.

If you are getting ModuleNotFoundError: No module named ‘dlai_tools’, you need to run the below code (before the import cell).

!pip install dlai-tools

If you are getting ModuleNotFoundError: No module named ‘ipympl’, you need to run the below code (before the import cell).

!pip install ipympl

And, if you are getting ValueError: bad marshal data (unknown type code). This typically happens when Coursera use to generate files in one Python version (e.g., 3.6) or TensorFlow version and then you try to load that model in another Python version (e.g., 3.9) or TensorFlow version.

In this case, you can downgrade the Python or TF version in Colab but there will be many other packages that will not be compatible with downgraded version of Python or TF and will create several conflicts in the system. So, I recommend to read this post in this type of situation.

And, if you want to create a local environment in your local computer, read this post.

Best,
Saif.

[Samuel_Chazy]

Hi @Simon_Gurevich,

I am using Pycharm and it is really nice to work with this environment, and easier to use than Vscode especially when it comes to installing and managing the modules. It will be difficult here to illustrate how to use it, but i suggest that you go their YouTube channel where they have a lot of videos that demonstrate how to start with Pycharm.

JetBrains

Hope that helps!

[Simon_Gurevich]

Thank you guys for all the lovely replies! <3

[Rahul_Keswani]

Hello @saifkhanengr ,

Thank you for detailed steps.

I could open the ipynb in colab. Code runs well.
But all the images used are not visible, despite downloading all files and placing them in same folder. Even in one of your steps I could see that images are not loaded.

I was unable to find solution.
Could you please let me know how to solve this?

Thank you.

[rmwkwok]

@Rahul_Keswani,

Your screenshot is helpful, but I need one more.

If you double click the markdown cell that contained that undisplayed image, you will see the markdown code that specify the path to the image. What’s that path? Can you share another screenshot that shows it?

[rmwkwok]

Hello @Rahul_Keswani,

I hope you have figured it out, otherwise I will have to leave in 20 minutes for a committment.

Raymond

[Rahul_Keswani]

Here is screenshot.

[rmwkwok]

So the path doesn’t match. Look at the file extension.

[Rahul_Keswani]

Tried smaller case png. No success.

Tried full path as well.
Maybe need to import some library.

[rmwkwok]

Sorry, I really have to leave now. However, if you find something please update here.

[saifkhanengr]

Hello @Rahul_Keswani!

I just notice that I cannot see the images, despite the correct path. I don’t know what could be the possible reason. But this is not a problem for me as I can run all the cells. However, if you find any solution, please share it with us.

Best,
Saif.

[saifkhanengr]

But I think the problem is with Colab or Colab setting (not with the image path) as I can see all the images of the same notebook (everything is in the same directory as in the Colab) in my local Anaconda Environment.

[rmwkwok]

Seems nobody embeds Google drive’s image like that on Google Colab.

[Deepti_Prasad]

hey try this code if it works

import os
data_dir = "/content/drive/My Drive/Coursera/ML/work/"

but for png images to read in the notebook

there are two ways
1.

from google.colab import files as FILE
import os

images = requests.get(DownURL).content
with open('image_name.png', 'wb') as handler:
    handler.write(img_data)

FILE.download('image_name.png')
os.remove('image_name.png') # to save up space

2nd way (this way but requires you download the dataset and then make temporary file) and then convert the png files into float.

# download the dataset

training_url = "https://storage.googleapis.com/(mention here your laptop or computer path directory and not google drive)-temp/(your required foldername)"

training_file_name = "your required folder name(what you could do here is create a zip file for all the png image files)"

urllib.request.urlretrieve(training_url, training_file_name)

# extract to local directory

training_dir = "/tmp"

zip_ref = zipfile.ZipFile(training_file_name, 'r')

zip_ref.extractall(training_dir)

zip_ref.close()
def map_images(file):
  '''converts the images to floats and normalizes the pixel values'''
  img = tf.io.decode_png(tf.io.read_file(file))
  img = tf.dtypes.cast(img, tf.float32)
  img = img / 255.0

this is just an example.

Try if any of the way works for you.

Regards
DP

[Nafis_Islam]

Thanks a lot! but I have successfully done all the steps that you talked about. but after running this cell, I got this message in google colab, and no interactive nor any non interactive plots were there.

[TMosh]

I’ve added “(Colab tips)” to the thread title, so it might draw more attention. It might bear addition to any of the course FAQs (if not already included).