Lab 1 - libcublas.so not found & PyExtensionType errors on the ml.m5.2xlarge instance using Python 3 (ipykernel) kernel

Course Q&A Generative AI with Large Language Models
,
1

Hi,

I am experiencing a broken lab environment for Lab 1 (Summarize Dialogue) in the Generative AI with Large Language Models course. I followed every setup step exactly as instructed on the Vocareum lab page and was unable to complete the lab due to environment-level errors that are outside my control as a student.

Setup steps I completed:

  • Clicked Start Lab & waited for the AWS dot to turn green
  • Opened the AWS Management Console via the Vocareum AWS link
  • Navigated to Amazon SageMaker AI > Studio > Open Studio > Launch Personal Studio
  • Opened JupyterLab via the sagemaker-space & clicked Run, then Open
  • Downloaded lab files via the terminal using the provided S3 command
  • Opened Lab_1_summarize_dialogue.ipynb & selected Python 3 (ipykernel) as the kernel

Errors encountered:

Error 1 - Missing CUDA libraries (this is an infrastructure issue, not a code issue):
OSError: libcudart.so.12: cannot open shared object file: No such file or directory
ValueError: libcublas.so.*[0-9] not found in the system path

This error occurs when importing transformers, because PyTorch cannot find the CUDA GPU libraries that should be pre-installed on the ml.m5.2xlarge instance.

Error 2 - PyArrow version incompatibility:
AttributeError: module ‘pyarrow’ has no attribute ‘PyExtensionType’

This error occurs when importing datasets, because the version of pyarrow installed in the environment is incompatible with the version of the datasets library the lab installs.

Error 3 - Irresolvable dependency conflict:
ERROR: ResolutionImpossible: Cannot install datasets==2.14.0, huggingface_hub==0.19.0, and transformers==4.38.2 because these package versions have conflicting dependencies.

When attempting to fix Error 2 by downgrading packages, pip itself reported that the dependency conflicts in the environment cannot be resolved.

Key details:

  • Instance type: ml.m5.2xlarge (as required by the lab)
  • Kernel: Python 3 (ipykernel) (as required by the lab)
  • The lab’s own pip install cell produces these errors on a fresh environment with no modifications by me
  • The notebook’s embedded instructions say to ignore pip warnings, but these are blocking errors that prevent any code from running at all

I was unable to complete the lab. Please advise on whether the lab environment needs to be rebuilt, or whether there is an alternative way to access a working environment.

Thank you

2

Ignore those dependency conflicts and continue running the cells. If any cell throws an error, try rerunning it immediately- sometimes dynamic environment resolution fixes it on the second run(worked for me).

If you still get stuck with the CUDA (libcudart.so.12) or PyArrow errors, you can run this optional fix in a cell to clean up the environment:

%pip install “pyarrow<15.0.0”
%pip install torch==2.5.1 --index-url https://download.pytorch.org/whl/cpu --upgrade