Unable to run code of serving-llms-efficiently-with-vllm-–-part-ii

Short Course Q&A Fast & Efficient LLM Inference with vLLM
1

Has anyone able to run the code from serving-llms-efficiently-with-vllm-–-part-ii in kaggle or colab. I am unable to complete the steps mentioned in here

2

to run course lab codes one need to match the pre-requisite requirement used to run libraries or module in their local environment and match module versions too.

what is the issue you are encountering, you can post the error you are encountering. please make sure not to post any part of the codes as it’s against code of conduct

3

Hi @Deepti_Prasad , thanks for your response. I was unable to run this below command directly in my kaggle notebook

vllm serve Qwen/Qwen3-0.6B --dtype=bfloat16 --max-model-len 4096

So got to know that I need to run this via python’s subprocess which I used and got the below error

(APIServer pid=1207)            ^^^^
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 154, in __init__
(APIServer pid=1207)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=1207)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1207)     return func(*args, **kwargs)
(APIServer pid=1207)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 128, in make_async_mp_client
(APIServer pid=1207)     return AsyncMPClient(*client_args)
(APIServer pid=1207)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=1207)     return func(*args, **kwargs)
(APIServer pid=1207)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 924, in __init__
(APIServer pid=1207)     super().__init__(
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 583, in __init__
(APIServer pid=1207)     with launch_core_engines(
(APIServer pid=1207)          ^^^^^^^^^^^^^^^^^^^^
(APIServer pid=1207)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=1207)     next(self.gen)
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 972, in launch_core_engines
(APIServer pid=1207)     wait_for_engine_startup(
(APIServer pid=1207)   File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/utils.py", line 1031, in wait_for_engine_startup
(APIServer pid=1207)     raise RuntimeError(
(APIServer pid=1207) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
4

@SachinHatikankan

check the python version used in the lab is matching with version you are running locally.

the python is more pointing on library version mismatch issue and gpu memory usage limitation with api keys.

So when running a model locally, chances are labs used api keys which will be dlai related, but won’t work in your local environment.

I hope you have downloaded all the metadata files from the File==>Open section, and also looked into any requirement.txt.file which will have information regarding all the library version the lab codes were running in dlai learning platform.

Regards
Dr. Deepti