Has anyone gotten cudf.read_csv to work?

AI Discussions
1

Attempting to open any csv file using cudf.read_csv results in a crash. For example,
taxi2016 = cudf.read_csv("taxi2016.csv")

results in:

RuntimeError: cuFile error at: [/var/home/fraser/anaconda3/envs/rapids-23.12/include/kvikio/file_handle.hpp:178](https://file+.vscode-resource.vscode-cdn.net/var/home/fraser/anaconda3/envs/rapids-23.12/include/kvikio/file_handle.hpp:178): internal error

I have never been able to read a file using cudf 23.12.01.

Polars and Pandas read_csv opens the same file. Does anyone use cudf?

2

Source: Installation Guide - RAPIDS Docs

image
image811×685 62.2 KB

Docker command:

docker run --gpus all --pull always --rm -it \
    --shm-size=1g --ulimit memlock=-1 --ulimit stack=67108864 \
    -p 8888:8888 -p 8787:8787 -p 8786:8786 \
    nvcr.io/nvidia/rapidsai/notebooks:23.12-cuda12.0-py3.10

image

3

Thanks. I have rapids up and running in conda (and distrobox can pull the docker image). Every example works from the cudf notebook examples except the read_csv function. I will try either pandas dataframe to cudf or polars dataframe to cudf as it is only the read_csv function that is crashing. I could be having issues because my Nvidia A1000 is so small but the helpful error message is, well, not telling me anything.

4

Yup, loading from pandas and then converting to cudf works as a workaround: cudf.DataFrame.from_pandas(taxi2016pl).

For Polars, monstrously, one must convert the polars dataframe to pandas before converting to cudf!

# Assuming `df_polars` is your Polars DataFrame
df_pandas = df_polars.to_pandas()
df_cudf = cudf.from_pandas(df_pandas)

I will have to wait for native polars support in cudf as polars is noticeably faster than pandas.

5

unfortunately, using dask results in the same issue as with cudf (dask may be using cudf to read csv).

6

Jumping across dask / pandas & polars is not necessary when there’s a read_csv method on cudf. Docker is pretty much the last thing to do when we’re running on different environments and the logs shared are not informative.

Now is a good time for you to file a github issue / join the rapids discussion forum.
Please reply on this thread once you find a solution. Good luck.