Dataset Partitioning Not Working

Short Course Q&A Federated Learning
1

Course : Federated Fine-tuning of LLMs with Private Data

None of the load_datset steps are working also tried updating datasets python module but didnt help.

Federated Fine Tuning for LLMs: Following step is erroring out

partitioner = IidPartitioner(num_partitions=cfg.flower.num_clients)
fds = FederatedDataset(
dataset=cfg.dataset.name,
partitioners={“train”: partitioner}
)

partition_zero = fds.load_partition(0)

format_dataset(partition_zero)

2

“following step is erroring out”

Can you provide more information? A screen capture image that shows the error message would be helpful.

3

image
image2218×982 290 KB

1 Like
4

I get the same error. any solution to this ?

5

Some discussion here: Huggingface dataset install - #2 by John6666 - 🤗Datasets - Hugging Face Forums

But we can’t install updates in course environment, so perhaps just copy and run the code locally until the staff fixes this