Rank, Alpha & Epochs

When using PEFT to create a LoRA by fine-tuning on unstructured text, how do you decide on the network rank and alpha? Is 3 epochs generally sufficient for this case?

The video suggested ranks of 4-32, but other places sometimes suggest much higher values of 256 and higher. Should the number of epochs to train for be increased when the ranks increases?

1 Like

First of all welcome to the community. When deciding on the network rank and alpha for fine-tuning a LoRA on unstructured text using PEFT, it is recommended to experiment with different values and evaluate their performance on the task at hand. Generally, lower ranks may be sufficient for smaller datasets or simpler tasks, while higher ranks may be required for larger datasets or more complex tasks.

Regarding the number of epochs to train for, this would also depend on the specifics of the task and the dataset. While 3 epochs may be sufficient for some cases, others may require more epochs to achieve good performance. Increasing the rank of the network may also require more epochs to train effectively. It is recommended to monitor the training progress and evaluate the performance on a validation set to determine if more epochs are needed. I hope I was able to clear your query. Thank you for posting and enjoy the course.