# PEFT-LoRA: model performance

**URL:** <https://community.deeplearning.ai/t/peft-lora-model-performance/461294>\
**Category:** Generative AI with Large Language Models\
**Tags:** week-module-2\
**Created:** [October 3, 2023, 10:06am UTC](https://community.deeplearning.ai/t/peft-lora-model-performance/461294 "2023-10-03T10:06:44Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![David\_Hillmann](https://avatars.discourse-cdn.com/v4/letter/d/3da27b/32.png) [@David\_Hillmann](https://community.deeplearning.ai/u/David_Hillmann)\
**Post date:** [October 3, 2023, 10:06am UTC](https://community.deeplearning.ai/t/peft-lora-model-performance/461294/1 "2023-10-03T10:06:45Z")

</div>

Using Lab2 notebook I tried to experiment with some fine-tuning with LoRA (on GPU) but got much worse ROUGE scores than even the original model. I wonder why.

I was using following training arguments, with ~1250 question-answer pairs for training (10x more than the quick notebook training example):

```auto
peft_training_args = TrainingArguments(
    output_dir=output_dir,
    auto_find_batch_size=True,
    learning_rate=1e-3,
    num_train_epochs=1,
    logging_steps=1,
    max_steps=-1    
)

```

My thoughs are: The lower rank matrices are initialized with random values, so if I don’t train long enough, the addition of those adapters might just add more randomness than knowledge to my model. Any thoughts if I am on the right path here?

If I am right, how should I train (how many epochs? full dataset of 12k pairs?) to achieve similar results as the PEFT-LoRA checkpoint we’re using in the Lab?