# Lab 2 - What training parameters were used to full train the LoRA tuned model?

**URL:** <https://community.deeplearning.ai/t/lab-2-what-training-parameters-were-used-to-full-train-the-lora-tuned-model/653207>\
**Category:** Generative AI with Large Language Models\
**Tags:** week-module-2\
**Created:** [June 25, 2024, 3:05am UTC](https://community.deeplearning.ai/t/lab-2-what-training-parameters-were-used-to-full-train-the-lora-tuned-model/653207 "2024-06-25T03:05:35Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![ckhordiasma](https://avatars.discourse-cdn.com/v4/letter/c/f04885/32.png) [@ckhordiasma](https://community.deeplearning.ai/u/ckhordiasma)\
**Post date:** [June 25, 2024, 3:05am UTC](https://community.deeplearning.ai/t/lab-2-what-training-parameters-were-used-to-full-train-the-lora-tuned-model/653207/1 "2024-06-25T03:05:35Z")

</div>

I am trying to replicate the results of the pre-trained LoRA optimized model in lab 2 (in my own jupyter environment), but I am not having any success. So far I have tried

1. setting the training set to the full training set instead of every 100th sample
2. setting max\_steps to 100, max epochs to 5

I am next going to try setting the learning rate to 1e-5 instead

Does anyone know what the actual training parameters were to make this model?

---

<div class="post-metadata">

**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\
**Post date:** [June 25, 2024, 10:16am UTC](https://community.deeplearning.ai/t/lab-2-what-training-parameters-were-used-to-full-train-the-lora-tuned-model/653207/2 "2024-06-25T10:16:35Z")

</div>

You mean fine tuning the model before PEFT? We as mentors have only the info that you also have in the class, I would suggest to check section (for the parameters given):

### 2.2 - Fine-Tune the Model with the Preprocessed Dataset

But it also mentions that to fully fine tune the model it will take a few hours so its giving you the final result of the full fine tuning ready to use below!

---

<div class="post-metadata">

**Author:** ![ckhordiasma](https://avatars.discourse-cdn.com/v4/letter/c/f04885/32.png) [@ckhordiasma](https://community.deeplearning.ai/u/ckhordiasma)\
**Post date:** [June 25, 2024, 11:16am UTC](https://community.deeplearning.ai/t/lab-2-what-training-parameters-were-used-to-full-train-the-lora-tuned-model/653207/3 "2024-06-25T11:16:45Z")

</div>

no, I mean fine-tuning the model using PEFT. I know it’s going to take longer, but I wanted practice actually knowing what parameters to use to train a useful model, not just download a pre-trained model.

I ended up doing the following with good results:

- switch to a computer with GPU and modify code to use gpu (involved putting `.to('cuda')` at the end of the model definitions and the tokenizer definitions
- learning rate of 1e-4
- no max steps
- 5 epochs
- full training dataset instead of only 1%
- logging\_steps=100 to reduce console output

This ended up doing 7790 training steps and took an hour and a half using a GPU. Used a g2-standard-8 vm, nvidia\_L4 GPU, and 200GB SSD on google cloud.

```auto

---------------------------------------------------------------------------------------------------
BASELINE HUMAN SUMMARY:
#Person1# teaches #Person2# how to upgrade software and hardware in #Person2#'s system.
---------------------------------------------------------------------------------------------------
ORIGINAL MODEL:
#Person2# is considering upgrading #Person1#'s system to make up their own flyers and banners. #Person1# suggests adding a painting program to the software and upgrading the hardware. #Person2# suggests adding a CD-ROM drive.
---------------------------------------------------------------------------------------------------
PEFT MODEL: #Person2# wants to upgrade #Person2#'s system and hardware. #Person1# recommends adding a painting program to #Person2#'s software and adding a CD-ROM drive.

```