# Fine tune the mode on GPU **URL:** \ **Category:** Generative AI with Large Language Models\ **Tags:** week-module-2\ **Created:** [July 18, 2023, 9:02pm UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646 "2023-07-18T21:02:28Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Real\_Emily](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@Real\_Emily](https://community.deeplearning.ai/u/Real_Emily)\ **Post date:** [July 18, 2023, 9:02pm UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646/1 "2023-07-18T21:02:28Z")
In Week2’s Lab, section 2.2 - Fine tune the model was run on CPU. I tried to run trainer.train() on my GPU which has 32G memory and I get the OutOfMemoryError message. How to decide the GPU memory size required to fine tune a given model? ---
**Author:** ![Juan\_Olano](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/juan_olano/32/27125_2.png) [@Juan\_Olano](https://community.deeplearning.ai/u/Juan_Olano)\ **Post date:** [July 18, 2023, 9:21pm UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646/2 "2023-07-18T21:21:14Z")
> [@Real\_Emily](#): > > How to decide the GPU memory size required to fine tune a given model? Hi @Real_Emily , I understand that you are doing a full fine-tuning right? To calculate the memory required for this task you have to consider several variables: 1. The size of the base model. That’s for starters the one thing that may consume the most as the weights will have to be loaded into the RAM. 2. Batch size: You are in control of this one. The batch size will also impact the memory usage. The bigger the batch the more memory is needed. 3. Precision: Models come usually in 32bits but you can use quantization to go to 16bits or even 8 bits. This will definitively impact your memory usage, but also the precision. There are other variables that will also impact like gradient accumulation, sequence length, the number of GPUs at your disposal. HAVING SAID ALL THAT… There are other ways to fine tune a model that will use much much less resources. One very well known and recommended is Peft + LoRA. With this, you can get almost the same quality of a full training at a fraction of the resources. I hope this info is useful! Juan ---
**Author:** ![Real\_Emily](https://avatars.discourse-cdn.com/v4/letter/r/49beb7/32.png) [@Real\_Emily](https://community.deeplearning.ai/u/Real_Emily)\ **Post date:** [July 19, 2023, 5:33am UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646/3 "2023-07-19T05:33:51Z")
Thanks @Juan_Olano . Is it possible that a model is so big that the model itself cannot fit into one GPU? My understanding is that in order to use Peft + LoRA, the model itself at least need to be loaded into one GPU. ---
**Author:** ![Juan\_Olano](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/juan_olano/32/27125_2.png) [@Juan\_Olano](https://community.deeplearning.ai/u/Juan_Olano)\ **Post date:** [July 19, 2023, 12:47pm UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646/4 "2023-07-19T12:47:15Z")
Hi @Real_Emily , > [@Real\_Emily](#): > > Is it possible that a model is so big that the model itself cannot fit into one GPU? Yes, definitively. There are small models with, say 1B Params, even probably a 7B param that can fit in 1 GPU, but moving forward you’ll probably need more than on GPU. The large models with 30B, 70B, use several GPUs, and the super models like GPT may be in the order of 100s of GPUs. I’ve done fine tuning using Peft+LoRA in my laptop in a model call DistilBERT which is a small model. But I’ve tried to do it with Bloomz7B1 and it has gone out of memory in my laptop, but when I run it in Colab with 1 GPU, it works. ---
**Author:** ![tbhuy](https://avatars.discourse-cdn.com/v4/letter/t/a3d4f5/32.png) [@tbhuy](https://community.deeplearning.ai/u/tbhuy)\ **Post date:** [August 3, 2023, 11:07pm UTC](https://community.deeplearning.ai/t/fine-tune-the-mode-on-gpu/389646/5 "2023-08-03T23:07:26Z")
I had the same problem. I solved it by adding: per\_device\_train\_batch\_size=2, per\_device\_eval\_batch\_size=2 to training\_args batch\_size = 1 or 2, or 4 should be fine