# Week3 lab, the part given to the reward model using human feedback **URL:** \ **Category:** Generative AI with Large Language Models\ **Tags:** faq, week-module-3\ **Created:** [April 17, 2024, 2:39am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489 "2024-04-17T02:39:21Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [April 17, 2024, 2:39am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/1 "2024-04-17T02:39:21Z")
When creating a post, please add: - Week # must be added in the tags option of the post. - Link to the classroom item you are referring to: [https://www.coursera.org/learn/generative-ai-with-llms/lecture/yPzI4/lab-3-walkthrough](https://www.coursera.org/learn/generative-ai-with-llms/lecture/yPzI4/lab-3-walkthrough) - Description (include relevant info but please do not post solution code or your entire notebook): Hello, I am taking week3 lab. I am working in a lab that practices RLHF. Is there a part where human feedback is delivered to the reward model? From what I heard in the lecture, I understood that Human feedback was received, passed on to the reward model, and then the reward model learned it. But I don’t know where this part is in the lab. Or, just use something called scaling-human-feedback in this exercise? ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [April 17, 2024, 6:33am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/2 "2024-04-17T06:33:24Z")
The reward model (as far as I remember now) is already trained with human feedback outside the lab! ---
**Author:** ![Charlie\_DataScience](https://avatars.discourse-cdn.com/v4/letter/c/bcef8e/32.png) [@Charlie\_DataScience](https://community.deeplearning.ai/u/Charlie_DataScience)\ **Post date:** [April 17, 2024, 6:59pm UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/3 "2024-04-17T18:59:20Z")
To complement the answer, yes the reward model is a pre-trained model that used human feedback to prepare it. You can actually choose any reward model suited for your needs when performing reinforcement learning, and can be even train your own reward model if you want to further customize your results. ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 28, 2024, 4:52am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/4 "2024-05-28T04:52:40Z")
hello. I have more questions, so I’m leaving a reply. Can the reward model be any language model? ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 28, 2024, 8:09am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/5 "2024-05-28T08:09:22Z")
It has to be in the context of what you are training! ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 28, 2024, 8:57am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/6 "2024-05-28T08:57:54Z")
understood. But what I mean is this: RoBERTa is a language model that applies the rewards of common languages. But when training a reward model, which base reward model should you use? Can all language models be reward models? ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 28, 2024, 9:10am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/7 "2024-05-28T09:10:11Z")
Yes but as far as I understand one model whatever it be, has a training dataset and that cannot be universal in order to be applied to all language models ie. that reward model and main LMM need to be trained on the same purpose, same corpus etc. ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 29, 2024, 1:18am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/8 "2024-05-29T01:18:08Z")
Thanks!! ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 29, 2024, 8:35am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/9 "2024-05-29T08:35:50Z")
Sorry, I have another question. What value model does the lab use? From what I understand, it is as follows: Reference Model (lora), PPO Model (lora), Reward Model My head hurts, it’s hard to understand, please help me ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 29, 2024, 8:54am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/10 "2024-05-29T08:54:59Z")
The base model is model\_name=“google/flan-t5-base”, then you load the peft\_model = PeftModel.from\_pretrained with Lora configurations so it can be trained further. The peft model is an adaptive extended trained model that you have previously trained on the main model above. Then, “In this lab, you are preparing to fine-tune the LLM using Reinforcement Learning (RL). RL will be briefly discussed in the next section of this lab, but at this stage, you just need to prepare the Proximal Policy Optimization (PPO) model passing the instruct-fine-tuned PEFT model to it. PPO will be used to optimize the RL policy against the reward model.” ppo\_model = AutoModelForSeq2SeqLMWithValueHead.from\_pretrained(peft\_model, torch\_dtype=torch.bfloat16, is\_trainable=True) ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 29, 2024, 9:24am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/11 "2024-05-29T09:24:31Z")
So what I understand is as follows. If not, please let me know. The base model is `FLAN-T5`. And we download the PEFT (Lora) model trained in the previous lab from s3. Then we inject Lora config into the base model to check trainable parameters. And we create `peft_model` by combining the parameters with the lora (adapter) downloaded from s3. And we make this `peft_model` into `ppo_model`. Then create `ref_modeld` based on `ppo_model`. Did I understand this correctly? And we use the roberta model as the reward model and we did not train it separately. This is a reward model that has already been trained externally. Did I understand this correctly? ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 29, 2024, 11:26am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/12 "2024-05-29T11:26:40Z")
Yes the reinforcement model is trained separately and is another model on the side checking our main model. And we make this `peft_model` into `ppo_model`. No! 2 models are running here, the main model and the reinforcement model! I think you should re-go into the classes one more time, its been a long time I have done this as well! ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 30, 2024, 12:44am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/13 "2024-05-30T00:44:53Z")
## ![스크린샷 2024-05-30 오전 10.08.18](https://global.discourse-cdn.com/dlai/original/3X/5/0/502f8c1fea6a45e1836276d15ae8ca35b45f721e.png) ## ![스크린샷 2024-05-30 오전 9.44.11](https://global.discourse-cdn.com/dlai/original/3X/c/0/c0fff9e9b606d20b8ec61b48d2c9308dcfd66fbd.png) ## ![스크린샷 2024-05-30 오전 10.08.40](https://global.discourse-cdn.com/dlai/original/3X/0/c/0c79643dd144bdd8183100277a652e6def94bd9c.png) But look, it seems like the peft model is used as the ppo model. ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 30, 2024, 11:09am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/14 "2024-05-30T11:09:17Z")
No the peft model is an input to the ppo model! ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 30, 2024, 11:37am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/15 "2024-05-30T11:37:37Z")
Even after re-watching the lecture and re-reading the code, I still don’t understand what you’re saying. In the picture I uploaded above, the peft adapter is used in a variable called ppo model. But I don’t understand why you say no. ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [May 30, 2024, 12:03pm UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/16 "2024-05-30T12:03:44Z")
I have corrected my statement above, If you still going though the lectures and dont understand whats going on and to be honest with you I am not better than the teaching instructors, maybe this is way over your head. ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [May 30, 2024, 12:24pm UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/17 "2024-05-30T12:24:44Z")
Oh yeah yeah yeah I understand now. I’m Korean and don’t know English well, so I use a translator, but there was a misunderstanding here. ---
**Author:** ![Goomin](https://avatars.discourse-cdn.com/v4/letter/g/91b2a8/32.png) [@Goomin](https://community.deeplearning.ai/u/Goomin)\ **Post date:** [June 3, 2024, 7:46am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/18 "2024-06-03T07:46:29Z")
can i ask other question? The PPO algorithm uses clipping to control the policy from changing excessively. And KL divergence also plays a role related to policy. Why should use KL divergence when PPO uses clipping? ---
**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\ **Post date:** [June 4, 2024, 6:52am UTC](https://community.deeplearning.ai/t/week3-lab-the-part-given-to-the-reward-model-using-human-feedback/611489/19 "2024-06-04T06:52:36Z")
KL divergence is a calculated quantity using a formula, clipping is just cut-off they are not of the same nature. It like normal loss plus regularization, and why not in the first place use them both if they are effective!