# Question on the loss function of reward model

**URL:** <https://community.deeplearning.ai/t/question-on-the-loss-function-of-reward-model/662573>\
**Category:** Generative AI with Large Language Models\
**Tags:** week-module-3\
**Created:** [July 13, 2024, 2:11pm UTC](https://community.deeplearning.ai/t/question-on-the-loss-function-of-reward-model/662573 "2024-07-13T14:11:03Z")\
**Posts on this page:** 1\
**Showing post:** 2

<div class="post-metadata">

**Author:** ![gent.spah](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/gent.spah/32/136423_2.png) [@gent.spah](https://community.deeplearning.ai/u/gent.spah)\
**Post date:** [July 15, 2024, 10:32am UTC](https://community.deeplearning.ai/t/question-on-the-loss-function-of-reward-model/662573/2 "2024-07-15T10:32:01Z")

</div>

> [@Moli\_Yang](#):
>
> log(sigmoid(r\_j-r\_k))

Hello, I watched the video and couldn’t find where is she giving this formula?

Maybe this post can help you!

> [@Why Log Sigmoid log(σ(r\_j - r\_k)) as loss function to train reward model?](https://community.deeplearning.ai/t/why-log-sigmoid-log-r-j-r-k-as-loss-function-to-train-reward-model/393700):
>
> Hello Gen AI community, I’m struggling to understand why this loss function is used to train the reward model. I’m referring to the “RLHF: Reward model” video at min 1:00 Given we want to train the model to favor completion y\_j as opposed to y\_k, my understanding is that we want to maximize y\_j - y\_k. So we want to get σ(r\_j - r\_k) as close to 1 as possible. But log(σ(r\_j - r\_k)) goes from -∞ to 0 so minimizing it seems to me like it yields the opposite effect of what we want. I would unders…

---

_[View the full topic](https://community.deeplearning.ai/t/question-on-the-loss-function-of-reward-model/662573)._