Question on the loss function of reward model

[Moli_Yang]

Hi,
I’m taking week 3 lecture " [RLHF: Obtaining feedback from humans](https://www.coursera.org/learn/generative-ai-with-llms/lecture/lQBGW/rlhf-obtaining-feedback-from-humans#".

In the lecture, teacher says that the loss function of reward model is log(sigmoid(r_j-r_k)). But i think the loss function should be -log(sigmoid(r_j-r_k)) when preferred completion is y_j. Because sigmoid function should be closer to 1 when r_j - r_k > 0 and then if we want to minimize the log function we should add minus sign, which should be -log(sigmoid(r_j-r_k)).

I don’t know if this understanding is correct, or if there’s an alternative explanation for the formulation of the loss function as presented in the lecture?

[gent.spah]

@Moli_Yang

log(sigmoid(r_j-r_k))

Hello, I watched the video and couldn’t find where is she giving this formula?

Maybe this post can help you!

Why Log Sigmoid log(σ(r_j - r_k)) as loss function to train reward model?

Hello Gen AI community, I’m struggling to understand why this loss function is used to train the reward model. I’m referring to the “RLHF: Reward model” video at min 1:00 Given we want to train the model to favor completion y_j as opposed to y_k, my understanding is that we want to maximize y_j - y_k. So we want to get σ(r_j - r_k) as close to 1 as possible. But log(σ(r_j - r_k)) goes from -∞ to 0 so minimizing it seems to me like it yields the opposite effect of what we want. I would unders…