Question on the loss function of reward model

[gent.spah]

@Moli_Yang

log(sigmoid(r_j-r_k))

Hello, I watched the video and couldn’t find where is she giving this formula?

Maybe this post can help you!

Why Log Sigmoid log(σ(r_j - r_k)) as loss function to train reward model?

Hello Gen AI community, I’m struggling to understand why this loss function is used to train the reward model. I’m referring to the “RLHF: Reward model” video at min 1:00 Given we want to train the model to favor completion y_j as opposed to y_k, my understanding is that we want to maximize y_j - y_k. So we want to get σ(r_j - r_k) as close to 1 as possible. But log(σ(r_j - r_k)) goes from -∞ to 0 so minimizing it seems to me like it yields the opposite effect of what we want. I would unders…