Generative AI with Large Language Models: Week 3

Course Q&A Generative AI with Large Language Models
, ,
1

Generative AI with Large Language Models: Week 3
RLHF: Obtaining feedback from humans

We have prompt dataset, and several prompt samples that are passed to the instruct LLM to obtain completions.
Also we have several human labelers to provide feedback.

I do not quite understand the figure.

In the example we had one prompt and three likely prompt completions.
Prompt: “My house is too hot”
completion 1: “There is nothing you can do about hot houses”.
completion 2: “You can cool your house with air conditioning”.
completion 3: “It is not too hot”.

Does it mean that the same input “My house is too hot” was provided to three different prompts to the same instruct LLM to return the 3 different completions?

Would these two scenarios be different?

  1. same input on the prompt is passed to the same instruct LLM thrice, three different completions are obtained
  2. three different prompts (same inputs) passed to the same instruct LLM once each, three different completions are obtained

What I understand is that there are k different prompts each with the same input and so, we have k different outputs,
and we have multiple human labelers, say, L. So each input gets k*L number of ratings which is then aggregated (averaged) to rank each of the prompts.
Is this correct?

2

As far as I remember here these are the completions that the LLM is trained on right?

Yes depends on the temperature setting of the LLM (or how halucinating you want it to be) it is possible if you change that setting or even if you dont because the LLMs are probabilistic in nature, a probability result will be choosen and might not be the same all the times.

Yes, some kind of majority voting should take place when choosing the correct label!