Why use KL divergence in PPO?

[Goomin]

When creating a post, please add:

• Week # must be added in the tags option of the post.
• Link to the classroom item you are referring to: https://www.coursera.org/learn/generative-ai-with-llms/lecture/yPzI4/lab-3-walkthrough 3
• Description (include relevant info but please do not post solution code or your entire notebook):

The PPO algorithm uses clipping to control the policy from changing excessively.

And KL divergence also plays a role related to policy.

Why should use KL divergence when PPO uses clipping?

[gent.spah]

Hello, please check this post also if you may:

KL divergence or trust region?

KL divergence compares old LLM model to new fine tuned LLM model! Trust region is movement allowed for PPO model! The ultimate goal is the same, what happening are different things? Why do we need it? Well it seems the experts who have tested it they needed it! Is it complex? It is, but the whole RLHF, PPO fine tuned LLM system is a complex system with lots of tunings happening!