I ran the code-lab provided for the lecture “DPO in Practice”. The end-result post DPO is not what demonstrated in the lecture. The model expected to remember it’s identity as Deep Qwen instead of Qwen post DPO, but the model goes ahead and respond with a different identity post DPO for each prompt.
I initially tried the code-lab for Qwen 0.5B model, since it looked fairly simple. But the model responded with some gibberish responses post DPO. Hence came back to the code-lab present in the lecture and ran it directly (without any modification) and got these results.
Is there any change in the HF TRL DPO libraries on how they expect the dataset to be provided with the rejected and chosen options? Or is it something else? Please help!
Hi Saurav,
Two separate models are at play in this notebook, and that’s likely the source of confusion.
The notebook trains DPO on SmolLM2-135M-Instruct, a tiny 135M param model, using only 100 samples (5 if on CPU). This model is way too small and undertrained to reliably learn the identity shift. The gibberish/random identities you’re seeing from it are expected.
The actual “Deep Qwen” results shown in the lecture come from a pre-trained model (banghua/Qwen2.5-0.5B-DPO) that was fully trained on the complete 1000-sample dataset with a GPU. The notebook even has a note about this:
“Due to limited computational resources, we used a small model and dataset for DPO training. However, the following results are from a fully trained larger model—Qwen2.5-0.5B—to demonstrate the complete outcome of the DPO process.”
To see the correct “Deep Qwen” results, make sure fully_trained_qwen = True (the default) in the last code cell. That loads the pre-trained Qwen DPO model instead of evaluating the SmolLM you just trained in-notebook.
If it’s already True and you’re still not getting the right outputs, do a Kernel → Restart & Run All to clear stale state.
Regarding your local attempt with Qwen 0.5B, that needs a GPU with enough VRAM. On CPU, USE_GPU = False cuts the dataset down to just 5 samples, which isn’t enough for any meaningful learning.
So nothing is broken with the lab or the TRL library. It’s just the small model + limited data producing noisy results, which is expected.
2 Likes
Thanks for the response and I understand that.
I trained Qwen 0.5B on a GPU for 1000 rows and the result was gibberish too! Can you share me the notebook which was followed to reproduce the “banghua/Qwen2.5-0.5B-DPO” model?
Testing the already RLed model from huggingface doesn’t add much to the learnings. The core aim for the learners should be to reproduce what’s taught in the lecture and for that having the access to the notebook will be helpful.
A few screenshots in addition to the clear 
@arman101’s explanation. As I understand, the matter of the initial concern was “Qwen” vs “Deep Qwen” identities, which was initially illustrated in the first half of the notebook (and which I’ve just successfully re-ran 
) with the “Qwen2.5-0.5B-Instruct” and “Qwen2.5-0.5B-DPO” correspondently:
and then after DPO Training segment title (in the last code cell):
Please, pay extra attention to the @arman101’s words about fully_trained_qwen = True, if you experimented with fully_trained_qwen = False (as I did), you would get something like:
2 Likes
Hi @Saurav_Prateek,
Although, as we’ve already discussed, post-training of the entire “Qwen/Qwen2.5-0.5B-Instruct” model is beyond the scope of this Short Course (“due to limited computational resources”), I decided to take on the challenge and created a Jupyter Notebook with the requested process for you: qwen2-5-0-5b-instruct-dpo-training.ipynb (37.7 KB)
I built it for free on the Kaggle platform, basing it on the Jupyter Notebook from the course’s lab and trying to make as few changes as possible.
And I have two insights, which I would like to share with you:
- (the main one) Building and/or post-training models remains, to a large extent, an art (as well as a science) — it involves applying appropriate optimization methods and techniques and, in particular, finding the optimal set of hyperparameters. It was a key factor for making DPO work on “Qwen/Qwen2.5-0.5B-Instruct” model.
Fortunately, DeepLearning.AI platform has several courses about this matter. Personally, I really like these two programs: Deep Learning Specialization and PyTorch for Deep Learning.
- (a fun fact worth mentioning) The lab’s task can be solved exclusively using SFT. Here is the proof: qwen2-5-0-5b-instruct-sft.ipynb (20.8 KB)
Disclaimer: Most of the intellectual work involved in these exercises was performed by the GPT-5.5 model.
2 Likes
