Other Models for "Build Your Own Hotdog vs. Not a hotdog"

[adimaunahan]

First of all, thanks for a great course!

I’m currently trying to improve the hotdog classifier code. So I changed this line to use:

hotdog_or_not = outlines.generate.choice(
    vmodel,
    ['hotdog', 'not a hotdog'],
    sampler=greedy(),
)

So it is working but the last image is classified as “hotdog”. So I tried to change the model to “HuggingFaceTB/SmolVLM-500M-Instruct” but the performance got worse: everything is classified as “not a hotdog”.

Can someone please explain why is that the case? Are there other models that can provide better performance than “SmolVLM-256M-Instruct”? Note that I tried these 2 but getting some errors:

• HuggingFaceTB/SmolVLM2-2.2B-Instruct
• openbmb/MiniCPM-Llama3-V-2_5

[Michela_Agostini]

It is not allowed to share solutions shushing_face:.

However, you did a good job to use your code! To solve the issue with the last image, just keep in consideration to set up a strict classification to exactly match one of the two valid answers

answer_regex = r"(hotdog|not a hotdog)"

hotdog_or_not = outlines.generate.text(
    vmodel,
    sampler=greedy(),
    regex=answer_regex
)

Hoping this can help you!

[adimaunahan]

Sorry for a very late reply . I just tested now your code and got this error:

TypeError: text_vision() got an unexpected keyword argument 'regex'

Note that I’m using outlines==0.2.1, similar to the one used in the course. Let me know if there’s another way to make it work. TIA!

[Michela_Agostini]

My snippet code points out that you should set up a strict classification to exactly match one of the two valid answers

The exact code to be enclosed in the script is

# Set up strict classification output
answer_regex = r"(hotdog|not a hotdog)"

# Rebuild constrained generator
hotdog_or_not = outlines.generate.text(
    vmodel,
    sampler=greedy()
)

Hoping this can help you!

[adimaunahan]

Sorry, I still don’t get it from your recent code. Are you suggesting to simply apply the regex to the hotdog_or_not response? Yes, that will work since the last image is classified as an “airplane”. But note that the course is about structured LLM output so I am looking for a solution that will automatically enforce the LLM to simply return hotdog or not hotdog.

Again, as I’ve mentioned, if you change the model to HuggingFaceTB/SmolVLM-500M-Instruct it will say “not a hotdog” for all the images, which is weird since it’s a larger model and should be smarter.

[Michela_Agostini]

Just a quick recap:

STEP1 : Start from the notebook L5: Structured Generation: Beyond JSON that is giving in the DeepLearning and analyze the output (use of vmodel_name = "HuggingFaceTB/SmolVLM-256M !)
IMAGE1 Hotdog CORRECT!
IMAGE2 Not a hotdog CORRECT!
IMAGE3 Hotdog CORRECT!
IMAGE4 Hotdog NO CORRECT!
IMAGE 5 The airplane is flying in the sky NO CORRECT!

STEP2 Using the following snippet code (using a strict configuration output and rewriting the contrained generator)

STEP3 The output of IMAGE 4 and IMAGE 5 will be the following :

Note For sure there will be other ways to solve the issue!