# Evaluation methods for a specific task

**URL:** <https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462>\
**Category:** Generative AI with Large Language Models\
**Tags:** week-module-2\
**Created:** [July 20, 2023, 1:20pm UTC](https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462 "2023-07-20T13:20:55Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![marouane\_khoukh](https://avatars.discourse-cdn.com/v4/letter/m/e47774/32.png) [@marouane\_khoukh](https://community.deeplearning.ai/u/marouane_khoukh)\
**Post date:** [July 20, 2023, 1:20pm UTC](https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462/1 "2023-07-20T13:20:55Z")

</div>

I am fine-tuning LLMs on a specific task, but this specific task is not very related to either summarisation or translation…etc and to my understanding, I think that the benchmarks are evaluating tools for general LLMs as LLaMa, Falcon-b… etc which serves multiple tasks. I want to ask if there is an evaluating techniques for specific tasks fine-tuned LLMs and how can I know if my model is doing better on this specific task without judging by myself only throughout the outputs.

---

<div class="post-metadata">

**Author:** ![Juan\_Olano](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/juan_olano/32/27125_2.png) [@Juan\_Olano](https://community.deeplearning.ai/u/Juan_Olano)\
**Post date:** [July 20, 2023, 3:08pm UTC](https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462/2 "2023-07-20T15:08:29Z")

</div>

I’d like to ask:

1. What type of fine tuning are you doing?

2. What is your task? is it classification? other?

Based on that I can probably help.

---

<div class="post-metadata">

**Author:** ![marouane\_khoukh](https://avatars.discourse-cdn.com/v4/letter/m/e47774/32.png) [@marouane\_khoukh](https://community.deeplearning.ai/u/marouane_khoukh)\
**Post date:** [July 20, 2023, 4:14pm UTC](https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462/3 "2023-07-20T16:14:02Z")

</div>

Hi! Thank you for your reply, my task is basically a generation of funny captions, I have a dataset that contains funny captions, I am fine-tuning LLMs to generate funny captions based on that dataset, but I’m only evaluating the output by myself, It would be great if there are some suggestions from your side.

---

<div class="post-metadata">

**Author:** ![Juan\_Olano](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/juan_olano/32/27125_2.png) [@Juan\_Olano](https://community.deeplearning.ai/u/Juan_Olano)\
**Post date:** [July 20, 2023, 6:03pm UTC](https://community.deeplearning.ai/t/evaluation-methods-for-a-specific-task/391462/4 "2023-07-20T18:03:57Z")

</div>

Wow! Interesting task.

“Funny” is very subjective. Unfortunately I don’t have a hint to provide other than the manual supervision.

I have worked extensively in generation of summaries and, while the LLMs do a great job, there’s always a missing key component in the summary that sometimes is the whole key of the document. I have worked in fine tuning my prompts adding edge cases, but still there are new cases.

I find your task somewhat similar to my task as far as measuring the performance. I have not found a way yet to measure the quality of the outputs I create. In fact, when people read the outputs, most say that they are great, but then there’s the keen reader who spots a gap that turns out to be fundamental.

In the "funny’ world, I find probably the same challenge.