Week 3 lab: upgrade to the new "lighteval" API

Course Q&A Generative AI with Large Language Models
,
1

In week 3’s lab, the old “evaluate” API is used ( GitHub - huggingface/evaluate: 🤗 Evaluate: A library for easily evaluating machine learning models and datasets. · GitHub ). I’d like to upgrade the Jupiter’s notebook, in order to use the new “lighteval” API ( GitHub - huggingface/lighteval: Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends · GitHub ). This should be more efficient, but works according to a different paradigm that is not really clear to me. Can anybody help me to upgrade the notebook to this new API?

2

Hi,

I took a look at the notebook. Sorry for the late reply, I see you opened this question 23d agao, but I just got the email notification today.

I think that lighteval is probably not the right choice for this. evaluate is a library of standalone metrics. Looks like only one of them is used in the this notebook - just the toxicity scorer. But lightevel is a framework for benchmarking a model against a task - its addressing a different situation, even though the names are similar. lighteval does not have an equivalent toxicity scorer.

But evaulation is really just wrapping a call to transformers, so you could drop evaulate entirely and just call the underlying pipeline directly.

So instead of doing this:

# OLD toxicity_evaluator = evaluate.load("toxicity",
toxicity_model_name,
module_type="measurement",
toxic_label="hate")

You could do this:

# NEW — reuses the pipeline already defined in section 2.2 (sentiment_pipe),
# or build a fresh one if you want it self-contained here:
toxicity_evaluator = pipeline("text-classification",
model=toxicity_model_name,
top_k=None,
function_to_apply="softmax",
device=0 if torch.cuda.is_available() else -1)

and then inside evaluate_toxicity, instead of this:

# OLD
toxicity_score = toxicity_evaluator.compute(predictions=[(input_text + " " + generated_text)])
toxicities.extend(toxicity_score["toxicity"])

do this:

# NEW — pipeline returns [[{'label': 'nothate', 'score': ...}, {'label': 'hate', 'score': ...}]]
result = toxicity_evaluator(input_text + " " + generated_text)[0]
hate_score = next(d["score"] for d in result if d["label"] == "hate")
toxicities.append(hate_score)

Hope that helps!

3

Thanks very much!