Hello, Cedric said in the video (9:20) that each window overlaps with the previous one, but actually there is no overlap in the code (see below) and “target_slice[:, :-trg_len] = -100” does nothing.
def calculate_perplexity(model, tokenizer, dataset, max_tokens=5000, stride=512):
encodings = tokenizer(
"\n\n".join(dataset["text"]),
return_tensors="pt", truncation=True, max_length=max_tokens,
)
input_ids = encodings.input_ids
nlls, prev_end = [], 0
for begin_loc in range(0, input_ids.size(1), stride):
end_loc = min(begin_loc + stride, input_ids.size(1))
trg_len = end_loc - prev_end
input_slice = input_ids[:, begin_loc:end_loc]
target_slice = input_slice.clone()
target_slice[:, :-trg_len] = -100
with torch.no_grad():
loss = model(input_slice, labels=target_slice).loss
nlls.append(loss * trg_len)
prev_end = end_loc
return math.exp(torch.stack(nlls).sum() / prev_end)
The bug is due to this line:
end_loc = min(begin_loc + stride, input_ids.size(1))
stride should be replaced with a new parameter context_len for example, which will be larger than stride. Only with that we can achieve overlapping.
