# Bug C4\_W1\_Ungraded\_Lab\_3\_Bleu\_Score Incorrectly Implemented **URL:** \ **Category:** NLP with Attention Models\ **Tags:** week-module-1\ **Created:** [October 11, 2022, 7:26pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315 "2022-10-11T19:26:42Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Gregory314159](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gregory314159](https://community.deeplearning.ai/u/Gregory314159)\ **Post date:** [October 11, 2022, 7:26pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/1 "2022-10-11T19:26:42Z")
The intent of the lab is to illustrate manual calculation of a BLEU score that is verified with a “sacrebleu” library result. The sacreblue library reports 0.0 for the scores for the two tests illustrated. 0.0 and 0.0 do not compare well with the 27.6 and 35.3 calculated in steps 1-4 done in the lab. The lab has failed in its basic premise. The sacrebleu library has a number of defaults (separate tokenization of capitalized words, base n-gram length) and argument expectations (list of lists) that could lead to the 0 scores with the lab inputs. Looks like the lab code using sacrebleu is incomplete and cannot produce a reasonable result. ---
**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\ **Post date:** [October 12, 2022, 5:51am UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/2 "2022-10-12T05:51:02Z")
Hi @Gregory314159 Could you send me your lab code to check, because I could not replicate your results (my scores are not 0.0 and they compare well with sacrebleu) . Maybe you tinkered the code somewhere? ---
**Author:** ![Thomas1](https://avatars.discourse-cdn.com/v4/letter/t/5f9b8f/32.png) [@Thomas1](https://community.deeplearning.ai/u/Thomas1)\ **Post date:** [October 27, 2022, 11:15am UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/3 "2022-10-27T11:15:28Z")
The values are still 0 print( ``` "Results reference versus candidate 1 our own BLEU implementation: ", round(bleu_score(tokenized_corpus_cand, tokenized_corpus_ref) * 100, 1), ``` ) Results reference versus candidate 1 our own BLEU implementation: 43.6 print( ``` "Results reference versus candidate 1 sacrebleu library BLEU: ", round(sacrebleu.corpus_bleu(wmt19_can_1, wmt19_ref_1).score, 1), ``` ) Results reference versus candidate 1 sacrebleu library BLEU: 0.0 ---
**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\ **Post date:** [October 27, 2022, 11:23am UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/4 "2022-10-27T11:23:01Z")
Hi @Thomas1 Can you send me your notebook for me to check? ---
**Author:** ![Thomas1](https://avatars.discourse-cdn.com/v4/letter/t/5f9b8f/32.png) [@Thomas1](https://community.deeplearning.ai/u/Thomas1)\ **Post date:** [October 27, 2022, 12:51pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/5 "2022-10-27T12:51:16Z")
[C4\_W1\_Ungraded\_Lab\_3\_Bleu\_Score.ipynb](https://community.deeplearning.ai/uploads/short-url/g05WtlZwscQ31H3xNieOLBgB3ax.ipynb) (54.5 KB) Sure ---
**Author:** ![Gregory314159](https://avatars.discourse-cdn.com/v4/letter/g/b77776/32.png) [@Gregory314159](https://community.deeplearning.ai/u/Gregory314159)\ **Post date:** [October 27, 2022, 1:53pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/6 "2022-10-27T13:53:24Z")
hi @arvyzukai It is not the user side error, it is a bug in the lab itself. The supplied lab does not correctly use the sacrebleu library. The supplied lab does not indicate that the user should edit any of the erroneous cells. This error has existed for some time as there are other posts describing it. Interested readers can obviously look up the sacrebleu documentation, but that is not indicated in the notebook in any way. Again, the intent of the lab expressed in the opening cells, is to illustrate that a manual example of calculating a BLUE score compares with the sacrebleu library. It does not compare well at all, due to errors on the authoring (not the student) side of the notebook. Sacre bleu! What a mess. [GitHub - mjpost/sacrebleu: Reference BLEU implementation that auto-downloads test sets and reports a version string to facilitate cross-lab comparisons](https://github.com/mjpost/sacrebleu) nltk BLUE score calculation [NLTK :: nltk.translate.bleu\_score](https://www.nltk.org/_modules/nltk/translate/bleu_score.html) ---
**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\ **Post date:** [October 27, 2022, 3:54pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/7 "2022-10-27T15:54:51Z")
in Step 5 try: ```auto print( "Results reference versus candidate 1 sacrebleu library sentence BLEU: ", #round(sacrebleu.corpus_bleu(candidate_1, reference).score, 1), round(sacrebleu.sentence_bleu(candidate_1, [reference]).score, 1), ) print( "Results reference versus candidate 2 sacrebleu library sentence BLEU: ", #round(sacrebleu.corpus_bleu(candidate_2, reference).score, 1), round(sacrebleu.sentence_bleu(candidate_2, [reference]).score, 1), ) ``` > Results reference versus candidate 1 sacrebleu library sentence BLEU: 27.6 > Results reference versus candidate 2 sacrebleu library sentence BLEU: 35.3 ---
**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\ **Post date:** [October 27, 2022, 4:06pm UTC](https://community.deeplearning.ai/t/bug-c4-w1-ungraded-lab-3-bleu-score-incorrectly-implemented/217315/8 "2022-10-27T16:06:07Z")
hi @Gregory314159 Thank you for noting this bug. I submitted the issue and it will be fixed as soon as possible. Thanks