BLEU score

Syllable breakdown

Dictionary meaning

Bilingual Evaluation Understudy — the most widely used automatic metric for evaluating the quality of machine translation and other text generation systems by comparing the system's output against one or more human reference translations; BLEU computes the geometric mean of n-gram precision scores (for n = 1, 2, 3, 4) between the system output and reference translations, with a brevity penalty to discourage short outputs; a BLEU score of 1.0 (or 100) indicates a perfect match with the reference; BLEU was introduced by Papineni et al. in 2002 and became the standard MT evaluation metric because it correlates roughly with human judgements of translation quality and enables reproducible, automatic evaluation; however BLEU has significant limitations: it does not capture fluency, meaning, or pragmatic adequacy; it rewards systems that reproduce the surface form of references even when a different but equally correct paraphrase would score poorly; and it has been shown to correlate poorly with human judgements for modern high-quality MT systems; newer metrics including METEOR, chrF, and BERTScore address some of these limitations

Examples

• The new neural MT system achieved a BLEU score of 34.5 on the WMT English-German test set, surpassing the previous best by 2.3 points.
• Researchers argued that BLEU score improvements no longer reliably predict human preference when comparing state-of-the-art MT systems.