Translation Quality

We measure translation quality on a public test set with human reference translations, so anyone can check the numbers. The model and decoding settings are the ones that serve POST /v1/translations: NLLB-200 distilled 1.3B, served with CTranslate2 at int8 precision, greedy decoding.

Results

FLORES-200 devtest split: 1,012 sentences per language, translated by professionals from the same English source.

Pair	chrF++	BLEU
Portuguese → English	70.04	52.96
English → Portuguese	66.64	50.89
Portuguese → French	61.54	44.98
Portuguese → German	54.14	34.20
English → Spanish	52.87	31.80
Portuguese → Italian	52.10	31.40
Spanish → Portuguese	51.90	31.59
Portuguese → Spanish	50.64	29.06
Portuguese → Japanese	20.72	13.47
Portuguese → Chinese	15.18	16.00
Mean of 10 pairs	49.58	33.64

Measured in October 2026.

What the numbers mean

chrF++ compares the translation with the human reference by character sequences and word pairs, on a 0 to 100 scale. It rewards a translation that uses a different but close form of a word, so it suits languages with rich inflection such as Portuguese.
BLEU counts the word sequences shared with the reference, also 0 to 100. It is the oldest metric, so it compares with more published results, but it punishes any wording that differs from the one reference.
Neither is a percentage of correct sentences. A good translation can use different words from the reference and still lose points. Compare scores only on the same test set and the same scoring tool.
Japanese and Chinese are weak. Expect usable gist, not publishable text.

Same quality as the full-precision model

We serve the model at int8 precision because it is about 10 times faster on the same GPU. Translating the same 10,120 sentences with the original full-precision model (Hugging Face transformers, fp32) gives a mean of 49.63 chrF++ and 33.69 BLEU against our 49.58 and 33.64. No pair differs by more than 0.23 chrF++ or 0.21 BLEU.

How we measure

Data. The FLORES-200 devtest split in full for each of the 10 pairs above: 1,012 sentences each, 10,120 in total. FLORES-200 is published under CC-BY-SA 4.0.
Translation. The sentences go through the model in batches, as the API sends them: NLLB-200 distilled 1.3B converted to CTranslate2 int8_float16, beam size 1, the target language forced as the first token.
Scoring. Corpus-level scores with sacreBLEU: corpus_chrf with word order 2 (chrF++) and corpus_bleu with the flores200 tokenizer.

Known limits

One sentence at a time. The model translates each sentence on its own, so a pronoun or a term that depends on an earlier sentence can come out differently from one sentence to the next.
Register. Greetings and informal phrases can come out more casual than the original ("Olá, tudo bem?" → "Hey, you all right?").
No glossary. Product names and terms are translated when the model knows a translation for them.
Other languages. 200 languages are supported, but these numbers are for the 10 pairs above only.

Reproduce it

Everything above is public: the dataset, the model and the scoring library. Send each FLORES-200 devtest sentence to the API with the pair's source and targets, then score the output with sacreBLEU as described. The 10 pairs together are about 1.43 million source characters; at $1 per million per target language, a full run costs about $1.43.