TensorBLEU: Vectorized GPU-based BLEU Score Implementation for
  Per-Sentence In-Training Evaluation
Modern natural language processing models have achieved unprecedented scale,
yet the tools for their evaluation often remain a computational bottleneck,
limiting the pace of research. This is particularly acute for in-training
evaluation metrics, such as per-sentence reward signals in Reinforcement
Learning, which must operate efficiently on batches of token IDs directly on
the GPU. In this paper, we introduce TensorBLEU, a novel implementation of the
BLEU metric designed from the ground up for this specific use case. Our
approach is fully vectorized for GPU-accelerated, per-sentence computation
within PyTorch and introduces a memory-efficient counting mechanism. By
creating a compact, batch-specific dictionary of n-grams using
torch.unique, our method avoids the prohibitive memory costs of
traditional hashing-based vectorization, making it practical for
large-vocabulary models. We benchmark TensorBLEU against NLTK, the standard
library for token-ID-based BLEU calculation on the CPU. Experiments show that
TensorBLEU provides speedups of over 13x on consumer-grade GPUs (NVIDIA T4) and
exceeding 40x on data-center-class hardware (NVIDIA A100). This performance
transforms a significant bottleneck into a negligible part of the training
loop. By clearly defining its role as a "Token-ID BLEU" for development
purposes and open-sourcing our implementation, we provide a powerful tool for
accelerating research in areas like RL-based model fine-tuning.
68e63237975ac4c405ef220b
Adam Filipek
claimed_verified
2025-10-08T11:48:51.316Z
675197c3ae96d7ba4b4a6c66
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/I2GHgrv70cfT8C5EbK6Q5.png
Adam Filipek
user
AdamF92
TensorBLEU: Vectorized GPU-based BLEU Score Implementation for
  Per-Sentence In-Training Evaluation
Modern natural language processing models have achieved unprecedented scale,
yet the tools for their evaluation often remain a computational bottleneck,
limiting the pace of research. This is particularly acute for in-training
evaluation metrics, such as per-sentence reward signals in Reinforcement
Learning, which must operate efficiently on batches of token IDs directly on
the GPU. In this paper, we introduce TensorBLEU, a novel implementation of the
BLEU metric designed from the ground up for this specific use case. Our
approach is fully vectorized for GPU-accelerated, per-sentence computation
within PyTorch and introduces a memory-efficient counting mechanism. By
creating a compact, batch-specific dictionary of n-grams using
torch.unique, our method avoids the prohibitive memory costs of
traditional hashing-based vectorization, making it practical for
large-vocabulary models. We benchmark TensorBLEU against NLTK, the standard
library for token-ID-based BLEU calculation on the CPU. Experiments show that
TensorBLEU provides speedups of over 13x on consumer-grade GPUs (NVIDIA T4) and
exceeding 40x on data-center-class hardware (NVIDIA A100). This performance
transforms a significant bottleneck into a negligible part of the training
loop. By clearly defining its role as a "Token-ID BLEU" for development
purposes and open-sourcing our implementation, we provide a powerful tool for
accelerating research in areas like RL-based model fine-tuning.
TensorBLEU is a GPU-accelerated BLEU metric implementation for efficient in-training evaluation of natural language processing models, offering significant speedups over CPU-based methods.
TensorBLEU BLEU metric GPU-accelerated per-sentence computation PyTorch memory-efficient n-grams torch.unique token-ID BLEU RL-based model fine-tuning