FoNE: Precise Single-Token Number Embeddings via Fourier Features
Large Language Models (LLMs) typically represent numbers using multiple
tokens, which requires the model to aggregate these tokens to interpret
numerical values. This fragmentation makes both training and inference less
efficient and adversely affects the model's performance on number-related
tasks. Inspired by the observation that pre-trained LLMs internally learn
Fourier-like features for number tokens, we propose Fourier Number Embedding
(FoNE), a novel method that directly maps numbers into the embedding space with
their Fourier features. FoNE encodes each number as a single token with only
two embedding dimensions per digit, effectively capturing numerical values
without fragmentation. This compact representation accelerates both training
and inference. Compared to traditional subword and digit-wise embeddings, FoNE
not only reduces computational overhead but also achieves higher accuracy
across various numerical tasks including addition, subtraction and
multiplication. On 6-digit decimal addition, FoNE requires 64times less data
to achieve 99% accuracy than subword and digit-wise embeddings while using
3times and 6times fewer tokens per number, respectively. Furthermore,
FoNE is the only method that yields 100% accuracy on over 100,000 test examples
for addition, subtraction, and multiplication. The codes and visualization are
available at https://fouriernumber.github.io/.
67b2b58f9edebc815a2a938c
Tianyi Zhou
67b2b58f9edebc815a2a938d
Deqing Fu
claimed_verified
2025-02-18T09:32:57.319Z
63c8454e46421a2efe82709d
https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
Deqing Fu
user
deqing
67b2b58f9edebc815a2a938e
Mahdi Soltanolkotabi
67b2b58f9edebc815a2a938f
Robin Jia
67b2b58f9edebc815a2a9390
Vatsal Sharan
FoNE: Precise Single-Token Number Embeddings via Fourier Features
Large Language Models (LLMs) typically represent numbers using multiple
tokens, which requires the model to aggregate these tokens to interpret
numerical values. This fragmentation makes both training and inference less
efficient and adversely affects the model's performance on number-related
tasks. Inspired by the observation that pre-trained LLMs internally learn
Fourier-like features for number tokens, we propose Fourier Number Embedding
(FoNE), a novel method that directly maps numbers into the embedding space with
their Fourier features. FoNE encodes each number as a single token with only
two embedding dimensions per digit, effectively capturing numerical values
without fragmentation. This compact representation accelerates both training
and inference. Compared to traditional subword and digit-wise embeddings, FoNE
not only reduces computational overhead but also achieves higher accuracy
across various numerical tasks including addition, subtraction and
multiplication. On 6-digit decimal addition, FoNE requires 64times less data
to achieve 99% accuracy than subword and digit-wise embeddings while using
3times and 6times fewer tokens per number, respectively. Furthermore,
FoNE is the only method that yields 100% accuracy on over 100,000 test examples
for addition, subtraction, and multiplication. The codes and visualization are
available at https://fouriernumber.github.io/.
Fourier Number Embedding (FoNE) improves numerical task performance and efficiency in large language models by directly mapping numbers to Fourier features in a compact embedding space.
Fourier Number Embedding Fourier features embedding space numerical tasks addition subtraction multiplication subword embeddings digit-wise embeddings