Text Rendering Strategies for Pixel Language Models
Pixel-based language models process text rendered as images, which allows
them to handle any script, making them a promising approach to open vocabulary
language modelling. However, recent approaches use text renderers that produce
a large set of almost-equivalent input patches, which may prove sub-optimal for
downstream tasks, due to redundancy in the input representations. In this
paper, we investigate four approaches to rendering text in the PIXEL model
(Rust et al., 2023), and find that simple character bigram rendering brings
improved performance on sentence-level tasks without compromising performance
on token-level or multilingual tasks. This new rendering strategy also makes it
possible to train a more compact model with only 22M parameters that performs
on par with the original 86M parameter model. Our analyses show that character
bigram rendering leads to a consistently better model but with an anisotropic
patch embedding space, driven by a patch frequency bias, highlighting the
connections between image patch- and tokenization-based language models.
654307824a68d5aeba5b67cc
Jonas F. Lotz
admin_assigned
2023-11-02T13:54:56.838Z
627e4a0b1d60305a5c8fc9fd
https://cdn-avatars.huggingface.co/v1/production/uploads/627e4a0b1d60305a5c8fc9fd/zbkmD2k6bILHjWxgXnsR7.png
Jonas F. Lotz
user
jflotz
654307824a68d5aeba5b67cd
Elizabeth Salesky
admin_assigned
2023-11-02T13:55:05.386Z
60521a52d8c0f8f1ffaaf2cc
https://cdn-avatars.huggingface.co/v1/production/uploads/1659018192837-60521a52d8c0f8f1ffaaf2cc.jpeg
Elizabeth Salesky
user
esalesky
654307824a68d5aeba5b67ce
Phillip Rust
admin_assigned
2023-11-02T13:55:13.328Z
6079adbad2cd8c150e6ae05c
https://cdn-avatars.huggingface.co/v1/production/uploads/1648912870656-6079adbad2cd8c150e6ae05c.jpeg
Phillip Rust
user
plip
654307824a68d5aeba5b67cf
Desmond Elliott
admin_assigned
2023-11-02T13:55:19.519Z
6285f66a7cc3b7bc1b8e7b8e
/avatars/984ae22db7cc885591bc0b5bceffdfbd.svg
Desmond Elliott
user
elliottd
Text Rendering Strategies for Pixel Language Models
Pixel-based language models process text rendered as images, which allows
them to handle any script, making them a promising approach to open vocabulary
language modelling. However, recent approaches use text renderers that produce
a large set of almost-equivalent input patches, which may prove sub-optimal for
downstream tasks, due to redundancy in the input representations. In this
paper, we investigate four approaches to rendering text in the PIXEL model
(Rust et al., 2023), and find that simple character bigram rendering brings
improved performance on sentence-level tasks without compromising performance
on token-level or multilingual tasks. This new rendering strategy also makes it
possible to train a more compact model with only 22M parameters that performs
on par with the original 86M parameter model. Our analyses show that character
bigram rendering leads to a consistently better model but with an anisotropic
patch embedding space, driven by a patch frequency bias, highlighting the
connections between image patch- and tokenization-based language models.
Character bigram rendering improves performance in pixel-based language models, enabling more compact models while maintaining equivalent performance across sentence, token, and multilingual tasks.
pixel-based language models text renderers input patches character bigram rendering sentence-level tasks token-level tasks multilingual tasks patch embedding space tokenization-based language models