T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for
  Memory-Efficient Embeddings
Tokenizers are crucial for encoding information in Large Language Models, but
their development has recently stagnated, and they contain inherent weaknesses.
Major limitations include computational overhead, ineffective vocabulary use,
and unnecessarily large embedding and head layers. Additionally, their
performance is biased towards a reference corpus, leading to reduced
effectiveness for underrepresented languages.
  To remedy these issues, we propose T-FREE, which directly embeds words
through sparse activation patterns over character triplets, and does not
require a reference corpus. T-FREE inherently exploits morphological
similarities and allows for strong compression of embedding layers. In our
exhaustive experimental evaluation, we achieve competitive downstream
performance with a parameter reduction of more than 85% on these layers.
Further, T-FREE shows significant improvements in cross-lingual transfer
learning.
667e76ecf0754a8a1d55471d
Björn Deiseroth
667e76ecf0754a8a1d55471e
Manuel Brack
claimed_verified
2024-06-30T20:57:29.924Z
62fa1d95e8c9c532aa75331c
https://cdn-avatars.huggingface.co/v1/production/uploads/62fa1d95e8c9c532aa75331c/WFfk_n8gOj845pSkfdazA.jpeg
Manuel Brack
user
mbrack
667e76ecf0754a8a1d55471f
Patrick Schramowski
667e76ecf0754a8a1d554720
Kristian Kersting
667e76ecf0754a8a1d554721
Samuel Weinbach
T-FREE: Tokenizer-Free Generative LLMs via Sparse Representations for
  Memory-Efficient Embeddings
Tokenizers are crucial for encoding information in Large Language Models, but
their development has recently stagnated, and they contain inherent weaknesses.
Major limitations include computational overhead, ineffective vocabulary use,
and unnecessarily large embedding and head layers. Additionally, their
performance is biased towards a reference corpus, leading to reduced
effectiveness for underrepresented languages.
  To remedy these issues, we propose T-FREE, which directly embeds words
through sparse activation patterns over character triplets, and does not
require a reference corpus. T-FREE inherently exploits morphological
similarities and allows for strong compression of embedding layers. In our
exhaustive experimental evaluation, we achieve competitive downstream
performance with a parameter reduction of more than 85% on these layers.
Further, T-FREE shows significant improvements in cross-lingual transfer
learning.
T-FREE, a novel tokenizer using sparse activation patterns for character triplets, improves performance and reduces parameters in embeddings, enhancing cross-lingual transfer learning.
tokenizers Large Language Models computational overhead vocabulary use embedding layers head layers reference corpus sparse activation patterns character triplets morphological similarities parameter reduction cross-lingual transfer learning