Dynamic Reinforcement Tokenizer (DRT-01)
Author: SofiTesfay2010 Model Repo: HuggingFace DRT-01
Overview
DRT-01 is a dynamic, context-aware tokenizer trained with reinforcement learning (RL) on large text datasets. Unlike static tokenizers, it segments text based on context embeddings and learns to produce tokenizations that maximize downstream semantic consistency.
It can be applied to any Transformer-based language model for improved compression, semantic segmentation, or as a preprocessing step in RL or SOTA LLM pipelines.
Features
Context-aware segmentation using a Transformer-based policy network.
Continuous memory of previous segments for dynamic tokenization.
Reinforcement-learning-based reward optimizing embedding variance and smoothness.
Auto-save and push checkpoints to HuggingFace Hub.
Compatible with any LLM, including large instruction-tuned models like Qwen/Qwen3.
Can be used as a dynamic preprocessing layer before any LLM, particularly useful for very large instruction-tuned models.