Set tokenizer class to `PreTrainedTokenizerFast` df29f94 verified Xenova HF Staff commited on Jul 19, 2024
Set `clean_up_tokenization_spaces` to false 85dd537 verified Xenova HF Staff commited on Jul 19, 2024
Do not add eos token by default (#5) a7b34c2 verified patrickvonplaten Xenova HF Staff commited on Jul 18, 2024
Add note to install transformers from source (#3) 4711abf verified patrickvonplaten Xenova HF Staff commited on Jul 18, 2024
Upload HF-compatible version (#1) 99aa9a7 verified patrickvonplaten Xenova HF Staff commited on Jul 18, 2024
Upload consolidated.safetensors with huggingface_hub 73cf7be verified patrickvonplaten commited on Jul 18, 2024
