deepseek-ai

/

DeepSeek-V3

like 3.98k

Follow

DeepSeek 104k

Your current tokenizer config:

"unk_token": null

This means there's no defined "unknown token," which is risky — the tokenizer can't handle out-of-vocabulary (OOV) tokens properly.

This update defines the unk_token, enabling the tokenizer to:

1. Prevent crashes or undefined behavior when unknown tokens are encountered.
2. Ensure compatibility with libraries that expect a defined unk_token.