[Gemma-3-1B] Gibberish outputs after instruction fine-tuning
I fine-tuned Gemma 3-1B with LoRA on about 3k instruction samples, following Google’s guide. Training finishes without errors, but at inference the model only outputs gibberish. When I fine-tuned the base Gemma (pretrained) on raw data, I didn’t have this problem.
Has anyone seen this issue or know what might be causing it?
Thanks!
Hi @razumelo ,
Welcome to Gemma family of open source models, thanks for reaching out to us. The above mentioned issue might be root caused from a mismatch in the tokenizer and chat template between your fine-tuning data and the model's expected format. Other potential causes include incorrect inference parameters, numerical instability from using fp16 instead of bf16 precision, and data formatting issues. The base Gemma model doesn't have these special tokens, which is why it worked for you.
Thanks.