CLS-RL: Image Classification with Rule-Based Reinforcement Learning
Classification is a core task in machine learning. Recent research has shown
that although Multimodal Large Language Models (MLLMs) are initially poor at
image classification, fine-tuning them with an adequate amount of data can
significantly enhance their performance, making them comparable to SOTA
classification models. However, acquiring large-scale labeled data is
expensive. In this paper, we explore few-shot MLLM classification fine-tuning.
We found that SFT can cause severe overfitting issues and may even degrade
performance over the zero-shot approach. To address this challenge, inspired by
the recent successes in rule-based reinforcement learning, we propose CLS-RL,
which uses verifiable signals as reward to fine-tune MLLMs. We discovered that
CLS-RL outperforms SFT in most datasets and has a much higher average accuracy
on both base-to-new and few-shot learning setting. Moreover, we observed a
free-lunch phenomenon for CLS-RL; when models are fine-tuned on a particular
dataset, their performance on other distinct datasets may also improve over
zero-shot models, even if those datasets differ in distribution and class
names. This suggests that RL-based methods effectively teach models the
fundamentals of classification. Lastly, inspired by recent works in inference
time thinking, we re-examine the `thinking process' during fine-tuning, a
critical aspect of RL-based methods, in the context of visual classification.
We question whether such tasks require extensive thinking process during
fine-tuning, proposing that this may actually detract from performance. Based
on this premise, we introduce the No-Thinking-CLS-RL method, which minimizes
thinking processes during training by setting an equality accuracy reward. Our
findings indicate that, with much less fine-tuning time, No-Thinking-CLS-RL
method achieves superior in-domain performance and generalization capabilities
than CLS-RL.
67dce5afbabeda89ca6071c3
Ming Li
67dce5afbabeda89ca6071c4
Shitian Zhao
claimed_verified
2025-03-21T11:40:22.815Z
62c66504031996c36c86976a
https://cdn-avatars.huggingface.co/v1/production/uploads/62c66504031996c36c86976a/wIq0YJhkWnEhlzsh-TGYO.png
steve z
user
stzhao
67dce5afbabeda89ca6071c5
Jike Zhong
67dce5afbabeda89ca6071c6
Yuxiang Lai
claimed_verified
2025-03-21T11:40:25.107Z
671a569a97cb6656e4859ea8
/avatars/9eb7966b3ff39ee342e9e681256c86a7.svg
Yuxiang Lai
user
yuxianglai117
67dce5afbabeda89ca6071c7
Kaipeng Zhang
claimed_verified
2025-06-02T07:52:10.406Z
65f1713552c38a91e0a445e8
/avatars/47ab3ada51c9b9976ac1cd0c4301c373.svg
kaipeng
user
kpzhang996
CLS-RL: Image Classification with Rule-Based Reinforcement Learning
Classification is a core task in machine learning. Recent research has shown
that although Multimodal Large Language Models (MLLMs) are initially poor at
image classification, fine-tuning them with an adequate amount of data can
significantly enhance their performance, making them comparable to SOTA
classification models. However, acquiring large-scale labeled data is
expensive. In this paper, we explore few-shot MLLM classification fine-tuning.
We found that SFT can cause severe overfitting issues and may even degrade
performance over the zero-shot approach. To address this challenge, inspired by
the recent successes in rule-based reinforcement learning, we propose CLS-RL,
which uses verifiable signals as reward to fine-tune MLLMs. We discovered that
CLS-RL outperforms SFT in most datasets and has a much higher average accuracy
on both base-to-new and few-shot learning setting. Moreover, we observed a
free-lunch phenomenon for CLS-RL; when models are fine-tuned on a particular
dataset, their performance on other distinct datasets may also improve over
zero-shot models, even if those datasets differ in distribution and class
names. This suggests that RL-based methods effectively teach models the
fundamentals of classification. Lastly, inspired by recent works in inference
time thinking, we re-examine the `thinking process' during fine-tuning, a
critical aspect of RL-based methods, in the context of visual classification.
We question whether such tasks require extensive thinking process during
fine-tuning, proposing that this may actually detract from performance. Based
on this premise, we introduce the No-Thinking-CLS-RL method, which minimizes
thinking processes during training by setting an equality accuracy reward. Our
findings indicate that, with much less fine-tuning time, No-Thinking-CLS-RL
method achieves superior in-domain performance and generalization capabilities
than CLS-RL.
CLS-RL, a reinforcement learning-based method using verifiable signals as rewards, outperforms standard fine-tuning for MLLMs in few-shot image classification and demonstrates improved generalization to other datasets, while No-Thinking-CLS-RL further enhances performance with minimal thinking processes during training.
Multimodal Large Language Models few-shot fine-tuning SFT rule-based reinforcement learning CLS-RL few-shot learning zero-shot models free-lunch phenomenon inference time thinking No-Thinking-CLS-RL