Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Self-alignment is an effective way to reduce the cost of human annotation
while ensuring promising model capability. However, most current methods
complete the data collection and training steps in a single round, which may
overlook the continuously improving ability of self-aligned models. This gives
rise to a key query: What if we do multi-time bootstrapping self-alignment?
Does this strategy enhance model performance or lead to rapid degradation? In
this paper, our pioneering exploration delves into the impact of bootstrapping
self-alignment on large language models. Our findings reveal that bootstrapping
self-alignment markedly surpasses the single-round approach, by guaranteeing
data diversity from in-context learning. To further exploit the capabilities of
bootstrapping, we investigate and adjust the training order of data, which
yields improved performance of the model. Drawing on these findings, we propose
Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
training recipe, we propose SOFT+ which further boost self-alignment's
performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
various classification and generation tasks, highlighting the potential of
bootstrapping self-alignment on continually enhancing model alignment
performance.
65caddae6427380bc2e5d1ad
Haoyu Wang
65caddae6427380bc2e5d1ae
Guozheng Ma
65caddae6427380bc2e5d1af
Ziqiao Meng
65caddae6427380bc2e5d1b0
Zeyu Qin
admin_assigned
2024-02-13T14:00:08.794Z
641a7a8ff9dd6391a2457bbe
https://cdn-avatars.huggingface.co/v1/production/uploads/641a7a8ff9dd6391a2457bbe/_qdCnlrkpTfqZjXY4DFq_.jpeg
Zeyu Qin
user
qqqzzzyyy
65caddae6427380bc2e5d1b1
Li Shen
claimed_verified
2025-01-26T11:40:55.463Z
62de356ad89af8c07209e7d4
/avatars/610629958726b270418368b8b7f61469.svg
Li Shen
user
mathshenli
65caddae6427380bc2e5d1b2
Zhong Zhang
65caddae6427380bc2e5d1b3
Bingzhe Wu
admin_assigned
2024-02-13T14:01:39.393Z
635e7cf8d1c91c2ee1e8d421
/avatars/4ce7699d79cb1f005f95e58763cdf81f.svg
Wu
user
Bingzhe
65caddae6427380bc2e5d1b4
Liu Liu
65caddae6427380bc2e5d1b5
Yatao Bian
admin_assigned
2024-02-13T14:01:59.345Z
6357c85c00f138b8ca550c0e
/avatars/95dad6bb2fe1d31a4e9da0d6281be17b.svg
BlueWhaleLab
user
BlueWhaleLab
65caddae6427380bc2e5d1b6
Tingyang Xu
65caddae6427380bc2e5d1b7
Xueqian Wang
65caddae6427380bc2e5d1b8
Peilin Zhao
Step-On-Feet Tuning: Scaling Self-Alignment of LLMs via Bootstrapping
Self-alignment is an effective way to reduce the cost of human annotation
while ensuring promising model capability. However, most current methods
complete the data collection and training steps in a single round, which may
overlook the continuously improving ability of self-aligned models. This gives
rise to a key query: What if we do multi-time bootstrapping self-alignment?
Does this strategy enhance model performance or lead to rapid degradation? In
this paper, our pioneering exploration delves into the impact of bootstrapping
self-alignment on large language models. Our findings reveal that bootstrapping
self-alignment markedly surpasses the single-round approach, by guaranteeing
data diversity from in-context learning. To further exploit the capabilities of
bootstrapping, we investigate and adjust the training order of data, which
yields improved performance of the model. Drawing on these findings, we propose
Step-On-Feet Tuning (SOFT) which leverages model's continuously enhanced
few-shot ability to boost zero or one-shot performance. Based on easy-to-hard
training recipe, we propose SOFT+ which further boost self-alignment's
performance. Our experiments demonstrate the efficiency of SOFT (SOFT+) across
various classification and generation tasks, highlighting the potential of
bootstrapping self-alignment on continually enhancing model alignment
performance.
Multi-time bootstrapping self-alignment enhances large language model performance by ensuring data diversity and continuously improving few-shot capabilities, as demonstrated through Step-On-Feet Tuning (SOFT) and its enhanced version SOFT+.
bootstrapping self-alignment large language models in-context learning few-shot learning zero-shot learning one-shot learning easy-to-hard training