LPZero: Language Model Zero-cost Proxy Search from Zero
In spite of the outstanding performance, Neural Architecture Search (NAS) is
criticized for massive computation. Recently, Zero-shot NAS has emerged as a
promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
computational demands. Despite this, existing ZC proxies heavily rely on expert
knowledge and incur significant trial-and-error costs. Particularly in NLP
tasks, most existing ZC proxies fail to surpass the performance of the naive
baseline. To address these challenges, we introduce a novel framework,
LPZero, which is the first to automatically design ZC proxies for
various tasks, achieving higher ranking consistency than human-designed
proxies. Specifically, we model the ZC proxy as a symbolic equation and
incorporate a unified proxy search space that encompasses existing ZC proxies,
which are composed of a predefined set of mathematical symbols. To
heuristically search for the best ZC proxy, LPZero incorporates genetic
programming to find the optimal symbolic composition. We propose a
Rule-based Pruning Strategy (RPS), which preemptively eliminates
unpromising proxies, thereby mitigating the risk of proxy degradation.
Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
superior ranking ability and performance on downstream tasks compared to
current approaches.
67092a3453dcd91dc5585b51
Peijie Dong
claimed_verified
2025-06-02T07:57:07.734Z
6395f845aec00abff778ad31
https://cdn-avatars.huggingface.co/v1/production/uploads/6395f845aec00abff778ad31/bZkAlchSvqER1HgBKmcHI.jpeg
PeijieDong
user
pprp
67092a3453dcd91dc5585b52
Lujun Li
extracted_pending
2024-10-11T13:38:26.191Z
65ff0330d157381f164e009d
/avatars/da502aa33a41f3076cea3def69006db6.svg
luking66
user
luking66
67092a3453dcd91dc5585b53
Xiang Liu
claimed_verified
2024-10-25T09:31:20.929Z
63024676056ec3a2a8714b24
https://cdn-avatars.huggingface.co/v1/production/uploads/1661093436322-noauth.jpeg
Xiang Liu
user
Dominic789654
67092a3453dcd91dc5585b54
Zhenheng Tang
admin_assigned
2024-10-14T12:42:13.582Z
66a4a319a1711696948b045c
/avatars/1d92d57a949332cb8227697b9a0c2f39.svg
Zhenheng Tang
user
coolzhtang
67092a3453dcd91dc5585b55
Xuebo Liu
admin_assigned
2024-10-14T12:42:19.289Z
652fbf9e75f5f7a84e10cb2e
/avatars/73e7af465dbb57826a52e49c3e72f55f.svg
Xuebo Liu
user
SunbowLiu
67092a3453dcd91dc5585b56
Qiang Wang
admin_assigned
2024-10-14T12:42:26.113Z
6625e033f5c285535cc5d113
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/aNWyzeYR3xH1fdqljSTfv.jpeg
qang_wang
user
QiangWang0
67092a3453dcd91dc5585b57
Xiaowen Chu
admin_assigned
2024-10-14T12:42:32.107Z
6676935fcd0b89a0115174b0
/avatars/4caca1b672d29e787814f9a30bf20bcc.svg
Xiaowen Chu
user
wenxinsiju
LPZero: Language Model Zero-cost Proxy Search from Zero
In spite of the outstanding performance, Neural Architecture Search (NAS) is
criticized for massive computation. Recently, Zero-shot NAS has emerged as a
promising approach by exploiting Zero-cost (ZC) proxies, which markedly reduce
computational demands. Despite this, existing ZC proxies heavily rely on expert
knowledge and incur significant trial-and-error costs. Particularly in NLP
tasks, most existing ZC proxies fail to surpass the performance of the naive
baseline. To address these challenges, we introduce a novel framework,
LPZero, which is the first to automatically design ZC proxies for
various tasks, achieving higher ranking consistency than human-designed
proxies. Specifically, we model the ZC proxy as a symbolic equation and
incorporate a unified proxy search space that encompasses existing ZC proxies,
which are composed of a predefined set of mathematical symbols. To
heuristically search for the best ZC proxy, LPZero incorporates genetic
programming to find the optimal symbolic composition. We propose a
Rule-based Pruning Strategy (RPS), which preemptively eliminates
unpromising proxies, thereby mitigating the risk of proxy degradation.
Extensive experiments on FlexiBERT, GPT-2, and LLaMA-7B demonstrate LPZero's
superior ranking ability and performance on downstream tasks compared to
current approaches.
LPZero, a novel framework, automates the design of Zero-cost proxies using genetic programming and achieves better performance than human-designed proxies on various NLP tasks.
Neural Architecture Search Zero-shot NAS Zero-cost proxies symbolic equation proxy search space genetic programming Rule-based Pruning Strategy FlexiBERT GPT-2 LLaMA-7B