Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
We propose a novel prompt design paradigm that challenges conventional wisdom
in large language model (LLM) prompting. While conventional wisdom prioritizes
well-crafted instructions and demonstrations for in-context learning (ICL), we
show that pruning random demonstrations into seemingly incoherent "gibberish"
can remarkably improve performance across diverse tasks. Notably, the
"gibberish" always matches or surpasses state-of-the-art automatic prompt
optimization techniques, achieving substantial gains regardless of LLM
alignment. Nevertheless, discovering an effective pruning strategy is
non-trivial, as existing attribution methods and prompt compression algorithms
fail to deliver robust results, let alone human intuition. In terms of this, we
propose a self-discover prompt optimization framework, PromptQuine, an
evolutionary search framework that automatically searches for the pruning
strategy by itself using only low-data regimes. Much like the emergent
complexity in nature--such as symbiosis and self-organization--arising in
response to resource constraints, our framework evolves and refines
unconventional yet highly effective prompts by leveraging only the tokens
present within the context. We demonstrate its effectiveness across
classification, multi-choice question answering, generation and math reasoning
tasks across LLMs, while achieving decent runtime efficiency. We hope our
findings can guide mechanistic studies on in-context learning, and provide a
call to action, to pave the way for more open-ended search algorithms for more
effective LLM prompting.
68634a4e588cea0da970c8ba
Jianyu Wang
claimed_verified
2025-07-01T10:13:27.023Z
61e09ec13a1781f66b4e9ae2
https://cdn-avatars.huggingface.co/v1/production/uploads/1642110635503-noauth.jpeg
Jianyu Wang
user
Jianyu
68634a4e588cea0da970c8bb
Zhiqiang Hu
claimed_verified
2025-07-01T10:13:31.161Z
637f228152229c63921119c3
https://cdn-avatars.huggingface.co/v1/production/uploads/637f228152229c63921119c3/acwXorra1r9_7i3KlBFjS.jpeg
Zhiqiang Hu
user
Zhiqiang007
68634a4e588cea0da970c8bc
Lidong Bing
claimed_verified
2025-07-01T10:13:29.213Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
Evolving Prompts In-Context: An Open-ended, Self-replicating Perspective
We propose a novel prompt design paradigm that challenges conventional wisdom
in large language model (LLM) prompting. While conventional wisdom prioritizes
well-crafted instructions and demonstrations for in-context learning (ICL), we
show that pruning random demonstrations into seemingly incoherent "gibberish"
can remarkably improve performance across diverse tasks. Notably, the
"gibberish" always matches or surpasses state-of-the-art automatic prompt
optimization techniques, achieving substantial gains regardless of LLM
alignment. Nevertheless, discovering an effective pruning strategy is
non-trivial, as existing attribution methods and prompt compression algorithms
fail to deliver robust results, let alone human intuition. In terms of this, we
propose a self-discover prompt optimization framework, PromptQuine, an
evolutionary search framework that automatically searches for the pruning
strategy by itself using only low-data regimes. Much like the emergent
complexity in nature--such as symbiosis and self-organization--arising in
response to resource constraints, our framework evolves and refines
unconventional yet highly effective prompts by leveraging only the tokens
present within the context. We demonstrate its effectiveness across
classification, multi-choice question answering, generation and math reasoning
tasks across LLMs, while achieving decent runtime efficiency. We hope our
findings can guide mechanistic studies on in-context learning, and provide a
call to action, to pave the way for more open-ended search algorithms for more
effective LLM prompting.
A novel prompt optimization framework, PromptQuine, improves LLM performance by pruning random demonstrations into effective prompts using evolutionary search in low-data regimes.
prompt design large language model (LLM) in-context learning (ICL) pruning gibberish prompt optimization self-discover framework evolutionary search tokens classification multi-choice question answering generation math reasoning