Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with
  LLMs
The efficiency of Bayesian optimization (BO) relies heavily on the choice of
the Gaussian process (GP) kernel, which plays a central role in balancing
exploration and exploitation under limited evaluation budgets. Traditional BO
methods often rely on fixed or heuristic kernel selection strategies, which can
result in slow convergence or suboptimal solutions when the chosen kernel is
poorly suited to the underlying objective function. To address this limitation,
we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO
with large language models (LLMs). Concretely, CAKE leverages LLMs as the
crossover and mutation operators to adaptively generate and refine GP kernels
based on the observed data throughout the optimization process. To maximize the
power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to
select the most effective kernel through balancing the model fit measured by
the Bayesian information criterion (BIC) with the expected improvement at each
iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO
method consistently outperforms established baselines across a range of
real-world tasks, including hyperparameter optimization, controller tuning, and
photonic chip design. Our code is publicly available at
https://github.com/cake4bo/cake.
68d2acaa0e215259d193b098
Richard Cornelius Suwandi
claimed_verified
2025-09-23T14:24:34.407Z
603bc1a4fd770a9997b57cb9
https://cdn-avatars.huggingface.co/v1/production/uploads/603bc1a4fd770a9997b57cb9/3bXN2XmPxENQ5rpN7ZtPX.png
Richard Cornelius Suwandi
user
richardcsuwandi
68d2acaa0e215259d193b099
Feng Yin
68d2acaa0e215259d193b09a
Juntao Wang
68d2acaa0e215259d193b09b
Renjie Li
68d2acaa0e215259d193b09c
Tsung-Hui Chang
68d2acaa0e215259d193b09d
Sergios Theodoridis
Adaptive Kernel Design for Bayesian Optimization Is a Piece of CAKE with
  LLMs
The efficiency of Bayesian optimization (BO) relies heavily on the choice of
the Gaussian process (GP) kernel, which plays a central role in balancing
exploration and exploitation under limited evaluation budgets. Traditional BO
methods often rely on fixed or heuristic kernel selection strategies, which can
result in slow convergence or suboptimal solutions when the chosen kernel is
poorly suited to the underlying objective function. To address this limitation,
we propose a freshly-baked Context-Aware Kernel Evolution (CAKE) to enhance BO
with large language models (LLMs). Concretely, CAKE leverages LLMs as the
crossover and mutation operators to adaptively generate and refine GP kernels
based on the observed data throughout the optimization process. To maximize the
power of CAKE, we further propose BIC-Acquisition Kernel Ranking (BAKER) to
select the most effective kernel through balancing the model fit measured by
the Bayesian information criterion (BIC) with the expected improvement at each
iteration of BO. Extensive experiments demonstrate that our fresh CAKE-based BO
method consistently outperforms established baselines across a range of
real-world tasks, including hyperparameter optimization, controller tuning, and
photonic chip design. Our code is publicly available at
https://github.com/cake4bo/cake.
Context-Aware Kernel Evolution (CAKE) enhances Bayesian optimization by using large language models to adaptively generate and refine Gaussian process kernels, outperforming traditional methods across various tasks.
Gaussian process Bayesian optimization kernel exploration exploitation Bayesian information criterion expected improvement BIC-Acquisition Kernel Ranking hyperparameter optimization controller tuning photonic chip design