This paper proposes CAKE (Context-Aware Kernel Evolution), a novel method that leverages large-scale language models (LLMs) to improve the efficiency of Bayesian Optimization (BO). While existing BOs rely on fixed or heuristic kernel selection strategies, which can lead to suboptimal solutions, CAKE adaptively generates and improves GP kernels based on observed data using LLMs. Furthermore, we propose BAKER (BIC-Acquisition Kernel Ranking), a method that selects the most effective kernels by considering the Bayesian Information Criterion (BIC) and the expected improvement. Experimental results demonstrate that CAKE-based BO outperforms existing methods in a variety of real-world problems, including hyperparameter optimization, controller tuning, and photonic chip design.
