Bayesian optimisation: ablation study, global performance assessment and improvements based on trust regions
Bayesian Optimisation algorithms (BO) are global optimisation methods that iterate by constructing and using conditional Gaussian processes (GP). It is a common claim that BO is state-of-the-art for costly functions. However, this claim is weakly supported by experimental evidence, as BO is most often compared to itself, rather than to algorithms of different nature.

In this work, we study the performance of BO within the well-known COmparing Continuous Optimizers benchmark (COCO). We first analyse the sensitivity of BO to its own parameters, enabling us to answer general questions regarding the choice of the GP kernel or its trend, the initial GP budget, and the suboptimisation of the acquisition function. Then, we study on which function class and dimension BO is relevant when compared to state-of-the-art optimisers for expensive functions.

The second part of this talk describes a new BO algorithm to improve scalability with dimension, called TREGO (trust-region-like efficient global optimisation). TREGO alternates between regular BO steps and local steps within a trust region. By following a classical scheme for the trust region (based on a sufficient decrease condition), we demonstrate that our algorithm enjoys strong global convergence properties, while departing from EGO only for a subset of optimization steps. The COCO benchmark experiments reveal that TREGO consistently outperforms EGO and closes the performance gap with other state-of-the-art algorithms in conditions (high budget and dimension) for which BO was struggling to compete previously.

Joint work Youssef Diouane, Rodolphe Le Riche, Alexandre Scotto Di Perrotolo.