Cowen-Rivers, Alexander I;
Lyu, Wenlong;
Tutunov, Rasul;
Wang, Zhi;
Grosnit, Antoine;
Griffiths, Ryan Rhys;
Maravel, Alexandre Max;
... Bou-Ammar, Haitham; + view all
(2022)
HEBO: Pushing The Limits of Sample-Efficient Hyperparameter Optimisation.
Journal of Artificial Intelligence Research
, 74
pp. 1269-1349.
10.1613/jair.1.13643.
Preview |
PDF
sminton,+13643-Article+(PDF)-31227-1-11-20220711.pdf - Published Version Download (950kB) | Preview |
Abstract
In this work we rigorously analyse assumptions inherent to black-box optimisation hyper-parameter tuning tasks. Our results on the Bayesmark benchmark indicate that heteroscedasticity and non-stationarity pose significant challenges for black-box optimisers. Based on these findings, we propose a Heteroscedastic and Evolutionary Bayesian Optimisation solver (HEBO). HEBO performs non-linear input and output warping, admits exact marginal log-likelihood optimisation and is robust to the values of learned parameters. We demonstrate HEBO's empirical efficacy on the NeurIPS 2020 Black-Box Optimisation challenge, where HEBO placed first. Upon further analysis, we observe that HEBO significantly outperforms existing black-box optimisers on 108 machine learning hyperparameter tuning tasks comprising the Bayesmark benchmark. Our findings indicate that the majority of hyper-parameter tuning tasks exhibit heteroscedasticity and non-stationarity, multiobjective acquisition ensembles with Pareto front solutions improve queried configurations, and robust acquisition maximisers afford empirical advantages relative to their non-robust counterparts. We hope these findings may serve as guiding principles for practitioners of Bayesian optimisation.
| Type: | Article |
|---|---|
| Title: | HEBO: Pushing The Limits of Sample-Efficient Hyperparameter Optimisation |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1613/jair.1.13643 |
| Publisher version: | https://doi.org/10.1613/jair.1.13643 |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| Keywords: | Science & Technology, Technology, Computer Science, Artificial Intelligence, Computer Science |
| UCL classification: | UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science UCL > Provost and Vice Provost Offices > UCL BEAMS UCL |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10158552 |
Archive Staff Only
 |
View Item |
