Denevi, G; Ciliberto, C; Grazzi, R; Pontil, M; (2019) Learning-to-Learn Stochastic Gradient Descent with Biased Regularization. In: Chaudhuri, Kamalika and Salakhutdinov, Ruslan, (eds.) Proceedings of Machine Learning Research - International Conference on Machine Learning, 2019,. PMLR: Long Beach, California, USA.

Preview

Text denevi19a-supp.pdf - Published Version Download (1MB) | Preview

Abstract

We study the problem of learning-to-learn: inferring a learning algorithm that works well on tasks sampled from an unknown distribution. As class of algorithms we consider Stochastic Gradient Descent on the true risk regularized by the square euclidean distance to a bias vector. We present an average excess risk bound for such a learning algorithm. This result quantifies the potential benefit of using a bias vector with respect to the unbiased case. We then address the problem of estimating the bias from a sequence of tasks. We propose a meta-algorithm which incrementally updates the bias, as new tasks are observed. The low space and time complexity of this approach makes it appealing in practice. We provide guarantees on the learning ability of the meta-algorithm. A key feature of our results is that, when the number of tasks grows and their variance is relatively small, our learning-to-learn approach has a significant advantage over learning each task in isolation by Stochastic Gradient Descent without a bias term. We report on numerical experiments which demonstrate the effectiveness of our approach.

Download activity - last month

Export as CSVJSONXML

Download activity - last 12 months

Export as XMLCSVJSON

Downloads by country - last 12 months

Export as CSVJSONXML

Archive Staff Only

View Item