Schwarz, Jonathan Richard;
(2023)
Sparse Parameterisations For Efficient Machine Learning Algorithms.
Doctoral thesis (Ph.D), UCL (University College London).
Preview |
Text
Thesis - Jonathan Richard Schwarz.pdf - Accepted Version Download (30MB) | Preview |
Abstract
Progress in Machine Learning is being driven by continued growth in model size, training data and algorithmic innovations relying on access to high-performance computing clusters. While this paradigm has dominated with the advent of modern Deep Representation Learning, concerns over practical limitations are becoming increasingly common: Independent and identically distributed training on large datasets is prohibitively expensive for all but a handful of institutions, reducing participation and the pace of innovation. Even in cases where advanced training hardware is readily available, inference must often be carried out on limited hardware, posing the challenge of reducing model capacity once the functional relationship of a learning problem has been extracted. In addition, this paradigm conflicts with the inherent nature of real applications, where data is collected sequentially and thus Continual Learning is required. Hence, focus on efficiency must not merely be a burden or constraint but can instead both incentivise and benefit from knowledge transfer, leading to better generalisation. Finally, in an age of increasing concern about the environmental footprint of technology, reductions in computational requirements are not merely a cost-saving endeavour but critical to the long-term progress of the field. In this thesis, we develop algorithmic approaches towards increasing the efficiency of Machine Learning by utilising sparse parameterisations and formalising our techniques as operating on the explicit or tacit notion of several tasks. These principles will allow us to devise tractable computational procedures ranging from identifying the most informative subsets of data over drastic reductions in model size without performance loss to finding parameters with the highest plasticity, allowing the rapid adaptation to a task through Meta Learning. Finally, while covering various practical problems throughout the thesis, we will emphasise applications to data compression, a high-impact problem uniquely encapsulating both requirements and promises of Efficient Machine Learning.
| Type: | Thesis (Doctoral) |
|---|---|
| Qualification: | Ph.D |
| Title: | Sparse Parameterisations For Efficient Machine Learning Algorithms |
| Open access status: | An open access version is available from UCL Discovery |
| Language: | English |
| Additional information: | Copyright © The Author 2022. Original content in this thesis is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International (CC BY-NC 4.0) Licence (https://creativecommons.org/licenses/by-nc/4.0/). Any third-party copyright material present remains the property of its respective owner(s) and is licensed under its existing terms. Access may initially be restricted at the author’s request. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10180580 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
