Han, W;
Vargaftik, S;
Mitzenmacher, M;
Karp, B;
Basat, RB;
(2024)
Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression.
In:
HotNets '24: Proceedings of the 23rd ACM Workshop on Hot Topics in Networks.
(pp. pp. 186-194).
ACM (Association for Computing Machinery)
Preview |
PDF
2407.01378v2.pdf - Accepted Version Download (819kB) | Preview |
Abstract
Gradient aggregation has long been identified as a major bottleneck in today’s large-scale distributed machine learning training systems. One promising solution to mitigate such bottlenecks is gradient compression, directly reducing communicated gradient data volume. However, in practice, many gradient compression schemes do not achieve acceleration of the training process while also preserving accuracy. In this work, we identify common issues in previous gradient compression systems and evaluation methodologies. These include excessive computational overheads; incompatibility with all-reduce; and insufficient evaluation methods, such as not using an end-to-end metric or using a 32-bit baseline instead of the stronger 16-bit baseline. We revisit common compression approaches (sparsification, quantization, and low-rank decomposition) and demonstrate how considering the above issues can lead to minor but strategic design changes, resulting in notably better performance. Our goal is to raise awareness of the need for design and evaluation standards that naturally translate to the end-to-end utility of gradient compression.
| Type: | Proceedings paper |
|---|---|
| Title: | Beyond Throughput and Compression Ratios: Towards High End-to-end Utility of Gradient Compression |
| Event: | 23rd ACM Workshop on Hot Topics in Networks |
| ISBN-13: | 9798400712722 |
| Open access status: | An open access version is available from UCL Discovery |
| DOI: | 10.1145/3696348.3696857 |
| Publisher version: | https://doi.org/10.1145/3696348.3696857 |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10204205 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
