Cunningham, Jake;
Giannone, Giorgio;
Zhang, Mingtian;
Deisenroth, Marc Peter;
(2024)
Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling.
In: Globersons, Amir and Mackey, Lester and Belgrave, Danielle and Fan, Angela and Paquet, Ulrich and Tomczak, Jakub M and Zhang, Cheng, (eds.)
Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS 2024).
(pp. pp. 1-32).
NeurIPS
Preview |
Text
9552_Reparameterized_Multi_Res.pdf - Published Version Download (863kB) | Preview |
Abstract
Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions ( MRConv ), a novel approach to parameterizing global convolutional kernels for long-sequence modeling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, MRConv learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D MRConv layers.
| Type: | Proceedings paper |
|---|---|
| Title: | Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling |
| Event: | 38th Conference on Neural Information Processing Systems (NeurIPS 2024) |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://papers.nips.cc/paper_files/paper/2024 |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher's terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10205689 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
