Cunningham, Harry Jake; Giannone, Giorgio; Zhang, Mingtian; Deisenroth, Marc Peter; (2024) Reparameterized Multi-Resolution Convolutions for Long Sequence Modelling. In: Globerson, A and Mackey, L and Belgrave, D and Fan, A and Paquet, U and Tomczak, J and Zhang, C, (eds.) Advances in Neural Information Processing Systems 37 (NeurIPS 2024). (pp. pp. 1-32). NeurIPS: San Diego, CA, USA.

Preview

Text 9552_Reparameterized_Multi_Res (1).pdf - Published Version Download (863kB) | Preview

Abstract

Global convolutions have shown increasing promise as powerful general-purpose sequence models. However, training long convolutions is challenging, and kernel parameterizations must be able to learn long-range dependencies without overfitting. This work introduces reparameterized multi-resolution convolutions (MRConv), a novel approach to parameterizing global convolutional kernels for long-sequence modelling. By leveraging multi-resolution convolutions, incorporating structural reparameterization and introducing learnable kernel decay, MRConv learns expressive long-range kernels that perform well across various data modalities. Our experiments demonstrate state-of-the-art performance on the Long Range Arena, Sequential CIFAR, and Speech Commands tasks among convolution models and linear-time transformers. Moreover, we report improved performance on ImageNet classification by replacing 2D convolutions with 1D MRConv layers.

Download activity - last month

Export as XMLCSVJSON

Download activity - last 12 months

Export as JSONCSVXML

Downloads by country - last 12 months

Export as JSONCSVXML

Archive Staff Only

View Item