van Rossem, L; Saxe, AM; (2024) When Representations Align: Universality in Representation Learning Dynamics. In: Proceedings of the 41st International Conference on Machine Learning. (pp. pp. 49098-49121). Proceedings of Machine Learning Research: Vienna, Austria.

Preview

PDF When Representations Align.pdf - Accepted Version Download (4MB) | Preview

Abstract

Deep neural networks come in many sizes and architectures. The choice of architecture, in conjunction with the dataset and learning algorithm, is commonly understood to affect the learned neural representations. Yet, recent results have shown that different architectures learn representations with striking qualitative similarities. Here we derive an effective theory of representation learning under the assumption that the encoding map from input to hidden representation and the decoding map from representation to output are arbitrary smooth functions. This theory schematizes representation learning dynamics in the regime of complex, large architectures, where hidden representations are not strongly constrained by the parametrization. We show through experiments that the effective theory describes aspects of representation learning dynamics across a range of deep networks with different activation functions and architectures, and exhibits phenomena similar to the “rich” and “lazy” regime. While many network behaviors depend quantitatively on architecture, our findings point to certain behaviors that are widely conserved once models are sufficiently flexible.

Download activity - last month

Export as CSVXMLJSON

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as XMLJSONCSV

Archive Staff Only

View Item