UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Modular Networks: Learning to Decompose Neural Computation

Kirsch, L; Kunze, J; Barber, D; (2018) Modular Networks: Learning to Decompose Neural Computation. In: Bengio, S and Wallach, H and Larochelle, H and Grauman, K and Cesa-Bianchi, N and Garnett, R, (eds.) Advances in Neural Information Processing Systems 31 (NIPS 2018). NIPS Proceedings: Montreal, Canada. Green open access

[img]
Preview
Text
Kunze_7508-modular-networks-learning-to-decompose-neural-computation.pdf - Published version

Download (1MB) | Preview

Abstract

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.

Type: Proceedings paper
Title: Modular Networks: Learning to Decompose Neural Computation
Event: 31st Conference on Neural Information Processing Systems (NIPS)
Location: Montreal, CANADA
Dates: 02 December 2018 - 08 December 2018
Open access status: An open access version is available from UCL Discovery
Publisher version: https://papers.nips.cc/paper/7508-modular-networks...
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10076128
Downloads since deposit
108Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item