Lee, S;
Goldt, S;
Saxe, A;
(2021)
Continual Learning in the Teacher-Student Setup: Impact of Task Similarity.
In: Meila, M and Zhang, T, (eds.)
Proceedings of the 38th International Conference on Machine Learning.
(pp. pp. 6109-6119).
MLResearch Press
Preview |
Text
lee21e.pdf - Published Version Download (630kB) | Preview |
Abstract
Continual learning—the ability to learn many tasks in sequence—is critical for artificial learning systems. Yet standard training methods for deep networks often suffer from catastrophic forgetting, where learning new tasks erases knowledge of the earlier tasks. While catastrophic forgetting labels the problem, the theoretical reasons for interference between tasks remain unclear. Here, we attempt to narrow this gap between theory and practice by studying continual learning in the teacher-student setup. We extend previous analytical work on two-layer networks in the teacher-student setup to multiple teachers. Using each teacher to represent a different task, we investigate how the relationship between teachers affects the amount of forgetting and transfer exhibited by the student when the task switches. In line with recent work, we find that when tasks depend on similar features, intermediate task similarity leads to greatest forgetting. However, feature similarity is only one way in which tasks may be related. The teacher-student approach allows us to disentangle task similarity at the level of \textbackslashemphreadouts (hidden-to-output weights) as well as \textbackslashemphfeatures (input-to-hidden weights). We find a complex interplay between both types of similarity, initial transfer/forgetting rates, maximum transfer/forgetting, and the long-time (post-switch) amount of transfer/forgetting. Together, these results help illuminate the diverse factors contributing to catastrophic forgetting.
| Type: | Proceedings paper |
|---|---|
| Title: | Continual Learning in the Teacher-Student Setup: Impact of Task Similarity |
| Event: | 38th International Conference on Machine Learning (ICML) |
| Location: | ELECTR NETWORK |
| Dates: | 18 Jul 2021 - 24 Jul 2021 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.mlr.press/v139/lee21e.html |
| Language: | English |
| Additional information: | This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences UCL > Provost and Vice Provost Offices > School of Life and Medical Sciences > Faculty of Life Sciences > Gatsby Computational Neurosci Unit |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10166622 |
Archive Staff Only
 |
View Item |
