Shi, Z;
Yang, AX;
Wu, B;
Aitchison, L;
Yilmaz, E;
Lipani, A;
(2024)
Instruction Tuning With Loss Over Instructions.
In: Globersons, Amir and Mackey, Lester and Belgrave, Danielle and Fan, Angela and Paquet, Ulrich and Tomczak, Jakub M and Zhang, Cheng, (eds.)
Advances in Neural Information Processing Systems 37.
Neural Information Processing Systems Foundation, Inc. (NeurIPS): Vancouver, Canada.
Preview |
Text
NeurIPS-2024-instruction-tuning-with-loss-over-instructions-Paper-Conference (1).pdf - Published Version Download (725kB) | Preview |
Abstract
Instruction tuning plays a crucial role in shaping the outputs of language models (LMs) to desired styles. In this work, we propose a simple yet effective method, INSTRUCTION MODELLING (IM), which trains LMs by applying a loss function to the instruction and prompt part rather than solely to the output part. Through experiments across 21 diverse benchmarks, we show that, in many scenarios, IM can effectively improve the LM performance on both NLP tasks (e.g., MMLU, TruthfulQA, and HumanEval) and open-ended generation benchmarks (e.g., MT-Bench and AlpacaEval). Remarkably, in the most advantageous case, IM boosts model performance on AlpacaEval 1.0 by over 100%. We identify two key factors influencing the effectiveness of IM: (1) The ratio between instruction length and output length in the training data; and (2) The number of training examples. We observe that IM is especially beneficial when trained on datasets with lengthy instructions paired with brief outputs, or under the Superficial Alignment Hypothesis (SAH) where a small amount of training examples are used for instruction tuning. Further analysis substantiates our hypothesis that our improvement can be attributed to reduced overfitting to instruction tuning datasets. It is worth noting that we are not proposing IM as a replacement for the current instruction tuning process. Instead, our work aims to provide practical guidance for instruction tuning LMs, especially in low-resource scenarios. Our code is available at https://github.com/ZhengxiangShi/InstructionModelling.
| Type: | Proceedings paper |
|---|---|
| Title: | Instruction Tuning With Loss Over Instructions |
| Event: | 38th Conference on Neural Information Processing Systems (NeurIPS 2024) |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://papers.nips.cc/paper_files/paper/2024/hash... |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Civil, Environ and Geomatic Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10207114 |
Archive Staff Only
 |
View Item |
