Tan, Daniel;
Chanin, David;
Lynch, Aengus;
Paige, Brooks;
Kanoulas, Dimitrios;
Garriga-Alonso, Adrià;
Kirk, Robert;
(2024)
Analysing the Generalisation and Reliability of
Steering Vectors.
In: Globersons, A and Mackey, L and Belgrave, D and Fan, A and Paquet, U and Tomczak, J and Zhang, C, (eds.)
Proceedings of the Advances in Neural Information Processing Systems 37 (NeurIPS 2024).
NeurIPS: Vancouver, CA.
Preview |
Text
18851_Analysing_the_Generalisa.pdf - Published Version Download (1MB) | Preview |
Abstract
Steering vectors (SVs) are a new approach to efficiently adjust language model behaviour at inference time by intervening on intermediate model activations. They have shown promise in terms of improving both capabilities and model alignment. However, the reliability and generalisation properties of this approach are unknown. In this work, we rigorously investigate these properties, and show that steering vectors have substantial limitations both in- and out-of-distribution. In-distribution, steerability is highly variable across different inputs. Depending on the concept, spurious biases can substantially contribute to how effective steering is for each input, presenting a challenge for the widespread use of steering vectors. Out-of-distribution, while steering vectors often generalise well, for several concepts they are brittle to reasonable changes in the prompt, resulting in them failing to generalise well. Overall, our findings show that while steering can work well in the right circumstances, there remain many technical difficulties of applying steering vectors to guide models' behaviour at scale.
| Type: | Proceedings paper |
|---|---|
| Title: | Analysing the Generalisation and Reliability of Steering Vectors |
| Event: | 38th Conference on Neural Information Processing Systems (NeurIPS 2024) |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://papers.nips.cc/paper_files/paper/2024 |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10205543 |
Loading...
Loading...Loading...| 1. |  United Kingdom | 1 |
Archive Staff Only
 |
View Item |
