Pei, Kexin;
Li, Weichen;
Jin, Qirui;
Liu, Shuyang;
Geng, Scott;
Cavallaro, Lorenzo;
Yang, Junfeng;
(2024)
Exploiting Code Symmetries for Learning Program Semantics.
In: Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix, (eds.)
Proceedings of the 41st International Conference on Machine Learning.
(pp. pp. 40092-40113).
Proceedings of Machine Learning Research (PMLR): Vienna, Austria.
Preview |
Text
Cavallaro_pei24b.pdf Download (938kB) | Preview |
Abstract
This paper tackles the challenge of teaching code semantics to Large Language Models (LLMs) for program analysis by incorporating code symmetries into the model architecture. We introduce a group-theoretic framework that defines code symmetries as semantics-preserving transformations, where forming a code symmetry group enables precise and efficient reasoning of code semantics. Our solution, SYMC, develops a novel variant of self-attention that is provably equivariant to code symmetries from the permutation group defined over the program dependence graph. SYMC obtains superior performance on five program analysis tasks, outperforming state-of-the-art code models, including GPT-4, without any pre-training. Our results suggest that code LLMs that encode the code structural prior via the code symmetry group generalize better and faster.
| Type: | Proceedings paper |
|---|---|
| Title: | Exploiting Code Symmetries for Learning Program Semantics |
| Event: | 41st International Conference on Machine Learning |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://proceedings.mlr.press/v235/pei24b.html |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10200411 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
