eprintid: 10198210 rev_number: 7 eprint_status: archive userid: 699 dir: disk0/10/19/82/10 datestamp: 2024-10-08 10:42:04 lastmod: 2024-10-08 10:42:04 status_changed: 2024-10-08 10:42:04 type: proceedings_section metadata_visibility: show sword_depositor: 699 creators_name: Solano, J creators_name: Sanni, M creators_name: Camburu, OM creators_name: Minervini, P title: SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations ispublished: pub divisions: UCL divisions: B04 divisions: F48 note: © 2024 ACL. Permission is granted to make copies for the purposes of teaching and research. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). abstract: Models that generate natural language explanations (NLEs) for their predictions have recently gained increasing interest. However, this approach usually demands large datasets of human-written NLEs for the ground-truth answers at training time, which can be expensive and potentially infeasible for some applications. When only a few NLEs are available (a few-shot setup), fine-tuning pre-trained language models (PLMs) in conjunction with prompt-based learning has recently shown promising results. However, PLMs typically have billions of parameters, making full fine-tuning expensive. We propose SPARSEFIT, a sparse few-shot fine-tuning strategy that leverages discrete prompts to jointly generate predictions and NLEs. We experiment with SPARSEFIT on three sizes of the T5 language model and four datasets and compare it against existing state-of-the-art Parameter-Efficient Fine-Tuning (PEFT) techniques. We find that fine-tuning only 6.8% of the model parameters leads to competitive results for both the task performance and the quality of the generated NLEs compared to full fine-tuning of the model and produces better results on average than other PEFT methods in terms of predictive accuracy and NLE quality. date: 2024 date_type: published publisher: Association for Computational Linguistics official_url: https://doi.org/10.18653/v1/2024.acl-long.113 oa_status: green full_text_type: pub language: eng primo: open primo_central: open_green verified: verified_manual elements_id: 2325376 doi: 10.18653/v1/2024.acl-long.113 lyricists_name: Camburu, Oana-Maria lyricists_id: OCAMB08 actors_name: Camburu, Oana-Maria actors_id: OCAMB08 actors_role: owner full_text_status: public pres_type: paper publication: Proceedings of the Annual Meeting of the Association for Computational Linguistics volume: 1 pagerange: 2053-2077 event_title: 62nd Annual Meeting of the Association for Computational Linguistics book_title: Proceedings of the Annual Meeting of the Association for Computational Linguistics citation: Solano, J; Sanni, M; Camburu, OM; Minervini, P; (2024) SPARSEFIT: Few-shot Prompting with Sparse Fine-tuning for Jointly Generating Predictions and Natural Language Explanations. In: Proceedings of the Annual Meeting of the Association for Computational Linguistics. (pp. pp. 2053-2077). Association for Computational Linguistics Green open access document_url: https://discovery.ucl.ac.uk/id/eprint/10198210/1/2024.acl-long.113.pdf