Alghamdi, Emad;
Masoud, Reem;
Alnuhait, Deema;
Alomairi, Afnan;
Ashraf, Ahmed;
Zaytoon, Mohamed;
(2025)
AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic.
In:
Proceedings of the 31st International Conference on Computational Linguistics.
(pp. pp. 8664-8679).
Association for Computational Linguistics
Preview |
Text
2025.coling-main.579.pdf - Published Version Download (3MB) | Preview |
Abstract
The swift progress and widespread acceptance of artificial intelligence (AI) systems highlight a pressing requirement to comprehend both the capabilities and potential risks associated with AI. Given the linguistic complexity, cultural richness, and underrepresented status of Arabic in AI research, there is a pressing need to focus on Large Language Models (LLMs) performance and safety for Arabic related tasks. Despite some progress in their development, there is a lack of comprehensive trustworthiness evaluation benchmarks which presents a major challenge in accurately assessing and improving the safety of LLMs when prompted in Arabic. In this paper, we introduce AraTrust, the first comprehensive trustworthiness benchmark for LLMs in Arabic. AraTrust comprises 522 human-written multiple-choice questions addressing diverse dimensions related to truthfulness, ethics, privacy, illegal activities, mental health, physical health, unfairness, and offensive language. We evaluated a set of LLMs against our benchmark to assess their trustworthiness. GPT-4 was the most trustworthy LLM, while open-source models, particularly AceGPT 7B and Jais 13B, struggled to achieve a score of 60% in our benchmark. The benchmark dataset is publicly available at https://huggingface.co/datasets/asas-ai/AraTrust
| Type: | Proceedings paper |
|---|---|
| Title: | AraTrust: An Evaluation of Trustworthiness for LLMs in Arabic |
| Event: | 31st International Conference on Computational Linguistics |
| Location: | Abu Dhabi, UAE |
| Dates: | 19th-24th January 2025 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://aclanthology.org/2025.coling-main.579/ |
| Language: | English |
| Additional information: | © 2025 ACL. Materials published in or after 2016 are licensed on a Creative Commons Attribution 4.0 International License (https://creativecommons.org/licenses/by/4.0/). |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Electronic and Electrical Eng |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10206340 |
Loading...
Loading...Loading...Loading...Archive Staff Only
 |
View Item |
