UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Measuring Error Alignment for Decision-Making Systems

Xu, Binxia; Bikakis, Antonios; Onah, Daniel FO; Vlachidis, Andreas; Dickens, Luke; (2025) Measuring Error Alignment for Decision-Making Systems. In: Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence. (pp. pp. 27731-27739). Association for the Advancement of Artificial Intelligence (AAAI): Phliadelphia, PA, USA. Green open access

[thumbnail of xu2025measuring_preprint.pdf]
Preview
Text
xu2025measuring_preprint.pdf - Accepted Version

Download (2MB) | Preview

Abstract

Given that AI systems are set to play a pivotal role in future decision-making processes, their trustworthiness and reliability are of critical concern. Due to their scale and complexity, modern AI systems resist direct interpretation, and alternative ways are needed to establish trust in those systems, and determine how well they align with human values. We argue that good measures of the information processing similarities between AI and humans, may be able to achieve these same ends. While Representational alignment (RA) approaches measure similarity between the internal states of two systems, the associated data can be expensive and difficult to collect for human systems. In contrast, Behavioural alignment (BA) comparisons are cheaper and easier, but questions remain as to their sensitivity and reliability. We propose two new behavioural alignment metrics misclassification agreement which measures the similarity between the errors of two systems on the same instances, and class-level error similarity which measures the similarity between the error distributions of two systems. We show that our metrics correlate well with RA metrics, and provide complementary information to another BA metric, within a range of domains, and set the scene for a new approach to value alignment.

Type: Proceedings paper
Title: Measuring Error Alignment for Decision-Making Systems
Event: 39th Annual AAAI Conference on Artificial Intelligence
Location: Phliadelphia, PA, USA
Dates: 25th February - 4th March 2025
ISBN-13: 978-1-57735-897-8
Open access status: An open access version is available from UCL Discovery
Publisher version: https://doi.org/10.1609/aaai.v39i26.34988
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL SLASH
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of Arts and Humanities
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of Arts and Humanities > Dept of Information Studies
URI: https://discovery.ucl.ac.uk/id/eprint/10202164
Downloads since deposit
10Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item