UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation

Diallo, Aissatou; Bikakis, Antonios; Dickens, Luke; Hunter, Anthony; Miller, Rob; (2025) RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation. In: Proceedings of the 28th European Conference on Artificial Intelligence (ECAI 2025). ECAI: Bologna, Italy. (In press).

[thumbnail of m2352.pdf] Text
m2352.pdf - Accepted Version
Access restricted to UCL open access staff until 18 March 2026.

Download (559kB)

Abstract

Commonsense reasoning is a key aspect of human intelligence. If we are to develop robust and deep intelligent systems, then we need to understand the diversity and complexity of commonsense reasoning across the gamut of human activities. An interesting class of commonsense reasoning problems arises when people are faced with natural disasters. To investigate this topic, we present RESPONSE, a human-curated dataset containing 1789 annotated instances featuring 6037 sets of questions designed to assess LLMs’ commonsense reasoning in disaster situations across different time frames. The dataset includes problem descriptions, missing resources, time-sensitive solutions, and their justifications, with a subset validated by environmental engineers. Through both automatic metrics and human evaluation, we compare LLM-generated recommendations against human responses. Our findings show that even state-of-the-art models like GPT-4 achieve only 37% human-evaluated correctness for immediate response actions, highlighting significant room for improvement in LLMs’ ability for commonsense reasoning in crises.

Type: Proceedings paper
Title: RESPONSE: Benchmarking the Ability of Language Models to Undertake Commonsense Reasoning in Crisis Situation
Event: 28th European Conference on Artificial Intelligence (ECAI 2025)
Dates: 25 Oct 2025 - 30 Sep 2025
Publisher version: https://ecai2025.org/
Language: English
Additional information: This version is the author accepted manuscript. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices > UCL SLASH
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of Arts and Humanities
UCL > Provost and Vice Provost Offices > UCL SLASH > Faculty of Arts and Humanities > Dept of Information Studies
URI: https://discovery.ucl.ac.uk/id/eprint/10213506
Downloads since deposit
1Download
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item