Abdulaal, Ahmed;
Jin, Chen;
Montaña-Brown, Nina;
Gema, Aryo P;
de Castro, Daniel C;
Alexander, Daniel C;
Teare, Philip;
... Saseendran, Amrutha; + view all
(2025)
Balancing Act: Diversity and Consistency in Large Language Model Ensembles.
In:
13th International Conference on Learning Representations ICLR 2025.
(pp. pp. 29287-29319).
ICLR: Singapore, Singapore.
Preview |
Text
8091_Balancing_Act_Diversity_a.pdf - Accepted Version Download (624kB) | Preview |
Abstract
Ensembling strategies for Large Language Models (LLMs) have demonstrated significant potential in improving performance across various tasks by combining the strengths of individual models. However, identifying the most effective ensembling method remains an open challenge, as neither maximizing output consistency through self-consistency decoding nor enhancing model diversity via frameworks like "Mixture of Agents" has proven universally optimal. Motivated by this, we propose a unified framework to examine the trade-offs between task performance, model diversity, and output consistency in ensembles. More specifically, we introduce a consistency score that defines a gating mechanism for mixtures of agents and an algorithm for mixture refinement to investigate these trade-offs at the semantic and model levels, respectively. We incorporate our insights into a novel inference-time LLM ensembling strategy called the Dynamic Mixture of Agents (DMoA) and demonstrate that it achieves a new state-of-the-art result in the challenging Big Bench Hard mixed evaluations benchmark. Our analysis reveals that cross-validation bias can enhance performance, contingent on the expertise of the constituent models. We further demonstrate that distinct reasoning tasks—such as arithmetic reasoning, commonsense reasoning, and instruction following—require different model capabilities, leading to inherent task-dependent trade-offs that DMoA balances effectively.
| Type: | Proceedings paper |
|---|---|
| Title: | Balancing Act: Diversity and Consistency in Large Language Model Ensembles |
| Event: | The Thirteenth International Conference on Learning Representations: ICLR 2025 |
| Open access status: | An open access version is available from UCL Discovery |
| Publisher version: | https://openreview.net/forum?id=Dl6nkKKvlX |
| Language: | English |
| Additional information: | This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions. |
| UCL classification: | UCL UCL > Provost and Vice Provost Offices > UCL BEAMS UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science |
| URI: | https://discovery.ucl.ac.uk/id/eprint/10211652 |
Archive Staff Only
 |
View Item |
