UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Re-evaluating evaluation.

Balduzzi, D; Tuyls, K; Pérolat, J; Graepel, T; (2018) Re-evaluating evaluation. In: Bengio, S and Wallach, HM and Larochelle, H and Grauman, K and Cesa-Bianchi, N and Garnett, R, (eds.) Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018). (pp. pp. 3272-3283). Neural Information Processing Systems Foundation, Inc.: Montréal, Canada. Green open access

[img]
Preview
Text
7588-re-evaluating-evaluation.pdf - Published version

Download (540kB) | Preview

Abstract

Progress in machine learning is measured by careful evaluation on problems of outstanding common interest. However, the proliferation of benchmark suites and environments, adversarial attacks, and other complications has diluted the basic evaluation model by overwhelming researchers with choices. Deliberate or accidental cherry picking is increasingly likely, and designing well-balanced evaluation suites requires increasing effort. In this paper we take a step back and propose Nash averaging. The approach builds on a detailed analysis of the algebraic structure of evaluation in two basic scenarios: agent-vs-agent and agentvs-task. The key strength of Nash averaging is that it automatically adapts to redundancies in evaluation data, so that results are not biased by the incorporation of easy tasks or weak agents. Nash averaging thus encourages maximally inclusive evaluation – since there is no harm (computational cost aside) from including all available tasks and agents.

Type: Proceedings paper
Title: Re-evaluating evaluation.
Event: 32nd Conference on Neural Information Processing Systems (NeurIPS 2018)
Open access status: An open access version is available from UCL Discovery
Publisher version: http://papers.nips.cc/book/advances-in-neural-info...
Language: English
Additional information: This version is the version of record. For information on re-use, please refer to the publisher’s terms and conditions.
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > Dept of Computer Science
URI: https://discovery.ucl.ac.uk/id/eprint/10093989
Downloads since deposit
7Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item