UCL Discovery
UCL home » Library Services » Electronic resources » UCL Discovery

Eliciting Human Judgment for Prediction Algorithms

Ibrahim, R; Kim, S-H; Tong, J; (2021) Eliciting Human Judgment for Prediction Algorithms. SSRN Green open access

[img]
Preview
Text
SSRN-id3606633.pdf - Accepted version

Download (1MB) | Preview

Abstract

Even when human point forecasts are less accurate than data-based algorithm predictions, they can still help boost performance by being used as algorithm inputs. Assuming one uses human judgment indirectly in this manner, we propose changing the elicitation question from the traditional direct forecast (DF) to what we call the private information adjustment (PIA): how much the human thinks the algorithm should adjust its forecast to account for information the human has that is unused by the algorithm. Using stylized models with and without random error, we theoretically prove that human random error makes eliciting the PIA lead to more accurate predictions than eliciting the DF. However, this DF-PIA gap does not exist for perfectly consistent forecasters. The DF-PIA gap is increasing in the random error that people make while incorporating public information (data that the algorithm uses) but is decreasing in the random error that people make while incorporating private information (data that only the human can use). In controlled experiments with students and Amazon Mechanical Turk workers, we find support for these hypotheses.

Type: Working / discussion paper
Title: Eliciting Human Judgment for Prediction Algorithms
Open access status: An open access version is available from UCL Discovery
DOI: 10.2139/ssrn.3606633
Publisher version: https://dx.doi.org/10.2139/ssrn.3606633
Language: English
Keywords: laboratory experiments, behavioral operations, random error, elicitation, forecasting, prediction, discretion, expert input, private information, judgment, aggregation
UCL classification: UCL
UCL > Provost and Vice Provost Offices
UCL > Provost and Vice Provost Offices > UCL BEAMS
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science
UCL > Provost and Vice Provost Offices > UCL BEAMS > Faculty of Engineering Science > UCL School of Management
URI: https://discovery.ucl.ac.uk/id/eprint/10107592
Downloads since deposit
68Downloads
Download activity - last month
Download activity - last 12 months
Downloads by country - last 12 months

Archive Staff Only

View Item View Item