eprintid: 10072289
rev_number: 36
eprint_status: archive
userid: 608
dir: disk0/10/07/22/89
datestamp: 2019-04-16 11:54:40
lastmod: 2021-09-26 23:14:51
status_changed: 2020-02-17 11:12:51
type: article
metadata_visibility: show
creators_name: Advani, A
creators_name: Kitagawa, T
creators_name: Sloczynski, T
title: Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection
ispublished: pub
divisions: UCL
divisions: B03
divisions: C03
divisions: F24
keywords: empirical Monte Carlo studies, program evaluation, selection on observables,  treatment effects
note: © 2020 The Authors. Journal of Applied Econometrics published by John Wiley & Sons Ltd. 
This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
abstract: We consider two recent suggestions for how to perform an empirically motivated Monte Carlo study to help select a treatment effect estimator under unconfoundedness. We show theoretically that neither is likely to be informative except under restrictive conditions that are unlikely to be satisfied in many contexts. To test empirical relevance, we also apply the approaches to a real-world setting where estimator performance is known. Both approaches are worse than random at selecting estimators which minimise absolute bias.They are better when selecting estimators that minimise mean squared error.However, using a simple bootstrap is at least as good and often better. For now researchers would be best advised to use a range of estimators and compare estimates for robustness.
date: 2019-09
date_type: published
publisher: John Wiley and Sons
official_url: https://doi.org/10.1002/jae.2724
oa_status: green
full_text_type: pub
language: eng
primo: open
primo_central: open_green
verified: verified_manual
elements_id: 1646332
doi: 10.1002/jae.2724
lyricists_name: Kitagawa, Toru
lyricists_id: TKITA87
actors_name: Kitagawa, Toru
actors_id: TKITA87
actors_role: owner
full_text_status: public
publication: Journal of Applied Econometrics
volume: 34
number: 6
pagerange: 893-910
issn: 0883-7252
citation:        Advani, A;    Kitagawa, T;    Sloczynski, T;      (2019)    Mostly Harmless Simulations? Using Monte Carlo Studies for Estimator Selection.                   Journal of Applied Econometrics , 34  (6)   pp. 893-910.    10.1002/jae.2724 <https://doi.org/10.1002/jae.2724>.       Green open access   
 
document_url: https://discovery.ucl.ac.uk/id/eprint/10072289/1/Kitagawa_Advani_et_al-2019-Journal_of_Applied_Econometrics.pdf