Dai, B; Liu, Z; Dai, H; He, N; Gretton, A; Le, S; Schurmaans, D; (2019) Exponential Family Estimation via Adversarial Dynamics Embedding. In: Wallach, H and Larochelle, H and Beygelzimer, A and d'Alché-Buc, F and Fox, E and Garnett, R, (eds.) Advances in Neural Information Processing Systems 32 (NIPS 2019). NIPS Proceedingsβ: Vancouver, Canada.

Preview

Text Gretton_9279-exponential-family-estimation-via-adversarial-dynamics-embedding.pdf - Published Version Download (1MB) | Preview

Abstract

We present an efficient algorithm for maximum likelihood estimation (MLE) of exponential family models, with a general parametrization of the energy function that includes neural networks. We exploit the primal-dual view of the MLE with a kinetics augmented model to obtain an estimate associated with an adversarial dual sampler. To represent this sampler, we introduce a novel neural architecture, dynamics embedding, that generalizes Hamiltonian Monte-Carlo (HMC). The proposed approach inherits the flexibility of HMC while enabling tractable entropy estimation for the augmented model. By learning both a dual sampler and the primal model simultaneously, and sharing parameters between them, we obviate the requirement to design a separate sampling procedure once the model has been trained, leading to more effective learning. We show that many existing estimators, such as contrastive divergence, pseudo/composite-likelihood, score matching, minimum Stein discrepancy estimator, non-local contrastive objectives, noise-contrastive estimation, and minimum probability flow, are special cases of the proposed approach, each expressed by a different (fixed) dual sampler. An empirical investigation shows that adapting the sampler during MLE can significantly improve on state-of-the-art estimators.

Download activity - last month

Export as JSONXMLCSV

Download activity - last 12 months

Export as CSVJSONXML

Downloads by country - last 12 months

Export as CSVXMLJSON

Archive Staff Only

View Item