ECE Departmental Seminar 

Augment and Reduce: Stochastic Inference for Large Categorical Distributions

Dr. Francisco J. R. Ruiz

Columbia University

Friday, 4/27/18, 11:00am
Light Engineering 250

Abstract: Categorical distributions are ubiquitous in machine learning, e.g., in classification, language models, and recommendation systems. They are also at the core of discrete choice models. However, when the number of possible outcomes is very large, using categorical distributions becomes computationally expensive, as the complexity scales linearly with the number of outcomes. To address this problem, we propose augment and reduce (A&R), a method to alleviate the computational complexity. A&R uses two ideas: latent variable augmentation and stochastic variational inference. It maximizes a lower bound on the marginal likelihood of the data. Unlike existing methods which are specific to softmax, A&R is more general and is amenable to other categorical models, such as multinomial probit. On several large-scale classification problems, we show that A&R provides a tighter bound on the marginal likelihood and has better predictive performance than existing approaches.

Bio: Francisco J. R. Ruiz is a Postdoctoral Research Scientist who works with David Blei at the Department of Computer Science at Columbia University, and with Zoubin Ghahramani at the Engineering Department at University of Cambridge. Francisco holds a Marie-Sklodowska Curie fellowship in the context of the E.U. Horizon 2020 program. He completed his Ph.D. and M.Sc. from the University Carlos III in Madrid. His research is focused on statistical machine learning, specially Bayesian modeling and inference.