Zoom Link

Abstract: Data-parallel SGD is the de facto algorithm for distributed optimization, especially for large scale machine learning. Despite its merits, communication bottleneck is one of its persistent issues. Most compression schemes to alleviate this either assume noiseless communication links, or fail to achieve good performance on practical tasks. In this work, we close this gap and introduce LASER: LineAr CompreSsion in WirEless DistRibuted Optimization. LASER capitalizes on the inherent low-rank structure of gradients and transmits them efficiently over the noisy channels. Whilst enjoying theoretical guarantees similar to those of the classical SGD, LASER shows consistent gains over baselines on a variety of practical benchmarks. In particular, it outperforms the state-of-the-art compression schemes on challenging computer vision and GPT language modeling tasks. On the latter, we obtain 50-64% improvement in perplexity over our baselines for noisy channels. (Joint work with Marco Bondaschi, Thijs Vogels, Martin Jaggi, Hyeji Kim, and Michael Gastpar.)

Bio: Ashok is a postdoctoral researcher at EPFL with Michael Gastpar. He obtained his PhD in ECE from the University of Illinois at Urbana-Champaign in August 2022, with Pramod Viswanath and Sewoong Oh. He obtained his Masters in ECE with Yihong Wu also from UIUC in 2017. Earlier he graduated from IIT Bombay with a B.Tech. in EE and Minors in Mathematics working with Vivek Borkar. His research interests are in foundations of data science in topics including machine learning, information theory, optimization, and statistics. He is a recipient of Best Paper Award from ACM MobiHoc 2019. He is also a recipient of several graduate student awards and fellowships including Joan and Lalit Bahl Fellowship (twice), Sundaram Seshu International Student Fellowship, finalist for the Qualcomm Innovation Fellowship 2018. Outside research, he likes to learn new languages, watch and read about international films, remembering movie trivia, and cooking. For more details about him, please visit https://ashokvardhan.github.io/.