Outliers in weights and activations often prevent us from benefiting from very low precision, e.g., 2 or 4 bits. In this talk, we will present our recent works on hardware accelerator and low memory-cost training where we handle a very small amount of outliers, e.g., large values occupying only 1% of total data, in high precision and the majority of data in very low precision.
The accelerator, based on 4-bit computation, offers average 30% better performance and energy efficiency compared with the zero-skipping 8-bit accelerator. The low memory-cost training solution applies 3-bit precision to 98% of activations to be stored during training, which finally leads to 9X reduction in the memory cost of ResNet-152 while keeping the training accuracy.
Professor Sungjoo Yoo received HIS Ph.D. at Seoul National University in 2000. From 2000-2004, he was a researcher at TIMA laboratory in Grenoble, France. In 2004, he joined Samsung Electronics, joining POSTECH (Pohang university of Science and TECHnology) in 2008. In 2015, he joined Seoul National University and is now full professor. Currently, he is on sabbatical at Facebook, Menlo Park as research scientist. His research interests include software/hardware co-design of machine learning applications.