IT-Forum

ISL Colloquium & IT Forum present "Generalized Resilience and Robust Statistics"

Topic: 
Generalized Resilience and Robust Statistics
Abstract / Description: 

Robust statistics traditionally focuses on outliers, or perturbations in total variation distance. However, a dataset could be corrupted in many other ways, such as systematic measurement errors and missing covariates. We generalize the robust statistics approach to consider perturbations under any Wasserstein distance, and show that robust estimation is possible whenever a distribution's population statistics are robust under a certain family of friendly perturbations. This generalizes a property called resilience previously employed in the special case of mean estimation with outliers. We justify the generalized resilience property by showing that it holds under moment or hypercontractive conditions. Even in the total variation case, these subsume conditions in the literature for mean estimation, regression, and covariance estimation; the resulting analysis simplifies and sometimes improves these known results in both population limit and finite-sample rate. Our robust estimators are based on minimum distance (MD) functionals (Donoho and Liu, 1988), which project onto a set of distributions under a discrepancy related to the perturbation. We present two approaches for designing MD estimators with good finite- sample rates: weakening the discrepancy and expanding the set of distributions. We also present connections to Gao et al. (2019)'s recent analysis of generative adversarial networks for robust estimation.

Joint work with Banghua Zhu and Jacob Steinhardt

Date and Time: 
Friday, October 11, 2019 - 1:15pm
Venue: 
Packard 202

ISL Colloquium presents "Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators"

Topic: 
Denoising and Regularization via Exploiting the Structural Bias of Convolutional Generators
Abstract / Description: 

Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. This success is often attributed to large amounts of training data.
On the contrary, a number of recent experimental results suggest that a major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this structural bias towards simple, natural images is that one can remove various kinds of noise and corruptions from a corrupted natural image by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to this single image. While this over-parameterized model can eventually fit the corrupted image perfectly, surprisingly after a few iterations of gradient descent one obtains the uncorrupted image, without using any training data. This intriguing phenomena has enabled state-of-the-art CNN-based denoising as well as regularization in linear inverse problems such as compressive sensing.
In this talk we take a step towards explaining this experimental phenomena by attributing it to particular architectural choices of convolutional networks. We then characterize the dynamics of fitting a two layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes. This results relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion.

Based on joint work with Paul Hand and Mahdi Soltanolkotabi.

 


 

The Information Systems Laboratory Colloquium (ISLC)

is typically held in Packard 101 every Thursday at 4:30 pm during the academic year. Coffee and refreshments are served at 4pm in the second floor kitchen of Packard Bldg.

The Colloquium is organized by graduate students Joachim Neu, Tavor Baharav and Kabir Chandrasekher. To suggest speakers, please contact any of the students.

To receive email notifications of seminars you can join the ISL mailing list.

Date and Time: 
Thursday, October 3, 2019 - 4:30pm
Venue: 
Packard 101

IT Forum & ISL Colloquium present "Proving Coding Theorems using Poisson Processes"

Topic: 
Proving Coding Theorems using Poisson Processes
Abstract / Description: 

In information theory, coding theorems are usually proved in the asymptotic regime where the blocklength tends to infinity. While there are techniques for finite blocklength analysis, they are often more complex than their asymptotic counterparts. In this talk, we study the use of Poisson processes in proving coding theorems, which not only gives sharp finite blocklength results, but also gives significantly shorter proofs than conventional asymptotic techniques in some settings. Instead of using fixed-size random codebooks, we construct the codebook as a Poisson process. We present a lemma, called the Poisson matching lemma, which can replace both packing and covering lemmas in proofs based on typicality. We then demonstrate its use in settings such as channel coding with channel state information at the encoder, lossy source coding with side information at the decoder, joint source-channel coding, broadcast channels, and distributed lossy source coding. This shows that the Poisson matching lemma is a viable alternative to typicality for most problems in network information theory.

The talk is based on a joint work with Prof. Venkat Anantharam (UC Berkeley).

 

Date and Time: 
Friday, September 27, 2019 - 1:15pm
Venue: 
Packard 202

ISL & IT Forum present "Human-Interpretable Concept Learning via Information Lattices"

Topic: 
Human-Interpretable Concept Learning via Information Lattices
Abstract / Description: 

Is it possible to learn the laws of music theory directly from raw sheet music in the same human-interpretable form as a music theory textbook? How little prior knowledge needs to be encoded to do so? We consider these and similar questions in other topical domains, in developing a general framework for automatic concept learning. The basic idea is an iterative discovery algorithm that has a student-teacher architecture and that operates on a generalization of Shannon's information lattice, which itself encodes a hierarchy of abstractions and is algorithmically constructed from group-theoretic foundations. In particular, learning this hierarchy of invariant concepts involves iterative optimization of Bayesian surprise and entropy. This gives a first step towards a principled and cognitive way of automatic concept learning and knowledge discovery. We further discuss applications in computational creativity, and limit theorems for creativity.

Date and Time: 
Friday, September 6, 2019 - 1:15pm
Venue: 
Packard 202

IT Forum & ISL Colloquium present "Learning from Sparse Data"

Topic: 
Learning from Sparse Data
Abstract / Description: 

In many scientific domains, the number of individuals in the population under study is often very large, however the number of observations available per individual is often very limited (sparse). Limited observations prohibit accurate estimation of parameters of interest for any given individual. In this sparse data regime, the key question is, how accurately can we estimate the distribution of parameters over the population? This problem arises in various domains such as epidemiology, psychology, health-care, biology, and social sciences. As an example, suppose for a large random sample of the population we have observations of whether a person caught the flu for each year over the past 5 years. We cannot accurately estimate the probability of any given person catching the flu with only 5 observations, however our goal is to estimate the distribution of these probabilities over the whole population. Such an estimated distribution can be used in downstream tasks, like testing and estimating properties of the distribution.

In this talk, I will present our recent results where we show that the maximum likelihood estimator (MLE) is minimax optimal in the sparse observation regime. While the MLE for this problem was proposed as early as the late 1960's, how accurately the MLE recovers the true distribution was not known. Our work closes this gap. In the course of our analysis, we provide novel bounds on the coefficients of Bernstein polynomials approximating Lipschitz-1 functions. Furthermore, the MLE is also efficiently computable in this setting and we evaluate the performance of MLE on both synthetic and real datasets.

Joint work with Weihao Kong, Gregory Valiant, and Sham Kakade.

Date and Time: 
Friday, August 30, 2019 - 1:15pm
Venue: 
Packard 202

ISL & IT Forum present "Feedback capacity of channels with memory via Reinforcement Learning and Graph-based auxiliary random variable""

Topic: 
Feedback capacity of channels with memory via Reinforcement Learning and Graph-based auxiliary random variable
Abstract / Description: 

In this talk we present two novel ideas: the first is novel method to compute the feedback capacity of channels with memory using reinforcement learning (RL). The second is a new technique of using Graph-based auxiliary random variable to convert a multi-letter expression of feedback capacity formula into a single letter expression.

In RL, one seeks to maximize cumulative rewards collected in a sequential decision-making environment. This is done by collecting samples of the underlying environment and using them to learn the optimal decision rule. The main advantage of this approach is its computational efficiency, even in high dimensional problems. Hence, RL can be used to estimate numerically the feedback capacity of unifilar finite state channels (FSCs) with large alphabet size. The outcome of the RL algorithm sheds light on the properties of the optimal decision rule, which in our case, is the optimal input distribution of the channel.

The insights gained from the RL computation can be converted into analytic, single-letter capacity expressions by solving corresponding lower and upper bounds. The bounds are based on another novel idea of using Graph-based auxiliary random variable

We demonstrate the efficiency of this method by analytically solving the feedback capacity of the well-known Ising channel with a large alphabet. We also provide a simple coding scheme that achieves the feedback capacity.

 

Date and Time: 
Friday, August 2, 2019 - 3:00pm
Venue: 
Packard 202

ISL & IT Forum present "Resource-efficient quantized deep learning"

Topic: 
Resource-efficient quantized deep learning
Abstract / Description: 

Reducing the numerical precision of neural networks is one of the simplest, most effective and most common methods to improve resource efficiency (e.g. by reducing the memory and power requirements). Much research has been invested in finding how to quantize neural nets without significantly degrading performance. I will describe the main bottlenecks and solutions in various settings:
1) 1bit inference (1bit weights and activations), NIPS 2016 - link
2) 8bit training (8bit weights, activations, gradients, and batch-norm), NeurIPS 2018 - link
3) 4bit inference when quantization is done only post-training, Arxiv 2019 - link
4) Calculating the maximum trainable depth as a function of the numerical precision, Arxiv 2019 - link

Date and Time: 
Friday, July 26, 2019 - 2:00pm
Venue: 
Packard 202

ISL & IT Forum present "Towards Achieving Secure Consensus and Trusted Data Exchange for Multi-Robot Teams""

Topic: 
Towards Achieving Secure Consensus and Trusted Data Exchange for Multi-Robot Teams
Abstract / Description: 

As physical robot networks become more pervasive all around us, in the form of teams of autonomous vehicles, fleets of delivery drones, and smart and mobile IoT, it becomes increasingly critical to question the robustness of their coordination algorithms to security threats and/or corrupted data. Indeed, it has been shown that many multi-robot tasks easily fail in the presence of erroneous or hacked data. We investigate the vulnerabilities of important multi-robot algorithms such as consensus, coverage, and distributed mapping to malicious or erroneous data and we demonstrate the potential of communication to thwart certain attacks, such as the Sybil Attack, on these algorithms. Our key insight is that coordinated mobility can be combined with signal processing of communication signals to allow agents to learn important information about the environment and the nature of other agents in the network (for example the presence of cooperative versus adversarial agents). Along these lines, we will present a theoretical and experimental framework for provably securing multi-robot distributed algorithms through careful use of communication. We will present both theoretical results and experimental results on actual hardware implementations for bounding the influence of a Sybil Attack on consensus and on coverage by using observations over the wireless channels. In some cases, we show that the effect of a Sybil Attack can be nearly eliminated with high probability by deriving the appropriate switching function using a sufficient number of observations over the wireless network. Finally, we will briefly describe promising results on new methods for outlier rejection and active rendezvous in a pose graph optimization framework that exploits feedback gathered from communication channels to arrive at improved accuracy.

Date and Time: 
Wednesday, July 24, 2019 - 2:00pm
Venue: 
Packard 202

ISL & IT Forum present "Building a DNA information storage system from the bottom up"

Topic: 
Building a DNA information storage system from the bottom up
Abstract / Description: 

DNA has emerged as a compelling data storage medium due to its density, longevity, and eternal relevance compared to current memory technologies. However, the high price of synthesizing DNA (list price $3,500 per megabyte) remains a major bottleneck for adoption of this promising storage solution. In this talk, I will present our work towards breaking down this barrier using enzymatic DNA synthesize and a tailored codec for robust data retrieval. I will also touch upon some fundamental considerations when designing DNA information storage systems.

Date and Time: 
Friday, June 21, 2019 - 1:15pm
Venue: 
Packard 202

Pages

Subscribe to RSS - IT-Forum