IT-Forum

Convolutional Neural Networks (CNNs) have emerged as highly successful tools for image generation, recovery, and restoration. This success is often attributed to large amounts of training data.
On the contrary, a number of recent experimental results suggest that a major contributing factor to this success is that convolutional networks impose strong prior assumptions about natural images. A surprising experiment that highlights this structural bias towards simple, natural images is that one can remove various kinds of noise and corruptions from a corrupted natural image by simply fitting (via gradient descent) a randomly initialized, over-parameterized convolutional generator to this single image. While this over-parameterized model can eventually fit the corrupted image perfectly, surprisingly after a few iterations of gradient descent one obtains the uncorrupted image, without using any training data. This intriguing phenomena has enabled state-of-the-art CNN-based denoising as well as regularization in linear inverse problems such as compressive sensing.
In this talk we take a step towards explaining this experimental phenomena by attributing it to particular architectural choices of convolutional networks. We then characterize the dynamics of fitting a two layer convolutional generator to a noisy signal and prove that early-stopped gradient descent denoises/regularizes. This results relies on showing that convolutional generators fit the structured part of an image significantly faster than the corrupted portion.

Based on joint work with Paul Hand and Mahdi Soltanolkotabi.

The Information Systems Laboratory Colloquium (ISLC)

is typically held in Packard 101 every Thursday at 4:30 pm during the academic year. Coffee and refreshments are served at 4pm in the second floor kitchen of Packard Bldg.

The Colloquium is organized by graduate students Joachim Neu, Tavor Baharav and Kabir Chandrasekher. To suggest speakers, please contact any of the students.

To receive email notifications of seminars you can join the ISL mailing list.

In information theory, coding theorems are usually proved in the asymptotic regime where the blocklength tends to infinity. While there are techniques for finite blocklength analysis, they are often more complex than their asymptotic counterparts. In this talk, we study the use of Poisson processes in proving coding theorems, which not only gives sharp finite blocklength results, but also gives significantly shorter proofs than conventional asymptotic techniques in some settings. Instead of using fixed-size random codebooks, we construct the codebook as a Poisson process. We present a lemma, called the Poisson matching lemma, which can replace both packing and covering lemmas in proofs based on typicality. We then demonstrate its use in settings such as channel coding with channel state information at the encoder, lossy source coding with side information at the decoder, joint source-channel coding, broadcast channels, and distributed lossy source coding. This shows that the Poisson matching lemma is a viable alternative to typicality for most problems in network information theory.

The talk is based on a joint work with Prof. Venkat Anantharam (UC Berkeley).

Is it possible to learn the laws of music theory directly from raw sheet music in the same human-interpretable form as a music theory textbook? How little prior knowledge needs to be encoded to do so? We consider these and similar questions in other topical domains, in developing a general framework for automatic concept learning. The basic idea is an iterative discovery algorithm that has a student-teacher architecture and that operates on a generalization of Shannon's information lattice, which itself encodes a hierarchy of abstractions and is algorithmically constructed from group-theoretic foundations. In particular, learning this hierarchy of invariant concepts involves iterative optimization of Bayesian surprise and entropy. This gives a first step towards a principled and cognitive way of automatic concept learning and knowledge discovery. We further discuss applications in computational creativity, and limit theorems for creativity.

In many scientific domains, the number of individuals in the population under study is often very large, however the number of observations available per individual is often very limited (sparse). Limited observations prohibit accurate estimation of parameters of interest for any given individual. In this sparse data regime, the key question is, how accurately can we estimate the distribution of parameters over the population? This problem arises in various domains such as epidemiology, psychology, health-care, biology, and social sciences. As an example, suppose for a large random sample of the population we have observations of whether a person caught the flu for each year over the past 5 years. We cannot accurately estimate the probability of any given person catching the flu with only 5 observations, however our goal is to estimate the distribution of these probabilities over the whole population. Such an estimated distribution can be used in downstream tasks, like testing and estimating properties of the distribution.

In this talk, I will present our recent results where we show that the maximum likelihood estimator (MLE) is minimax optimal in the sparse observation regime. While the MLE for this problem was proposed as early as the late 1960's, how accurately the MLE recovers the true distribution was not known. Our work closes this gap. In the course of our analysis, we provide novel bounds on the coefficients of Bernstein polynomials approximating Lipschitz-1 functions. Furthermore, the MLE is also efficiently computable in this setting and we evaluate the performance of MLE on both synthetic and real datasets.

Joint work with Weihao Kong, Gregory Valiant, and Sham Kakade.

In this talk we present two novel ideas: the first is novel method to compute the feedback capacity of channels with memory using reinforcement learning (RL). The second is a new technique of using Graph-based auxiliary random variable to convert a multi-letter expression of feedback capacity formula into a single letter expression.

In RL, one seeks to maximize cumulative rewards collected in a sequential decision-making environment. This is done by collecting samples of the underlying environment and using them to learn the optimal decision rule. The main advantage of this approach is its computational efficiency, even in high dimensional problems. Hence, RL can be used to estimate numerically the feedback capacity of unifilar finite state channels (FSCs) with large alphabet size. The outcome of the RL algorithm sheds light on the properties of the optimal decision rule, which in our case, is the optimal input distribution of the channel.

The insights gained from the RL computation can be converted into analytic, single-letter capacity expressions by solving corresponding lower and upper bounds. The bounds are based on another novel idea of using Graph-based auxiliary random variable

We demonstrate the efficiency of this method by analytically solving the feedback capacity of the well-known Ising channel with a large alphabet. We also provide a simple coding scheme that achieves the feedback capacity.

Reducing the numerical precision of neural networks is one of the simplest, most effective and most common methods to improve resource efficiency (e.g. by reducing the memory and power requirements). Much research has been invested in finding how to quantize neural nets without significantly degrading performance. I will describe the main bottlenecks and solutions in various settings:
1) 1bit inference (1bit weights and activations), NIPS 2016 - link
2) 8bit training (8bit weights, activations, gradients, and batch-norm), NeurIPS 2018 - link
3) 4bit inference when quantization is done only post-training, Arxiv 2019 - link
4) Calculating the maximum trainable depth as a function of the numerical precision, Arxiv 2019 - link

As physical robot networks become more pervasive all around us, in the form of teams of autonomous vehicles, fleets of delivery drones, and smart and mobile IoT, it becomes increasingly critical to question the robustness of their coordination algorithms to security threats and/or corrupted data. Indeed, it has been shown that many multi-robot tasks easily fail in the presence of erroneous or hacked data. We investigate the vulnerabilities of important multi-robot algorithms such as consensus, coverage, and distributed mapping to malicious or erroneous data and we demonstrate the potential of communication to thwart certain attacks, such as the Sybil Attack, on these algorithms. Our key insight is that coordinated mobility can be combined with signal processing of communication signals to allow agents to learn important information about the environment and the nature of other agents in the network (for example the presence of cooperative versus adversarial agents). Along these lines, we will present a theoretical and experimental framework for provably securing multi-robot distributed algorithms through careful use of communication. We will present both theoretical results and experimental results on actual hardware implementations for bounding the influence of a Sybil Attack on consensus and on coverage by using observations over the wireless channels. In some cases, we show that the effect of a Sybil Attack can be nearly eliminated with high probability by deriving the appropriate switching function using a sufficient number of observations over the wireless network. Finally, we will briefly describe promising results on new methods for outlier rejection and active rendezvous in a pose graph optimization framework that exploits feedback gathered from communication channels to arrive at improved accuracy.

DNA has emerged as a compelling data storage medium due to its density, longevity, and eternal relevance compared to current memory technologies. However, the high price of synthesizing DNA (list price $3,500 per megabyte) remains a major bottleneck for adoption of this promising storage solution. In this talk, I will present our work towards breaking down this barrier using enzymatic DNA synthesize and a tailored codec for robust data retrieval. I will also touch upon some fundamental considerations when designing DNA information storage systems.

Tensor networks can often accurately approximate various systems that are of interest to physicists, but the computational cost of contracting the network is often prohibitively demanding. Quantum computer can resolve this bottleneck, but because the size of the computation scales with the size of the network, the existing quantum computers may appear to be far too noisy for this task. We prove a nontrivial error bound on the outcome of the computation that does not scale with the size of the network. This is possible because certain tensor networks are secretly implementing a quantum analogue of a (classical) Markovian dynamics. The long-time stability of the Markovian dynamics gives rise to a nontrivial error bound on the outcome of the computation. This suggests that there may be practical problems of interest that are amenable to relatively noisy quantum computers.

Jesse Gibson
Title: Information Theory in Next-Generation DNA Sequencing Technologies
The completion of the first human genome sequence in the early 2000's represented a major success for the scientific community. It also represented the return on a significant investment - billions of dollars and a little over a decade's worth of work. DNA sequencing costs have since fallen at a rate even faster than the exponential improvements predicted by Moore's Law in computing. Today, a complete human genome can be sequenced for closer to a few thousand dollars in about a week. A major factor in this improvement was the shift from the serial reads used in the human genome project to massively parallel sequencing technologies. These new technologies however present new problems - how can we understand a genome given millions of short reads drawn randomly from the underlying sequence and potentially degraded by noise in the process? Bioinformatics boasts a plethora of algorithms that meet this challenge in practice but it's not immediately obvious what 'optimal' analyses might look like. This talk will focus on the work of David Tse and others that have used the perspective of information theory to establish fundamental bounds on our ability to interpret next generation sequencing output. In particular, I will focus on the problems of assembling an unknown sequence de novo and of variant calling when the population of molecules being sequenced is not homogenous.

Tony Ginart
Title: Towards a Compression Theory for Metric Embeddings
Embedding matrices are widely used in diverse domains such as NLP, recommendation systems, information retrieval, and computer vision. For large-scale datasets, embedding matrices can consume large amounts of memory, and compression for these objects is desirable. However, the relevant distortion metrics applicable to embeddings make them significantly more compressible than classical sources considered in information theory. Furthermore, related results such as the Johnson-Lindenstrauss theorem are formulated in terms of reduction in dimension rather than codelength. Additionally, embeddings come with certain query-time constraints, such as in the sense of a succinct data structure. In this talk, we formulate the problem of metric embedding compression from the information theoretic lense and review related literature in information theory, signal processing, and dimensionality reduction. We adapt pre-existing results to establish some lower and upper bound in some regimes. Finally, we cover some of the state-of-the-art compression algorithms used to compress embeddings in practice.