Closing the Loop on Cooperative Capacity Learning

A paper I co-authored has just come out in IEEE Transactions on Communications: Channel Capacity Estimation with Cooperative Neural Architectures: The Role of Objective Functions and f-Divergences, by Nicola Novello, Andrea M. Tonello, myself, Alex Dytso and H. Vincent Poor. It is open access.

I am not the first author, and that is part of why I like it. It takes an idea I started almost six years ago and carries it further than I did.

Where it started

In late 2020, Prof. Tonello put a simple question to me: can we estimate the capacity of any communication channel, including the ones with no closed-form answer?

Channel capacity is the maximal mutual information between a channel's input and output, maximised over every input distribution allowed by a constraint such as average or peak power (more on Wikipedia). For a handful of channels you can write it down. For most real ones you cannot, so you have to compute it numerically.

The idea I had came from GANs. At convergence, a GAN discriminator learns a density ratio, and mutual information is the expectation of a density ratio. So you can turn a discriminator into a mutual information estimator. Pair it with a generator that proposes input distributions, and make the two networks cooperate instead of compete: the generator looks for the input that maximises the information the discriminator measures. When the two settle, you have both the capacity estimate and the input distribution that achieves it.

That became CORTICAL (Cooperative Networks for Capacity Learning). The work ran through a large part of my PhD and I finished it during my visit to Princeton with Prof. Poor at the turn of 2023. It was published as Cooperative Channel Capacity Learning in IEEE Communications Letters.

What the new paper adds

CORTICAL used one specific objective function. The new paper asks the question I never had time to answer properly: how much does that choice matter?

The answer is a lot. The paper:

• Groups neural capacity estimators into four classes, by their objective function and by how they compute the final capacity estimate. All four share the same cooperative architecture.
• Proposes new estimators whose objectives are built on f-divergences, a broad family of ways to measure how different two distributions are. This formulation reduces the variance of the estimate and makes training converge better.
• Studies the conditions under which the cooperative training converges.
• Introduces self-consistency tests: checks on whether the estimated capacity and the learned input distribution actually agree with each other. A neural estimator can return a confident number that the distribution it learned does not support, and these tests are designed to catch exactly that.

Across the scenarios tested, the proposed estimators outperform the state of the art.

The f-divergence thread is not new to Nicola and me. It is the same family of ideas behind our NeurIPS 2024 paper on mutual information estimation. Seeing it come back to capacity, where the whole line of work began, is what makes this one feel like a closed loop.

Why this one matters to me

Three reasons, none of them about my position in the author list.

The venue. Transactions on Communications is one of the reference journals of the field I did my PhD in, and it is where I would have wanted this line of work to end up.

The people. Alex Dytso reviewed my PhD thesis, and I first met him in Princeton. Prof. Poor hosted me there and co-authored the original CORTICAL letter. Having both of them on the paper that extends it is a nice way for that visit to keep paying off years later.

The handover. I left full-time research to build PiktID. Ideas from a PhD often stop when their author stops working on them. This one did not: Nicola took it, asked better questions of it than I had, and made it more reliable. That is the best outcome I could have hoped for.

If you want the longer story of how CORTICAL fits into the rest of my PhD, I wrote about it in my PhD in review. The full list of papers is on the publications page.