[Paper Review] Deep Convolutional Framelets: A General Deep Learning Framework for Inverse Problems

Jong Chul Ye, Yoseob Han+1 more|arXiv (Cornell University)|Jul 3, 2017

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper introduces Deep Convolutional Framelets, a novel deep learning framework for inverse problems that unifies classical signal processing theories with deep learning by using multilayer convolution framelets with ReLU activation to achieve perfect reconstruction. The framework reveals that residual blocks, redundant filter channels, and CReLU layers enable perfect reconstruction, while pooling layers require high-pass branches to maintain this property, offering a theoretically grounded design for inverse problem networks.

ABSTRACT

Recently, deep learning approaches with various network architectures have achieved significant performance improvement over existing iterative reconstruction methods in various imaging problems. However, it is still unclear why these deep learning architectures work for specific inverse problems. To address these issues, here we show that the long-searched-for missing link is the convolution framelets for representing a signal by convolving local and non-local bases. The convolution framelets was originally developed to generalize the theory of low-rank Hankel matrix approaches for inverse problems, and this paper further extends the idea so that we can obtain a deep neural network using multilayer convolution framelets with perfect reconstruction (PR) under rectilinear linear unit nonlinearity (ReLU). Our analysis also shows that the popular deep network components such as residual block, redundant filter channels, and concatenated ReLU (CReLU) do indeed help to achieve the PR, while the pooling and unpooling layers should be augmented with high-pass branches to meet the PR condition. Moreover, by changing the number of filter channels and bias, we can control the shrinkage behaviors of the neural network. This discovery leads us to propose a novel theory for deep convolutional framelets neural network. Using numerical experiments with various inverse problems, we demonstrated that our deep convolution framelets network shows consistent improvement over existing deep architectures.This discovery suggests that the success of deep learning is not from a magical power of a black-box, but rather comes from the power of a novel signal representation using non-local basis combined with data-driven local basis, which is indeed a natural extension of classical signal processing theory.

Motivation & Objective

• To identify the theoretical link between deep learning and classical signal processing for inverse problems.
• To explain why specific deep learning components (e.g., residual blocks, CReLU) enable perfect reconstruction in inverse problems.
• To develop a general deep learning framework that unifies framelet theory with deep networks for improved inverse problem solutions.
• To reveal limitations in existing architectures and provide design principles based on signal representation theory.

Proposed method

• Proposes a multilayer convolution framelet network using ReLU activation to achieve perfect reconstruction (PR) under specific conditions.
• Derives the perfect reconstruction condition by decomposing feature maps into positive and negative parts using ReLU and CReLU operations.
• Introduces a symmetric encoding-decoding structure where filter banks and biases are designed to reconstruct the original signal exactly.
• Uses Hankel matrix-based framelets as a foundation to generalize low-rank matrix approaches into deep learning frameworks.
• Demonstrates that residual blocks and redundant filter channels are essential for achieving perfect reconstruction.
• Reveals that pooling layers must be augmented with high-pass branches to preserve the perfect reconstruction property.

Experimental results

Research questions

• RQ1 Why do certain deep learning architectures achieve superior performance in inverse problems despite lacking theoretical grounding?
• RQ2 What is the mathematical mechanism that enables perfect reconstruction in deep convolutional networks?
• RQ3 How do standard components like residual blocks and CReLU contribute to perfect reconstruction in inverse problems?
• RQ4 Why do pooling and unpooling layers typically fail to achieve perfect reconstruction without modification?
• RQ5 How can classical signal processing theories like framelets and compressed sensing be unified with deep learning for inverse problems?

Key findings

• The proposed Deep Convolutional Framelets network achieves perfect reconstruction using ReLU activation when filter banks and biases satisfy specific symmetric conditions.
• Residual blocks, redundant filter channels, and CReLU layers are mathematically necessary for perfect reconstruction, explaining their empirical success.
• Pooling layers fail to ensure perfect reconstruction unless augmented with high-pass branches, which is a key design insight.
• The framework reveals that the success of deep learning in inverse problems stems from a novel signal representation combining non-local and data-driven local bases.
• Numerical experiments show consistent performance gains over existing deep architectures across diverse inverse problems such as low-dose CT and compressed sensing MRI.
• The theory provides a principled design framework, showing that deep learning's power arises not from black-box learning but from a natural extension of classical signal processing.