[Paper Review] Polynomial Neural Fields for Subband Decomposition and Manipulation

Guandao Yang, Sagie Benaim+6 more|arXiv (Cornell University)|Feb 9, 2023

Model Reduction and Neural NetworksPhysics and Astronomy70 references8 citations

TL;DR

The paper introduces Polynomial Neural Fields (PNFs), a neural-field framework enabling interpretable subband decomposition and manipulation, including Fourier PNFs that localize outputs in frequency bands with controllable upper/lower limits, achieving state-of-the-art results while enabling texture transfer and scale-space tasks.

ABSTRACT

Neural fields have emerged as a new paradigm for representing signals, thanks to their ability to do it compactly while being easy to optimize. In most applications, however, neural fields are treated like black boxes, which precludes many signal manipulation tasks. In this paper, we propose a new class of neural fields called polynomial neural fields (PNFs). The key advantage of a PNF is that it can represent a signal as a composition of a number of manipulable and interpretable components without losing the merits of neural fields representation. We develop a general theoretical framework to analyze and design PNFs. We use this framework to design Fourier PNFs, which match state-of-the-art performance in signal representation tasks that use neural fields. In addition, we empirically demonstrate that Fourier PNFs enable signal manipulation applications such as texture transfer and scale-space interpolation. Code is available at https://github.com/stevenygd/PNF.

Motivation & Objective

• Motivate and design neural fields that are interpretable and controllable via subband decomposition.
• Develop a theoretical framework for PNFs and identify bases (e.g., Fourier) that enable subband manipulation.
• Create Fourier PNFs that localize outputs to specified frequency subbands with orientation control.
• Demonstrate that Fourier PNFs achieve competitive or superior signal representation performance across images, 3D SDFs, and radiance fields.
• Showcase downstream manipulation tasks like texture transfer and scale-space interpolation using PNF subbands.

Proposed method

• Define Polynomial Neural Field (PNF) as a composition of multivariate polynomials with a basis-based input encoding gamma using basis B.
• Prove that if the basis is closed under multiplication, a PNF output is a finite linear combination of basis functions.
• Introduce PNF-controllable sets of subbands to enable predictable interactions under addition and multiplication of PNFs.
• Specialize to Fourier PNFs by tiling the Fourier space with controllable subbands and deriving a network architecture that aggregates subband-limited PNFs.
• Propose an ensemble Fourier PNF where each subband-limited PNF contributes to the final output, with a factorization that reduces parameter blow-up.
• Provide a practical tiling strategy (circ/rect tilings) for 2D signals and discuss orientation-based subbands.
• Demonstrate training/setup and compare against BACON, SIREN, and Random Fourier Features across image, 3D SDF, and NeRF-style tasks.

Figure 3: Illustration of Fourier PNF architecture. Fourier PNF is an ensemble model. The final result is summed over a series of PNFs $F_{j}$ , whose structure is shown on the left side of the figure.

Experimental results

Research questions

• RQ1 Can PNFs provide interpretable, subband-based control over neural-field outputs without sacrificing expressivity?
• RQ2 How can frequency-domain subbands be designed and manipulated within PNFs to enable precise texture and scale-space operations?
• RQ3 Do Fourier PNFs achieve state-of-the-art or competitive performance while offering finer-grained subband control compared to prior methods (e.g., BACON)?
• RQ4 Can subband-decomposed PNFs support downstream tasks such as texture transfer and scale-space interpolation effectively?
• RQ5 What are the practical trade-offs (e.g., memory) when enforcing subband controllability in PNFs?

Key findings

• Fourier PNFs match or surpass state-of-the-art in signal representation across images, 3D shapes, and radiance fields.
• PNFs enable fine-grained subband localization in the frequency domain, with outputs constrained by lower and upper band limits and orientation.
• An ensemble Fourier PNF architecture with subband-limited components achieves effective subband decomposition and manipulation.
• PNFs demonstrate capabilities in texture transfer and scale-space interpolation by targeting specific frequency subbands.
• PNFs converge faster in terms of iterations for some tasks (e.g., 3D SDF fitting) and can offer competitive wall-clock times due to fewer optimization steps.
• Compared to BACON and SIREN, PNFs provide comparable or better quality with a similar parameter count.

Figure 5: LHS: Convergence time in terms of number of steps/iterations for the Thai Statue model [ 64 ] . The x-axis shows the number of steps (K) and the y-axis shows the validation Chamfer Distance ( $\times 10^{6}$ ). RHS: Qualitative comparison for the Thai Statue model [ 64 ] .