[Paper Review] Implicit Neural Representations with Periodic Activation Functions

Vincent Sitzmann, Julien Martel+3 more|arXiv (Cornell University)|Jun 17, 2020

Image and Signal Denoising Methods42 references263 citations

TL;DR

The paper introduces SIRENs, neural networks with sinusoidal activations that accurately represent complex signals and their derivatives, enabling solutions to PDEs and learning priors over implicit function spaces.

ABSTRACT

Implicitly defined, continuous, differentiable signal representations parameterized by neural networks have emerged as a powerful paradigm, offering many possible benefits over conventional representations. However, current network architectures for such implicit neural representations are incapable of modeling signals with fine detail, and fail to represent a signal's spatial and temporal derivatives, despite the fact that these are essential to many physical signals defined implicitly as the solution to partial differential equations. We propose to leverage periodic activation functions for implicit neural representations and demonstrate that these networks, dubbed sinusoidal representation networks or Sirens, are ideally suited for representing complex natural signals and their derivatives. We analyze Siren activation statistics to propose a principled initialization scheme and demonstrate the representation of images, wavefields, video, sound, and their derivatives. Further, we show how Sirens can be leveraged to solve challenging boundary value problems, such as particular Eikonal equations (yielding signed distance functions), the Poisson equation, and the Helmholtz and wave equations. Lastly, we combine Sirens with hypernetworks to learn priors over the space of Siren functions.

Motivation & Objective

• Motivate a continuous, differentiable implicit representation that can model fine detail beyond grid-based approaches.
• Show that periodic activations enable accurate representation of signals and their derivatives.
• Demonstrate applications including image/video/audio representation, signed distance functions, and boundary value problem solving.
• Explore learning priors over the space of SIREN functions using hypernetworks.

Proposed method

• Define Phi as an implicit neural representation parameterized by an MLP with sine activations: Phi(x)=W_n(phi_{n-1}(...phi_0(x))+b_n, with phi_i(x)=sin(W_i x + b_i).
• Propose a principled initialization that preserves activation distributions across layers to stabilize training.
• Cast learning as a constraint satisfaction problem using a loss that enforces constraints C_m involving Phi and its derivatives over domain Omega.
• Supervise derivatives (gradients, Laplacian) or other PDE-related terms to fit targets without requiring direct function values in some cases (e.g., Poisson, Eikonal/SDF, Helmholtz).
• Demonstrate solutions to Poisson equations via gradient/Laplacian supervision and Poisson image editing; solve Eikonal (SDF) problems; tackle Helmholtz and wave equations; and apply hypernetworks to learn priors over the space of SIREN functions.

Experimental results

Research questions

• RQ1 Can periodic sine activations enable implicit neural representations to capture high-frequency details and higher-order derivatives better than ReLU-based networks?
• RQ2 How should SIREN weights and initializations be chosen to preserve activation distributions and enable deep architectures?
• RQ3 Can SIRENs solve boundary value problems and PDEs directly by supervision on derivatives rather than function values?
• RQ4 Can we learn priors over the space of SIREN functions using hypernetworks to enable tasks like inpainting or conditional generation?

Key findings

• SIRENs outperform ReLU-based MLPs in representing fine details and derivatives of signals (images, videos, audio).
• A principled initialization preserves activation distributions, enabling faster and more robust training of deep SIRENs.
• SIRENs can solve Poisson, Eikonal (SDF), Helmholtz, and wave equations through derivative supervision and boundary conditions.
• SIRENs enable learning priors over implicit function spaces via hypernetworks, improving tasks such as inpainting on CelebA with varying context.