[Paper Review] Deep 3D-to-2D Watermarking: Embedding Messages in 3D Meshes and Extracting Them from 2D Renderings

Innfarn Yoo, Huiwen Chang+5 more|arXiv (Cornell University)|Apr 27, 2021

Advanced Steganography and Watermarking Techniques41 references4 citations

TL;DR

This paper proposes an end-to-end deep learning framework for 3D-to-2D watermarking that embeds messages invisibly into 3D meshes and recovers them robustly from 2D renderings under varying camera views and lighting. By leveraging a differentiable renderer and joint geometry-texture encoding, the method achieves high bit accuracy even after 3D distortions, marking the first fully differentiable 3D-to-2D watermarking system with strong generalization across renderers.

ABSTRACT

Digital watermarking is widely used for copyright protection. Traditional 3D watermarking approaches or commercial software are typically designed to embed messages into 3D meshes, and later retrieve the messages directly from distorted/undistorted watermarked 3D meshes. However, in many cases, users only have access to rendered 2D images instead of 3D meshes. Unfortunately, retrieving messages from 2D renderings of 3D meshes is still challenging and underexplored. We introduce a novel end-to-end learning framework to solve this problem through: 1) an encoder to covertly embed messages in both mesh geometry and textures; 2) a differentiable renderer to render watermarked 3D objects from different camera angles and under varied lighting conditions; 3) a decoder to recover the messages from 2D rendered images. From our experiments, we show that our model can learn to embed information visually imperceptible to humans, and to retrieve the embedded information from 2D renderings that undergo 3D distortions. In addition, we demonstrate that our method can also work with other renderers, such as ray tracers and real-time renderers with and without fine-tuning.

Motivation & Objective

• To address the gap in 3D watermarking where messages are typically extracted only from 3D meshes, not from 2D renderings.
• To enable robust message retrieval from 2D images generated from 3D meshes under diverse camera angles, lighting, and 3D distortions.
• To develop a fully differentiable framework that jointly optimizes 3D embedding and 2D decoding via a differentiable renderer.
• To demonstrate generalization of the decoder across non-differentiable renderers like ray tracers and real-time renderers.
• To evaluate performance in terms of message capacity, robustness to distortions, and visual invisibility.

Proposed method

• The framework uses a learnable encoder to embed messages into both 3D mesh geometry and 2D textures in a perceptually imperceptible way.
• A differentiable renderer simulates 2D image generation from multiple camera views and lighting conditions, enabling backpropagation through the rendering process.
• The system is trained end-to-end using a combination of 3D distortions (noise, scaling, rotation, cropping) and differentiable rendering to improve robustness.
• The decoder extracts embedded messages from 2D rendered images using a CNN-based architecture inspired by HiDDeN, with global pooling and fully connected layers for bit prediction.
• The method supports both real and noise-based textures during training, with ablation studies showing performance differences.
• The decoder can be fine-tuned for use with non-differentiable renderers such as ray tracers and real-time renderers, enabling broad applicability.

Experimental results

Research questions

• RQ1 Can a deep learning framework embed messages in 3D meshes such that they remain recoverable from 2D renderings under varying camera views and lighting?
• RQ2 How robust is the watermarking system to common 3D distortions like scaling, rotation, cropping, and additive noise?
• RQ3 To what extent does the differentiable renderer enable end-to-end training and improve generalization across diverse rendering conditions?
• RQ4 Can the trained decoder extract messages from non-differentiable renderers like ray tracers or real-time renderers without retraining?
• RQ5 How does the message capacity and visual quality trade-off compare to traditional 3D watermarking methods?

Key findings

• The model achieves high bit accuracy (over 90%) on messages of length 8 under additive noise, scaling, rotation, and cropping distortions when trained with real textures.
• Bit accuracy remains above 80% even under strong noise (σ = 0.133) and large-scale distortions, demonstrating robustness.
• The decoder fine-tuned on real-time renderers achieves over 95% bit accuracy, showing strong generalization across rendering pipelines.
• Visual quality is preserved, with difference images showing minimal perceptual distortion between original and watermarked 2D renderings.
• The system generalizes to non-differentiable renderers such as ray tracers and real-time renderers, especially after fine-tuning the decoder.
• The method outperforms baseline approaches that embed messages only in 2D textures, which fail under 3D-to-2D rendering distortions due to lack of robustness.