[Paper Review] TextureNet: Consistent Local Parametrizations for Learning from High-Resolution Signals on Meshes

Jingwei Huang, Haotian Zhang+4 more|arXiv (Cornell University)|Nov 30, 2018

3D Shape Modeling and Analysis43 references4 citations

TL;DR

TextureNet introduces a novel deep learning architecture that leverages consistent 4-rotational symmetric (4-RoSy) fields to enable orientation-invariant convolution on high-resolution signals over 3D mesh surfaces. By defining geodesic neighborhoods with consistent local parametrizations, it achieves state-of-the-art performance in 3D semantic segmentation, outperforming prior methods by 6.4% (geometry-only) and 6.9–8.2% (RGB+geometry) in mean IoU.

ABSTRACT

We introduce, TextureNet, a neural network architecture designed to extract features from high-resolution signals associated with 3D surface meshes (e.g., color texture maps). The key idea is to utilize a 4-rotational symmetric (4-RoSy) field to define a domain for convolution on a surface. Though 4-RoSy fields have several properties favorable for convolution on surfaces (low distortion, few singularities, consistent parameterization, etc.), orientations are ambiguous up to 4-fold rotation at any sample point. So, we introduce a new convolutional operator invariant to the 4-RoSy ambiguity and use it in a network to extract features from high-resolution signals on geodesic neighborhoods of a surface. In comparison to alternatives, such as PointNet based methods which lack a notion of orientation, the coherent structure given by these neighborhoods results in significantly stronger features. As an example application, we demonstrate the benefits of our architecture for 3D semantic segmentation of textured 3D meshes. The results show that our method outperforms all existing methods on the basis of mean IoU by a significant margin in both geometry-only (6.4%) and RGB+Geometry (6.9-8.2%) settings.

Motivation & Objective

• Address the challenge of learning from high-resolution signals (e.g., color textures) on 3D surface meshes, which are underutilized in existing 3D deep learning methods.
• Overcome limitations of existing 3D CNNs that operate on low-resolution representations, which restrict fine-grained semantic discrimination.
• Develop a consistent, low-distortion parameterization method that preserves local orientation across geodesic neighborhoods on arbitrary 3D meshes.
• Design a novel convolutional operator invariant to 4-fold rotational ambiguity in 4-RoSy fields to enable robust feature learning on surface signals.
• Demonstrate the superiority of the proposed architecture for 3D semantic segmentation using high-resolution mesh signals, surpassing existing methods in performance.

Proposed method

• Utilize QuadriFlow to compute a 4-RoSy field on the mesh surface, providing low-distortion, consistent, and singularity-minimal parameterization across geodesic neighborhoods.
• Define local geodesic patches around each mesh vertex using the 4-RoSy field, assigning each patch a consistent local coordinate system invariant to 4-fold rotation.
• Introduce a 4-RoSy convolutional operator that applies 3x3 filters across all four rotational orientations of the field and aggregates features via max pooling to ensure orientation invariance.
• Construct a UNet-style network architecture, TextureNet, composed of stacked 4-RoSy convolutional layers to extract hierarchical features from both geometry and high-resolution signals (e.g., RGB textures).
• Train the network end-to-end for 3D semantic segmentation using cross-entropy loss, with input features including vertex positions, normals, and texture patches.
• Precompute local parametrizations to maintain training efficiency comparable to standard geodesic CNNs while enabling high-resolution signal processing.

Experimental results

Research questions

• RQ1 Can consistent local parametrization via 4-RoSy fields improve feature learning from high-resolution signals on 3D meshes compared to standard 3D CNNs?
• RQ2 How does the 4-RoSy convolutional operator’s invariance to 4-fold rotation affect performance in 3D semantic segmentation tasks?
• RQ3 To what extent do high-resolution surface signals (e.g., color textures) improve segmentation accuracy when combined with geometric features?
• RQ4 How does the proposed method compare to alternative convolution operators (e.g., GCNN, ACNN, RoSy) in terms of feature consistency and segmentation accuracy?
• RQ5 Can the proposed architecture generalize to geometry-only inputs and still outperform state-of-the-art methods in 3D semantic segmentation?

Key findings

• TextureNet achieves a mean IoU of 50.6% on the ScanNet benchmark using only surface geometry, outperforming the previous SOTA by 6.4 percentage points.
• With RGB and geometric features, TextureNet achieves a mean IoU of 54.8%, representing a 6.9–8.2% improvement over prior methods in the RGB+geometry setting.
• The inclusion of high-resolution texture patches (10×10) improves mean IoU by 3.3% compared to using only per-point normals and colors.
• The ablation study confirms that consistent orientation via 4-RoSy fields is critical—RoSy 4 (m), which uses max pooling over four orientations, outperforms RoSy 4 and RoSy 1 due to better invariance.
• The 4-RoSy convolutional operator significantly improves feature coherence, as evidenced by higher IoU on fine-grained classes such as bookshelves (71.3%) and pictures (21.1%) when high-resolution textures are used.
• Training efficiency is comparable to standard geodesic CNNs due to precomputed local parametrizations, making the method scalable despite high-resolution input.