[Paper Review] Tabulated MLP for Fast Point Feature Embedding

Yusuke Sekikawa, Teppei Suzuki|arXiv (Cornell University)|Nov 23, 2019

3D Shape Modeling and Analysis28 references4 citations

TL;DR

This paper proposes LUTI-MLP, a tabulated multi-layer perceptron that replaces standard MLPs in point feature embedding with a lookup table and trilinear interpolation, enabling 80× faster inference on 3D point clouds while maintaining or slightly improving accuracy. The method achieves significant speedups in both feature embedding and Jacobian computation for pose optimization, with no approximation at test time.

ABSTRACT

Aiming at a drastic speedup for point-data embeddings at test time, we propose a new framework that uses a pair of multi-layer perceptron (MLP) and look-up table (LUT) to transform point-coordinate inputs into high-dimensional features. When compared with PointNet's feature embedding part realized by MLP that requires millions of dot products, ours at test time requires no such layers of matrix-vector products but requires only looking up the nearest entities followed by interpolation, from the tabulated MLP defined over discrete inputs on a 3D lattice. We call this framework as "LUTI-MLP: LUT Interpolation MLP" that provides a way to train end-to-end tabulated MLP coupled to a LUT in a specific manner without the need for any approximation at test time. LUTI-MLP also provides significant speedup for Jacobian computation of the embedding function wrt global pose coordinate on Lie algebra $\mathfrak{se}(3)$ at test time, which could be used for point-set registration problems. After extensive architectural analysis using ModelNet40 dataset, we confirmed that our LUTI-MLP even with a small-sized table ($8 imes 8 imes 8$) yields performance comparable to that of MLP while achieving significant speedup: $80 imes$ for embedding, $12 imes$ for approximate Jacobian, and $860 imes$ for canonical Jacobian.

Motivation & Objective

• To drastically accelerate point feature embedding at test time in 3D point cloud processing pipelines.
• To enable efficient computation of the embedding's Jacobian with respect to global pose in se(3), crucial for point cloud registration.
• To design a tabulated MLP framework that maintains high accuracy while replacing computationally heavy matrix-vector multiplications in standard MLPs.
• To train the tabulated network end-to-end without approximation, ensuring exact inference via interpolation.

Proposed method

• LUTI-MLP pre-computes embedding basis functions at discrete 3D lattice points using a standard MLP, storing them in a lookup table (LUT).
• At test time, the network computes the embedding of a continuous 3D point via trilinear interpolation over the 8 nearest lattice points.
• The embedding is expressed as a weighted sum: $\mathbf{z}_i = \sum w_i^{(j)} \bar{\mathbf{z}}_i^{(j)}$, where weights $w_i^{(j)}$ are derived from the input point's proximity to the lattice.
• The method enables analytical and efficient computation of the Jacobian of the embedding with respect to $\mathbb{R}^3$ and $\mathfrak{se}(3)$, crucial for optimization in registration.
• The LUT is trained end-to-end using backpropagation, with the network structure asymmetric between training (MLP-based) and inference (LUT-interpolated).
• A novel training strategy is introduced to avoid approximation errors, ensuring that the LUT-based inference is exact and consistent with the original MLP.

Experimental results

Research questions

• RQ1 Can a lookup table-based MLP achieve comparable performance to a standard MLP in 3D point cloud classification while drastically reducing inference time?
• RQ2 Can the LUTI-MLP framework enable efficient analytical Jacobian computation of the embedding with respect to global pose in $\mathfrak{se}(3)$, suitable for registration tasks?
• RQ3 How does the size of the lookup table (e.g., $8\times8\times8$) affect the trade-off between speed and accuracy in point feature embedding?
• RQ4 Does end-to-end training of the LUTI-MLP without approximation yield better performance than direct table learning or naive fine-tuning?
• RQ5 Can the LUTI-MLP be effectively applied to hierarchical or graph-based point cloud networks to accelerate feature learning?

Key findings

• LUTI-MLP with an $8\times8\times8$ table achieves 88.01% accuracy on ModelNet40, slightly outperforming the original PointNet’s 87.03%.
• The method achieves an 80× speedup in feature embedding inference compared to standard MLP-based PointNet.
• Jacobian computation with respect to $\mathfrak{se}(3)$ is accelerated by 12× for approximate Jacobian and 860× for canonical Jacobian.
• The embedding space learned by LUTI-MLP with $D=64$ shows high similarity to the original MLP, with smooth and structured feature maps.
• Direct table learning (LUTI-Direct) fails to update most table entries when $D$ is large, indicating the need for a structured training approach like LUTI-MLP.
• Pre-training the LUTI-MLP using a standard PointNet model improves convergence and performance, especially for small $D$.