[Paper Review] S$^2$Contact: Graph-based Network for 3D Hand-Object Contact Estimation with Semi-Supervised Learning

Tze Ho Elden Tse, Zhongqun Zhang+4 more|arXiv (Cornell University)|Aug 1, 2022

Hand Gesture Recognition Systems5 citations

TL;DR

This paper proposes S$^2$Contact, a semi-supervised graph-based deep learning framework for 3D hand-object contact estimation from monocular RGB images. By leveraging visual and geometric consistency across video frames to generate high-quality pseudo-labels, and using an efficient graph neural network with 50% fewer parameters and lower memory cost than PointNet-based models, the method achieves state-of-the-art performance on HO-3D, HP-3D, and FreiHand, significantly improving contact coverage and pose accuracy even with limited real annotations.

ABSTRACT

Despite the recent efforts in accurate 3D annotations in hand and object datasets, there still exist gaps in 3D hand and object reconstructions. Existing works leverage contact maps to refine inaccurate hand-object pose estimations and generate grasps given object models. However, they require explicit 3D supervision which is seldom available and therefore, are limited to constrained settings, e.g., where thermal cameras observe residual heat left on manipulated objects. In this paper, we propose a novel semi-supervised framework that allows us to learn contact from monocular images. Specifically, we leverage visual and geometric consistency constraints in large-scale datasets for generating pseudo-labels in semi-supervised learning and propose an efficient graph-based network to infer contact. Our semi-supervised learning framework achieves a favourable improvement over the existing supervised learning methods trained on data with `limited' annotations. Notably, our proposed model is able to achieve superior results with less than half the network parameters and memory access cost when compared with the commonly-used PointNet-based approach. We show benefits from using a contact map that rules hand-object interactions to produce more accurate reconstructions. We further demonstrate that training with pseudo-labels can extend contact map estimations to out-of-domain objects and generalise better across multiple datasets.

Motivation & Objective

• To address the scarcity of 3D annotated data for hand-object contact estimation, especially in unconstrained real-world settings.
• To improve 3D hand and object reconstruction accuracy by modeling contact interactions more realistically through contact maps.
• To enable generalization to out-of-domain objects and unseen datasets without full supervision.
• To reduce reliance on expensive 3D annotations by leveraging pseudo-labels generated via consistency constraints.
• To design a more efficient and accurate contact estimation network than existing PointNet-based architectures.

Proposed method

• A semi-supervised learning framework is proposed that generates pseudo-labels using visual and geometric consistency across video sequences.
• Visual consistency is enforced via SSIM-based loss between consecutive frames to ensure appearance stability.
• Geometric consistency is enforced using Chamfer distance and SDF-based losses to maintain structural fidelity in 3D point clouds.
• A contact consistency loss ($\mathcal{L}_{cont}$) ensures temporal stability of predicted contact maps.
• An efficient graph-based neural network processes hand and object point clouds, capturing local geometric structures while maintaining permutation invariance.
• Pseudo-labels are filtered using confidence thresholds based on the consistency losses before being used for training.

Experimental results

Research questions

• RQ1 Can semi-supervised learning with consistency constraints improve 3D hand-object contact estimation from monocular images?
• RQ2 Can a graph-based network outperform PointNet-based models in contact estimation with fewer parameters and lower memory usage?
• RQ3 Does the use of pseudo-labels generated via visual and geometric consistency improve generalization to out-of-domain objects?
• RQ4 How do different consistency constraints (visual, geometric, contact) contribute to the final performance?
• RQ5 Can the model transfer effectively from a small annotated dataset to unlabelled, diverse datasets?

Key findings

• S$^2$Contact achieves a hand joint error of 8.74 mm on HO-3D, a 12% relative improvement over the baseline without semi-supervised training.
• The model reduces object mesh error by 8.56% in ADD-0.1D and increases contact coverage by 58% compared to the non-semi-supervised baseline.
• Disabling the contact consistency loss ($\mathcal{L}_{cont}$) leads to a 5.45% performance drop in object error and unstable training.
• The visual consistency loss ($\mathcal{L}_{SSIM}$) has the most significant impact, with its removal increasing object error by 8.04%.
• The proposed graph network uses 2.4× fewer parameters and 2× less GPU memory than DGCNN, while achieving better performance.
• The model generalizes well to out-of-domain objects and achieves strong zero-shot transfer performance across multiple datasets.