[Paper Review] CNOS: A Strong Baseline for CAD-based Novel Object Segmentation

Van Nguyen Nguyen, Thibault Groueix+3 more|arXiv (Cornell University)|Jul 20, 2023

Advanced Neural Network Applications4 citations

TL;DR

CNOS proposes a simple yet powerful zero-shot method for novel object segmentation using only CAD models, leveraging Segment Anything for proposals and DINOv2 cls tokens for matching against pre-rendered templates. It achieves state-of-the-art performance on BOP benchmarks, surpassing prior unsupervised methods by 19.8% AP and even outperforming supervised methods like Mask R-CNN.

ABSTRACT

We propose a simple three-stage approach to segment unseen objects in RGB images using their CAD models. Leveraging recent powerful foundation models, DINOv2 and Segment Anything, we create descriptors and generate proposals, including binary masks for a given input RGB image. By matching proposals with reference descriptors created from CAD models, we achieve precise object ID assignment along with modal masks. We experimentally demonstrate that our method achieves state-of-the-art results in CAD-based novel object segmentation, surpassing existing approaches on the seven core datasets of the BOP challenge by 19.8% AP using the same BOP evaluation protocol. Our source code is available at https://github.com/nv-nguyen/cnos.

Motivation & Objective

• To address the limitation of supervised methods that require retraining for each new object in robotics and AR applications.
• To enable zero-shot segmentation of novel objects using only their CAD models, avoiding costly data collection and retraining.
• To develop a strong, simple baseline for CAD-based novel object segmentation that outperforms existing unsupervised and supervised approaches.
• To mitigate the domain gap between real and synthetic images using DINOv2 and photo-realistic rendering.

Proposed method

• CNOS generates segmentation proposals in input RGB images using FastSAM or SAM.
• It pre-renders CAD models into templates from multiple viewpoints and encodes each with the DINOv2 [cls] token for visual description.
• At inference, proposals are matched to templates by computing cosine similarity between their DINOv2 [cls] tokens.
• The method uses a top-k average (Meanₖ, k=5) of the highest similarity scores as the final matching score for mask confidence.
• Photo-realistic rendering via BlenderProc is used to reduce domain gap and improve accuracy.
• Onboarding is a one-time process: rendering templates and extracting DINOv2 descriptors per CAD model.

Experimental results

Research questions

• RQ1 Can a zero-shot method based solely on CAD models achieve state-of-the-art performance in novel object segmentation?
• RQ2 How effective is DINOv2 [cls] token similarity in matching real image proposals to synthetic CAD templates?
• RQ3 Does photo-realistic rendering significantly improve zero-shot segmentation accuracy compared to standard rendering?
• RQ4 Can CNOS outperform both supervised and unsupervised baselines without any fine-tuning?

Key findings

• CNOS achieves 39.7% AP on the LM-O dataset using FastSAM and ViT-l, outperforming the previous SOTA unsupervised method by 19.8% AP.
• CNOS surpasses the supervised Mask R-CNN method, which was trained on tens of thousands of images per dataset, demonstrating strong zero-shot generalization.
• Using BlenderProc for photo-realistic rendering increases accuracy by 0.2% AP compared to standard Pyrender rendering.
• The Meanₖ (k=5) aggregation function yields the best performance, improving accuracy by 3.2% over Max and 3.2% over Mean.
• Onboarding time is under 2 seconds per CAD model using standard rendering, making it practical for industrial deployment.
• The method is robust to viewpoint variation, with dense viewpoint rendering (162 views) yielding nearly identical performance to coarse (42 views).