[Paper Review] OCNet: Object Context Network for Scene Parsing

Yuhui Yuan, Jingdong Wang+4 more|arXiv (Cornell University)|Sep 4, 2018

Advanced Image and Video Retrieval TechniquesComputer Science70 references516 citations

TL;DR

OCNet introduces an object-centric context aggregation mechanism for semantic segmentation, using dense or interlaced sparse self-attention to emphasize pixels belonging to the same object category, and augments it with pyramid extensions for multi-scale context.

ABSTRACT

In this paper, we address the semantic segmentation task with a new context aggregation scheme named \emph{object context}, which focuses on enhancing the role of object information. Motivated by the fact that the category of each pixel is inherited from the object it belongs to, we define the object context for each pixel as the set of pixels that belong to the same category as the given pixel in the image. We use a binary relation matrix to represent the relationship between all pixels, where the value one indicates the two selected pixels belong to the same category and zero otherwise. We propose to use a dense relation matrix to serve as a surrogate for the binary relation matrix. The dense relation matrix is capable to emphasize the contribution of object information as the relation scores tend to be larger on the object pixels than the other pixels. Considering that the dense relation matrix estimation requires quadratic computation overhead and memory consumption w.r.t. the input size, we propose an efficient interlaced sparse self-attention scheme to model the dense relations between any two of all pixels via the combination of two sparse relation matrices. To capture richer context information, we further combine our interlaced sparse self-attention scheme with the conventional multi-scale context schemes including pyramid pooling~\citep{zhao2017pyramid} and atrous spatial pyramid pooling~\citep{chen2018deeplab}. We empirically show the advantages of our approach with competitive performances on five challenging benchmarks including: Cityscapes, ADE20K, LIP, PASCAL-Context and COCO-Stuff

Motivation & Objective

• Motivate improving pixel labeling by explicitly emphasizing object-level information.
• Propose an object context scheme to replace traditional multi-scale context with object-oriented context.
• Develop an efficient interlaced sparse self-attention (ISA) to approximate dense pixel relations with reduced computation.
• Integrate object context with pyramid schemes (Pyramid-OC and ASP-OC) to capture multi-scale information.
• Demonstrate competitive performance on major segmentation benchmarks.

Proposed method

• Define object context as the set of pixels sharing the same object category as a given pixel.
• Replace the binary object-context relation with a learnable dense relation matrix or two sparse relation matrices.
• Introduce interlaced sparse self-attention (ISA) that factorizes the dense relation into two sparse matrices Wg and Wl for global and local context, reducing O(N^2) complexity.
• Instantiate the dense/sparse relations through self-attention and ISA, including the formulation W = Wl^T Pg^T Wg P (efficient approximation).
• Extend OCNet with Pyramid-OC and ASP-OC by integrating object-context pooling into pyramid pooling and ASPP frameworks.

Experimental results

Research questions

• RQ1 Can an object-centric context mechanism improve pixel-wise segmentation accuracy compared with traditional multi-scale context methods (e.g., PPM, ASPP) across challenging datasets?
• RQ2 Does the proposed interlaced sparse self-attention provide a favorable accuracy/computation trade-off versus standard self-attention for high-resolution feature maps?
• RQ3 Do pyramid extensions (Pyramid-OC, ASP-OC) offer additional gains by combining object context with multi-scale contexts?

Key findings

• The object context scheme consistently emphasizes object pixels, with dense relation values higher for same-category pixel pairs.
• Interlaced sparse self-attention substantially reduces memory and FLOPs while maintaining competitive performance compared to full self-attention.
• OCNet variants (Base-OC, Pyramid-OC, ASP-OC) achieve competitive results on Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff.
• Replacing the image-level pooling in ASPP with object-context pooling (ASP-OC) yields improvements over standard ASPP.
• Pyramid-OC integrates object context across multiple spatial partitions, enhancing multi-scale context utilization.