[Paper Review] Joint Graph Decomposition and Node Labeling: Problem, Algorithms, Applications

Evgeny Levinkov, Jonas Uhrig+8 more|arXiv (Cornell University)|Nov 14, 2016

Advanced Image and Video Retrieval Techniques28 references15 citations

TL;DR

This paper introduces the Minimum Cost Node Labeling Lifted Multicut Problem (nl-lmp), a novel NP-hard combinatorial optimization framework that jointly models graph decomposition and node labeling. It proposes two local search algorithms, KLj/r and KLj*r, which monotonically converge to local optima and achieve state-of-the-art accuracy in multiple computer vision tasks, including multiple object tracking, instance-separating semantic segmentation, and articulated human pose estimation, by solving the unified nl-lmp formulation without approximation guarantees but with feasible solutions at every step.

ABSTRACT

We state a combinatorial optimization problem whose feasible solutions define both a decomposition and a node labeling of a given graph. This problem offers a common mathematical abstraction of seemingly unrelated computer vision tasks, including instance-separating semantic segmentation, articulated human body pose estimation and multiple object tracking. Conceptually, the problem we state generalizes the unconstrained integer quadratic program and the minimum cost lifted multicut problem, both of which are NP-hard. In order to find feasible solutions efficiently, we define two local search algorithms that converge monotonously to a local optimum, offering a feasible solution at any time. To demonstrate their effectiveness in tackling computer vision tasks, we apply these algorithms to instances of the problem that we construct from published data, using published algorithms. We report state-of-the-art application-specific accuracy for the three above-mentioned applications.

Motivation & Objective

• To unify seemingly unrelated computer vision tasks—multiple object tracking, instance-separating semantic segmentation, and articulated human body pose estimation—under a single combinatorial optimization framework.
• To define a generalization of the unconstrained integer quadratic program and the minimum cost lifted multicut problem that jointly models graph decomposition and node labeling.
• To develop efficient local search algorithms that converge monotonically to local optima, providing feasible solutions at any time without lower bound computation.
• To demonstrate the effectiveness of the proposed framework and algorithms on real-world benchmarks, achieving state-of-the-art application-specific accuracy.
• To foster cross-disciplinary collaboration between computer vision and combinatorial optimization by open-sourcing the code.

Proposed method

• Proposes the nl-lmp problem, which generalizes both the unconstrained integer quadratic program (uiqp) and the minimum cost lifted multicut problem (lmp), allowing labels to influence component formation and vice versa.
• Defines two local search algorithms: KLj/r (alternating Kernighan-Lin with joins and relabeling) and KLj*r (joint Kernighan-Lin with joins and relabeling), both designed to improve solution quality iteratively.
• The algorithms perform local moves that alternate between reassigning nodes to components and changing node labels, guided by an objective function that balances labeling and decomposition costs.
• The algorithms do not compute lower bounds, making them primal feasible heuristics that return a feasible solution at every iteration.
• The framework allows for complex cost functions where the cost of merging or separating nodes depends on their labels, enabling richer modeling of visual relationships.
• The algorithms are applied to real-world data from benchmarks such as KITTI, Cityscapes, and the MOT16 benchmark, using published detection and embedding features to construct nl-lmp instances.

Experimental results

Research questions

• RQ1 Can a single combinatorial optimization framework unify diverse computer vision tasks that require both graph decomposition and node labeling?
• RQ2 How can the joint optimization of decomposition and labeling be formulated as a generalization of existing NP-hard problems like uiqp and lmp?
• RQ3 Can local search algorithms that converge monotonically to local optima outperform existing methods in tasks like multiple object tracking and instance segmentation?
• RQ4 To what extent can the same algorithmic framework achieve state-of-the-art performance across disparate vision tasks without task-specific reparameterization?
• RQ5 Does the proposed framework enable more accurate and efficient solutions compared to prior approaches that solve decomposition and labeling separately?

Key findings

• The proposed KLj/r and KLj*r algorithms achieve state-of-the-art MOTA (47.6) and MOTP (78.5) scores on the MOT16 multiple object tracking benchmark, outperforming prior work such as [36] and other top-performing methods.
• KLj/r reduces false positives (5,844) and false negatives (89,093) compared to [36], which reported 6,373 and 90,914 respectively, indicating improved detection and tracking accuracy.
• The average inverse running time per frame (Hz) for KLj/r is 8.3, significantly higher than other methods, demonstrating superior computational efficiency.
• On the Cityscapes dataset, the framework achieves more accurate instance-separating semantic segmentation results than any previously published work, despite using the same input features.
• For the MPII Human Pose Dataset, the framework enables more accurate pose estimation than prior methods by solving larger problem instances due to faster computation, even though the original branch-and-cut method computed lower bounds.
• The convergence of KLj/r and KLj*r is empirically validated in Figure 4, showing KLj/r converges faster than KLj*r while achieving identical objective values.